"If You Can't See It Break, You Can't Fix It Before Someone's Money Disappears" — Ankit Rawat On Engineering AI You Can Trust

Ankit Rawat spent 13 years building payment and logistics systems at Amazon, Wayfair, and Meta, environments where a software failure is a financial event, not a learning opportunity. Here, he explains why production-ready AI requires observability built in from day one, and what most engineering teams only discover after something has already gone wrong.

On March 25, 2026, the UK's Financial Conduct Authority published its Payments Regulatory Priorities report, the first time the regulator formally named agentic AI payments as a live policy question. The admission was telling: regulators are now scrambling to govern technology that is already ahead of them. In payments, that gap between capability and trustworthy production deployment is not theoretical. It is where failures happen.

The technology is ahead of the governance. And the governance is still ahead of the engineering discipline required to make any of it trustworthy in production. 99% of companies plan to put AI agents into production, but only 11% have done so, with implementation challenges around data, governance, and security accounting for most of the gap. In payments specifically, that gap is not an abstract metric. It is the distance between a system that behaves reliably when real money is moving and one that fails quietly while nobody is watching.

To understand what closing that gap requires, we spoke with Ankit Rawat, a Senior Software Engineer at Meta Platforms in Seattle. Across thirteen years at Yodlee, Snapdeal, Amazon, Salesforce, Wayfair, and Meta, he has built software in environments where a failure is not a learning opportunity; it is a financial event. At Meta, the observability methodology he developed now spans 100+ teams and underpins payment flows serving hundreds of millions of users. His career has been built on a single conviction: if you cannot see a system break, you cannot fix it before someone's money disappears.

In the conversation that follows, Ankit Rawat draws on 13 years of production experience across financial technology, logistics, and payments to explain where engineering discipline separates AI that survives deployment from AI that creates incidents.

Ankit, your observability work at Meta started as a four-team initiative and expanded to 100+ teams, directly surfacing a multi-year payments opportunity that existing tooling had never detected. How does something like that happen, and what does it tell us about how most teams think about system visibility?

Most teams are watching the wrong thing. Their dashboards show healthy infrastructure, uptime, latency, error rates, and everything looks fine right until a user is already affected. But dashboards can't show what users experience: the payment step that keeps stalling, the flow where people quietly drop off, the error that never triggers an alert because technically nothing broke.

We fixed that by instrumenting real user journeys instead of infrastructure. We defined success and failure at every step, collected structured events along those paths, and built dashboards that showed where people struggled and why. That's how the payment opportunity surfaced, a surface nobody had instrumented before. The 100-team adoption wasn't a planned rollout. When engineers finally saw what they had been missing, they came to us.

That kind of visibility doesn't develop in a vacuum. You built the thinking behind it across environments where failure had no grace period. At Amazon, you worked across freight logistics and physical store checkout systems, environments where load spikes, mid-transaction failures, and unpredictable traffic are the norm. What did those environments teach you about how systems fail that you cannot learn in a testing environment?

Systems don't fail the way you expect. They fail at the intersection of things you tested separately but never together. Freight routing spikes while checkout degrades simultaneously, and no test scenario ever covered that combination.

What those environments taught me about AI specifically is that stress doesn't just slow a system down. It changes how it behaves in ways testing never surfaces. Under real load, pipelines deliver inconsistently, services respond slowly, and agents start deciding on incomplete information. No errors thrown, just different outcomes. If you haven't instrumented what the agent is seeing at each decision point, you won't know something is wrong until the damage is already done.

Knowing where things break before they do, that same instinct shaped one of your most consequential pieces of work. At Wayfair, you led a performance review across senior engineers in multiple organisations, identified why B2B cart processing was capped at 30 items, and ultimately enabled roughly $150 million in annual business with seven percent year-over-year growth. Walk us through what that process looked like and what it produced.

Wayfair's B2B customers were hitting a hard ceiling. Professional buyers couldn't complete real commercial orders because the cart simply couldn't handle them. The question was why, and the answer lived in several places simultaneously, which is why we had to look across teams rather than inside any single one.

What that process produced was a precise map of where the system would break under real B2B load. Once you know that, optimisation becomes a structured problem instead of guesswork. Cart capacity went from 30 to 90+ items, and the business impact followed.

But the transferable lesson is this: before you optimise anything, understand where things will break first. That discipline applies directly to AI on payment infrastructure. The model inherits whatever sits beneath it; weak foundations don't disappear, they amplify.

Wayfair wasn't only about foundational infrastructure. You also built and shipped two GenAI tools there: a conversation summarisation capability that reduced call abandonment by fourteen percent, and an agent assist tool that cut customer discovery time by eighty-five percent, both without replacing a single human agent. Why is that framing, augmenting rather than replacing, especially important in payments?

In payments, when full automation fails, recovery is extraordinarily difficult. If a human agent makes a mistake, there is a path: the customer calls back, and someone fixes it. But when an automated system quietly makes thousands of wrong calls, you are cleaning up damage you did not even know was happening. Users whose transactions failed do not give the platform a second chance.

The summarisation work came from understanding why agents were slow; they were reading through long conversation histories before they could help anyone. Collapsing those into structured summaries meant they got the situation instantly, and their decisions improved. The agent assist tool solved the same problem differently: agents had good judgment, but were spending most of their time hunting through disconnected systems for context. We surfaced it instantly. Their expertise remained intact; the friction around it was removed. The principle holds across both tools: the best AI doesn't automate the decision. It removes the friction around it.

You have spent thirteen years across Amazon, Wayfair, and Meta, building in environments where code quality directly determines whether a financial system survives production. Most engineers think about failure after something breaks. You seem to think about it before a single line ships. What does that look like at the code review stage, and where do most teams underinvest?

Code review is treated as a quality gate, a checkpoint before something ships. That framing misses the point. In financial systems, review is where you stress-test assumptions before they reach production. Asking "Does this work?" is less than half the job. The harder questions are: what happens when the data isn't what we expect? What fails silently at volume?

Most teams underinvest in the quality of review, not the quantity. They catch obvious bugs and miss race conditions that only surface under load. Across Amazon, Wayfair, and Meta, the teams with the strongest production records all ask the same question during review: if this goes wrong, will we know?

Payments AI is moving fast globally, with new infrastructure layers, agentic transaction flows, and domain-specific financial models. From the perspective of someone who has shipped AI into production payment systems, what should engineering teams build before anything else goes live?

Build the mechanism that tells you when things break before users find out. Not server alerts. User-facing failure signals tied to real transaction journeys. Know which payment step causes abandonment when it degrades. Know what an AI agent is seeing at each decision point.

Most teams only build that visibility after something goes wrong in production. They ship, it breaks, and then they start instrumenting. By that point, users whose money was moving have already made their decision about the platform.

The teams that get this right don't treat observability as a follow-up task. They treat it the same way they treat security, something you design in from the first line, not something you bolt on when the incident report lands on your desk.

Your career has been almost entirely defined by industry practice at scale. But you also engage with the broader research community through academic work. Can you tell us about that side of your professional life, and why it matters to you, alongside the engineering work?

Engineering at scale generates problems that research alone cannot solve, and questions that practice alone cannot answer. I contribute to peer-reviewed work because academic writing forces a rigour that production pressures can erode. Articulating a methodology clearly enough for others to evaluate sharpens your understanding of what you know versus what you assume. My published research focuses on observability, distributed systems reliability, and AI in high-stakes environments, areas that remain genuinely unsolved. There is still a significant gap between what researchers understand and what practitioners encounter. I try to contribute from both directions.

That's a perspective you don't often hear from engineers working at this scale. Does academic research change how you approach problems at work, and what is the case for engineers earlier in their careers to engage with it alongside industry work?

They are not separate. Much of my published work began as a production problem without a clean answer. Building the observability methodology at Meta meant constructing an approach from first principles, and formalising that through academic writing made it something others could engage with and build on. The reverse is equally true: staying current with research introduces conceptual frameworks that make diagnosis faster. For earlier-career engineers, the core value is learning to distinguish what works in one context from what is genuinely generalisable. Industry alone does not reliably develop that skill, and in payments AI, it matters enormously.

Published on: Tuesday, August 04, 2026, 03:18 PM IST