AI-generated code velocity mismatch creating high blast radius production incidents, and why review burden should increase not decrease with AI assistance
The Blast Radius Problem Nobody Wants to Talk About
Amazon is holding mandatory internal meetings about AI-generated code causing production incidents. The briefing note, surfaced publicly this week, describes a pattern of incidents with “high blast radius” caused by “Gen-AI assisted changes” and includes this sentence: “best practices and safeguards are not yet established.”
That last part is the admission most engineering organizations are too proud to make out loud.
🔥 What “High Blast Radius” Actually Means
Blast radius is a term borrowed from incident response. It describes how much of your system a single failure can take down. A misconfigured feature flag that affects one tenant is low blast radius. A subtle race condition baked into a shared data pipeline that gets deployed across every service simultaneously, that is another thing entirely.
The problem with AI-assisted code is not that the code is wrong in obvious ways. If it were obviously wrong, someone would catch it. The problem is that it is plausible. It compiles. It passes the tests that exist. It looks like something a senior engineer might have written on a Tuesday afternoon. And because the engineer reviewing it is now moving faster than they were six months ago, they apply less scrutiny, not more.
That gap between velocity and verification is where production incidents live.
Why Review Burden Should Increase, Not Shrink
There is a seductive logic that has taken hold in a lot of teams. The AI writes the code, so maybe it can also review the code. Anthropic shipped Claude Code’s new PR review feature this week, which dispatches a “team of agents to hunt for bugs” when a pull request opens. The demos look impressive.
But automated review catching automated code is not the same as a human engineer who owns the system asking hard questions about it. What is the failure mode if this cache is cold? What happens to this migration script if it runs twice? Does this change assume ordering guarantees that do not exist in production?
Those questions require context that lives in human heads, not in the diff. The faster you ship, the more important those questions become, because the blast radius of a wrong answer scales with deployment frequency.
My actual position: when a team adopts an AI coding assistant, their code review checklist should get longer, not shorter. The bar for what gets approved should go up, not down. The AI handles volume. The human has to handle consequence.
The Confidence Problem Is the Real Risk
Models write code the way confident junior engineers write code. Fluent, complete, internally consistent, and sometimes wrong in ways that only show up under conditions the writer did not fully consider. The difference is that a junior engineer usually has some hesitation visible in their PR description. The model does not hedge. It ships complete, confident solutions to problems it may have partially misunderstood.
This is not a criticism of the models. It is a description of how they work. Understanding that is the engineer’s job.
The Amazon situation is not unique. It is just the first time I have seen a large organization put the phrase “best practices and safeguards are not yet established” in an official internal document and have it leak out. Most organizations are in the same place. They have adopted the tools. They have not adapted the processes.
Where This Goes
The teams that get this right will be the ones that treat AI assistance as a reason to invest more in their review culture, their deployment controls, their rollback capabilities, and their incident runbooks. Not less.
The teams that get this wrong will keep showing up in post-mortems wondering how something so plausible-looking took down three services at once.
Amazon’s mandatory meeting is not a cautionary tale about AI. It is a cautionary tale about adoption without infrastructure. The model did exactly what it was asked to do. The process around it was not ready.
Build the process.
Sources
#AIEngineering #SoftwareEngineering #DevOps #CodeReview #ProductionIncidents
