Hot take: most AI system failures are state management failures, not model failures
The Real Reason Your AI System Is Broken (It’s Not the Model)
Every week I watch engineers spend hours debating which model to use. GPT-4o versus Claude 3.5 versus Gemini 1.5 Pro. Benchmark comparisons, pricing calculators, latency tests. It’s not that those things don’t matter. They do, at the margins. But in my experience shipping AI systems that actually run in production, model choice is rarely what breaks things. State management is.
This is my hot take, and I’ll defend it.
The Stateless Model Problem
Here’s the architectural truth nobody puts in their system design docs: every model call is stateless by design. You send tokens in, you get tokens out, and the model retains nothing. The slate is wiped clean on every request.
That would be fine if the problems we’re solving were also stateless. They’re not. Real AI systems have users with histories, multi-step workflows, tool outputs that feed into later reasoning steps, long-running agent loops, and sessions that span hours or days. So what do we do? We bolt state on from the outside. We stuff context windows until they’re bloated. We build RAG pipelines that retrieve the “right” history. We serialize agent checkpoints to Redis between steps and hope the deserialization logic stays consistent.
Then we wonder why the system behaves unpredictably.
Where I’ve Actually Seen Things Break
Let me be concrete, because vague complaints don’t help anyone.
The failure modes I keep seeing in production AI systems tend to cluster around a few patterns. Context poisoning is one: early tool outputs or incorrect intermediate reasoning steps get carried forward in the context window and corrupt every subsequent decision. The model isn’t wrong on its own terms, it’s reasoning correctly from a broken premise that lives in state, not weights.
Then there’s agent amnesia, where a long-running workflow serializes its state to a database between steps but the schema drifted, or a field got silently dropped, and the agent picks up from a position that doesn’t match where it left off. You’ll see this show up as bizarre non-sequiturs in agent output that look like model hallucination but are actually a state reconstruction failure.
Session bleed is a subtler one. Shared infrastructure, misconfigured session isolation, and you get fragments of one user’s context leaking into another’s. The model does exactly what it’s told. What it’s told is wrong, because state management failed upstream.
The Numbers That Should Bother You
When Google published internal findings on their agent reliability work, they noted that the majority of unexpected agent behaviors traced back to context construction and state retrieval problems rather than base model errors. The model was doing what you asked. You asked based on broken state.
Microsoft’s research on AutoGen-style multi-agent systems found that inter-agent communication failures, essentially state passing between agents, accounted for a disproportionate share of task completion failures. Not reasoning failures. Handoff failures.
This pattern holds in my own work. When I do a post-mortem on a production AI system failure, the checklist I go through first is: what was in the context window, how was history retrieved, what intermediate state was passed between steps, and was that state validated at the boundary. The model gets interrogated last, not first.
What Good State Management Actually Looks Like
Building well here means treating AI state with the same discipline you’d apply to any distributed system’s data layer.
That means explicit state schemas with versioning. It means validation at every boundary where state is read or written. It means designing context construction as a first-class function, not an afterthought you assemble inline. It means being ruthless about what actually needs to be in the context window versus what can be retrieved on demand. And it means building observability around state, not just around model outputs, so you can actually see what the model was given, not just what it said.
The engineers I’ve seen build reliable AI systems think of the context window as a managed resource. The ones who build fragile systems treat it like a scratch pad.
Where This Leaves the Model Debate
None of this means model selection is irrelevant. Reasoning quality, instruction following, tool use accuracy, these matter and they differ across models. But they’re the last mile. A better model running on broken state is still a broken system.
The dirty secret of AI engineering right now is that we’ve over-invested in model evaluation and under-invested in the infrastructure around it. We have excellent benchmarks for what models can do. We have almost no standard tooling for auditing state management in agent systems.
That gap is where most production failures live. Until the industry closes it, choosing the “right” model is going to keep solving the wrong problem.
Sources & Further Reading
#AIEngineering #MLOps #AgentSystems #SoftwareArchitecture #MachineLearning
