Prediction: termination logic is the underrated design problem in agentic AI systems, not model quality or prompt design
The Hardest AI Problem Isn’t the Model. It’s Knowing When to Stop.
Most teams building agentic AI systems are focused on the wrong problem. They spend weeks on prompt engineering. They benchmark models. They agonize over latency. And then they ship an agent that loops forever, or halts too early, or confidently hands back a result that is subtly wrong in ways nobody catches until production.
The real problem is termination logic. And almost nobody is talking about it.
Why This Is Harder Than It Looks
When you write a function, done is obvious. Tests pass. Return value is correct. You move on.
When an agent works through a multi-step task, done is fuzzy. The output looks reasonable. The model is confident. But confidence from a language model is not a signal about ground truth. It is a signal about fluency. Those are completely different things.
A model can be maximally confident while being completely wrong. This is not a bug you can prompt your way out of.
So teams do the obvious thing. They add more checks. More validation layers. More human review gates. What they have built, without naming it, is a termination condition. A set of rules that answers the question: is this done?
The problem is they built it reactively, after things broke, rather than designing it upfront as a first-class concern.
The Karpathy Example Gets This Right
Andrej Karpathy just open-sourced an autonomous research agent that runs overnight experiments on a single GPU. The repo is around 630 lines of code. The agent edits training code, runs a small language model for exactly five minutes, checks validation loss, keeps or discards the result, and loops again. It runs roughly 12 experiments per hour, about 100 overnight, with no human in the loop.
Here is what I think is genuinely smart about the design, and it has nothing to do with the model or the prompts. It is the fixed five-minute clock. Every single run, regardless of what the agent changed, gets compared on equal footing. That constraint is a termination condition built into the architecture. The agent does not decide when a run is good enough. The clock decides. The score decides. The human who wrote the research prompt decided, before the agent ever started.
That is the right order of operations. https://x.com/karpathy/status/2030371219518931079
The Design Pattern Nobody Is Teaching
OpenAI’s Codex Security agent is another example worth looking at. The framing in their announcement is that it “finds vulnerabilities, validates them, and proposes fixes you can review and patch.” The human review gate at the end is not an afterthought. It is the termination condition. The agent stops at proposal. It does not self-merge. https://x.com/OpenAIDevs/status/2029983809652035758
Both of these systems have something in common. The scope boundary was decided before the agent ran. Not during. Not after.
This is what I try to get teams to internalize: termination logic is not a post-processing step. It is architecture. You need to answer three questions before you write a single line of agent code.
What does done look like in measurable terms, not vibes?
What happens when the agent cannot reach done?
Who or what has authority to make the final call?
If you cannot answer all three, you do not have an agent design. You have a loop waiting to go sideways.
Where Teams Actually Get Into Trouble
The failure mode I see most often is what I call confidence laundering. The agent produces output. The output looks polished and complete. A human skims it, sees no obvious errors, and approves it. The agent’s confident tone did the heavy lifting that actual verification should have done.
This is especially dangerous in code generation, legal document drafting, and anything touching financial data. The cost of a wrong answer that sounds right is much higher than the cost of an uncertain answer that signals it needs review.
A termination condition that catches this is not complicated. It can be as simple as a mandatory diff review before any external action, or a confidence threshold below which the agent flags uncertainty explicitly rather than papering over it. The mechanism matters less than the decision to build one deliberately.
What I Would Do Differently
If I were starting a new agentic system today, termination logic would be in the design doc before anything else. Specifically: what is the measurable exit criterion, what is the fallback when that criterion is not met within N steps, and where does human judgment re-enter the loop.
The teams that get this right tend to ship agents that are actually useful in production. The teams that get it wrong ship demos that work great until the first edge case, then require a human to babysit every run anyway.
The unsexy truth is that a well-defined stopping condition is worth more than a better model.
Sources & Further Reading
- Karpathy autoresearch repo announcement | https://x.com/karpathy/status/2030371219518931079
2. OpenAI Codex Security agent announcement | https://x.com/OpenAIDevs/status/2029983809652035758
#AIEngineering #AgenticAI #MachineLearning #SoftwareArchitecture #LLM
