Insight on context window management as an architectural discipline for agentic AI builders, prompted by Karpathy’s /compact observation
Context Windows Are an Architectural Problem, Not a Prompt Problem
Most builders treat context windows the way they treat browser tabs. Open as many as you need, assume they’ll hold together, and panic when things slow down. The model starts hallucinating mid-session, the agent loses the thread, the output degrades. And the fix is usually to restart or cram in a better system prompt. Neither of those is a solution. They’re avoidance.
Karpathy posted something this week that I haven’t been able to stop thinking about. He mentioned that when he informs a model he’s about to /compact its context, the model is starting to “get” what that means. His framing: “its self-awareness gradually builds up and falls out from pretraining on tokens of us talking about them, but it’s laggy and a bit incomplete.” That’s a throwaway observation from a late-night session. It also points at something most agentic builders have completely missed.
The Boundary Problem
We spend serious engineering time optimizing what enters the context. Prompt design, memory retrieval, RAG pipelines, tool output formatting. All of it focused on the fill operation. Almost nobody has a deliberate strategy for what happens at the boundary, when the window is saturated and the session has to continue anyway.
The naive approaches are a sliding window that drops old tokens, a hard reset that drops all state, or just hoping the model muddles through. None of those are architectural choices. They’re defaults. And in agentic systems, those defaults will wreck you. An agent mid-task doesn’t just need new tokens. It needs a coherent handoff of what it knew, what it decided, and what it was about to do.
Context Management as Discipline
Here’s my actual take. Context management is becoming a real engineering discipline, roughly equivalent to memory management in systems programming. Not glamorous, not the thing that gets demo’d at conferences, but load-bearing.
That means thinking about a few things explicitly. What is the minimum viable state representation that survives a compaction? If you can’t answer that, your agent is fragile by design. What signals tell you degradation is approaching before performance actually drops? Context length is a lagging indicator. You want earlier warnings. And how do you summarize work-in-progress in a way the model can actually reconstruct from, rather than a way that satisfies the human reader?
Karpathy’s /compact moment is interesting precisely because it suggests models can begin to participate in this process if they’ve seen enough tokens about it. That’s not magic. That’s the training distribution at work. Which means builders who document their compaction protocols well, inside their own agent traces and logs, may be teaching the next generation of models how to handle this more gracefully.
What Agentic Systems Actually Need
The longer I work with agentic pipelines, the more I think the field is underestimating session continuity as a core infrastructure problem. OpenAI is pushing Codex and multi-agent orchestration hard right now. Their Presence product for enterprise is explicitly about agents that “improve over time” across customer workflows. Google is launching model after model optimized for agentic tasks, including Gemini 3.6 Flash, explicitly marketed as better at “not getting stuck in loops.”
Getting stuck in loops is a context problem. It almost always is.
When a model loops, it has usually lost the plot of what it was doing. The working memory of the session has decayed or conflated. A better model helps, but it doesn’t fix the underlying architecture if you’re not managing the window deliberately.
What I’d Actually Build
If I were designing an agentic system from scratch today, context lifecycle management would be a first-class component, not an afterthought. I’d define explicit checkpoints where the agent produces a structured state summary before any compaction. I’d log those summaries and use them to evaluate drift over sessions. I’d treat the context boundary as an event in the system, not a failure condition.
The models are starting to understand what /compact means. The question is whether the builders are.
The real architectural work in agentic AI right now isn’t building better tools or smarter retrieval. It’s figuring out how to make systems that don’t lose their minds partway through a long job. That’s a harder problem than it looks, and almost nobody is talking about it directly.
Sources & Further Reading
#AIEngineering #AgenticAI #LLM #ContextWindow #MachineLearning #AIArchitecture
