The Black Box Problem: Why LLM Internals Should Change How You Build Most engineers treating LLMs like black boxes are building on sand. That sounds harsh, but I’ve watched it happen enough times to say it plainly. The system works, demos great, ships to production, and then fails in ways that feel random but absolutely…
Context Windows Are Not Scratch Pads Most engineers I talk to treat a 200K token context window like a gift. More space means more context, more context means better results, better results mean ship it. That logic feels intuitive. It is also how you end up with a production agent that confidently contradicts its own…
The hidden cost nobody talks about when building AI agents. It’s not compute. It’s not the API bill. It’s decision fatigue at the architecture level. Every week I talk to engineers who are drowning in choices before they’ve written a single line of agent logic. Which orchestration framework. Which memory layer. Whether to use tool-calling…
The RAG Problem Nobody Wants to Admit Every company I talk to is convinced that dumping more documents into their vector store makes their AI smarter. It doesn’t. Past a certain point, it makes things worse. And the teams building these systems are skipping the one engineering step that would tell them that. This is…
The RAG Problem Nobody Wants to Admit Every company I talk to is convinced that dumping more documents into their vector store makes their AI smarter. It doesn’t. Past a certain point, it actively makes things worse. And the engineering teams building these systems either don’t know it yet, or they know it and nobody…
The RAG Problem Nobody Wants to Admit Every company I talk to is convinced that dumping more documents into their vector store makes their AI smarter. It doesn’t. Past a certain point, it actively makes things worse. The math here is not subtle, and the engineering community is mostly pretending it isn’t happening. Let me…
Inference Cost Is an Architecture Problem Most AI engineers I know have never seriously thought about inference cost until it destroyed their unit economics in production. I’ve watched it happen more times than I’d like to admit. The pattern is always the same: weeks of benchmarking, heated Slack debates about GPT-4o versus Claude versus Gemini,…
Gemini 2.5 Pro Just Changed What “Usable Context” Means The AI coding race has a new front-runner, at least this week, and it’s worth paying attention to why. Google shipped Gemini 2.5 Pro recently, and the benchmark numbers are genuinely hard to dismiss. It’s sitting at the top of the LMSys leaderboard for coding tasks,…
Google just shipped Gemini 2.5 Pro, and the benchmark numbers are hard to ignore. It’s sitting at the top of the LMSys leaderboard for coding tasks, outperforming GPT-4o and Claude 3.7 Sonnet on several software engineering benchmarks. On SWE-bench Verified, it’s hitting numbers that weren’t realistic from any model twelve months ago. But here’s what…
Plausible Is Not the Same as Correct There is a distinction circulating in engineering circles right now that I think deserves more attention than it’s getting. It comes from a piece by @KatanaLarp that’s been making rounds on X, and the framing is sharp enough that I want to expand on it here. LLMs don’t…
