LLM Evals Are an Engineering Discipline. Start Treating Them Like One. There’s a moment I’ve seen play out on too many AI teams. The demo works. The vibes are good. Someone says “ship it.” And everything is fine, right up until it isn’t. Users hit edge cases nobody tested. Outputs drift. Trust erodes fast and…
The Black Box Problem: Why LLM Internals Should Change How You Build Most engineers treating LLMs like black boxes are building on sand. That sounds harsh, but I’ve watched it happen enough times to say it plainly. The system works, demos great, ships to production, and then fails in ways that feel random but absolutely…
The Real Skill Gap in AI Engineering (And It’s Not Prompting) Everyone is learning to prompt. Courses, threads, workshops, a 24-minute video from Anthropic’s applied AI team promising to teach you “the 6 elements” you’ve been missing. The tutorials are multiplying faster than anyone can watch them. And I think most of them are solving…
Context Windows Are Not Scratch Pads Most engineers I talk to treat a 200K token context window like a gift. More space means more context, more context means better results, better results mean ship it. That logic feels intuitive. It is also how you end up with a production agent that confidently contradicts its own…
Stop Building Agents. Start Packaging Knowledge. I’ve been thinking about something Barry Zhang and Mahesh Murag said at the AI Engineer Code Summit, and I can’t shake it. Sixteen minutes. Two Anthropic engineers. One idea that cuts through most of the noise in agentic AI right now. “Stop building agents. Build Skills instead.” Most people…
The Real Reason Your AI System Is Broken (It’s Not the Model) Every week I watch engineers spend hours debating which model to use. GPT-4o versus Claude 3.5 versus Gemini 1.5 Pro. Benchmark comparisons, pricing calculators, latency tests. It’s not that those things don’t matter. They do, at the margins. But in my experience shipping…
Gemini 2.5 Pro Just Changed What “Usable Context” Means The AI coding race has a new front-runner, at least this week, and it’s worth paying attention to why. Google shipped Gemini 2.5 Pro recently, and the benchmark numbers are genuinely hard to dismiss. It’s sitting at the top of the LMSys leaderboard for coding tasks,…
Google just shipped Gemini 2.5 Pro, and the benchmark numbers are hard to ignore. It’s sitting at the top of the LMSys leaderboard for coding tasks, outperforming GPT-4o and Claude 3.7 Sonnet on several software engineering benchmarks. On SWE-bench Verified, it’s hitting numbers that weren’t realistic from any model twelve months ago. But here’s what…
Plausible Is Not the Same as Correct There is a distinction circulating in engineering circles right now that I think deserves more attention than it’s getting. It comes from a piece by @KatanaLarp that’s been making rounds on X, and the framing is sharp enough that I want to expand on it here. LLMs don’t…
Google just shipped Gemini 2.5 Pro, and the benchmark numbers are hard to ignore. It’s sitting at the top of the LMSys leaderboard for coding tasks, outperforming GPT-4o and Claude 3.7 Sonnet on several software engineering benchmarks. On SWE-bench Verified, it’s hitting numbers that weren’t realistic from any model twelve months ago. But here’s what…
