Hot take: inference cost optimization is an architecture problem, not a model selection problem
Inference Cost Is an Architecture Problem
Most AI engineers I know have never seriously thought about inference cost until it destroyed their unit economics in production. I’ve watched it happen more times than I’d like to admit. The pattern is always the same: weeks of benchmarking, heated Slack debates about GPT-4o versus Claude versus Gemini, careful prompt engineering, then a launch that burns through budget in days because nobody asked the more important question first.
The model is usually the cheapest part of your system.
That sentence makes people uncomfortable, but it’s true. The real cost isn’t per-token pricing. It’s the accumulated weight of every architectural decision you made before the model ever saw a request.
Where the Real Leverage Is
The decisions that actually determine your inference bill happen upstream. Before you write a single line of model-calling code, you should be asking whether this request needs a call at all.
Semantic caching is the most underused lever in production AI systems right now. Exact-match caching is obvious and most teams do it. But if your application is answering structurally similar questions repeatedly, and most applications do exactly that, you’re paying for 400 identical inference calls when you could pay for one. Semantic caching catches near-duplicates by embedding the incoming query and checking it against a vector index of previous requests. The threshold tuning matters and takes some work, but the cost reduction on high-traffic endpoints can be dramatic.
The Model Routing Problem
Not every request needs a 70B+ frontier model. This sounds obvious, but teams almost never act on it. A well-tuned 7B or 13B model handling the 60% of your traffic that’s routine and well-defined will cost a fraction of routing everything to the flagship API. The hard part is building the classifier that decides which requests go where, and accepting that you need to instrument your production traffic first before you can make that call intelligently.
The teams that get this right think of their AI backend the way a database engineer thinks about query routing. You don’t run every query against your primary replica with full resources. You profile, you classify, you route.
Batching Is Not Optional
Synchronous, one-at-a-time inference is the default because it’s easy to implement and matches how humans think about requests. It’s also expensive. Batching requests where latency tolerance allows it can cut costs significantly, and for a lot of workloads, users don’t actually need sub-second responses. A document processing pipeline, a nightly report generator, an async research task, these all have room to batch. The question teams rarely ask is which of our use cases actually require real-time inference and which ones just got built that way because that was the path of least resistance.
Prompt Engineering Is Infrastructure
Your prompt length is a cost driver. A bloated system prompt that grew organically over six months of iteration, with examples and caveats and edge case handling piled on top of each other, is money leaving your account on every single call. Treating prompt optimization as a first-class engineering concern, with versioning, cost tracking, and regular audits, is something almost no team does until they’re forced to.
A 500-token reduction in your system prompt at 10 million daily calls is not a rounding error.
What This Means Going Forward
Model prices are dropping fast. OpenAI, Anthropic, and Google are all competing aggressively on per-token cost, and that trend will continue. But cheaper tokens don’t fix a wasteful architecture. They just mean you’re wasting less money per bad decision.
The teams building durable AI products are the ones who’ve internalized that inference optimization is a systems engineering discipline, not a model selection exercise. They have cost dashboards. They route by request complexity. They cache aggressively. They treat every token as a resource to be managed.
Pick your model last. Design your architecture first.
Sources & Further Reading
- vast.ai RTX 4090 inference pricing reference | https://x.com/vast_ai/status/2043151667357437973
#AIEngineering #MachineLearning #LLMOps #InferenceCost #AIArchitecture #MLEngineering
