Contrarian take on ‘assumption debt’ in AI product building, the silent killer that optimizes for the wrong objective
Assumption Debt: The Silent Killer of AI Products
Most AI products don’t fail because the model was bad. They fail because the team spent six months optimizing for the wrong thing, and by the time anyone noticed, the cost of changing course felt unbearable. I’ve watched this happen repeatedly, and I’ve done it myself. The technical debt gets all the press, but assumption debt is what actually kills products.
What Assumption Debt Actually Is
Assumption debt accumulates the moment you make an early call about what your user wants and then build everything downstream of that call without questioning it again. Unlike technical debt, it’s invisible. Your metrics look clean. Your eval suite is green. Your pipeline hums. The system is performing brilliantly, just on the wrong objective.
Here’s a classic example. You’re building a recommendation engine. Click-through rate is measurable, so you optimize for it. Six months later, users are clicking but not converting, not returning, not satisfied. The model learned to generate curiosity-bait, not value. You didn’t build a bad recommendation system. You built a perfect one for a goal nobody actually cared about.
The assumption was that engagement equals value. That’s a reasonable first guess. It’s also wrong in most real product contexts.
Why AI Makes This Worse Than Traditional Software
In traditional software, a bad assumption usually surfaces quickly because the feature either works or it doesn’t. In AI systems, the feedback loop is softer. The model adjusts. It finds patterns in your proxy metric and exploits them. It gets better at the thing you measured rather than the thing you wanted. And because the numbers improve, the assumption never gets challenged.
This is sometimes called Goodhart’s Law, and it’s been documented extensively in ML research. When a measure becomes a target, it ceases to be a good measure. The model is just doing what you told it to do.
The Moment You Should Be Most Worried
The dangerous moment isn’t when your product is struggling. It’s when everything is going well. Strong early metrics create organizational momentum. Engineers optimize the existing system rather than questioning the objective. Product managers point to the dashboard numbers. The assumption gets baked deeper and deeper into the architecture.
By month six, you have thousands of lines of code, a full data pipeline, and an eval suite, all built around a belief nobody ever verified with real users under real conditions.
How to Actually Fight This
First, write your assumptions down before you write any code. Not your requirements, your assumptions. What do you believe users want? What behavior are you treating as a proxy for success? What would it look like if you were wrong? This sounds simple. Almost nobody does it consistently.
Second, treat your objective function as a hypothesis, not a decision. The moment you pick a metric to optimize, you should be scheduling a review to challenge it. I’d suggest doing this aggressively early, before you have sunk cost pulling you toward defending the original choice.
Third, talk to users about outcomes, not features. Don’t ask if they like the recommendations. Ask what they did after they followed one. Ask if they came back. Ask what they wish the system had done differently. The distance between what users say in a feature interview and what they actually need from a working system is enormous.
The Harder Truth
The reason assumption debt accumulates isn’t laziness or bad engineering. It’s that changing your objective function mid-build is genuinely painful. You have to throw out evals. You might have to restructure training data. You have to have uncomfortable conversations about sunk cost. Most teams choose not to. Most products pay for that choice eventually.
The best AI builders I know treat the objective function as the most fragile part of the system, not the model weights, not the architecture. Because you can retrain a model in a week. Rebuilding organizational alignment around a new objective takes much longer.
If your product is performing well right now, I’d ask one question before celebrating: performing well at what, exactly, and did you ever prove that thing is what the user actually needed?
That question is either comforting or terrifying. The answer tells you a lot.
Sources
#AIProducts #MachineLearning #ProductStrategy #MLEngineering #AIEngineering #TechLeadership
