The Technical Debt Unique to AI Products (And How to Pay It Down)
Prompts in string literals, no evals, three model providers, and a vector index nobody can rebuild. Sound familiar?
AI features rot differently from normal code. Nothing breaks loudly. Quality just drifts downward until a customer notices before you do.
Debt 1: prompts scattered across the codebase
Prompts inlined in components and server functions cannot be versioned, diffed, or A/B tested. Move every prompt to a single module with a version tag, and log which version produced each output. Half of all "why did it answer that?" investigations end the moment you can answer "which prompt ran?".
Debt 2: no evaluation harness
Without a golden set, every prompt change is a guess. This is the most expensive debt on the list because it compounds — it makes paying down every other item unsafe. Fix it first, even if the harness is 50 examples and a script.
Debt 3: provider coupling
Provider-specific SDK calls sprinkled through the app make model migration a rewrite. Wrap inference in one internal interface: input, schema, model tier, budget. Then swapping providers is a config change and a golden-set run.
Debt 4: index drift
Your vector index was built by a script somebody ran once, with a chunking strategy nobody documented, using an embedding model that has since been deprecated. Make reindexing a repeatable job with the chunker and model version recorded in metadata, and run it on a schedule so it never becomes scary.
Debt 5: unbounded cost paths
An agent loop without a step cap, a retry without a ceiling, a context window that grows with conversation length forever. Put hard budget caps at the inference wrapper, not in each call site.
Debt 6: no output logging
If you do not store inputs, outputs, model version, and cost per call, you cannot debug, cannot evaluate, and cannot forecast spend. Log them with a retention window that respects your privacy policy.
Paydown order
- Eval harness — makes everything else safe.
- Output logging — makes everything else measurable.
- Prompt registry.
- Inference wrapper with budget caps.
- Reproducible reindexing.
That sequence takes a competent team about two weeks and removes most of the fear from touching AI code.
FAQ
Frequently asked questions
Do I need a prompt management platform?+
Not early. A versioned module in your repo with logged version ids covers most teams until you have non-engineers editing prompts.
How often should I reindex?+
Incrementally on source change, and a full rebuild whenever the embedding model or chunking strategy changes.
Is multi-provider support worth the effort?+
The abstraction is, on day one. Actually running two providers in production usually is not, until you need failover or pricing leverage.
Building something similar?
Let's talk in 30 minutes.