How to Test LLM Features Before They Embarrass You (2026 Playbook)
Unit tests do not catch hallucinations. Here is the evaluation stack we put behind every AI feature we ship.
Every AI feature that has embarrassed a company in public passed its code review. The failure was never in the code — it was in the absence of an evaluation harness.
Start with 50 examples, not a platform
Collect 50 real inputs with the output you would accept. That is your golden set. It takes an afternoon and catches more than any tool you will buy in month one. Grow it every time production surprises you — a bug report becomes a test case, permanently.
Three layers of checks
- Deterministic assertions. Valid JSON, required fields, enum membership, no PII leakage, length bounds. Cheap, fast, catches most breakage.
- Reference comparison. For extraction and classification, exact or fuzzy match against the labelled answer. Report precision and recall, not a vague pass rate.
- LLM-as-judge. For open-ended output only. Give the judge a rubric with explicit scoring criteria and few-shot examples, and audit its verdicts against human labels monthly.
Put a gate in CI
Run the golden set on every prompt or model change. Block the merge if accuracy drops more than a threshold you agreed on. A prompt is production code; treat an untested prompt edit the same way you would treat an untested schema migration.
Watch cost and latency as first-class metrics
An eval run should report p50/p95 latency and cost per task alongside accuracy. A prompt change that raises quality 2% and cost 60% is usually a bad trade, and you will only notice if the harness measures it.
Production monitoring
- Sample 1–5% of live traffic and score it with the same rubric.
- Track refusal rate, empty output rate, and tool-error rate — they move before accuracy does.
- Alert on distribution shift in input length and language, the two most common causes of silent quality collapse.
Model upgrades
Never swap a model because a benchmark says it is better. Run your golden set, compare cost and latency, and ship behind a flag with a 10% rollout. Benchmarks measure the average task; you only care about yours.
FAQ
Frequently asked questions
How many test cases do I need?+
Fifty real examples gets you most of the value. Two hundred covers the long tail for a mature feature.
Is LLM-as-judge reliable?+
With an explicit rubric it correlates well with human raters on relative comparisons. Audit it against human labels regularly and never use it as the only signal.
Where should evals run?+
In CI on every prompt or model change, plus a sampled scoring job against production traffic.
Building something similar?
Let's talk in 30 minutes.