AI Evals for Production Apps, Explained Without the Research Papers
How to know your AI feature is actually working, and how to know when it stops. A working setup for teams that don't have an ML research team.
The version of "AI evals" you'll find in research papers isn't the version you need to ship a product. Papers care about benchmark scores. You care about whether the feature works well enough that customers don't churn, and whether it's still working six weeks after launch.
This post is the working version of AI evals for a small team shipping a real product. Not academic. Practical. What to measure, how to build the eval set, and how to catch regressions before customers do.
Why AI needs evals in the first place
Traditional code has tests. You change a function, run the tests, and know if it works. AI features are different. The model is a black box, the prompt is fragile, and small changes can silently break behavior in ways unit tests don't catch.
Evals are the AI version of tests. They measure whether a system produces the right output on inputs you care about. Without them, you're shipping AI features and hoping.
What to actually measure
The trap is measuring what's easy instead of what matters. Perplexity scores are easy. Whether the AI feature actually helps a user is not, but it's what counts.
Task-level metrics
Does the output accomplish the task. Was the email actually summarized correctly? Did the classifier put the ticket in the right bucket? Did the extraction pull the right fields from the document?
Task-level metrics require human judgment on a sample, and that's fine. You don't need to eval every request — you need to eval a representative set.
Constraint metrics
Did the output follow the rules. Did it respect the format? Did it stay within the character limit? Did it avoid making up citations?
Constraints are easier to measure automatically than task quality. Length checks, JSON schema validation, regex for prohibited content — all cheap to run on every request in production.
User-signal metrics
What did the user do after the AI responded. Did they accept it, edit it, redo it, or bounce? User signals are the ground truth for whether the AI is actually useful. Every AI feature should collect them from day one.
Our post on AI observability covers the instrumentation for this.
Building an eval set that isn't a joke
The single biggest determinant of whether your evals are useful is the quality of the eval set. Bad eval sets create false confidence.
Start with real production data
Not synthetic examples. Real inputs from actual users. Sanitize PII, of course. Our post on PII handling covers how.
Fifty to two hundred real examples is enough to start. It's not a research benchmark — it's a signal detector. You need enough to catch regressions, not enough to publish a paper.
Include the hard cases
Every AI feature has edge cases where it fails. Weird inputs, ambiguous phrasing, adversarial users. Include them in the eval set explicitly. If you only test on well-behaved input, you'll ship a system that breaks on real users.
Include cases where the correct answer is "no answer"
A common failure mode: the model always answers, even when it shouldn't. Test cases where the correct response is "I don't have enough information" or "this isn't something I can help with." Otherwise you're rewarding hallucination.
Update the eval set continuously
When users hit a case that breaks, add that case to the eval set. When a customer complains about a specific kind of response, add examples of both the bad response and the desired one. The eval set is a living document, not a snapshot.
Running evals — automated where possible, human where it isn't
Three flavors of eval, in order of cost.
Rule-based checks
Assertions that run automatically. "Output must be valid JSON." "Response must be under 500 characters." "Output must not contain the substring 'as an AI language model.'"
Cheap, fast, run on every commit. First line of defense against regressions.
LLM-as-judge
A second model evaluates the first model's output. "Given this input and this response, is the response helpful, correct, and on-topic?" The judge model is often a stronger, more expensive model — you're spending inference cost to grade cheaper inference.
Not perfect — LLM judges have their own biases, especially favoring longer or more polite responses. But calibrated against a small human-graded set, LLM judges give you scalable evaluation for the ambiguous cases rule-based checks can't cover.
Human evaluation
An actual person grades a sample. Most expensive, most trustworthy. Reserve for the metrics that matter most and for calibrating your LLM judge.
Twenty examples graded by a human every two weeks is enough for many teams. Not glamorous. Effective.
The setup that catches regressions before customers
The workflow that works in practice.
- An eval set of 50-200 real, sanitized production examples with known good outputs.
- A CI job that runs the eval set against any prompt or model change and reports the delta.
- A weekly human review of 20 recent production requests, with edits captured back into the eval set.
- Rule-based constraint checks on every production request in real time.
- A dashboard tracking user-signal metrics — acceptance rate, edit rate, retry rate — over time.
Total setup: a couple of engineering weeks. Ongoing maintenance: a few hours a week. Worth it the first time a silent model regression would have shipped without it.
What triggers a real problem
Not every eval score drop is a regression. Some are noise. Distinguishing which is which is skill.
Real regressions look like
- A meaningful drop on the eval set that persists across runs.
- A pattern of failures in a specific category of input.
- A jump in the edit rate or retry rate in production.
- User complaints citing specific behaviors.
Noise looks like
- A small drop that recovers on the next run.
- A change in a single edge case.
- Slightly different phrasing on the same essentially-correct answer.
The eval set should be large enough that noise averages out but small enough to run frequently. 100 examples is a good balance for most teams.
Common misconceptions
"Evals are only for research teams." Every AI feature in production needs evals, whether or not it's called that. If you can't measure quality, you can't defend it against regressions.
"We can just use a benchmark." Public benchmarks measure general model capabilities. They tell you almost nothing about how your specific feature works on your specific data. Build a task-specific eval set.
"LLM-as-judge is unreliable." On its own, yes. Calibrated against a small human-graded set, and used for the right metrics, it's a reasonable scalable option. Not perfect. Good enough for most product decisions.
Trade-offs
Evals cost time to build and maintain. That cost is real. The alternative — shipping without evals and finding out about regressions from customers — is more expensive.
Automated evals miss subtle quality issues that humans catch. Human evals don't scale. The right mix depends on the stakes. High-stakes features get more human review. Low-stakes ones lean on automation.
Evals also can't cover everything. A model can pass every eval and still feel wrong to users in ways your eval set didn't capture. Real user signals — acceptance rate, edit rate — are the ground truth. Evals are the leading indicator.
How this fits with prompt changes
The workflow for changing a prompt or swapping a model:
- Make the change on a branch.
- Run the eval set. Note the delta.
- If numbers dropped in any category, investigate. Fix or revert.
- If numbers held or improved, merge and deploy behind a flag.
- Monitor production user signals for the next week.
- If production signals hold, remove the flag.
This is what "prompt engineering as engineering" actually looks like. Not vibes. A process. Our prompt ops post covers the surrounding infrastructure.
Frequently asked questions
Final note
Evals aren't glamorous. They don't make the demo better; they make sure the demo still works six months from now. Most AI features fail not because they were built badly but because they silently drifted and nobody was watching.
If you're shipping an AI feature and want a review of your eval strategy before launch, book a call. We'll walk through the setup and tell you what's missing.
FAQ
Frequently asked questions
What are AI evals?+
Evals are the AI version of tests — a set of representative inputs with known good outputs, used to measure whether a system produces the right output when prompts or models change. They exist to catch regressions that unit tests can't detect and to give teams confidence in shipping AI changes.
How big should an eval set be?+
For most product teams, fifty to two hundred real, sanitized production examples is enough. Small enough to run frequently and grade by humans occasionally, large enough that random noise averages out. Public benchmarks are much larger but measure the wrong thing for a specific product.
What is LLM-as-judge and does it work?+
LLM-as-judge uses a second, usually stronger model to evaluate the output of the primary model. Calibrated against a small human-graded set, it's a reasonable scalable option for ambiguous metrics that rule-based checks can't cover. It has known biases toward longer and more polite outputs, so calibration matters.
How do I catch AI regressions before customers do?+
Combine an eval set that runs in CI, rule-based constraint checks on every production request, and a dashboard of user-signal metrics like acceptance and edit rate. When any of the three shifts meaningfully, investigate. This is the same discipline as monitoring any production system, applied to AI-specific failure modes.
Do I need a research team to build evals?+
No. A product team can build a working eval setup in a couple of engineering weeks. The core skills are picking the right metrics, curating a real production sample, and running the set on every change. Academic evaluation techniques are unnecessary for most product decisions.
Building something similar?
Let's talk in 30 minutes.