How to Ship AI Features Safely Without Breaking Production

Traditional feature rollout doesn't cut it for AI. Prompts change model behavior in unpredictable ways. Here's the deployment discipline we use to ship LLM features every week without waking anyone up at 3am.

The 5-layer safety stack

1. Eval gate — CI runs a golden set of 100–500 tasks. PR fails if quality drops.
2. Feature flag — every AI surface behind a per-tenant flag (LaunchDarkly, PostHog, Statsig).
3. Staged rollout — 1% → 5% → 25% → 100% over 5–10 days.
4. Kill switch — flip a single env var to disable the AI path and fall back to the pre-AI UI.
5. Cost cap — hard ceiling per tenant per day.

What "eval gate" actually means

A CI job runs your prompt against a fixed dataset of inputs with graded expected outputs. LLM-as-judge grades most tasks, humans review the borderline ones. If pass rate drops more than 2 points, the PR blocks.

Kill switch pattern

if (env.AI_KILL_SWITCH === "1") {
  return legacyClassifier(input);
}
return await aiClassifier(input);

Every AI path has a non-AI fallback. If you can't ship a fallback, you can't safely ship the feature.

Observability minimums

• Log every prompt + completion (sampled if volume is huge).
• Track latency p50/p95/p99 per prompt version.
• Track cost per request, per user, per tenant.
• Alert on 5xx from the model provider > 1% for 5 minutes.

The rollout schedule we default to

1. Day 0: internal team only.
2. Day 2: 1% of prod traffic.
3. Day 4: 5%.
4. Day 7: 25% + human eval sample.
5. Day 10: 100% if metrics hold.

Prompt versioning

Treat prompts like code. Version them, code-review them, and pin every prod deploy to a specific prompt hash. "Someone edited the prompt in the dashboard" should never be a valid postmortem.

How we help

Our AI Engineering engagements ship this stack as table stakes — evals, flags, observability, and rollback from day one.