Handling AI Provider Outages in Production

Every major AI provider has had public outages. OpenAI, Anthropic, Google — none of them are exempt. If your AI feature is on the critical path, its uptime is at best equal to your provider's, and usually worse because of your own bugs on top.

This post is the working plan for handling AI provider outages in production. Not a heroic multi-cloud architecture. Just the small set of decisions that keep users from noticing when the provider hiccups for six minutes on a Wednesday afternoon.

What "outage" actually looks like

Providers rarely hard-fail. They soft-fail. Requests hang for 30 seconds. Latency doubles. A subset of requests return 500s. The rate limit tightens without a status-page update. The model returns garbage for reasons you can't reproduce.

Designing for outages means designing for the messy version, not the "provider is completely down" version. Most incidents are partial.

The first line of defense is timeouts

The single most useful thing to configure is a request timeout. Without one, a hanging LLM request holds your worker, holds the user's tab open, and cascades into a bad time.

Sensible defaults: 30 seconds for interactive requests, 60 seconds for batch. Anything longer and you're probably in the "should be async" category anyway. Tune down from there based on your feature's tolerance.

Timeouts should be enforced both client-side (the fetch call) and server-side (the request handler). One catches network hangs, the other catches upstream stalls.

Retries, done carefully

Not every failure is worth retrying. A 429 (rate limit) with a Retry-After header — retry after that delay. A 500 or 502 — retry with backoff, once or twice. A 400 (bad request) — do not retry, you have a bug.

Retry loops without a ceiling turn a small outage into a self-inflicted DDoS on your own bill. Cap retries at two or three. Add jitter to the backoff. Log every retry so you can see the pattern in metrics.

Circuit breakers

If the provider has returned errors for the last 20 requests, the 21st is unlikely to succeed. A circuit breaker fails fast for a short window, then tries a single probe, then reopens if the probe succeeds.

The value is protecting the rest of your system. When the LLM path is slow, everything else queued behind it slows too. Circuit-breaking the LLM path frees the queue for the endpoints that still work.

Fallback strategies

Second provider

Route to Anthropic if OpenAI is down (or vice versa). This sounds elegant and is fine for stateless calls with similar prompts. It gets complicated for anything that depends on model-specific behavior — different prompt style, different structured output support, different token counts.

A working middle ground: route your critical, prompt-simple calls (classification, extraction, embedding) across providers. Keep chat and complex flows on their primary and accept the risk.

Cached response

If the request is one you've seen recently, return the cached response instead of an error. This is not a real answer to novel questions but it's the right move for high-repeat features.

Rule-based fallback

For classifications: fall back to a deterministic rules engine that handles the common cases. For search: fall back to keyword search. For summaries: fall back to a truncated version of the input. Users get something instead of nothing.

Graceful degradation in the UI

The AI-powered draft button becomes "We're having trouble generating a draft. Try again in a minute." Not an error toast. Not a spinner forever. Not silently broken. An honest, small message that reflects what actually happened.

Where teams get outage handling wrong

Assuming the provider is up

The happy path in code. No timeouts, no retries, no fallback. Works fine until the day it doesn't.

Retrying too aggressively

Exponential retry with no ceiling. Provider recovers and gets hammered by all your queued retries at once. Rate limit re-triggers. You're now part of the incident.

Failing loud when you should fail quiet

A stack trace in the UI is worse than a small "try again" message. Users don't need to know about the provider.

Not tracking incidents in your own metrics

Provider status pages lag by minutes to hours. Your own error rate is the first place you'll see an incident. Alert on your own rates, not on their status page.

The observability piece

Track per-provider latency and error rate as first-class metrics. When latency doubles, you want to know before users do. When error rate ticks up, you want a graph that answers "is this us or them?"

Every LLM call should log: provider, model, latency, token counts, status, and error class if any. Aggregate this into a dashboard you actually look at. Our AI observability post covers the instrumentation.

Async where possible

If the feature doesn't require instant response, don't make it require instant response. Move it behind a job queue. Retry the job. Notify the user when it's done.

Async fundamentally changes the outage story. A queue can hold work for 20 minutes and drain it once the provider recovers. Users see "processing…" instead of "error." That's a much easier UX to defend.

Mistakes teams make

Skipping status-code discipline. Every provider returns different codes. Read their error taxonomy once and handle each class deliberately.

No load testing. An outage often reveals your own capacity issues at the same time. Load test the fallback path, not just the happy path.

Fallback that's never exercised. A fallback that's never been tested is worse than no fallback. Run monthly game-days where you force a failure and confirm the fallback works.

No customer communication plan. When the outage is long, users notice. Have a status page or a support macro ready. Not on the day.

Trade-offs

Multi-provider adds engineering complexity. Aggressive fallback risks lower quality. Deep retries risk higher cost. Async breaks the illusion of instant AI.

The right posture depends on your product. A B2B tool users tolerate. A consumer chat product they don't. Match the resilience investment to the tolerance.

Common misconceptions

"Provider SLAs cover me." They cover credits, not user trust. Users don't care about SLA math.

"I'll add fallbacks when I hit scale." Outages happen at any scale. The first one always feels expensive.

"Multi-provider is the answer." It's an answer. Not always the right one. Async and caching often win by a wider margin.

Frequently asked questions

Should I use two providers by default?

Only where the calls are simple, stateless, and worth the extra complexity. Don't dual-provider a chat product on day one.

What's a good retry policy?

Two or three retries with exponential backoff and jitter. Respect Retry-After headers. Never retry 4xx errors.

How do I know if I'm rate-limited vs down?

Watch the status codes. 429 is rate limit. 5xx is provider trouble. 408/504 is timeout. Distinguish in metrics.

Should I show users the error?

Show a friendly "try again in a minute" message. Log the technical detail. Never expose stack traces.

Can I use my provider's status page instead of monitoring?

Not really. Status pages lag. Alert on your own error rate.

Conclusion

Handling AI provider outages is the boring reliability work that separates products that feel professional from products that feel fragile. Timeouts, honest retries, circuit breakers, thoughtful fallbacks, and clear customer messaging.

None of this is exotic. Most teams just haven't sat down to design it because the happy path is working. The good news is a small amount of deliberate work here pays off enormously the first time the provider has a bad day.

If you'd like a review of where your AI features are quietly single-pointed on a provider, the AI Audit covers this as part of the reliability layer.