Load Testing an AI API Before Launch, So It Doesn't Fold on Day One
The pre-launch step every AI product should do and most skip. A practical setup for teams without a dedicated performance engineer.
The moment an AI product hits Hacker News, five things happen in order. Traffic spikes. Model provider rate limits kick in. Timeouts pile up. Users see broken responses. And the founder learns, in public, what happens when nobody load-tested the API.
Load testing an AI API is different from load testing a normal web service. The costs are higher, the timing is weirder, and the failure modes are unusual. This post is the setup we use for teams launching an AI product and can't afford a performance engineer full-time.
Why AI APIs need special attention
Standard web services fail predictably under load: CPU pegs, memory grows, response times climb. AI APIs fail in messier ways.
Requests are much slower than normal — a single call can take five to thirty seconds. Concurrency limits at the model provider hit long before your own servers strain. And a single retry storm from clients can make a small blip into a cascading outage.
The result: an AI API that handles 1,000 requests per hour comfortably might collapse at 200 requests per minute, because the concurrency shape is different from raw volume.
What to measure
Four numbers matter more than the rest.
1. Time to first token
If your product streams responses, users see the first token in a fraction of the total response time. Under load, this is the number that changes first — total time might still be acceptable while first-token times climb into "the product feels broken" territory.
2. Total response time at the 95th percentile
Average response time is a lie. The 95th percentile — the slowest 5% — is what users complain about. On social media. Loudly. If your p95 is more than 2x your average, you have a variance problem before you even start scaling.
3. Error rate by cause
Not just "did requests fail" but why. Was it a model provider rate limit? A timeout? Your own database? Each has a different fix. A dashboard that only shows "5% of requests failed" is useless during a launch.
4. Concurrent request ceiling
The number of simultaneous in-flight requests your system can hold before it starts dropping. This is usually much lower than founders expect, especially when the model provider is the bottleneck.
How to simulate real AI traffic
Standard load-testing tools generate artificial traffic that doesn't look like real AI users. AI traffic has specific characteristics that matter.
Real traffic is bursty
Users don't send requests at a steady rate. They come in clusters — a Product Hunt spike, a Reddit thread, a demo. Your load test should include bursts, not just steady state. Ramp from 0 to 500 concurrent users over 30 seconds and see what breaks.
Real prompts vary in size
A 200-token prompt and a 5,000-token prompt have very different costs and latencies. If your test uses one fixed prompt, you're not testing your actual workload. Use a distribution of realistic prompt sizes drawn from staging or early usage.
Real users retry
When a request fails, clients retry. When retries fail, they retry again. This is where retry storms come from. A load test without retry behavior will underestimate the real failure cascade.
Tooling that works for small teams
You don't need enterprise tooling for this. A couple of open-source tools cover the basics.
k6 is the modern default for HTTP load testing. It runs from a laptop, scripts in JavaScript, and handles streaming responses reasonably well. Artillery is another option with similar capabilities.
For AI-specific testing, we usually write a small Python or Node script that pulls sample prompts from a JSONL file and hits the API with a configurable concurrency level. It's not fancy, but it lets you swap prompt distributions and retry logic in a few lines.
For observability during the test, whatever you're using in production works — Datadog, Grafana, or the model provider's built-in dashboards. The point is to have a real-time view of both your system and the model provider's response.
The failure modes we see every launch
These are the ones that show up in the first hour after a real spike.
Model provider rate limits
Every provider has request-per-minute and tokens-per-minute limits per API key. Under a launch spike, you'll hit them fast. The fix isn't just "get a higher tier" — it's a combination of tier upgrade, backoff logic, and a queue in front of the model that absorbs the spike.
Our post on rate limiting goes into detail on the queue design.
Database contention
AI APIs often write more per request than typical APIs — logging the prompt, the response, the tokens used, the model version. Under load, this write volume can exceed what a small Postgres instance handles. The fix is batching or moving the writes to a background queue.
Serverless cold starts
If your API runs on serverless, the first request after idle has a cold-start penalty. Under a spike, dozens of new instances spin up at once, each paying that penalty. The fix is provisioned concurrency, warm pings, or moving to always-warm instances.
Client-side retry storms
The one that gets people. When your API returns 500s, clients retry, often aggressively. Retries pile on top of the original spike and take down anything that survived. The fix is exponential backoff with jitter, on both client and server.
A pre-launch load test that actually works
The sequence we run before an AI product launch.
- Build a script that hits the production API with a realistic prompt distribution.
- Start at 10 concurrent users. Verify latency and error rate at that level.
- Ramp to 50. Watch the model provider dashboard.
- Ramp to expected launch peak, times two. This is your safety margin.
- Introduce a 60-second burst that hits 4x expected peak.
- Do it again with retry logic that mimics real clients.
- Record the failure modes. Fix the top three. Repeat.
The goal isn't to prove the system is invincible. The goal is to know exactly where and how it breaks, so you can decide which failures are acceptable and which aren't.
Trade-offs to expect
Every load-test finding presents a choice. Some are cheap, some aren't.
You might learn that your API can only handle 40 concurrent requests. Options: pay for a higher model provider tier ($$), add a queue and accept slower launches ($), or cap sign-ups at launch to keep quality high (free but slower growth).
You might learn that first-token latency is fine but total latency is slow. Options: switch to streaming responses (engineering week), pick a faster model (quality trade-off), or set user expectations in the UI ("this may take a moment").
None of these decisions is right or wrong in isolation. They depend on your buyer, your quality bar, and your budget.
Common misconceptions
"We'll test it when we have real traffic." Real traffic is where you learn about failures that make the news, not the ones you can quietly fix. Load-test before you're the story.
"Our cloud auto-scales." Auto-scaling helps with your infrastructure. It doesn't help with the model provider's rate limits, which are usually the actual ceiling.
"We can't afford enterprise load testing." A one-day setup with k6 and a laptop covers 80% of what you need. This isn't a budget problem.
Frequently asked questions
Final note
An AI product's launch day is when a lot of decisions get tested at once — infrastructure, provider limits, retry logic, and UI expectations. Load testing is the cheapest way to find out which decisions were wrong before customers do.
If you're launching an AI product and want a pre-launch review of your architecture and load profile, book a call. We'll run through the shape and tell you what breaks first.
FAQ
Frequently asked questions
How do you load-test an AI API?+
Write a script that sends realistic prompts at controlled concurrency, ramp from low volume up through expected launch peak plus a safety margin, and include bursts and retries to simulate real user behavior. k6 or a small Python script covers most of what small teams need without dedicated tooling.
What are the most common failure modes for an AI API under load?+
Model provider rate limits, database contention from heavy per-request logging, serverless cold starts under spike, and client-side retry storms cascading into full outages. Each has a distinct fix; the load test's job is to tell you which one hits first.
What metrics matter most in an AI load test?+
Time to first token, 95th-percentile total response time, error rate broken down by cause, and concurrent request ceiling. Average latency alone is misleading. The variance and the failure attribution are what tell you where to spend engineering time.
Do I need to worry about model provider rate limits?+
Yes, and usually before you need to worry about your own servers. Every managed provider has per-minute request and token limits per key. Under a launch spike, those limits hit first. The fix combines a higher tier, backoff logic, and a queue to smooth the burst.
When should I load-test my AI product?+
Two weeks before a public launch, and again the day before. Two weeks is enough time to fix the top failure modes; the day-before test is to verify the fixes held. Testing on launch day is testing on your users.
Building something similar?
Let's talk in 30 minutes.