Cutting LLM Inference Costs Without Making the Product Worse

The first month of an AI product is exciting. The second month is when the finance person forwards you the model bill and asks a very direct question.

We've watched enough of those emails to compile the playbook that actually moves the number. This is what we do at Augere Labs when a client's inference cost has quietly become their second-largest infrastructure line item.

Where LLM costs actually come from

Before optimizing, know what you're optimizing. On a typical RAG-based feature, cost breaks down roughly as:

• 50–70% — output tokens (usually the biggest lever)
• 20–35% — input tokens (system prompt + retrieval context)
• 5–15% — embeddings and reranking

Most teams optimize the wrong slice. They spend a week shortening the system prompt and save 4%. Meanwhile the output is a 900-token essay when the UI only shows the first 150. That's where the money is.

1. Model routing (biggest win, usually)

Not every request needs your best model. In practice, 60–80% of production calls are simple enough for a smaller, cheaper model.

A pattern we see work: a small classifier — either a cheap model or a rules-based check — decides whether the request goes to Haiku/Mini or to Sonnet/GPT-4o. Simple queries land on the cheap tier; complex ones escalate. Real cost drops of 40–65% on features we've retrofitted this into.

The trap: don't let the router itself become expensive. Keep it under 200 tokens per call, or use a real classifier model.

2. Cap output length ruthlessly

Every model provider lets you set max_tokens. Almost no team sets it aggressively enough. If the UI can only render 3 sentences, set the limit at 120 tokens. The model doesn't know how much space you have; you have to tell it.

Combined with an explicit instruction in the prompt ("Reply in under 3 sentences. No preamble."), we routinely cut output cost 30–50% with zero user-visible quality loss.

3. Prompt caching (free money if your provider supports it)

Anthropic, OpenAI, and Google all support prompt caching now. If your system prompt and retrieved context are stable across requests, you pay a fraction of the input cost after the first call.

Put your stable content first (system prompt, tool definitions, retrieved docs) and the volatile user turn last. Turn on caching. Expect 20–40% savings on input tokens for a chat product, more for high-repetition workflows like batch classification.

4. Retrieve smaller, rerank harder

Teams routinely stuff 20 chunks of 500 tokens each into every RAG call. That's 10,000 input tokens per request. The model uses maybe 3 of those chunks.

Better shape: retrieve 30–50 chunks with a fast vector search, rerank down to the top 5 with a cross-encoder or Cohere Rerank ($0.001 per call), send only those. Total input tokens drop from 10K to 2.5K. Quality goes up. See our vector DB post for the retrieval side.

5. Cache the answer, not just the prompt

For classification, tagging, categorization — anything with a bounded input space — cache the full response by input hash. First hit pays the model bill, every subsequent hit is a Redis lookup.

On support-ticket categorization for a client, we hit 43% cache hit rate after two weeks. That's 43% of the calls that never touched the model at all.

6. Batch when you can

Most providers offer batch APIs (asynchronous, 24h SLA) at roughly half price. If the workload isn't user-facing — nightly summarization, backfills, evals — put it on the batch endpoint. Half your bill on those workloads goes away.

Teams forget this exists. It exists.

7. Streaming is a cost tool, not just a UX one

When you stream and let the user cancel, some percentage of requests end early. On features where users interrupt frequently (agent tasks, long generations), you pay only for tokens generated up to the abort. Real savings: 5–15% depending on the feature.

Combined with a UI that surfaces a stop button, this compounds nicely with cap-output-length.

8. Fine-tune only when the math works

Fine-tuning cuts per-request cost only if two things are true: request volume is high (>50K/month on the same task), and the task is narrow enough that a smaller model + fine-tune matches your current model on the eval set.

When both are true, fine-tuning can cut cost 3–10×. When either is false, you'll spend more on training than you save. Our RAG vs fine-tuning guide covers when the math flips.

9. Kill the multi-turn agent loops

Agent frameworks that "think step by step" for 8 turns to answer a question that a single prompt could answer are the single most expensive anti-pattern we see. Every extra turn is another full input pass, another output, another chance to loop.

Rules of thumb: cap agent loops at 3 turns hard. Log the turn count. Alert if p95 goes above 2. Most tasks that "need" an agent don't — they need better prompting and better retrieval.

What we do first on a cost-audit engagement

When a client hands us a bloated model bill, we don't guess. We instrument every call for a week:

• Model name, input tokens, output tokens, latency
• Feature and endpoint
• Tenant / user ID
• Cache hit or miss
• Cost in cents

Then we sort by feature and by tenant. The winning move is almost always visible in the first row: one feature or one user is 40% of the bill. Fix that first.

Common mistakes we see

Optimizing before measuring. Teams spend two weeks compressing prompts to save $80/mo when routing would save $2,000/mo. Instrument first.

Switching models mid-week. A model swap that "seems fine" on 20 dev examples can tank quality on 20,000 production ones. Always run the eval set before rolling.

Assuming cheaper is worse. Modern small models (Haiku, Gemini Flash, GPT-4o-mini) are astonishingly capable. On many production tasks they match Sonnet or GPT-4o. Test, don't assume.

Ignoring per-tenant caps. One customer with a script can 10× your monthly bill. Ship rate caps day one.

Trade-offs no one tells you

• Prompt caching locks you to a provider. If you cache heavily on Anthropic, a switch to OpenAI resets the savings.
• Routing adds latency unless the classifier is cheap and parallel. Measure the added p95, not just the average.
• Fine-tuning is a maintenance liability. Every base-model update means retraining. Budget for it or don't start.

A realistic outcome

On a project earlier this year we walked in with a $9,400/month OpenAI bill. Over six weeks we shipped routing, prompt caching, output caps, and batch-mode for background jobs. Same product, same acceptance rate. Bill: $2,600/month. That's a normal outcome, not a magic one.

Frequently asked questions

What's the fastest way to reduce LLM cost?

Model routing. Send simple queries to a cheap model and only escalate to your premium model when needed. Typical savings: 40–65% within a week.

Is prompt caching worth setting up?

Yes, if your system prompt or retrieved context is stable across requests. Setup takes an afternoon and typically cuts input token cost 20–40%.

Should I fine-tune to save money?

Only if you have consistent volume above ~50K requests per month on a narrow task. Below that, prompting and retrieval will beat fine-tuning on both cost and flexibility.

How do I stop one user from blowing up my bill?

Per-tenant rate limits and monthly caps, enforced in your backend before the model call. Add them from day one, not after the first surprise invoice.

Are cheaper models actually good enough?

On 60–80% of production tasks, yes. Run your own eval set — vendor benchmarks lie. See our LLM comparison for current picks.

Want a cost audit?

We do this exact work at Augere Labs — one-week engagement, instrumented audit, prioritized fix list. If your inference bill is bigger than it should be, book a call and bring the last invoice. We'll tell you what would move the number.