7 ways to cut LLM inference cost without changing your model

1. Route to a cheaper model (saves 80% to 85%)

This is the single largest lever available and it requires no prompt changes. Moving the same task from GPT-5.5 to GPT-5.4 mini takes it from $0.06750 to $0.01012, a saving of 85%. Claude Opus 4.8 to Claude Haiku 4.5 goes from $0.06000 to $0.01200, a saving of 80%. Gemini 3.1 Pro to Gemini 2.5 Flash goes from $0.02700 to $0.00510, a saving of 81%.

Nothing else on this list comes close, because nothing else changes the price per token. The objection is always quality, and it is a fair objection, but it is testable: build twenty representative cases from your real traffic, run both models, and look. Most production workloads are a mix, where a majority of requests are routine and a minority are genuinely hard.

That observation leads to cascading: send everything to the cheap model first, and escalate to the expensive one only when a check fails or confidence is low. If 80% of traffic never escalates, you capture most of the 80% saving while keeping the frontier model for the cases that need it.

2. Cap output length (saves 31% to 37%)

Output is priced above input on every model we track, by 2x to 8.33x. Halving the output tokens on our profile saves 36.8% on Gemini 2.5 Flash, 33.3% on GPT-5.5, 31.3% on Claude Opus 4.8 and 31.2% on Claude Sonnet 5.

The saving is largest exactly where the output multiple is largest, which is why Gemini 2.5 Flash tops the list despite already being cheap. Set max_tokens deliberately rather than leaving it at the default, ask for the format you want, and use stop sequences so generation ends when the useful part is done.

Much long output is unrequested. A model asked for a classification that returns a paragraph of reasoning followed by the label is billing you for the paragraph. Asking for the label alone, or a compact JSON object, often cuts output by more than half with no loss of information.

3. Make fewer calls per task (saves 33%)

Cost scales linearly with calls, so dropping an agent loop from three calls to two saves exactly 33.3% on every model. It is the most predictable lever on the list.

Look for calls that exist for structural reasons rather than useful ones: a separate classification step that could be a field in the main response, a validation pass that a schema constraint would enforce for free, a summarisation step feeding a model that could have read the original.

This lever compounds with the others, because it multiplies the whole per-call cost rather than one side of it.

4. Cache the repeated prefix (saves 24% to 34%)

A long system prompt resent on every call is billed on every call. Where a provider offers cached-input pricing, eliminating 90% of billed input saves 33.7% on Claude Opus 4.8, 33.8% on Claude Sonnet 5, 30.0% on GPT-5.5 and 23.8% on Gemini 2.5 Flash.

The saving tracks how much of the bill was input in the first place, which is why it is smallest on Gemini 2.5 Flash, where output already dominates at 74% of cost. Caching helps most on models with a low output multiple and long shared prefixes.

Realising it requires structuring prompts so the stable part comes first and varies last, which is a small change that mostly costs discipline rather than engineering.

5. Send less context

Retrieval systems commonly stuff a fixed number of chunks into the prompt regardless of whether they help. Cutting retrieved context from ten chunks to four reduces input tokens by roughly 60% for that portion of the prompt, and often improves answer quality by removing distractors.

This lever is bounded by the same ceiling as caching: it can only ever recover the input share of the bill, which on our profile is 26% to 60% depending on model. On an output-heavy model it is worth little.

Measure before trimming. If your input is already small, this is effort spent on the wrong half of the invoice.

6. Use batch and asynchronous tiers

Most providers offer a substantially discounted rate for work you are willing to wait for. Anything not in front of a waiting user is a candidate: overnight enrichment, backfills, evaluation runs, bulk classification.

This changes the rate rather than the token count, so it stacks cleanly with every other lever here. It costs latency, which is why it applies to a subset of traffic rather than all of it.

Our dataset carries standard rates only, so we do not quote batch figures. Check your provider's current discount and apply it to the per-task numbers here.

7. Stop paying for failures

Retries, refusals and discarded responses are billed exactly like successful ones, and they rarely appear in cost models because they do not appear in the happy path.

A malformed response that triggers an automatic retry costs double. A pipeline that generates a response and then discards it because a downstream validator rejected it paid full price for nothing. An agent that loops until a timeout can bill many times the expected amount for a single task.

Instrument cost per completed task, not cost per call. The gap between the two is your failure tax, and on unstable pipelines it is often larger than any optimisation on this list.

Ranking the levers

Do them in return order. Model routing first, at 80% to 85%, because nothing else is close and it requires no prompt work. Then output caps at 31% to 37%, then call reduction at a predictable 33%, then caching at 24% to 34%.

The savings are multiplicative, not additive. Routing to a cheaper model and then halving output on it captures both: 85% then 33% of what remains leaves roughly a tenth of the original bill.

Note the asymmetry in effort. Routing is a configuration change plus an evaluation run. Caching and context trimming are architectural. Start where the return is highest and the change is smallest.

FAQ

What is the single biggest lever on inference cost?

Changing which model you call. On our task profile, moving from GPT-5.5 to GPT-5.4 mini saves 85%, and Claude Opus 4.8 to Claude Haiku 4.5 saves 80%. Every prompt-level optimisation combined is worth 20% to 40%, because only model choice changes the price per token.

Why does capping output save more than trimming the prompt?

Because output is priced 2x to 8.33x higher than input on every model we track. On our profile, output accounts for 40% to 74% of the bill despite being a quarter of the tokens, so halving output saves 31% to 37% while the same proportional cut to input saves considerably less.

How much does prompt caching actually save?

Eliminating 90% of billed input saves 33.7% on Claude Opus 4.8, 30.0% on GPT-5.5 and 23.8% on Gemini 2.5 Flash. The saving is bounded by how much of your bill was input to begin with, so it returns least on output-heavy models.

Do these savings add up if I do all of them?

They multiply rather than add. Routing to a model 85% cheaper and then halving output on it leaves roughly a tenth of the original bill. Apply them in return order: model routing, output caps, fewer calls, then caching.

Why track cost per completed task instead of per call?

Because retries, refusals and discarded responses are billed exactly like successful calls but produce nothing. The difference between cost per call and cost per completed task is your failure tax, and on an unstable pipeline it can exceed every optimisation on this list.