The Invoice That Made Me Rethink Everything
A few months ago, a team I worked with shipped a feature that summarized customer support tickets using an LLM. The demo was beautiful. The first month's bill was not. They had built a pipeline that sent the entire ticket history — every reply, every internal note — into a frontier model on every request. It worked, but it was burning through budget at a rate that made the finance team ask uncomfortable questions.
This is the moment most engineering teams hit: the prototype works, the value is real, but token pricing turns out to be the thing that decides whether the feature survives contact with production. Cost optimization isn't a nice-to-have you get to later. It's part of shipping.
This post walks through how token pricing actually works, where the money really goes, and a set of concrete strategies you can apply today to bring your LLM costs under control — without degrading the experience your users signed up for.
How Token Pricing Actually Works
Every major LLM API prices in tokens, but the pricing structure is rarely as simple as "X dollars per token." Understanding the mechanics is the prerequisite for optimizing anything.
Input vs. Output Tokens
Almost every provider charges separately for input tokens (what you send) and output tokens (what the model generates). Output tokens typically cost 3-5x more than input tokens. This asymmetry matters enormously: a prompt that produces a 2,000-word essay costs far more than a prompt that produces a one-line classification, even if the inputs are identical.
The practical takeaway: when you're optimizing, look first at what the model is writing. Trimming verbose system prompts helps, but capping output length, using structured formats, and asking for concise answers often moves the needle more.
The Tiered Pricing Reality
Model tiers create enormous price spreads. A small, fast model might cost a fraction of a cent per thousand tokens, while a frontier reasoning model can cost orders of magnitude more. The same request routed to the wrong tier can be 10-50x more expensive — and frequently, the cheaper model would have done the job just as well.
"The most expensive model isn't the one with the highest price per token. It's the one you didn't need to call."
Where Your Token Budget Actually Goes
Before optimizing, instrument. Most teams are surprised by which parts of their pipeline dominate cost.
The Usual Suspects
- System prompts: A 1,500-token system prompt sent on every request adds up fast at scale. Ten thousand requests a day means 15 million input tokens just for instructions.
- Retrieved context: RAG pipelines that stuff 20 documents into the prompt "just in case" pay for every irrelevant chunk.
- Conversation history: Chat apps that resend the full transcript on every turn grow quadratically in cost as conversations get longer.
- Retries and fallbacks: Failed calls still cost money. A flaky validation loop that retries three times triples your bill for that request.
Measure Cost Per Successful Task
Token-level metrics are useful, but the number that matters is cost per successful task. A cheap model that fails 30% of the time and triggers a fallback to an expensive model is often more costly than just calling the expensive model directly. Track success rate, retry rate, and fallback rate alongside raw token spend.
Cost Optimization Strategies That Actually Work
Here are the levers that consistently deliver the biggest savings, roughly in order of impact.
1. Prompt Caching
Many providers now offer prompt caching: if you send the same prefix repeatedly, cached input tokens cost a fraction of fresh ones. This is transformative for applications with long, stable system prompts or shared context. Restructure your prompts so the static portion comes first and the variable portion comes last — that's what makes caching effective.
2. Model Routing
Not every request needs your most capable model. Build a routing layer that classifies requests by complexity and sends simple ones to a smaller model. A support-ticket classifier doesn't need a frontier reasoning model. Reserve the expensive tier for tasks that genuinely require it.
A common pattern: try the small model first, validate the output, and escalate to a larger model only on failure. This gives you the cost profile of the small model with a safety net for hard cases.
3. Prompt Compression and Context Discipline
Every token in your prompt should earn its place. Techniques that work:
- Summarize long conversation histories instead of resending them verbatim.
- Retrieve fewer, more relevant documents — quality over quantity.
- Replace verbose instructions with examples (few-shot often beats paragraphs of rules).
- Strip boilerplate, formatting artifacts, and redundant context before sending.
4. Cap Output Length
Since output tokens cost more, set max_tokens deliberately. Ask for structured output (JSON, bullet lists) instead of prose. Instruct the model to be concise. These changes often cut output cost by 40-60% with no loss in usefulness.
5. Batch and Debounce
If your workload isn't latency-sensitive — analytics, nightly summaries, bulk classification — batch requests where providers offer discounts, and coalesce small requests into larger ones to amortize fixed prompt overhead.
Guardrails: Don't Learn Cost Lessons the Hard Way
Optimization is proactive; guardrails are defensive. You need both.
- Per-user and per-tenant rate limits to prevent a single runaway client from dominating spend.
- Budget alerts at 50%, 80%, and 100% of expected monthly spend.
- Anomaly detection on token consumption — a sudden spike usually means a bug, not growth.
- Kill switches for expensive endpoints so you can disable them without a deploy.
None of this is glamorous, but the teams that ship LLM features sustainably are the ones that treat cost as an operational concern from day one.
The Trade-Offs Nobody Talks About
Cost optimization isn't free. Aggressive prompt compression can hurt quality. Model routing adds latency and complexity. Caching requires restructuring prompts, which can make them harder to reason about. Every optimization trades something — usually developer time, latency, or a small amount of quality — for money.
The right balance depends on your product. A consumer app with razor-thin margins should optimize hard. An enterprise tool where a single successful task is worth dollars can afford to be generous. There's no universal answer, only the answer that fits your unit economics.
"Optimize for cost per outcome, not cost per token. The cheapest pipeline is the one that solves the problem on the first try."
Where to Start Tomorrow
If you take nothing else from this post, take this: instrument first, optimize second. You can't fix what you can't see. Add token accounting to your logging, break down cost by feature and by user, and identify your top three cost drivers. Then apply the levers above in order of impact — caching, routing, compression, output caps.
Token pricing will keep evolving, and so will the models. The teams that treat cost as a first-class engineering concern will be the ones still shipping ambitious features a year from now, while everyone else is explaining to finance why the demo can't go to production.
So here's the question worth sitting with: if you had to cut your LLM spend in half tomorrow without degrading the user experience, which of these levers would you pull first — and what does your answer reveal about where you've been spending without thinking?