Token optimization for LLMs: advanced techniques that actually work

Token optimization for LLMs means reducing the number of tokens consumed per API call while maintaining output quality. The real goal isn’t fewer tokens — it’s lower cost per outcome. A well-optimized pipeline spends 40-60% less than an unoptimized one handling the same workload.

This guide covers four advanced techniques — prompt compression, semantic caching, model routing, and benchmarking — with real numbers from production deployments.

Pillar guide: FinOps for AI: the complete cost control guide

Why token count is the wrong metric (cost per outcome matters)

Most optimization guides focus on reducing token count. That’s the wrong target.

Here’s why. You switch from GPT-5 ($10/$30 per million tokens) to GPT-5-nano ($0.05/$0.20). Token count stays the same — maybe even increases, because the smaller model needs more explicit instructions. But cost drops 99%. Token count went up. Cost went down. Which metric matters?

The reverse is also true. A team compresses their prompts by 40%, cutting token count dramatically. But they’re using a model where input tokens are cheap ($0.05/M) and the compression doesn’t affect output length. They saved $2/month on a $10,000/month bill. The effort wasn’t worth it.

Cost per outcome captures what actually matters:

Cost per customer ticket resolved
Cost per document classified
Cost per report generated
Cost per code review completed

This metric accounts for model choice, token count, retries, cache hits, and fallbacks. It’s the one number that maps to business value.

When optimizing, always ask: “Did cost per outcome go down?” If yes, the optimization worked — regardless of what happened to raw token counts.

The token count trap

Teams that optimize for token count alone make three common mistakes:

Over-compressing prompts. They strip so much context that quality drops. The model produces wrong answers. Users retry. Total cost goes up.
Ignoring output tokens. They obsess over input compression but let the model generate 2,000-token responses when 300 tokens would suffice. Output tokens cost 3-6x more than input. A max_tokens parameter is the simplest optimization most teams miss.
Counting tokens across different tokenizers. 1,000 tokens on GPT-5 is not the same as 1,000 tokens on Claude. Tokenizers differ. Comparing raw counts across models is comparing apples to orangutans.

Prompt compression: reducing input tokens by 40% without losing quality

Prompt compression removes unnecessary tokens from your input without degrading the model’s ability to understand and respond correctly.

What can be compressed

1. System prompt redundancy. Most production system prompts contain instructions the model already follows by default. “Respond in a helpful manner” adds tokens and changes nothing. “Be accurate” — the model is already trying.

Audit your system prompt. For each instruction, ask: “If I remove this, does the output change?” Test it. Many teams find 30-50% of their system prompt is inert.

2. Few-shot examples. Few-shot prompting is powerful but expensive. Each example costs hundreds of tokens. Advanced compression techniques:

Reduce from 5 examples to 2. Research shows diminishing returns after 2-3 examples for most tasks.
Shorten examples to the minimum that demonstrates the pattern. If the model needs to see input/output format, a 50-token example works as well as a 200-token one.
Use structured formats. JSON examples compress better than prose examples because models parse structure efficiently.

3. Context window stuffing. RAG pipelines often retrieve 10 chunks of 500 tokens each — 5,000 tokens of context. But only 2-3 chunks are usually relevant. Better retrieval relevance scoring reduces context tokens by 50-70% while improving answer quality.

4. Formatting overhead. Markdown headers, horizontal rules, excessive whitespace, XML tags for structure — they cost tokens. In many cases, minimal formatting performs identically.

Real-world compression results

Technique	Token reduction	Quality impact
System prompt audit	20-40% of system prompt	None (verified)
Few-shot reduction (5→2)	60% of examples	<2% accuracy drop
RAG chunk pruning	40-60% of context	+5% relevance (better retrieval)
Format stripping	5-10% overall	None
Combined	30-40% total input	Neutral to positive

The combined effect: 30-40% fewer input tokens, with no measurable quality loss. On GPT-5, that’s $3-4 saved per million calls on input alone.

The Anthropic tokenizer lesson

When Anthropic released Opus 4.7, they updated their tokenizer. The same prompts — identical text — consumed roughly 27% more tokens. Teams that had optimized their prompts for Claude woke up to a 27% cost increase with zero code changes.

This is why token count is a fragile metric. Tokenizers change. Models change. What doesn’t change is the outcome you need. Optimize for cost per outcome, and tokenizer changes become just another variable your routing layer handles — shifting traffic to the model with the best cost-quality ratio at any given moment.

Semantic caching: avoid paying twice for the same answer (35% average hit rate)

Semantic caching stores LLM responses and serves them for semantically similar queries — without calling the model again.

Unlike traditional caching (exact string match), semantic caching uses embedding similarity. “What’s your return policy?” and “How do I return an item?” are different strings but the same question. A semantic cache serves the stored answer for both.

How it works

New query arrives.
The cache layer generates an embedding of the query.
It compares the embedding against stored query embeddings using cosine similarity.
If similarity exceeds the threshold (typically 0.92-0.95), return the cached response.
If not, forward to the LLM, store the response, and return it.

The embedding generation costs about $0.0001 per query (using a lightweight embedding model). An LLM call costs $0.01-0.05. That’s a 100-500x cost difference for a cache hit.

Hit rates by use case

Use case	Typical hit rate	Why
Customer support	55-70%	High repetition — same questions from different users
FAQ / knowledge base	60-75%	By definition, frequently asked
Code generation	10-20%	Each request is contextually unique
Content writing	5-15%	Creative tasks rarely repeat
Document classification	40-55%	Similar documents, similar classifications
Data extraction	30-45%	Structured patterns repeat across documents

Average across production deployments: 35%.

That means 35% of your LLM calls cost essentially zero. On a $150,000/month LLM budget, that’s $52,500/month saved by caching alone.

Cache configuration that matters

Similarity threshold. Too low (0.85) and you serve wrong answers for different-enough queries. Too high (0.98) and you barely get hits. 0.92-0.95 is the sweet spot for most use cases. Tune per task type.

TTL (time to live). Static content (company policies, product specs) can cache for days. Dynamic content (stock prices, live inventory) should cache for minutes at most. Set TTLs per content category, not globally.

Cache size. Embedding storage is cheap. A million cached entries with 1536-dimension embeddings costs ~6GB. Response storage depends on average response length. Budget 10-50GB for a production cache layer.

Invalidation. When source data changes, invalidate related cache entries. This requires tagging cached responses with their data sources — if the return policy updates, all cached return-policy responses should expire.

Model routing: GPT-4 for reasoning, GPT-4o-mini for extraction

Model routing directs each request to the optimal model based on task complexity, required quality, and cost constraints. It’s the single highest-impact optimization available.

The routing matrix

Task complexity	Best model class	Typical cost (input/M)	Examples
Complex reasoning	GPT-5, Claude Opus	$5-10	Multi-step analysis, nuanced writing, complex code
Standard generation	GPT-4.1, Claude Sonnet	$2-3	Summarization, translation, moderate code
Simple extraction	GPT-5-nano, Haiku	$0.05-0.25	Classification, entity extraction, formatting
Embeddings	text-embedding-3-small	$0.01	Search, caching, similarity

How routing works in practice

A routing engine classifies incoming requests before they reach a model. Classification methods:

1. Rule-based routing. Simplest approach. Route by endpoint: /api/classify goes to nano, /api/analyze goes to GPT-5. Works when task types map cleanly to API endpoints.

2. Keyword/pattern routing. Scan the prompt for complexity signals. Long context windows, multi-step instructions, or reasoning keywords route to premium models. Short, structured prompts route to cheap models.

3. ML-based routing. A small classifier model (costs nearly nothing) analyzes the prompt and predicts which model will produce acceptable quality at the lowest cost. This is what production-grade AI gateways like Veltrix implement.

The cost impact

A typical enterprise sends 100% of traffic to GPT-5 “because it’s the best.” After implementing routing:

15-20% of calls genuinely need GPT-5 (complex reasoning)
25-30% work fine on GPT-4.1 (standard generation)
50-60% perform identically on GPT-5-nano (extraction, classification)

Weighted cost drops from $10/M input (GPT-5 for everything) to roughly $2.50/M input (blended). That’s a 75% reduction on input tokens alone.

Quality gates

Routing without quality monitoring is gambling. Every routing decision should be validated:

A/B test routing changes. Route 10% of traffic through the new path. Compare quality metrics.
Set quality thresholds per task. “Classification accuracy must stay above 95%.” If the cheaper model drops below, escalate to the next tier.
Monitor continuously. Model performance drifts. A model that handled classification well last month might struggle after a provider update.

Benchmarking your optimization: before/after framework

You can’t prove optimization worked without measurement. Here’s the framework.

Step 1: Establish the baseline (1 week)

Before changing anything, instrument your current system to capture:

Total daily spend by model
Cost per task by task type
Token count per call (input and output separately)
Quality metrics per task type (accuracy, satisfaction, etc.)
Cache hit rate (0% if no cache exists)
Model distribution (likely 100% one model)

Run for one full week to capture daily and weekly patterns.

Step 2: Implement one lever at a time

Don’t deploy all optimizations simultaneously. You won’t know what worked.

Week 2: Implement model routing. Measure cost per task, quality, model distribution. Week 3: Add semantic caching. Measure cache hit rate, cost per task, quality. Week 4: Apply prompt compression. Measure token reduction, cost per task, quality. Week 5: Enable batching for eligible workloads. Measure batch volume, cost per task. Week 6: Configure fallback chains. Measure fallback rates, cost per task, quality.

Step 3: Compare before/after

Metric	Baseline	After routing	+ Cache	+ Compression	+ Batch	+ Fallback
Daily spend	$X
Cost per task (support)	$X
Cost per task (classify)	$X
Quality score	X%
Cache hit rate	0%	0%	X%	X%	X%	X%

Fill in each column as you deploy each lever. The cumulative effect tells the complete story.

Step 4: Report ROI

Monthly savings = baseline monthly spend - current monthly spend Implementation cost = engineering hours x hourly rate + platform fees Payback period = implementation cost / monthly savings Annual ROI = (annual savings - annual cost) / annual cost x 100

Most teams see payback within 30-60 days when using a managed platform. Building in-house extends payback to 3-6 months due to engineering investment.

Complete framework: FinOps for AI: the complete cost control guide

Summary

Token optimization for LLMs is about reducing cost per outcome — not raw token counts. The four techniques that consistently deliver results: prompt compression (30-40% input reduction), semantic caching (35% average hit rate), model routing (75% input cost reduction through smart distribution), and disciplined benchmarking to prove it all works.

Start with routing. It has the highest impact. Add caching next. Then compress. Measure everything.

Platforms like Veltrix automate these techniques through an AI gateway — routing, caching, compression, and cost tracking without changing your application code.

FAQ

How much can prompt compression reduce token usage without losing quality?

Semantic prompt compression typically reduces input tokens by 30-40% without measurable quality loss. The biggest gains come from auditing system prompts (removing instructions the model already follows), reducing few-shot examples from 5 to 2, and improving RAG retrieval to include fewer, more relevant context chunks. Quality actually improves in some cases — less noise in the prompt means more focused responses.

What’s a realistic cache hit rate for production LLM applications?

The average across production deployments is 35%. Customer support and FAQ-heavy applications see 55-75% hit rates due to high query repetition. Creative generation and code tasks see 5-20%. The key variables are query diversity, similarity threshold tuning (0.92-0.95 is the sweet spot), and TTL configuration. Even a 20% hit rate saves significant money at scale.

Is it worth building custom model routing or should we use a platform?

Rule-based routing (route by endpoint) takes a day to build and works for simple cases. ML-based routing that dynamically selects the optimal model based on prompt analysis, quality monitoring, and cost constraints is significantly more complex. Platforms like Veltrix provide production-grade routing with quality gates, real-time cost tracking, and automatic rebalancing as new models launch. For most teams, the platform approach delivers faster ROI — typically payback within 30 days versus 3-6 months for a custom build.