AI cost optimization: a practical guide for engineering teams

AI cost optimization means spending less on LLM calls without degrading output quality. The average engineering team overspends by 40-60% on AI APIs — mostly because nobody tracks cost per task, only cost per token.

This guide covers the five optimization levers that actually move the needle, how AI gateways automate them, and how to measure ROI in terms your CFO understands. If you’re running GPT-5, Claude, Gemini, or any mix of models in production, this is the playbook.

Pillar guide: FinOps for AI: the complete cost control guide

The real cost of LLM calls at scale

A single GPT-5 call costs fractions of a cent. A million calls per day costs a house payment.

Here’s the math most teams skip. GPT-5 charges $10 per million input tokens and $30 per million output tokens. A typical enterprise API call sends 800 input tokens and receives 400 output tokens. That’s $0.02 per call. At 500,000 calls per day — normal for a mid-size SaaS company with AI features — that’s $10,000/day. $300,000/month. From one model.

And GPT-5 isn’t even the most expensive option. Claude Opus 4 runs $5/$25 (input/output per million tokens). Smaller models like GPT-5-nano sit at $0.05/$0.20. The spread between cheapest and most expensive is 500x.

Three facts about LLM costs at scale:

1. Output tokens are the silent killer. Output costs 3-6x more than input across every major provider. If your model generates verbose responses without a max_tokens cap, you’re burning money on words nobody reads. A customer support bot that returns 800-token answers when 200 tokens would suffice costs 4x more than necessary — on the output side alone.

2. Token count varies by model. The same English sentence generates different token counts depending on the tokenizer. Anthropic’s tokenizer update with Opus 4.7 increased token counts by roughly 27% for the same prompts. That’s a 27% cost increase with zero change in your code.

3. Most teams don’t know their real spend. According to a 2026 Zylo report, 78% of IT leaders experienced unexpected charges from AI tools and APIs. Not because they’re careless — because there’s no equivalent of a cloud billing dashboard for LLM usage. Each team has its own API key. Nobody aggregates.

The global waste is staggering. Gartner and Flexera estimate $4.2 billion wasted annually on inefficient AI consumption. Duplicate calls without caching. Premium models used for simple extraction tasks. Prompts stuffed with unnecessary context. Outputs without length limits.

That $4.2B isn’t a projection. It’s happening now.

5 optimization levers: model routing, caching, prompt compression, batching, fallback chains

There are exactly five techniques that reliably reduce LLM spend. Everything else is a variation of these.

Lever 1: Model routing

Not every task needs your most expensive model. Classification tasks, entity extraction, and structured data formatting work just as well on GPT-5-nano ($0.05/$0.20) as on GPT-5 ($10/$30). That’s a 200x price difference for equivalent accuracy on simple tasks.

Smart model routing analyzes the incoming request and sends it to the cheapest model that can handle it at the required quality level. In practice, this means:

Reasoning and complex analysis → GPT-5 or Claude Opus ($5-10 input)
Summarization and rewriting → GPT-4.1 or Claude Sonnet ($2-3 input)
Classification and extraction → GPT-5-nano or Haiku ($0.05-0.25 input)
Simple completions → GPT-4.1-mini ($0.10 input)

Most companies route 100% of traffic to one model. When you implement routing, 60-70% of calls typically shift to cheaper models. That alone cuts spend by 30-50%.

The key is quality gates. You need an evaluation framework that measures output quality per model per task type. Without it, routing is just guessing.

Lever 2: Semantic caching

If two users ask “What’s our refund policy?” three minutes apart, should you pay for two LLM calls? No.

Semantic caching stores model responses and matches new queries against previous ones using embedding similarity — not exact string matching. “What’s the refund policy?” and “How do I get a refund?” hit the same cache entry.

Real-world numbers:

Average cache hit rate: 35% across production deployments
Best case (customer support, FAQ-heavy): 60-70% hit rate
Worst case (creative generation, unique queries): 10-15% hit rate

A 35% hit rate means 35% of your LLM calls cost zero. The cache lookup costs a fraction of a cent (embedding comparison), versus $0.02+ for a full model call.

The economics are compelling. If you spend $100,000/month on LLM calls and achieve a 35% hit rate, you save $35,000/month. The cache infrastructure costs maybe $500/month.

Cache invalidation matters. Stale responses degrade user experience. Good implementations set TTLs based on content type — product info might cache for 24 hours, while market data caches for 15 minutes.

Lever 3: Prompt compression

Most prompts contain redundant tokens. System prompts repeated on every call. Verbose instructions that could be condensed. Examples that demonstrate patterns already learned through fine-tuning.

Semantic prompt compression reduces input tokens by 30-40% without measurable quality loss. It works by:

Removing redundant instructions. If your system prompt says “You are a helpful assistant” followed by 500 tokens of behavioral guidelines, the compression engine identifies which instructions are already embedded in the model’s training.
Condensing few-shot examples. Instead of providing 5 full examples at 200 tokens each (1,000 tokens total), compression can distill them to 2 examples at 120 tokens each (240 tokens). Same pattern demonstration, 76% fewer tokens.
Stripping formatting overhead. Markdown headers, excessive whitespace, decorative separators — these tokens cost money and add nothing to model comprehension.

At scale, 40% input compression on GPT-5 saves $4 per million calls. Multiply by your daily volume.

Lever 4: Request batching

Most LLM providers offer batch APIs at 50% discount. OpenAI’s batch API charges half price for requests that can tolerate 24-hour latency.

Not every use case qualifies. Real-time chat needs instant responses. But many production workloads are batch-compatible:

Nightly content generation
Document classification pipelines
Data enrichment jobs
Scheduled report generation
Embedding generation for search indexes

If 30% of your workload can shift to batch processing, that’s a 15% reduction in total spend — with zero quality impact.

Lever 5: Fallback chains

When your primary model is overloaded or down, most systems either fail or wait. A fallback chain routes to the next best model automatically.

This isn’t just about reliability. It’s about cost. When GPT-5 hits rate limits during peak hours, a fallback to GPT-4.1 ($2 input vs $10) serves most requests at 80% lower cost. If quality monitoring confirms equivalent results, you’ve found a permanent optimization.

Fallback chains also protect against price changes. When a provider raises prices (and they do — often), your system automatically shifts traffic to alternatives.

Combined impact of all five levers:

Lever	Typical savings	Implementation effort
Model routing	30-50%	Medium
Semantic caching	25-35%	Medium
Prompt compression	15-25%	Low
Request batching	10-15%	Low
Fallback chains	5-15%	Low

These compound. A team that implements all five typically sees 40-60% total cost reduction.

How AI gateways reduce spend by 40-60%

An AI gateway sits between your application and the LLM providers. Every API call passes through it. This single chokepoint enables all five optimization levers automatically — without changing your application code.

Think of it as a reverse proxy for AI. Your app sends requests to the gateway. The gateway decides: which model handles this? Is there a cached response? Can the prompt be compressed? Should this be batched? What’s the fallback if the primary model fails?

What a gateway does that SDKs don’t

Using the OpenAI SDK directly gives you access to one provider, one model at a time. You handle retries, fallbacks, and caching in your own code. Every team builds their own version. None of them are consistent.

A gateway centralizes:

Routing logic — one policy engine, not scattered if/else blocks
Cache layer — shared across all teams, all endpoints
Cost tracking — per team, per project, per model, per call
Rate limiting — prevent any single team from blowing the budget
Audit trail — who called what, when, and how much it cost

The 40-60% number

Where does the 40-60% savings claim come from?

Real deployments across multiple companies show consistent patterns:

Model routing alone: 30-40% (shifting 60-70% of calls to cheaper models)
Semantic caching on top: additional 15-20% (35% average hit rate)
Prompt compression on top: additional 5-10%
Net result: 40-60% total reduction

A company spending $200,000/month on LLM APIs typically drops to $80,000-$120,000/month after deploying a gateway. The gateway itself costs a fraction of the savings.

Build vs. buy

Some teams build their own gateway. It works for simple cases — a basic proxy with model routing takes a few weeks. But maintaining it is the problem. New models launch weekly. Pricing changes constantly. Cache invalidation bugs cost money. Quality monitoring requires ML infrastructure.

Platforms like Veltrix handle this out of the box. Smart routing, semantic cache, prompt compression, real-time cost dashboard, team governance — all managed. The engineering team focuses on product, not infrastructure.

Related: FinOps for AI: the complete cost control guide

Building cost awareness into your CI/CD pipeline

Most teams discover cost problems after deployment. By then, the damage is done. A prompt change that doubles token count ships to production, runs for two weeks, and shows up on next month’s invoice.

Cost awareness belongs in CI/CD — alongside tests, linting, and security checks.

Step 1: Cost estimation in pull requests

Add a CI step that estimates the cost impact of prompt changes. When a developer modifies a system prompt, the CI pipeline:

Tokenizes the old prompt and the new prompt
Calculates the delta in tokens
Multiplies by expected call volume
Comments the estimated monthly cost change on the PR

Example output: “This prompt change adds 340 input tokens per call. At current volume (850K calls/month), estimated monthly cost increase: +$2,890.”

That single comment changes the conversation. The developer might find a way to achieve the same result with fewer tokens. Or the team might decide the cost is justified. Either way, the decision is informed.

Step 2: Cost budgets per environment

Set token budgets for staging and production separately. Staging shouldn’t run at production scale — but it often does, because nobody sets limits.

A staging budget of $500/month catches runaway experiments before they reach production. Alert at 80%. Hard-stop at 100%.

Step 3: Cost regression tests

Just like performance regression tests catch slowdowns, cost regression tests catch spend increases.

Define a baseline: “This workflow should cost no more than $0.08 per execution.” Run the workflow in CI against live models (or cost-equivalent mocks). If the cost exceeds the baseline by more than 15%, fail the build.

Step 4: Deploy-time cost alerts

When a new version deploys, monitor cost metrics for the first hour. If cost-per-call increases by more than 20% compared to the previous version, trigger an alert. Optionally, auto-rollback.

This is the AI equivalent of canary deployments for latency. The metric is just different: dollars instead of milliseconds.

Step 5: Weekly cost reviews

Automated doesn’t mean unattended. A weekly Slack digest showing:

Total spend by team and project
Top 10 most expensive endpoints
Week-over-week change
Models used and cost per model
Cache hit rate trend

This creates accountability without blame. Teams see their own numbers. Optimization becomes a habit, not a fire drill.

Measuring ROI: cost per task, not cost per token

“We reduced our average cost per token by 15%.”

That statement is meaningless without context. If you switched to a cheaper model that requires 3x more tokens to produce the same result, you spent more — not less.

The right metric: cost per task

A task is a unit of business value. Examples:

Answering one customer support ticket
Classifying one document
Generating one product description
Summarizing one meeting transcript

Cost per task = total LLM spend / number of tasks completed.

This metric captures everything: model choice, token count, cache hits, retries, fallbacks. It’s the only number that correlates with business impact.

How to calculate cost per task

Define your tasks. List every distinct use case that calls an LLM.
Tag API calls. Each call includes metadata: task type, team, project.
Aggregate by task. Sum all costs (including retries and fallbacks) per task type.
Divide. Total cost / total tasks completed = cost per task.

Benchmark: what “good” looks like

Task type	Typical cost per task	Optimized cost per task
Customer support response	$0.08-0.15	$0.02-0.05
Document classification	$0.01-0.03	$0.002-0.005
Content generation (500 words)	$0.05-0.12	$0.02-0.04
Code review suggestion	$0.10-0.25	$0.03-0.08
Meeting summary	$0.06-0.15	$0.02-0.05

The “optimized” column reflects teams using all five levers: routing, caching, compression, batching, and fallbacks.

Building the ROI case

To justify investment in AI cost optimization (whether building internally or using a platform like Veltrix), frame it as:

Monthly savings = (current cost per task - optimized cost per task) x monthly task volume

A team processing 500,000 customer support interactions per month at $0.12 each spends $60,000/month. Optimizing to $0.04 per task saves $40,000/month — $480,000/year.

The ROI calculation is straightforward: optimization cost vs. annual savings. Most teams see payback within 30-60 days.

Quality must be part of the equation

Cost optimization without quality monitoring is a race to the bottom. Every optimization should be paired with a quality check:

Accuracy metrics per task type (does the output match human evaluation?)
User satisfaction scores (are customers noticing degradation?)
Fallback rates (how often does the cheaper model fail and escalate?)

If cost per task drops 50% but customer satisfaction drops 10%, you’ve made a bad trade. The goal is maintaining quality at lower cost — not sacrificing quality for savings.

Deep dive: FinOps for AI: the complete cost control guide

Summary

AI cost optimization isn’t optional at scale. It’s infrastructure. The five levers — model routing, semantic caching, prompt compression, request batching, and fallback chains — compound to reduce spend by 40-60%.

The teams that get this right share three habits: they measure cost per task (not cost per token), they embed cost awareness in CI/CD, and they automate optimization through a gateway layer.

Whether you build or buy, the starting point is visibility. You can’t optimize what you can’t see.

FAQ

How much can AI cost optimization actually save?

Most engineering teams reduce LLM spend by 40-60% by combining model routing, semantic caching, prompt compression, batching, and fallback chains. The exact savings depend on your workload mix — customer support and FAQ-heavy use cases save more (due to higher cache hit rates), while creative generation saves less. A team spending $100,000/month typically drops to $40,000-$60,000/month.

What’s the fastest optimization to implement?

Model routing delivers the biggest impact with moderate effort. Most companies send 100% of traffic to one model. Simply routing classification and extraction tasks to GPT-5-nano ($0.05 input) instead of GPT-5 ($10 input) cuts 30-50% of spend for those task types. You can implement basic routing in a day; a production-grade system with quality monitoring takes 2-3 weeks.

Should we build our own AI gateway or use a platform?

Building a basic proxy with routing takes a few weeks. But maintaining it — keeping up with weekly model launches, pricing changes, cache invalidation, quality monitoring — is a full-time job. Platforms like Veltrix provide routing, caching, compression, cost dashboards, and team governance out of the box. Most teams find that the engineering time saved exceeds the platform cost within the first month.