There's a recurring scene we've watched play out across every AI startup we've spoken with in the last year: the CTO opens the AWS or provider billing dashboard, sees a five-figure monthly LLM bill, and realizes the application was architected without any cost engineering. The default everything-in-flagship-model implementation works fine at development scale and becomes financially uncomfortable at production scale. By that point the architecture is hard to change.
This article is the foundation that should have been laid before the first production deployment. We work through what we call the Cost-Per-Useful-Output (CPUO) framework: tokens, pricing, usage patterns, optimization levers, and the self-host crossover point. By the end you'll be able to estimate cost for any LLM application before you build it, which is materially harder once it's already running.
Tokens: The Unit That Determines Cost
LLMs don't process characters or words. They process tokens — typically 3-4 characters or about 0.75 words in English. The token-to-word ratio matters more than people expect: a 1,000-word document is roughly 1,300-1,400 tokens, not 1,000.
Five behaviors most cost models miss:
- Whitespace and punctuation count. Each space, newline, and punctuation mark consumes tokens. JSON output is token-expensive for this reason.
- Languages tokenize unevenly. Chinese and Japanese run 2-3× tokens per character. Code runs 1.5-2× tokens vs natural English. Translation costs vary dramatically by language pair.
- Special tokens add overhead. System prompts, role tags, and formatting markers all count toward the bill even though they don't reach the user.
- Reasoning models burn invisible tokens. Models with internal reasoning (o3, DeepSeek-R1, etc.) consume "reasoning tokens" that aren't returned to you but appear on your invoice — sometimes 5-20× the visible output count.
- The first cost surprise is usually JSON. Structured output formats inflate token counts by 30-50% vs prose. Worth modeling explicitly.
Quick estimate: when scoping cost, count words in your prompts and expected outputs, multiply by 1.4, and that's your token estimate. Add 20% buffer for special tokens and formatting.
Input vs Output: The Asymmetry That Drives Cost
Output tokens cost 3-5× more than input tokens on every major provider. This asymmetry is the single most important pricing dynamic to understand because it determines which optimizations matter.
| Model (2026) | Input ($/M) | Output ($/M) | Ratio |
|---|---|---|---|
| GPT-5 | $10.00 | $40.00 | 4× |
| Claude Opus 4.7 | $15.00 | $75.00 | 5× |
| Claude Sonnet 4.6 | $3.00 | $15.00 | 5× |
| Gemini 2.5 Pro | $1.25 | $10.00 | 8× |
| GPT-5-mini | $0.60 | $2.40 | 4× |
The implication: minimizing output verbosity has a 4-8× bigger impact on bill than minimizing input length. A common antipattern is feeding minimal context to the model and letting it write long responses — exactly backwards from what cost engineering would suggest. Better: give it rich context (cheaper) and constrain output verbosity (more expensive per token).
For a comprehensive provider-by-provider cost matrix including 23 models and 5 production scenarios, see our 2026 LLM pricing comparison.
Context Windows: The Hidden Cost Multiplier
Context window size determines how much information you can send per call. But every token in the context window is also a token you pay for on every call. Long-context use cases (200K+ tokens) get expensive fast because the same context gets re-sent every interaction.
The math: a 100K-token system prompt sent on every call at GPT-5 pricing costs $1.00 per call (input only). Multiply by 10,000 calls per day and that's $10,000/day in input cost — just for the system prompt. Same system prompt with prompt caching enabled costs $0.25 per call — a 4× reduction for what is essentially a configuration change.
The structural lesson: caching is no longer optional. If your application has any reusable prompt prefix (system instructions, RAG context, conversation history), enable caching before deployment. Skipping it is leaving 60-90% on the table.
The Four Usage Patterns and How They Cost
Most LLM applications fall into one of four cost-archetype patterns. Each has different optimization levers:
Pattern 1: Single-Shot Q&A
Short input, short output, one call per user query. Examples: factual questions, code completions, simple classifications. Lowest cost-per-call of any pattern. Optimization lever: use the smallest capable model. Mini-tier almost always works.
Typical cost: $0.001-0.01 per call. 100K queries/day at $0.005 = $500/day = $15K/month.
Pattern 2: Multi-Turn Conversation
Conversation history grows with each turn; cost-per-turn rises non-linearly. By turn 10, you may be sending 8,000+ tokens of accumulated history every turn. Optimization levers: aggressive prompt caching on history, conversation summarization at length thresholds, history truncation strategies.
Typical cost: starts at $0.01 per turn, grows to $0.10+ by turn 20. Cache reduces this by 60-80% for the common case.
Pattern 3: Document Processing
Large input (entire documents), structured output (summaries, extractions, classifications). Output is small but variable. Optimization levers: batching for non-realtime workloads (50% discount), output schema design to minimize tokens, classification rather than generation where possible.
Typical cost: $0.05-0.50 per document, depending on size. Bulk classification of 1M documents in batch mode on a mini-tier model: $100-300 total.
Pattern 4: Agentic Workflows
Multiple tool-calling iterations per user request, each consuming reasoning tokens. Cost-per-task rises with task complexity. Optimization levers: limit max iterations explicitly, validate tool outputs before continuing, use cheaper models for sub-tasks while reserving frontier models for synthesis.
Typical cost: $0.10-2.00 per agentic task. Reasoning-model agents (o3-pro, DeepSeek-R1) can cost $5-20 per task on complex workflows.
The Six Optimization Levers (Ranked by Impact)
From highest to lowest impact in our experience:
- Right-size the model. Most workloads run fine on mini-tier. Default to frontier only when measured quality difference justifies the 10-50× cost increase.
- Enable prompt caching. 60-90% input cost reduction for workloads with reusable prefixes. Free win that most teams skip.
- Use batch API for offline workloads. Automatic 50% discount on any non-realtime processing. Bulk classification, scheduled reports, data labeling, content moderation.
- Constrain output verbosity. Output costs 4-8× more than input; controlling response length is high-leverage.
- Architect for RAG over context-stuffing. Retrieve relevant chunks rather than passing entire knowledge bases. Reduces input tokens by 90%+ at the cost of retrieval infrastructure.
- Self-host at scale. Above $10K/month consistent workload, self-hosted open models become cost-competitive. Below that, operational complexity exceeds savings.
Self-Hosted vs API: Where the Crossover Sits
Self-hosting open models (Llama 3.3, DeepSeek V3.5, Mixtral) sounds expensive — GPUs cost real money — but at scale the math often inverts. The crossover depends on three variables: workload volume, workload consistency, and engineering capacity.
| Monthly API spend | Self-host viable? | Reasoning |
|---|---|---|
| < $5K | No | GPU rental floor exceeds savings; operational complexity not worth it |
| $5K - $15K | Maybe | Depends on workload consistency; spiky traffic kills self-host economics |
| $15K - $50K | Often yes | Bare-metal GPU rentals (~$3K/month for 4xA100) become cost-competitive |
| > $50K | Usually yes | Multi-GPU infrastructure pays for itself; need DevOps capacity |
The hidden cost is operational: self-hosting requires inference framework expertise (vLLM, TensorRT-LLM, SGLang), monitoring, autoscaling, GPU procurement. For a startup without ML infrastructure engineers, API hosting wins even at $30K/month because hiring time exceeds API spend.
A Complete Worked Example
SaaS application with 50,000 monthly active users. Average user has 3 conversations/week, 6 turns/conversation. System prompt is 2,000 tokens. Average user message 100 tokens; average response 400 tokens.
Monthly volume calculation:
- Conversations: 50,000 users × 3/week × 4.3 weeks = 645,000 conversations
- Turns: 645,000 × 6 = 3.87M turns
- Per turn: 2,000 (system) + 600 (avg user history) + 100 (current message) + 400 (response) = 3,100 tokens
- Monthly tokens: 3.87M × 3,100 = ~12B tokens (10B input, 2B output average)
| Approach | Monthly cost |
|---|---|
| GPT-5, no optimization | $180,000 |
| GPT-5 with caching | $48,000 |
| Claude Sonnet 4.6 with caching | $15,300 |
| GPT-5-mini with caching | $2,580 |
| Llama 3.3 70B self-hosted (2×A100 dedicated) | $2,400 + ops |
The spread between worst-case and best-case is 75×. The decisions that produce that spread are: model tier (5× impact), caching configuration (3-4× impact), and self-host crossover (1-2× impact at this scale).
Common Cost Mistakes
- Defaulting to flagship. "Quality first, optimize later" usually means "optimize never." Right-size from day one.
- Skipping caching. Free 60-90% reduction left on the table because the docs aren't read.
- Real-time everything. Workloads that don't need synchronous response belong on batch API.
- Letting output run unbounded. No max_tokens parameter, verbose system prompts asking for "detailed" answers, etc.
- Re-sending the same context every call. Cache it. Always.
- Underestimating reasoning tokens. Reasoning models charge for invisible tokens; check the actual usage report.
Run the Numbers
The framework above is general. Apply it to your specific architecture with the tools below:
- AI Inference Cost Calculator — model API spend across providers with custom workload profiles.
- ChatGPT Token Counter — estimate token counts from prompt text before committing.
- AI Training Cost Calculator — for fine-tuning or full training cost projections.
- AI Chatbot Total Cost Calculator — comprehensive cost model for conversational applications.
- AI ROI Calculator — when AI features pay for themselves.
The reason most AI startups discover cost engineering after deployment is that the cost dimension wasn't part of the architectural conversation. Make it part of the conversation upfront. Run the cost estimate before writing the integration code. Most of the savings from cost engineering compound; doing it early is dramatically cheaper than doing it later.