When you build an application on top of an LLM API, you're essentially renting computational reasoning. The pricing structure of major AI providers — OpenAI, Anthropic, Google — follows a deceptively simple model: pay per token consumed. But that simplicity hides significant complexity in how those costs accumulate, why they vary across models, and how to control them in production.

The Fundamental Unit: Tokens

1 token ≈ 4 characters of English ≈ 0.75 words

Every API call is priced based on tokens consumed: tokens you send in (input/prompt) and tokens the model generates back (output/completion). Practical conversions: 100 tokens ≈ 75 words ≈ a short paragraph. 1,000 tokens ≈ 750 words ≈ 1.5 standard pages. Code, structured data (JSON), and non-English languages tokenize differently — sometimes 2-3× more tokens for equivalent information. Use our ChatGPT Token Counter to estimate token counts.

Input vs Output Pricing: The Critical Distinction

Input and output tokens are priced separately, with output typically costing 2-5× more than input. This reflects the actual computational cost — generating tokens requires the model to run a forward pass through its weights for each output token, while input tokens are processed in parallel during the prefill phase.

For most LLM applications, cost breakdown roughly follows: input cost × ratio of input tokens + output cost × ratio of output tokens. A typical chatbot exchange of 500 input tokens (system prompt + user message) and 300 output tokens (response) heavily weights toward output cost despite output being only 37% of total tokens.

Context Windows: Hard Cost Multipliers

Each model has a maximum context window — total tokens (input + output) it can process in one request. Modern frontier models support 128K-1M tokens. Older or cheaper models support 4K-32K.

Context window matters subtly. The longer your context, the more you pay per request — sending 50K tokens of context to extract one piece of information costs 50× more than sending 1K tokens. Hitting the context window limit causes errors or silently truncated responses. Conversational chatbots that retain message history hit ballooning costs as conversations extend.

The Real Cost: API Calls × Usage Pattern

Token-level pricing is meaningless without understanding usage patterns.

Pattern 1: Single-Shot Q&A

User sends a question, gets an answer, conversation ends. A help-bot serving 10,000 questions/day at 500 input + 300 output tokens averages ~8M tokens daily.

Pattern 2: Multi-Turn Conversation

Each new user message includes all previous messages and assistant responses as context. After 10 turns, context per request might be 8,000+ tokens before the new message arrives. Costs grow superlinearly with conversation length.

Pattern 3: Document Processing

A 50-page document might be 30,000+ tokens. Processing one document costs as much as 60-100 chatbot exchanges. Document QA over thousands of documents requires retrieval-augmented generation (RAG) to avoid sending the whole corpus on every query.

Pattern 4: Agentic Workflows

Multi-step automated tasks where the model calls tools, observes results, calls more tools. A single complex task can involve 10-20 LLM calls, each with growing context. Costs spiral if loops aren't bounded. Use our AI Inference Cost Calculator to model your usage.

The Comparative Cost Question

Frontier models from OpenAI, Anthropic, and Google offer similar capabilities at different price points. Smaller, cheaper variants (mini class, Haiku class, Flash class) typically cost 5-15× less than flagship models while handling many real-world tasks adequately.

The right model isn't always the most capable one — it's the cheapest one that meets your quality bar. A customer service bot doesn't need GPT-4 for every interaction; a smaller model handles 90% of queries fine, with escalation to flagship only for complex cases. This "cascading model" strategy reduces costs by 70-90%. Use our LLM Comparison Calculator.

Cost Reduction Strategies

Prompt Caching

When prompts share common prefixes (system instructions, examples, retrieval context), the cached portion costs 10-50% of standard input pricing. The cache typically expires after 5-10 minutes of inactivity. For high-traffic applications with consistent system prompts, this represents major savings.

Batching

Multiple providers offer batch APIs at 50% discount. Trade-off: batch jobs complete asynchronously over hours rather than seconds. For non-real-time use cases (data processing, content generation, classification), batch APIs cut costs in half with zero quality impact.

Smaller Models for Most Tasks

Flagship models handle complex reasoning. Smaller models handle routine tasks at 5-15× lower cost. Most production applications can use small models for 70-90% of traffic, reserving flagships for genuinely hard cases.

Output Token Limits

Setting `max_tokens` parameters prevents runaway generation. A bug that causes the model to generate 10,000 tokens of repetitive output instead of 300 doesn't ruin your bill if you've capped output at 500.

Structured Outputs

Using JSON mode or structured output features prevents the model from wasting tokens on conversational filler. "Output a JSON object with these fields" produces 50% fewer tokens than verbose natural language responses.

Embeddings and RAG

Retrieval-augmented generation reduces context costs by retrieving only relevant snippets instead of sending entire documents. Embedding generation (one-time cost) plus retrieval (very low per query) is dramatically cheaper than including full documents on every query. See our Embedding Cost and RAG System Cost calculators.

Self-Hosted vs API: The Crossover Point

For sufficiently high-volume applications, self-hosting open-source models on your own GPUs becomes cheaper than API calls. The crossover depends on:

  • API pricing for your chosen model
  • GPU hourly costs ($1-15/hour depending on hardware)
  • Throughput (tokens/second your hardware can produce)
  • Utilization (idle GPUs are wasted money)
  • Engineering overhead (deployment, monitoring, scaling)

Rough rule: self-hosting becomes interesting around 100M+ tokens/day of consistent traffic. Below that, API pricing wins. Above that, per-token economics shift in favor of self-hosting for applications that can use open-source models. Use our Self-Hosted vs API Calculator.

A Complete Cost Example

Customer service chatbot serving 10,000 conversations/day, each averaging 4 turns.

Per-conversation tokens: system prompt (300, cached after first call), accumulated user messages (200 average), accumulated assistant responses (300 average). After 4 turns: 300 cached + (800 × 4) + (300 × 4) = 4,700 tokens.

Daily volume: 10,000 × 4,700 = 47M tokens. Monthly: ~1.4B tokens.

Cost (flagship model): ~$5,000-15,000/month depending on provider.

Cost (small model): ~$400-1,500/month for the same workload.

Cost (small model + caching): additional 10-20% savings.

Same application can have 10-20× cost spread depending on model and optimization. Without intentional cost engineering, AI applications start expensive and stay that way.

Common Mistakes

Defaulting to flagship models for everything: small models handle most tasks at 10-15× lower cost.

Not capping output tokens: a bug or prompt injection can produce wildly long outputs that wreck your bill.

Sending full conversation history every turn: summarize older context after a threshold to keep request size bounded.

Ignoring prompt caching: high-traffic apps with consistent system prompts leave 30-50% of cost on the table without it.

Not using batch APIs for non-real-time work: 50% discount for asynchronous processing.

Including entire documents in context instead of using RAG: 10-100× cost overhead vs retrieval-based approaches.

Tools to Use

AI API costs are predictable when you understand the patterns. They're also extremely manageable with the right architecture — but it takes intentional design choices, not default everything-in-flagship-model implementation. The savings from thoughtful cost engineering on a high-volume AI application typically pay for the engineering time many times over.