AI Inference Cost Calculator

API inference costs look small per request — usually fractions of a cent. Then your app gets ten thousand users and the monthly bill is $4,000. The math is straightforward but unintuitive: cost scales with token volume, not with API calls. Two short requests can cost less than one long one. Knowing your tokens-per-request matters more than knowing your request count.

How API pricing actually works

Every major LLM provider charges by tokens — pieces of text roughly equal to 4 characters in English. A token is not a word. "Calculator" is roughly 2 tokens. "AI" is 1. A typical English word averages 1.3-1.4 tokens. Languages with more characters per word (German) or with non-Latin scripts (Chinese, Arabic) often have very different ratios.

Pricing splits into two halves: input tokens (everything you send — your prompt, conversation history, RAG context) and output tokens (what the model generates). Output is almost always more expensive per token than input, often 2-5×. This means a chatbot that takes a short user message and produces a long response costs roughly the same as one that takes a long context and produces a short response.

Prices change every few months — always verify against the provider's current pricing page before building cost projections. The gap between flagship and lighter models keeps narrowing, and the lighter models often handle 80-90% of real-world tasks well enough.

The per-request math

For each API call, cost equals (input tokens × input rate) + (output tokens × output rate), both per million. A typical chatbot request with 500 input tokens (system prompt + user message) and 300 output tokens through GPT-4o costs:

(500 × $2.50 / 1,000,000) + (300 × $10.00 / 1,000,000) = $0.00125 + $0.003 = $0.004 per request

That same conversation through Gemini Flash: $0.000038 + $0.00009 = $0.000128 per request. Same task, 30× cheaper. The difference compounds at scale.

Why output tokens cost more

The asymmetry between input and output pricing isn't arbitrary. Input tokens get processed once during the "prefill" phase — the model reads them all at once and they can be batched efficiently across requests. Output tokens are generated one at a time during "decoding" — each new token requires a full forward pass through the model, can't be batched the same way, and uses GPU compute differently.

The practical implication: if you can reduce output length without losing utility, you save more than reducing input length. A response that's 50% shorter saves more money than a prompt that's 50% shorter at most providers.

Context caching: the biggest cost lever most teams miss

Most providers now support context caching (Anthropic prompt caching, OpenAI prompt caching, Gemini context caching). The idea: when you send the same large prefix repeatedly — system prompts, RAG context, long documents — the provider caches it after the first request and charges much less for re-using it.

For RAG applications where the same long context gets reused across user queries, context caching can reduce inference costs by 60-80%. It's free money sitting in the API for teams that haven't implemented it yet.

Streaming responses and what they actually cost

Streaming (where tokens arrive one at a time as they're generated) doesn't change the total cost per request — you pay for the same tokens whether they arrive all at once or piece by piece. But streaming does change something important: it lets users see partial responses and choose to stop generation early.

For chat-style applications, this matters more than people realize. If users can hit "stop" or close the response when they've seen what they need, output token consumption can drop 20-40% in practice. The UX win and the cost win align.

The catch: streaming adds latency at the start (slight setup overhead) and complicates error handling (a stream that fails midway is different from a completed request that fails). For non-realtime workloads where users wait for completion anyway, batch processing is the better cost play.

Vision and audio tokens: more expensive than they look

Multimodal models charge for images and audio inputs, and the token counts are often surprising. An image isn't "free input" — it's converted into a sequence of visual tokens that get counted just like text tokens.

A vision-heavy application analyzing 100 images per day at standard resolution uses 76,500 input tokens just for the images — before any text prompt. At GPT-4o rates, that's $0.19/day or $5.74/month for the visual content alone. Scale to 10,000 images/day and it becomes $570/month for vision inputs.

Function calling and tool use overhead

Function calling (also called tool use) adds tokens beyond the conversation itself. Every available tool gets defined in the system context, and every tool call gets counted as output tokens.

A typical function-calling setup with 5 tools defined adds roughly 800-1,500 input tokens per request (the tool schemas) plus 50-200 output tokens per tool call. For a tool-heavy agent making 3-5 tool calls per user interaction, this can be 30-50% overhead on the base text generation cost.

Optimization tactics: only include the tools relevant to the current task (use a router/classifier to narrow down), keep tool descriptions tight, and use structured output when possible instead of free-form function arguments.

Common mistakes

• Using flagship models for simple tasks. Classification, summarization of short text, basic Q&A — GPT-4o mini or Gemini Flash handle these at 1/15th the cost of GPT-4o.
• Not setting max_tokens. Letting the model run to its natural stop can produce twice the output you needed. Set max_tokens conservatively — you can always increase if responses get truncated.
• Sending full chat history every turn. A 20-message conversation eventually sends 19 previous messages with each new request. Implement context summarization or sliding window for long conversations.
• Ignoring batch APIs. Most providers offer 50% discount on async batch processing (24-hour turnaround). For non-realtime workloads (overnight processing, daily reports), batch is much cheaper.
• Premature optimization. Building elaborate cost-reduction infrastructure before you know your actual usage patterns. Run for a month, look at the bill, then optimize the top 20% of cost drivers.

Cache TTL and storage strategy

Prompt caching isn't permanent. Each provider has different time-to-live (TTL) for cached content. Anthropic's prompt cache expires after 5 minutes by default. OpenAI's automatic cache is shorter. Gemini's explicit cache can be configured up to 1 hour.

For RAG applications where the same context might be needed multiple times within an hour, configuring longer cache TTL pays for itself quickly. For chat applications with bursts of similar queries from one user, even the default 5-minute cache provides substantial savings.

Latency vs cost tradeoffs

Pricing isn't the only consideration. Different models have very different latency characteristics, which matters for user-facing applications.

For chatbot applications where users see responses streaming, faster TTFT and higher throughput beat lower cost. For background processing where users wait for completion anyway, batch APIs and slower models win on cost.

Questions and answers

How many tokens is 1,000 words?

Roughly 1,300-1,500 tokens for typical English prose. Code is denser (often 2,500+ tokens per 1,000 words) because of symbols and short variable names. Non-Latin scripts can be 2-3× more tokens per word.

Do I pay for prompts that return errors?

Generally no, if the error is on the provider's side (rate limits, server errors). You pay if your request was malformed but reached the model. Check the specific API docs for edge cases.

What's the cheapest production-ready API?

Currently Gemini 1.5 Flash for most tasks, with DeepSeek V3 close behind. For very high volume, self-hosting open models on dedicated GPUs (via providers like Together, Modal, Replicate) can beat any API at scale above 100M tokens/month.

Sources

• OpenAI API pricing documentation (openai.com/api/pricing)
• Anthropic API pricing (anthropic.com/pricing)
• Google AI pricing for Gemini models
• Artificial Analysis: model comparison benchmarks

Related: AI Tokens Per Word · Vector Database Cost · RAG System Cost · GPU Memory Required