AI Latency Calculator

Understanding the AI Latency Calculator

An AI latency calculator estimates how long a language model response takes, splitting it into time to first token and generation time. The split matters more than the total, because the two feel completely different to a user and are optimised in different ways.

How it actually works

Enter input tokens, output tokens, and a model speed tier. The calculator applies a time-to-first-token figure and a tokens-per-second rate for the chosen tier, then adds generation time to the initial delay. One thousand output tokens at the medium tier gives 0.50 seconds to first token and 20.50 seconds total at 50 tokens per second.

The deeper context most people miss

Time to first token and tokens per second have different causes. The first is dominated by the prefill phase, where the model processes your entire input before producing anything, so it scales with input length. The second is the decode phase, where tokens are generated one at a time, and it scales with output length. Long prompts hurt the first, long answers hurt the second.

Why prefill and decode behave so differently

Transformer inference has two distinct phases with different computational characteristics, and understanding the difference explains most latency behaviour. Prefill processes the entire input prompt in a single forward pass, and because all input tokens are known in advance, this work parallelises well across the available compute. It is therefore compute-bound: it uses the hardware efficiently and its duration scales roughly with input length, though sublinearly on modern hardware. Decode generates output tokens one at a time, each conditioned on all previous tokens, which makes it inherently sequential and impossible to parallelise across the sequence. Each token requires reading the entire model's weights from memory to compute a single step, which makes decode memory-bandwidth-bound rather than compute-bound. This is the key insight: generation speed is limited by how fast weights can be streamed from memory, not by raw computational throughput, which is why memory bandwidth is the specification that matters most for inference hardware and why batching multiple requests together improves throughput dramatically without proportionally increasing latency, since the same weight read serves many requests. It also explains why a model with twice the parameters is roughly half as fast to generate, and why techniques including quantisation, which reduces the bytes per parameter, improve generation speed substantially. Speculative decoding, where a small fast model drafts tokens that the large model verifies in parallel, attacks the sequential bottleneck directly and can produce meaningful speedups.

A worked example: where the time actually goes

The default gives 0.5 seconds to first token and 20 seconds of generation for 1,000 output tokens, so 97% of the wait is generation. That distribution has clear design implications. Streaming the response so tokens appear as they are produced transforms the experience: the user sees output after 0.5 seconds rather than staring at a blank screen for 20, and because reading speed for most people is somewhere around 200 to 300 words per minute, which is roughly 4 to 7 tokens per second, a model generating at 50 tokens per second produces text far faster than anyone reads it. Once generation outpaces reading, further speed improvements stop being perceptible for that use case. This is why streaming is close to mandatory for conversational interfaces and why the perceived latency of a streamed response bears little relation to its total duration. Now change the scenario: a classification task returning a single token has essentially no generation time, so the entire latency is time to first token, and reducing prompt length becomes the only lever that matters. A long document summarisation with a 50,000 token input and a 500 token output inverts it again, with prefill dominating. The optimisation that helps depends entirely on the shape of the workload, and applying a general latency figure without knowing that shape produces poor decisions.

Deciding how to reduce latency in practice

Several levers exist and they address different parts of the problem. Reducing output length is the most direct for generation-heavy workloads: instructing the model to be concise, using structured output formats, and avoiding requests for lengthy explanations all cut tokens generated. Reducing input length helps prefill, and prompt caching, offered by several providers, stores the processed state of a repeated prefix so subsequent requests skip reprocessing it, which is substantial for applications sending a large system prompt or document with every request. Choosing a smaller model is often the largest single improvement, and many tasks that appear to need a frontier model perform adequately on a smaller one, which is worth testing rather than assuming. Streaming does not reduce latency but dramatically improves perceived latency, and it is usually the highest-value change for interactive applications. Parallelising independent requests helps throughput where the workload allows. Moving work off the critical path matters: anything that can be precomputed, cached, or done asynchronously while the user does something else effectively costs nothing. For applications making multiple sequential model calls, each one adds its full latency, so reducing the number of round trips frequently matters more than optimising any single call. Provider and region selection affects network round-trip time, which is small relative to generation but not negligible for short responses.

What the tokens-per-second figure hides

Published throughput numbers are measured under specific conditions and real performance varies considerably. Load is the largest factor: shared inference endpoints serve many customers, and latency rises during peak periods, sometimes substantially, which is why providers offer provisioned throughput options that reserve capacity at higher cost. Context length affects generation speed, since attention over a longer context requires more work per token, so a request with 100,000 tokens of context generates more slowly than one with 1,000 even at the same output length. Batch size affects the trade-off between throughput and latency, with larger batches improving tokens per second across all requests while increasing the latency of any individual one. Output variability matters: a model asked an open-ended question may generate 200 or 2,000 tokens depending on the response, so latency planning needs distributions rather than point estimates, and the tail matters more than the mean for user experience. Reasoning models complicate this further, since they generate substantial intermediate reasoning tokens before producing a visible answer, and that reasoning is often not streamed, so the user experiences a long silent wait followed by an answer, which is a different and worse experience than progressive streaming despite potentially better output. Measuring your own workload against your own provider under realistic load is the only reliable approach, and published benchmarks are a starting point rather than a specification.

Variations: reasoning models, batch processing, and local inference

Reasoning models trade latency for capability, generating extended internal reasoning before answering, which can multiply response time by a large factor and changes the latency calculation fundamentally since reasoning token counts are variable and often not visible. Batch APIs, offered by several providers at reduced cost, process requests asynchronously with completion times measured in hours rather than seconds, which suits offline workloads and is dramatically cheaper. Local inference on consumer hardware has become viable for smaller models, with generation speed governed by memory bandwidth, so unified memory architectures and high-bandwidth GPUs perform well while systems bottlenecked on memory perform poorly regardless of raw compute. Quantisation reduces model precision to fit more parameters in memory and stream them faster, with modest quality cost at moderate quantisation levels. Speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel. For applications where latency is critical, a smaller specialised model, a fine-tuned model, or a non-model approach entirely will frequently outperform a large general model on both speed and cost.

Managing AI latency

Split your thinking between time to first token and generation time, since prefill scales with input length and decode with output length, and the optimisations differ. Stream responses for anything interactive, since it transforms perceived latency and most models generate faster than people read at around 4 to 7 tokens per second. Reduce output length first for generation-heavy workloads, since it is the most direct lever. Use prompt caching where a large prefix repeats across requests, which eliminates redundant prefill work. Test smaller models rather than assuming a frontier model is needed, since this is frequently the largest single improvement available. Minimise sequential round trips, since each model call adds its full latency and reducing the number often matters more than optimising any one. Plan for distributions rather than point estimates, since output length varies and the tail matters more than the mean for user experience. And measure your own workload under realistic load rather than relying on published benchmarks.

What people get wrong

• Optimising total latency without splitting prefill from decode, when long prompts and long outputs cause different problems requiring different fixes.
• Chasing faster generation for interactive use beyond reading speed, when most people read at roughly 4 to 7 tokens per second and streaming makes further gains imperceptible.
• Treating published tokens-per-second figures as reliable, when load, context length, and batch size all shift real performance substantially.
• Planning around average latency, when output length varies widely and the tail of the distribution determines how the application actually feels.

Where the math comes from

Total Latency = Time to First Token + (Output Tokens / Tokens per Second). Speed tiers apply 0.3 seconds and 80 tokens per second for fast, 0.5 and 50 for medium, and 1.0 and 30 for slow. Time to first token reflects the prefill phase and scales with input length, while generation time reflects the sequential decode phase and scales with output length.

Questions and answers

Related calculators

AI ROI · AI Energy Use · AI Chatbot Total Cost · AI Training Cost · GPU Memory Required