AI Tokens Per Word Calculator

Tokens are not words. They're chunks of text that LLMs use as their basic unit, and the ratio between tokens and words varies more than most developers realize. A 1,000-word document in English is roughly 1,300 tokens. The same content translated to Chinese might be 1,800 tokens. As JSON, it might be 2,500. As code, it could hit 3,000. Knowing your token ratio matters for cost projection and context window planning.

What tokenizers actually do

LLMs don't read text directly. Before the model sees anything, a tokenizer splits the text into pieces — usually subword chunks based on statistical frequency. Common words become single tokens. Rare words get split into multiple tokens. Whitespace and punctuation sometimes get their own tokens, sometimes get merged with adjacent words.

Different models use different tokenizers. OpenAI's GPT-4 uses tiktoken with the cl100k_base or o200k_base vocabulary. Claude uses its own tokenizer. Llama uses SentencePiece. These produce different token counts for the same input text — sometimes by 20-30%.

Tokens-per-word by language

English is unusually efficient for tokenizers — most common English words map to single tokens, and the tokenizer vocabularies were primarily trained on English text. Other languages can be substantially less efficient.

This matters for two reasons. First, multilingual applications cost more per "equivalent meaning" when operating in non-English languages. A French user message and an English user message conveying the same idea use different token counts. Second, context windows fill faster in non-English languages. A "128K token" context window holds about 95K English words but only 35-40K Hindi words.

Special cases that surprise developers

Why context windows matter

Every model has a maximum context window — the total tokens (input + output combined) it can process in one request. Going beyond the limit produces an error. The numbers sound large but fill up faster than you'd expect.

For perspective: a typical novel is 70,000-100,000 words. Claude can hold an entire novel in context. Gemini Pro can hold an entire textbook. These large context windows enable use cases (analyzing entire codebases, long legal documents, full book Q&A) that weren't possible with 8K or 32K windows.

Counting tokens before sending

For production cost projection, count tokens before sending. Each provider offers tools:

• OpenAI: tiktoken library (Python, JS). Returns exact token counts for any model.
• Anthropic: count_tokens API endpoint. Free, returns Claude-specific token count.
• Google: count_tokens method on the Gemini API. Free.
• Hugging Face Transformers: tokenizer libraries for all major open models.

For rough estimation without an API call, use 0.75 words per token as a default for English. So 4,000 tokens ≈ 3,000 words. This estimate is good enough for planning but not for billing-accurate calculations.

A brief history of tokenization

Modern LLM tokenization is built on Byte-Pair Encoding (BPE), an algorithm Phillip Gage published in 1994 for data compression. Researchers at Edinburgh and OpenAI adapted it for neural machine translation in 2015-2016, and it became the standard for transformer models. The basic idea: start with single characters, then iteratively merge the most common pairs into single tokens until you reach a target vocabulary size.

GPT-2 used a 50,257-token vocabulary. GPT-3 expanded slightly to 50,257 (same). GPT-4 grew to 100,277 tokens (cl100k_base). GPT-4o pushed to 200,019 (o200k_base). Each expansion improved efficiency for non-English text and code — what took 5 tokens in GPT-3 might take 3 tokens in GPT-4o.

Different tokenization algorithms have different strengths. BPE (used by OpenAI and many others) excels at general text. WordPiece (BERT, ELECTRA) handles morphology better — useful for languages with complex word forms. SentencePiece (Llama, T5) treats whitespace as just another character, which works better across languages with different word-boundary conventions.

Prompt compression: getting more from fewer tokens

For applications hitting context window limits or paying high token costs, prompt compression is the technical frontier. Several approaches work:

The newest approaches use "soft prompts" — embedding-level representations that aren't human-readable but encode instructions more densely than text. For applications doing the same task repeatedly (a SaaS feature with fixed system instructions), soft prompts can replace 500-2,000 text tokens with 10-20 embedding tokens.

Edge cases that produce surprising token counts

• Trailing whitespace. Some tokenizers count trailing spaces as separate tokens. Two text strings that look identical can have different token counts.
• Unicode characters with combining marks. Characters like emoji with skin-tone modifiers can use 5-10 tokens each.
• Repeated punctuation. "!!!!" is usually 1-2 tokens but "?????" might be 4. Inconsistent across tokenizers.
• URLs. A URL like "https://example.com/very/long/path/with/many/segments?query=string" can use 15-25 tokens for what looks like a single "word."
• UUIDs and hashes. Random strings can't be tokenized efficiently. A UUID is typically 12-18 tokens despite being a single identifier.
• Code with single-letter variables. Compact code (i, j, x, y) uses fewer tokens than verbose code with full names.
• JSON keys. JSON keys get tokenized separately from values. Verbose key names ("customerFirstName" vs "name") substantially increase token count.

Practical token budgeting

For production applications, build a token budget per request type. Don't just track "average tokens per request" — track p50, p95, and p99. Long-tail requests (rare cases with much longer context) often drive surprise costs.

A typical chatbot might have: p50 = 1,800 tokens, p95 = 4,500 tokens, p99 = 12,000 tokens. Budgeting based on p50 underestimates actual costs by 15-25%. Use weighted average ((p50 × 0.7) + (p95 × 0.25) + (p99 × 0.05)) for realistic projections.

Tokenizer playground tools

To see exactly how text gets tokenized, several free playground tools let you paste text and see the token breakdown visually. OpenAI's tokenizer page (platform.openai.com/tokenizer) shows GPT tokenization. Hugging Face hosts tokenizer demos for most open models. These are invaluable for debugging surprising token counts in production prompts.

One quick exercise that teaches more than reading about tokenization: paste a typical user prompt from your app into a tokenizer playground. The visual breakdown often reveals 10-20% of tokens you didn't realize were there (extra whitespace, hidden Unicode, structural overhead).

Common mistakes

• Assuming "tokens = words" for cost math. You'll undershoot by 30-40%. Always multiply word count by 1.3-1.4 minimum.
• Ignoring system prompts in token count. The system prompt (often 200-2000 tokens) is counted in every request. A long system prompt at high volume is a real cost driver.
• Forgetting conversation history. Multi-turn chats include all previous messages in each new request. A 20-message conversation eventually costs more per turn than the first message did.
• Using one tokenizer for all models. tiktoken counts are accurate for OpenAI but wrong for Claude or Gemini by 5-15%.
• Not accounting for tool calls. Function calling adds tokens for the function schema, tool calls, and tool results. Tool-heavy applications can have 50%+ overhead.

Questions and answers

Why doesn't 1 word = 1 token?

Because tokenizers optimize for compression across the entire training corpus. Common words become single tokens; rare words split into multiple tokens. "The" is 1 token. "Antidisestablishmentarianism" is probably 6.

Can I write more efficiently for tokens?

Slightly. Shorter common words, avoiding unnecessary punctuation, and removing redundant whitespace help marginally. For real cost reduction, focus on system prompt length and conversation history management — those have much bigger impacts.

Do images and audio count as tokens?

Yes, in multimodal models. Images convert to "tokens" representing visual patches — a typical image is 1,000-3,000 tokens depending on resolution. Audio in models like GPT-4o is similar. Multimodal inputs aren't "free."

Sources

• OpenAI: tiktoken library documentation (github.com/openai/tiktoken)
• Anthropic Claude tokenizer reference
• Google Gemini count_tokens documentation
• Sennrich et al. (2016): "Neural Machine Translation of Rare Words with Subword Units"

Related: AI Inference Cost · AI API Cost · ChatGPT Token Counter · RAG System Cost