RAG System Cost Calculator

RAG (Retrieval-Augmented Generation) costs more than people expect because they only price the last step. The visible cost is the LLM inference at the end. The hidden costs are embedding all your documents up front, storing them in a vector database, re-embedding when content changes, and retrieving relevant chunks for every query. The retrieval and generation phases each have their own pricing models that compound.

The three cost layers of any RAG system

Every RAG pipeline has three cost layers. Skipping any of them in your math gives you a wildly low estimate.

Example: cost breakdown for a real system

Consider a customer support chatbot indexing 50,000 documents (avg 500 tokens each = 25M tokens total), serving 1,000 queries per day, using OpenAI GPT-4o for generation and OpenAI text-embedding-3-small for embeddings.

Notice how the generation phase dominates: $375 of $375 monthly cost, the retrieval infrastructure is rounding error. This is typical. For most RAG systems at moderate scale, the LLM inference is 90-98% of total cost. Vector database costs only become significant at very large scale (10M+ vectors) or very low query volume.

Chunking strategy and its cost impact

How you split documents into chunks affects retrieval quality and cost. Larger chunks mean more context per retrieval but more tokens sent to the LLM. Smaller chunks mean cheaper individual retrievals but you need to retrieve more chunks per query to get the same coverage.

The current best practice is hierarchical or hybrid chunking: small chunks (300-500 tokens) for retrieval, then expanding to larger surrounding context (1,000-2,000 tokens) when feeding to the LLM. This gets precision from small chunks for retrieval and coverage from larger chunks for generation.

When to switch models or strategies

If RAG generation costs become a significant line item, three changes typically save 50-80%:

1. Switch to a lighter generation model. GPT-4o mini or Gemini Flash handle most RAG tasks well. The retrieval already filters down to relevant content — the model often just needs to summarize and cite.
2. Implement context caching. If you have a long system prompt or stable retrieval results across queries, providers offer 50-90% off cached input tokens. See your provider's prompt caching docs.
3. Reduce retrieved chunk count. Most RAG systems retrieve top 5-10 chunks. Many tasks work fine with top 3. Cutting from 5 to 3 chunks reduces context tokens by 40% with minimal quality loss.

Re-ranking: the often-missed cost layer

Many production RAG systems add a re-ranking step between vector retrieval and LLM generation. The pattern: retrieve top-50 candidates with cheap vector search, then re-rank with a more accurate (and more expensive) model to find the actual top-5 to send to the LLM. This significantly improves answer quality at moderate cost.

Re-ranking models like Cohere Rerank, Voyage rerank-2, or open models like BGE-reranker cost per token of (query + candidate document) pairs evaluated. For a 50-candidate re-rank with 500-token documents and a 100-token query, you're evaluating roughly 30K tokens of pairings.

For a system doing 100K queries/month with 50-candidate re-rank, this is roughly $200-300/month for the re-ranking layer — significant but often worth it for accuracy-sensitive use cases.

Hybrid search: BM25 + vector cost dynamics

Hybrid search combines traditional keyword search (BM25) with semantic vector search. The retrieval quality improvement is substantial — typically 15-25% better recall — but the cost implications depend on the implementation.

If your vector database supports hybrid natively (Weaviate, Qdrant, Vespa, OpenSearch), the marginal cost is small. If you have to run two separate systems (Pinecone for vectors + Elasticsearch for keywords), you're doubling infrastructure.

The practical recommendation: pick a vector database with native hybrid search if you'll need it. Retrofitting hybrid search after the fact often costs more in engineering time and infrastructure than picking the right vendor up front.

Evaluation costs: the often-forgotten line item

Production RAG systems need ongoing quality monitoring. "LLM-as-judge" evaluation (using another LLM to score response quality) is the current standard, and it has real costs.

A typical eval setup: for each production query, store the query, retrieved context, and generated response. Periodically run a judge LLM on a sample (5-10% of queries) to score answer quality. The judge typically uses 3,000-5,000 tokens per evaluation, mostly input.

At 100K queries/month with 10% sampling, that's 10K evaluations × 4K avg tokens = 40M tokens/month for evaluation. At Claude Sonnet prices, that's roughly $120-180/month just for evaluation. Often skipped in initial cost projections, but essential for quality assurance.

When to stop optimizing RAG cost

RAG systems can be optimized to extreme degrees, but at some point the engineering time exceeds the savings. Stop optimizing when:

• Total RAG cost is under 5% of your product's revenue or operations cost
• Each engineer-week of optimization saves less than $2,000-5,000/month
• Optimization starts degrading user experience (slower responses, lower quality)
• You're spending more time on cost optimization than on actual product features

Most teams over-optimize too early or under-optimize too late. The sweet spot is monitoring monthly costs, optimizing when they grow past a defined threshold (often 10-20% of revenue or budget), and otherwise focusing on product work.

Tracking cost per query type

Not all RAG queries cost the same. Simple factual queries might use 1,500 tokens total. Complex multi-document synthesis queries can use 8,000+ tokens. The average tells you total cost; the distribution tells you what to optimize.

Production RAG systems benefit from logging query category alongside token usage. After a month of data, you'll typically find that 10-20% of query types account for 50-60% of cost. Those are the targets for caching, model downgrades, or context length limits.

Common mistakes

• Pricing only the LLM call. The first cost projection most teams build ignores embedding, storage, and re-indexing. Add 10-20% to the LLM-only estimate.
• Re-embedding everything on every change. Some teams re-embed entire knowledge bases when a few documents change. Implement document-level change detection to only re-embed modified content.
• Using oversized embedding models. text-embedding-3-large (3,072 dim) costs 2× text-embedding-3-small (1,536 dim) and the small model is sufficient for 90% of use cases.
• Retrieving too many chunks "to be safe." Each extra chunk adds 500-2,000 input tokens, costs real money, and often degrades response quality by diluting relevant content with marginal matches.
• No metadata filtering. Vector similarity alone often retrieves wrong content. Adding metadata filters (date, category, source) before similarity search cuts irrelevant retrievals dramatically.

Questions and answers

When is RAG cheaper than just sending everything to the LLM?

Almost always above 10K tokens of source material. A 100-page document is roughly 30K-40K tokens. Sending that with every query costs $0.10-0.50 per request through GPT-4o. RAG retrieves only the relevant 2-3K tokens per query, cutting cost 90%+.

Do I need a vector database, or can I use Postgres?

For under 1 million vectors with modest query volume, pgvector (Postgres extension) works fine and is much cheaper. Above 5-10 million vectors or with strict latency requirements, dedicated vector databases (Pinecone, Weaviate, Qdrant) pull ahead.

How often should I re-embed my corpus?

Only when you change embedding models. The vectors don't degrade over time. Re-embed when you switch from text-embedding-ada-002 to text-embedding-3, or when adding new content.

Sources

• Lewis et al. (2020): "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"
• OpenAI embedding model pricing documentation
• Pinecone, Weaviate, Qdrant: published pricing and architecture docs
• LangChain and LlamaIndex: documentation on RAG patterns and costs

Related: Vector Database Cost · AI Inference Cost · AI Tokens Per Word · AI API Cost