Home » RAG Pipelines » RAG Pipeline Cost

How Much Does a RAG Pipeline Cost to Run

A RAG pipeline serving 100,000 queries per month costs between $75 and $2,500 depending on the LLM choice, vector database, and retrieval configuration. The LLM generation step accounts for 60% to 80% of the total cost, making model selection and context size the highest-leverage cost decisions. Embedding, vector storage, and retrieval infrastructure together account for the remaining 20% to 40%.

The Full Cost Breakdown

A RAG pipeline has five cost components: document embedding during ingestion, vector database hosting, query-time embedding, optional reranking, and LLM generation. Each scales differently, and understanding the cost structure lets you make informed trade-offs between quality and spend.

The numbers below use July 2026 API pricing. Self-hosted alternatives reduce marginal costs but add fixed infrastructure costs. All figures assume a knowledge base of 1 million chunks (roughly 500,000 pages of text at 512 tokens per chunk) and 100,000 queries per month with an average of 4 retrieved chunks per query at 512 tokens each.

Embedding Costs

Ingestion embedding (one-time). Embedding 1 million chunks at 512 tokens each is 512 million tokens total. Using OpenAI's text-embedding-3-small at $0.02 per million tokens, the total cost is $10.24. Using text-embedding-3-large at $0.13 per million tokens, it is $66.56. Using Cohere embed-v4 at $0.10 per million tokens, it is $51.20. Using Voyage AI's voyage-3 at $0.06 per million tokens, it is $30.72. These are one-time costs that only recur when documents are added or updated. Even for large corpora, embedding is not a significant cost driver.

Query-time embedding. Each query must be embedded using the same model before vector search. A typical query is 20 to 50 tokens. At 100,000 queries per month with an average of 35 tokens per query, that is 3.5 million tokens. At $0.02 per million tokens (text-embedding-3-small), the monthly cost is $0.07. At $0.13 per million tokens (text-embedding-3-large), it is $0.46. Query-time embedding cost is negligible at any reasonable scale.

Self-hosted embedding. Running an open-source embedding model (BGE-large-en-v1.5, GTE-large, E5-large-v2) on a single NVIDIA T4 GPU costs roughly $75 to $150 per month for the instance. The marginal cost per embedding is effectively zero. This makes sense if your total embedding spend exceeds the GPU instance cost, which happens at roughly 1 billion tokens per month of ingestion throughput or if you need sub-10ms embedding latency for query-time performance.

Does re-embedding the entire corpus when switching models cost a lot?
Switching embedding models requires re-embedding every chunk because the vector spaces are incompatible. For 1 million chunks, this costs $10 to $67 with API models, which is modest. The real cost is the engineering effort to rebuild the index and validate retrieval quality, not the API bill. Plan embedding model changes as deliberate migrations, not casual swaps.

Vector Database Costs

Vector database hosting is the primary fixed cost in a RAG pipeline. It scales with the number of vectors stored and the performance requirements (query latency and throughput).

Pinecone pricing starts at $0.096 per GB per month on the Standard tier. One million 1536-dimensional vectors (float32) occupy roughly 6 GB, costing about $0.58 per month for storage alone. But Pinecone charges primarily by pod type: a p1.x1 pod at $0.096/hr (~$70/month) handles 1 million vectors with single-digit millisecond latency. For 1 million vectors with moderate traffic, expect $70 to $100 per month.

Qdrant Cloud pricing starts at $0.035 per GB-hour for the starter tier. A 4 GB node (~$100/month) comfortably handles 2 million 1024-dimensional vectors. For 1 million vectors, a 2 GB node at roughly $50 per month is sufficient.

Weaviate Cloud pricing starts at $25 per month for the Standard tier (1 million objects). Enterprise tiers scale based on object count and compute requirements.

pgvector (self-hosted) runs as a PostgreSQL extension. A db.r6g.large RDS instance ($150/month) or equivalent EC2 instance handles 5 million vectors with adequate performance. For 1 million vectors, a db.t4g.medium ($60/month) is sufficient if you configure shared_buffers to keep the HNSW index in memory. This is the cheapest option for teams that already manage PostgreSQL infrastructure.

Self-hosted Qdrant or Milvus on a dedicated instance (4 vCPU, 16 GB RAM, $80 to $120/month on most clouds) handles 5 to 10 million vectors. The marginal cost per vector is near zero, with the instance cost being the floor.

At what scale does self-hosting the vector database save money?
Self-hosting becomes cost-effective at around 2 to 5 million vectors, where managed service pricing rises above the cost of a dedicated instance. Below 1 million vectors, managed services cost $25 to $100 per month and save significant operational effort. Above 10 million vectors, the cost savings of self-hosting compound, but so does the operational burden of managing replication, backups, and scaling.

Reranking Costs

Reranking is optional but improves answer quality measurably. The cost depends on the model and the number of candidates reranked per query.

Cohere Rerank v3.5 costs $1 per 1,000 search queries (each query reranks up to 100 documents). At 100,000 queries per month, the cost is $100. This is the simplest managed option with consistently strong results.

Jina Reranker v2 costs $0.60 per 1,000 queries for up to 50 documents per query. At 100,000 queries per month, the cost is $60.

Self-hosted reranking using an open-source cross-encoder (BGE-reranker-v2-m3, or the smaller BGE-reranker-v2-m3-FP16) on a GPU instance adds the GPU cost ($75 to $150/month for a T4) with zero marginal cost per query. This makes sense at around 200,000+ queries per month, where the managed API cost exceeds the GPU instance cost.

Skipping reranking saves 100% of the reranking cost and 50 to 200 milliseconds of latency per query. If your retrieval is already high quality (recall@5 above 90% on your evaluation set), skipping the reranker may be acceptable. If recall is below 85%, the reranker's quality improvement typically justifies the cost.

LLM Generation Costs

This is where the money goes. LLM generation is the dominant cost in almost every RAG deployment, accounting for 60% to 80% of the total monthly spend. The cost scales with the number of queries, the amount of context per query, and the length of generated responses.

For a typical RAG query with 4 retrieved chunks of 512 tokens each, a system prompt of 200 tokens, and a conversation history of 500 tokens, the input is approximately 2,750 tokens. The average generated response is 300 to 500 tokens. Here is the monthly cost for 100,000 queries at these sizes:

GPT-4o at $2.50 per million input tokens and $10 per million output tokens: input cost = $687.50, output cost = $400 to $500. Monthly total: roughly $1,100 to $1,200.

GPT-4o mini at $0.15 per million input tokens and $0.60 per million output tokens: input cost = $41.25, output cost = $18 to $30. Monthly total: roughly $60 to $70.

Claude Sonnet 4 at $3.00 per million input tokens and $15 per million output tokens: input cost = $825, output cost = $450 to $750. Monthly total: roughly $1,275 to $1,575.

Claude Haiku 3.5 at $0.80 per million input tokens and $4 per million output tokens: input cost = $220, output cost = $120 to $200. Monthly total: roughly $340 to $420.

Gemini 2.0 Flash at $0.10 per million input tokens and $0.40 per million output tokens: input cost = $27.50, output cost = $12 to $20. Monthly total: roughly $40 to $48.

Self-hosted Llama 3.1 70B on 2x A100 GPUs ($3,000 to $4,000/month on cloud): zero marginal cost per query, but the fixed infrastructure cost is only justified at very high volume (500,000+ queries per month) or when data privacy requirements prohibit API usage.

Can I use a cheap model for simple queries and an expensive model for complex ones?
Yes, this is called model routing and it is one of the most effective cost optimizations. A lightweight classifier examines the query and the retrieved context, then routes simple factual lookups to a small model (GPT-4o mini, Gemini Flash) and complex analytical questions to a larger model (GPT-4o, Claude Sonnet). In practice, 60% to 80% of queries are simple enough for the small model, reducing average cost per query by 50% to 70% with minimal quality impact on the simple queries.

Total Cost Scenarios

Combining all components for 100,000 queries per month against 1 million chunks:

Budget configuration: text-embedding-3-small ($0.07/month queries), pgvector on db.t4g.medium ($60/month), no reranker, GPT-4o mini ($65/month). Total: roughly $125 per month. This is a production-viable configuration for internal tools where answer quality does not need to be best-in-class.

Balanced configuration: text-embedding-3-small ($0.07/month queries), Qdrant Cloud ($50/month), Cohere Rerank ($100/month), GPT-4o mini for simple queries + GPT-4o for complex queries with 70/30 routing ($65 + $360 = $425/month). Total: roughly $575 per month. This is the sweet spot for most production RAG applications.

Premium configuration: Voyage voyage-3 ($0.20/month queries), Qdrant Cloud ($100/month), Cohere Rerank ($100/month), GPT-4o for all queries ($1,150/month). Total: roughly $1,350 per month. This is for applications where answer quality is the primary concern and cost is secondary.

Enterprise self-hosted: Self-hosted BGE embedding on T4 ($100/month), self-hosted Qdrant on dedicated instance ($120/month), self-hosted BGE-reranker on T4 ($100/month), GPT-4o API for generation ($1,150/month). Total: roughly $1,470 per month but with better latency and no vendor dependencies for the retrieval stack.

Where to Optimize First

Since LLM generation dominates cost, every optimization that reduces token consumption has outsized impact.

Reduce context size. Retrieve 3 chunks instead of 5 or 10. Better retrieval precision (through hybrid search and reranking) means you can use fewer chunks without losing answer quality. Each chunk removed saves 512 tokens of input per query. Removing 2 chunks from every query saves roughly $250 per month on GPT-4o at 100,000 queries.

Compress retrieved context. Instead of including full chunks in the prompt, extract only the sentences relevant to the query from each chunk. A 512-token chunk might contain only 2 to 3 sentences (80 to 120 tokens) that are relevant to the specific question. Context compression reduces input tokens by 70% to 80% with minimal quality loss, because the irrelevant sentences in each chunk are noise that can confuse the model. LLMLingua and similar compression tools automate this extraction.

Cache frequent queries. Semantic caching stores recent answers and returns them for semantically similar future queries. If 20% of your queries are near-duplicates (common in customer support), caching eliminates 20% of LLM calls entirely. At $1,150/month for GPT-4o, that saves $230 per month.

Route by complexity. Use a small model for simple queries and a large model for complex ones. A lightweight router that examines query length, entity count, and retrieval confidence can route 70% of queries to GPT-4o mini at $0.65 per 1,000 queries instead of GPT-4o at $12.50 per 1,000 queries, reducing average generation cost by 60%.

Shorten system prompts. Many RAG system prompts include verbose instructions that consume hundreds of tokens on every query. Trim the system prompt to the essential constraints and cache it (most LLM APIs support prompt caching that reduces cost for repeated prefixes by 50% to 90%).

Is it cheaper to fine-tune a model than to use RAG?
Fine-tuning eliminates the retrieval infrastructure cost (vector database, embedding, reranking) and reduces input token count (no context chunks in the prompt). But fine-tuning has its own costs: training runs ($50 to $500+ per run), data preparation, evaluation, and the need to retrain when the knowledge changes. For static knowledge that rarely updates, fine-tuning can be cheaper at high query volumes. For dynamic knowledge that changes weekly or monthly, RAG is almost always cheaper because updating documents is free, while retraining is not. Most production systems use both: fine-tuning for behavioral patterns and RAG for factual grounding.

Cost Scaling Behavior

RAG costs scale differently across components. Embedding and database costs scale with corpus size (number of chunks), while generation and reranking costs scale with query volume. This means that growing your knowledge base from 1 million to 10 million chunks increases database cost (roughly 5x to 10x) but does not increase per-query generation cost. Conversely, growing from 100,000 to 1 million queries per month increases generation cost by 10x but barely affects database cost.

At high query volumes (1 million+ per month), the generation cost so thoroughly dominates that the entire retrieval stack, database, embedding, reranking, is a rounding error. At this scale, every optimization effort should focus on reducing tokens per query: fewer chunks, compressed context, cached responses, and model routing. The retrieval infrastructure could double in cost and the total monthly spend would barely change.

At low query volumes (under 10,000 per month), the fixed costs, vector database hosting and GPU instances for self-hosted components, dominate. At this scale, use managed services with usage-based pricing (Pinecone serverless, Qdrant free tier) and API-based models rather than self-hosted infrastructure. The production RAG guide covers infrastructure sizing in more detail.

Key Takeaway

LLM generation is 60% to 80% of RAG pipeline cost. A budget setup (GPT-4o mini, pgvector, no reranker) runs at $125/month for 100,000 queries. A balanced setup with model routing costs roughly $575/month. Optimize by reducing context size, compressing chunks, caching frequent queries, and routing simple questions to cheaper models. Retrieval infrastructure costs are relatively fixed and minor compared to generation at any meaningful query volume.

Related Articles

Production RAG: Scaling and Reliability

Infrastructure decisions that affect cost at scale.

How to Build a RAG Pipeline from Scratch

The architecture whose cost this guide quantifies.

Hybrid Search for RAG

Better retrieval precision that enables fewer, more relevant chunks.

AI Cost Optimization

Broader strategies for reducing AI infrastructure spend.

How to Evaluate RAG Pipeline Quality

Measuring quality to know when cost cuts go too far.