RAG Implementation Consulting: Scope, Cost and Partners

A retrieval-augmented generation (RAG) implementation connects a language model to your own documents. The system ingests and chunks the content, embeds it, stores it in a searchable index, retrieves and ranks passages for each question, and generates an answer from them. Consultants add the most value in retrieval quality, evaluation, security, and run cost, because the model call itself is the easy part.

This guide is part of our AI consulting firms research. Menlo Ventures' 2024 survey of 600 US IT decision-makers reported RAG adoption at 51%, up from 31% a year earlier, against fine-tuning in 9% of production models. Its 2025 report says prompt design remained the dominant customization technique, followed by RAG.

What does a RAG implementation involve?

Seven stages: ingestion and parsing, chunking, embedding, indexing, retrieval and ranking, generation, and evaluation with guardrails. Each has a failure you can test for. Lewis and colleagues introduced the pattern in 2020 as a way to combine a model's built-in knowledge with an external index.

OpenAI's embeddings guide lists 1,536 dimensions for text-embedding-3-small and 3,072 for text-embedding-3-large, and says vectors can be shortened with a dimensions parameter, for example to fit a store limited to 1,024. The pgvector README says its indexes cover standard vector columns up to 2,000 dimensions and half-precision columns up to 4,000, so a 3,072-dimension embedding needs half precision or shortening. A partner should have met these limits before.

Do you need RAG at all, and should you build or buy?

Start with the cheapest option that meets the requirement: put a small knowledge base in the prompt, use managed retrieval in the platform you already run, or build custom retrieval only when you need control over ranking, permissions, or data location. Anthropic's 2024 guidance was that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt with prompt caching. Context limits and prices have changed since, so test current figures.

RAG and fine-tuning solve different problems: RAG supplies knowledge, and fine-tuning changes behavior or format. In a Microsoft study, RAG consistently outperformed unsupervised fine-tuning for knowledge-intensive tasks, both for facts the model had seen in training and for new ones (Ovadia et al., 2024). Fine-tuning access is also narrowing: OpenAI's pricing page says its self-serve fine-tuning platform is being wound down and is closed to new users.

Which retrieval techniques improve answer quality?

Hybrid search, which combines keyword and vector retrieval, plus reranking is the best-documented improvement. In Anthropic's 2024 tests across codebases, fiction, arXiv papers, and science papers, the share of relevant passages missed in the top 20 fell from 5.7% to 3.7% with contextual embeddings (a 35% reduction), to 2.9% when contextual keyword search was added (49%), and to 1.9% with a reranker (67%). Contextual embeddings prepend a short, chunk-specific description of where the passage sits in its document before indexing. These are one vendor's results on its own datasets, so replicate them on your documents.

Platforms now ship this pattern. Bedrock's managed knowledge bases always use hybrid search with reranking on by default. Azure AI Search merges BM25 and vector results with reciprocal rank fusion and offers a semantic ranker. Snowflake's Cortex Search combines vector and keyword retrieval with semantic reranking, and Databricks AI Search supports hybrid search and reranking.

More passages are not always better. Anthropic found 20 passages beat 10 or 5 in its tests, while Liu et al. found that performance can drop significantly when the relevant passage sits in the middle of a long input. Tune the number and order of passages on your own evaluation set. Agentic retrieval, where an LLM plans sub-queries, is in preview in Azure AI Search, and graph-based retrieval adds a knowledge-graph layer. Both add components and calls, so adopt them only if evaluation shows hybrid retrieval with reranking falls short.

What drives RAG cost?

People cost dominates the build. At run time, generation tokens dominate and scale with query volume, while embedding and vector storage are usually small. Prices change often; the list prices below are as of October 2026 and link to each vendor's page.

Token counts depend on the tokenizer. Anthropic notes that its newer models' tokenizer produces about 30% more tokens for the same text than earlier ones, so estimate with each vendor's own counter.

DCF Research estimate: an internal assistant over 200,000 documents averaging 5,000 tokens, answering 5,000 questions a day. All volumes are assumptions.

• Corpus: 200,000 × 5,000 = 1.0 billion tokens. Chunks of 500 tokens with 10% overlap give about 2.2 million chunks and 1.1 billion tokens to embed.
• One-time embedding: 1,100 million tokens × $0.13 per million = $143 with text-embedding-3-large, or $22 with text-embedding-3-small.
• Storage: 2.2 million chunks × 3,072 dimensions × 4 bytes ≈ 27 GB, about $9 a month at $0.33 per GB; at 1,024 dimensions, about 9 GB and $3 a month. The $50 plan minimum applies either way.
• Per query: 6 passages × 500 tokens + 700 tokens of instructions + 300 tokens of question and history = 4,000 input tokens, plus 400 output tokens.
• Generation: (4,000 × $2 + 400 × $10) ÷ 1 million = $0.012 per query on Claude Sonnet 5.5, so 150,000 queries a month cost $1,800. On Claude Haiku 5.5 the cost is $0.0006 per query, or $90 a month.
• Reranking: 150,000 × $2 ÷ 1,000 = $300 a month.
• Run total: $1,800 + $300 + $50 ≈ $2,150 a month with the mid-tier model, about $26,000 a year. At ten times the volume, generation alone is $18,000 a month.
• Build: 3 engineers × 10 weeks × 40 hours × $175 per hour (an assumed blended rate) = $210,000, about eight times the annual infrastructure bill.

At modest volume the infrastructure bill is small next to the build team. As volume grows, model choice, passage count, prompt length, caching, and routing routine questions to a smaller model decide the run cost.

How do you evaluate a RAG system and a RAG partner?

Require a labeled question set built from your own documents, and metrics that separate retrieval from generation. The open-source RAGAS library, for example, defines context precision, context recall, faithfulness, and response relevancy as separate metrics. A partner that cannot show retrieval metrics from a prior project has not yet shown it can improve retrieval.

Barnett and colleagues catalogued seven failure points from three case studies: missing content, the right document ranked too low, the right document lost when passages are consolidated into the prompt, the answer present but not extracted, the wrong output format, the wrong level of specificity, and incomplete answers. They also concluded that a RAG system can only be validated in operation and that its reliability evolves after launch, so budget for calibration.

1. How do you measure retrieval quality separately from answer quality, and on whose questions? Expect a labeled set from your documents, not a generic benchmark.
2. Show metric changes across chunking, embedding, and reranking experiments on a prior project.
3. How do you extract tables and scans, and how do you test that extraction?
4. How are permissions enforced at query time, and how do you test for leakage between users?
5. What happens when the answer is not in the sources, and how is the no-answer rate measured?
6. How do updates and deletions reach the index, and how stale can it get?
7. What does cost per query look like at pilot and at target volume, and which levers reduce it?
8. What do you hand over: evaluation set, harness, dashboards, and runbooks?

For monitoring, track retrieval hit rate on the labeled set, faithfulness and response relevancy on sampled traffic, latency by stage at the median and the 95th percentile, index freshness against how often sources change, cost per query, no-answer rate, user feedback, and periodic permission tests. Set thresholds from the baseline measured at acceptance rather than from a generic target.

What security and data-governance controls should a partner provide?

Ask for permission-aware retrieval, isolation between tenants, validation of what enters the index, retrieval logging, and a deletion path. The OWASP Top 10 for LLM Applications (2025) lists vector and embedding weaknesses as a risk (LLM08). It covers unauthorized access and cross-context leakage when groups share a vector database, embedding inversion, and poisoned data. It also notes that RAG does not fully mitigate prompt injection (LLM01).

• Enforce access control at query time. Azure AI Search documents security filters and document-level access control, with ACL and RBAC support in preview. On any platform, ask how permissions sync from the source system and how long changes take to apply.
• Separate tenants and groups logically in the index, and test for cross-user leakage.
• Validate ingested content for hidden instructions and untrusted sources. OWASP's example is hidden text in a résumé that redirects the model.
• Check what guardrails cover. Bedrock's documentation says guardrails apply to the input and the generated response, not to retrieved references.
• Log retrievals immutably, and treat embeddings as sensitive data, since OWASP notes they can be inverted to recover source information.
• Propagate deletions from source documents to chunks and embeddings, and confirm each vendor's retention, training-use, and regional-processing terms.
• If you use the NIST AI Risk Management Framework, ask the partner to map controls to its generative AI profile, NIST AI 600-1.

What should the engagement look like?

Plan four phases with gates: discovery and evaluation set, ingestion and baseline retrieval, tuning against the evaluation set, and hardening with hand-over. If a quote omits evaluation or hardening, compare it line by line before signing.

DCF Research estimate: the durations below are planning assumptions for a team of three working on one language, a few internal sources, and an existing cloud account. They are built from the deliverables listed, not from a survey of projects. Multi-tenant or customer-facing systems, strict latency targets, and regulated data take longer.

The phases sum to 10 weeks, which gives the $210,000 build estimate above (3 × 10 × 40 × $175). Replace the rate with quotes you receive. For a feasibility phase before committing, see GenAI consulting proof of concept; for pricing models, see AI consulting pricing; and for registry and monitoring practices that carry over, see MLOps consulting.

Our AI consulting firms directory tracks 50 data and AI consulting firms using public evidence: partner-directory listings, each firm's own service pages, and public case studies, checked on 2026-10-23. Firms cannot pay to change their score or rank, and we have not audited client projects or tested firms' work.

Sources

• 2024: The State of Generative AI in the Enterprise — Menlo Ventures (2024). https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/
• 2025: The State of Generative AI in the Enterprise — Menlo Ventures (2025). https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS (2020). https://arxiv.org/abs/2005.11401
• Seven Failure Points When Engineering a Retrieval Augmented Generation System — Barnett et al., CAIN (2024). https://arxiv.org/abs/2401.05856
• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Ovadia et al. (2024). https://arxiv.org/abs/2312.05934
• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., Transactions of the ACL (2024). https://arxiv.org/abs/2307.03172
• Introducing Contextual Retrieval — Anthropic (2024). https://www.anthropic.com/news/contextual-retrieval
• Text splitter integrations — LangChain (2026). https://docs.langchain.com/oss/python/integrations/splitters
• Vector embeddings — OpenAI API documentation (2026). https://developers.openai.com/api/docs/guides/embeddings
• pgvector: open-source vector similarity search for Postgres — pgvector (2026). https://github.com/pgvector/pgvector
• Query a knowledge base and retrieve data — Amazon Bedrock User Guide, Amazon Web Services (2026). https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-retrieve.html
• RAG Engine on Gemini Enterprise Agent Platform overview — Google Cloud Documentation (2026). https://docs.cloud.google.com/gemini-enterprise-agent-platform/build/rag-engine/rag-overview
• Hybrid search overview — Azure AI Search, Microsoft Learn (2026). https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview
• Document-level access control — Azure AI Search, Microsoft Learn (2026). https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview
• Databricks AI Search — Databricks (2026). https://docs.databricks.com/aws/en/ai-search/ai-search
• Cortex Search overview — Snowflake Documentation (2026). https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview
• Pricing — OpenAI API documentation (accessed October 2026). https://developers.openai.com/api/docs/pricing
• Pricing — Pinecone (accessed October 2026). https://www.pinecone.io/pricing/
• Vector Database Pricing — Weaviate (accessed October 2026). https://weaviate.io/pricing
• Qdrant Cloud Pricing — Qdrant (accessed October 2026). https://qdrant.tech/pricing/
• Pricing — Anthropic (accessed October 2026). https://platform.claude.com/docs/en/about-claude/pricing
• List of available metrics — Ragas documentation (2025). https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/
• OWASP Top 10 for LLM Applications 2025 — OWASP Gen AI Security Project (2025). https://genai.owasp.org/llm-top-10/
• LLM08:2025 Vector and Embedding Weaknesses — OWASP Gen AI Security Project (2025). https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/
• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — National Institute of Standards and Technology (2024). https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence