Without rigorous evaluation, your RAG is a black box
A RAG (Retrieval-Augmented Generation) chain in production without evaluation metrics is like an e-commerce site without analytics: you don't know what works or what fails. Companies that deploy a RAG without an evaluation protocol discover problems through user complaints — too late. The key metrics to track: faithfulness (fidelity to sources), answer relevancy (relevance of the answer), context recall (ability to retrieve the right documents) and context precision (absence of noise in the results). With a structured protocol and the right tools, you move from an opaque system to a measurable and improvable pipeline.
The problem
You've built an AI assistant connected to your document base. The first tests are encouraging. You deploy it internally. And then the feedback comes in: "The AI made up a figure that exists nowhere," "The answer doesn't match what the document says," "It ignored the procedure that was updated last month." These problems are predictable — and measurable — if you have an evaluation protocol in place.
The fundamental challenge of a RAG is that it combines two fallible systems: a vector search engine that can retrieve the wrong documents, and an LLM that can misinterpret or hallucinate from the right documents. Without distinct metrics for each component, you don't know which one to fix when the answer is wrong.
The three most common symptoms of an unevaluated RAG:
- Silent hallucinations — The LLM generates plausible but false answers. Without systematic verification against the sources, no one notices until a business decision is made on erroneous information.
- Gradual degradation — As documents are added, search quality degrades. New documents introduce noise, embeddings lose precision, but no alert is triggered.
- Blind optimization — You modify the prompt, change the embedding model, adjust the number of retrieved chunks… without knowing whether these changes improve or degrade the results. Each iteration is a gamble.
The cost of inaction is real: based on our field observations, a significant share of unevaluated RAG answers (often around a third) contain at least one inaccuracy. After implementing an evaluation and correction protocol, this rate drops sharply.
The AI solution
A RAG evaluation protocol rests on three pillars: automated metrics for continuous monitoring, a reference dataset for regression tests, and human evaluations for critical cases. Here are the three essential components to put in place.
Automated metrics (LLM-as-Judge)
Use an evaluator LLM (GPT-4o, Claude) to automatically score the faithfulness, relevance and completeness of each answer. Ragas and DeepEval provide ready-to-use implementations. Cost: about $0.02 per evaluation. Run daily on a sample of real queries.
Reference dataset (Golden Set)
Build a set of 200 to 500 question/answer pairs validated by business experts. This dataset serves as a reference for non-regression tests after each pipeline change. Include factual, comparative, multi-document questions as well as out-of-scope trick questions.
Targeted human evaluation
Each week, a business expert evaluates 20 critical answers on a scale of 1 to 5. This human feedback loop detects the false positives of the automated metrics and feeds the reference dataset. Use Argilla or Label Studio to structure the annotation process.
Implementation
Setting up a RAG evaluation pipeline takes 3 to 6 weeks depending on the maturity of your infrastructure. Here are the three key phases.
Build the Golden Set (weeks 1-2)
Collect 200 questions representative of your real users. For each question, write the expected answer and identify the relevant source passages. Categorize the questions by type (factual, comparative, procedural, out of scope). Involve at least 2 business experts to validate the reference answers. Store everything in a structured format (JSON or CSV) versioned in Git.
Implement automated metrics (weeks 3-4)
Integrate Ragas or DeepEval into your CI/CD pipeline. Configure the 4 essential metrics: faithfulness (threshold > 0.85), answer relevancy (threshold > 0.80), context recall (threshold > 0.75) and context precision (threshold > 0.70). Connect the pipeline to LangSmith or Weights & Biases to visualize trends. Automate daily execution on 50 queries sampled from production logs.
Continuous improvement loop (weeks 5-6)
Analyze the first results: identify the question categories where the score is lowest. Prioritize fixing retrieval problems (poorly split chunks, missing metadata) before generation problems (prompt engineering). Set up a monitoring dashboard with automatic alerts when a metric drops below the threshold. Schedule a weekly review of the 20 worst answers with the business team.
Results
Here are the results measured at our clients after implementing a structured RAG evaluation protocol, on enterprise RAG projects.
FAQ
What is a faithfulness metric in RAG?
Faithfulness measures whether the answer generated by the LLM is faithful to the retrieved source documents. Concretely, every claim in the answer must be traceable to a passage in the context. A score of 0.85 means that 85% of the claims are verifiable in the sources. It is the most critical metric for avoiding hallucinations.
How many questions are needed in a RAG evaluation dataset?
A minimum of 50 questions for a pilot, ideally 200 to 500 for a robust production evaluation. The questions must cover the different types of user queries: factual, comparative, multi-document and out-of-scope. Plan for 20% trick questions (out of scope) to measure the appropriate refusal rate.
Which tools should be used to evaluate a RAG chain?
The three reference tools are Ragas (open source, standardized metrics), DeepEval (Python framework with 14 metrics) and LangSmith (LangChain platform for tracing and evaluation). For human evaluations, Argilla lets you annotate the results collaboratively. Allow 2 to 4 weeks to set up a complete evaluation pipeline.
How often should a RAG chain be evaluated?
In production, run a daily automated evaluation on a sample of 50 real queries. Complement this with a weekly human evaluation on 20 critical cases. Re-run a complete evaluation (200+ questions) after each pipeline change: changing the embedding model, adding documents, modifying the prompt or updating the LLM.
For technical profiles
Architecture of a RAG evaluation pipeline
The evaluation pipeline integrates in parallel with your existing RAG chain. It intercepts the triplets (question, retrieved context, generated answer) and computes the metrics asynchronously. The reference implementation uses Ragas as the metrics engine, LangSmith for tracing, and Weights & Biases for longitudinal monitoring.
The 4 essential metrics to implement:
- Faithfulness — Breaks the answer down into claims, verifies each claim against the context. Implementation: NLI (Natural Language Inference) or LLM-as-Judge.
- Answer Relevancy — Generates N questions from the answer, measures the cosine similarity with the original question. High score = focused answer, low score = vague or off-topic answer.
- Context Recall — Compares the retrieved passages with the reference passages from the Golden Set. Identifies the documents missed by the retriever.
- Context Precision — Measures the proportion of retrieved passages that are actually useful for answering. A low score indicates too much noise in the retrieval.
Comparison of evaluation frameworks
| Criterion | Ragas | DeepEval | LangSmith | TruLens |
|---|---|---|---|---|
| Native RAG metrics | 8 metrics | 14 metrics | 5 metrics | 6 metrics |
| Open source | Yes (Apache 2) | Yes (Apache 2) | Partial | Yes |
| CI/CD integration | Native pytest | Native pytest | REST API | REST API |
| Dashboard | Basic | Confident AI | Complete | Complete |
| Cost | Free + LLM cost | Free + LLM cost | Freemium ($1/dev/day) | Free + LLM cost |