RAG Evaluation Set Building Tutorial: Questions, Answers, Evidence, and Hard Example Design

A RAG evaluation set is used to validate the effectiveness of retrieval-augmented generation. Samples are organized around questions, ground truth answers, and evidence snippet annotations. Through hard example design and metric evaluation, it helps locate problems such as inaccurate retrieval and answers lacking supporting evidence.

What Is a RAG Evaluation Set?

A RAG evaluation set is a collection of sample data used to evaluate the quality of a retrieval-augmented generation system. It is usually designed around questions, corresponding ground truth answers, and whether the answers can be obtained from retrieved documents. It does not merely judge whether one model response “looks reasonable”; instead, it helps teams verify whether retrieval, evidence support, and generated results are consistent with one another.

In RAG applications, evaluation results can be used both in the experimentation and tuning stages and for continuous monitoring after launch. As knowledge base content, prompts, retrieval strategies, or model versions change, the evaluation set can serve as a stable reference for observing whether system quality fluctuates.

Key point: A usable RAG evaluation sample should be able to answer at least three questions: what the user asked, what the ground truth answer is, and whether the retrieved documents support this answer.

Minimum Field Design for an Evaluation Set

When building a RAG evaluation set, it is not recommended to store only the two columns “question” and “answer.” To evaluate the retrieval pipeline, evidence snippets, domain topics, and task types need to be included in the sample structure, making it easier to later import evaluation metrics, conduct manual review, and perform grouped analysis.

FieldPurposeDesign recommendation
Sample IDUniquely locate the sampleUse stable identifiers to facilitate retesting, troubleshooting, and version management
QuestionRepresent the user queryTry to derive it from real intent, not merely copy document titles
Ground truth answerServe as the reference answerIt should correspond to the question and be clear and verifiable
Evidence snippetSupport the ground truth answerRecord the document content or snippet location that can lead to the answer
Domain topicMark the knowledge scopeSuch as product documentation, internal policies, technical knowledge bases, financial knowledge, etc.
RAG taskMark the task typeSuch as factual Q&A, summary extraction, intent recognition, document-based answer generation
Evaluation metric or checkpointConnect to the evaluation processCan record checks such as answer correctness, evidence support, and intent match

Among these, the question and corresponding ground truth answer are the basic fields; the evidence snippet is used to determine whether the answer comes from retrievable documents rather than “plausible but incorrect” content generated by the model based on language ability.

Split the Sample Scope by Domain Topic and RAG Task

An evaluation set must first define its coverage scope. A relatively sound approach is to split it from two dimensions: one is the domain topic, and the other is the RAG task. This prevents the evaluation set from covering only high-frequency simple questions, which would make it impossible to discover poor system performance in boundary scenarios.

DimensionQuestion to answerExamples
Domain topicWhat content does the knowledge base cover?Product documentation, financial knowledge, internal policies, technical support documents
RAG taskWhat task does the user want the system to complete?Find facts, extract summaries, recognize intent, generate answers based on documents
Sample distributionWhich topic and task combinations need key coverage?High-frequency questions, easily confused topics, questions with scattered evidence

A simple coverage matrix can be used to check whether the samples are balanced:

Domain topic / RAG taskFactual Q&ASummary extractionIntent recognitionDocument answer generation
Product documentationCoverage neededOptional coverageCoverage neededCoverage needed
Internal policiesCoverage neededOptional coverageCoverage neededCoverage needed
Technical knowledge baseCoverage neededCoverage neededOptional coverageCoverage needed

The matrix does not require every cell to have the same number of samples, but it should reflect real business needs and system risk points. For enterprise knowledge base Q&A, in addition to high-frequency questions, attention should also be paid to samples that are easily affected by similar documents, similar concepts, or incomplete evidence.

Start Question Design from Real Intent

The core of question design is not to “make the question match a certain document title,” but to make the sample represent the user’s real query intent. A good question should reflect the task the user wants to solve and make it easy to judge whether the system has understood the intent.

During the evaluation preparation stage, a language model can be used to rephrase, summarize, or simplify questions in order to check whether the question intent is clear. For the same intent, different expressions can also be designed to observe whether the system remains stable when facing rewritten questions, colloquial questions, and simplified questions.

Design methodPurposeKey check
Original questionPreserve the real user expressionWhether the system can understand natural queries
Rewritten questionChange the wording while preserving the intentWhether the retrieval results are stable
Simplified questionRemove redundant informationWhether the intent can still be recognized
Summarized questionCompress a long question into the core requestWhether the system captures the main task

After question design is completed, it is recommended to check each item one by one:

  • Whether the question corresponds to a clear intent;
  • Whether the question can find supporting evidence within the knowledge base scope;
  • Whether the question avoids ambiguity unrelated to the ground truth answer;
  • Whether different expressions of the same intent are all included in the evaluation samples;
  • Whether the question can support subsequent answer and evidence alignment checks.

How to Align Ground Truth Answers with Evidence Snippets

A ground truth answer is the reference content used to evaluate whether a model response is correct, while an evidence snippet is supporting material used to determine whether the answer has a documentary basis. The two need to be aligned; otherwise, the evaluation results may only reflect the fluency of generated text and cannot determine whether the RAG system truly used the retrieval results.

The ground truth answer should satisfy three points as much as possible: it should be verifiable, reproducible, and directly relevant to the question. The evidence snippet should be able to support the key conclusions in the ground truth answer, especially conditions, scope, limitations, and conclusive statements.

Check itemJudgment questionRisk
Whether the answer can be derived from the evidenceCan the conclusions in the ground truth answer be obtained from the evidence snippet?The answer is correct but lacks retrieval evidence
Whether the evidence covers key conditionsAre time, scope, objects, and limitations complete?The model omits constraints
Whether the evidence is ambiguousCould the same snippet support multiple interpretations?It is difficult to judge right or wrong during evaluation
Whether the answer is overextendedDoes it add information outside the evidence?The generated content looks reasonable but is unverifiable
Whether the question and answer correspond one-to-oneDoes the answer truly answer this question?Metric scores are inconsistent with the actual experience

Key point: In RAG evaluation, evidence snippets are not auxiliary information. They are the key material for determining “whether the retrieved documents contain the intent or the basis for the answer.”

Hard Example Design Exposes Retrieval and Generation Weaknesses

The goal of hard examples is not to artificially increase complexity, but to expose weaknesses in the system. You can select samples that are more easily confused at the intersections of domain topics and RAG tasks, and then design checkpoints around situations such as intent rewriting, closely related topics, and insufficient evidence support.

Hard example directionDesign ideaMain observation point
Intent rewritingUse different expressions for the same intentWhether the system can still recognize the correct intent
Closely related topicsThe question involves similar concepts or adjacent knowledge scopesWhether retrieval hits the correct document
Insufficient evidence supportThe document contains only partial information or cannot directly lead to the answerWhether the model generates a response that lacks supporting evidence
Task switchingMix tasks such as Q&A, summarization, and intent recognition under the same topicWhether the system answers according to task requirements

When designing hard examples, “whether the answer can be obtained from retrieved documents” can be used as the core checkpoint. If the evidence is insufficient, the system should not give an answer that appears certain but lacks support; if the evidence exists but is distributed across different snippets, the evaluation should focus on whether the system has found enough material to support the answer.

The number of hard examples does not need to be large at the beginning, but they should be accumulated continuously. Every manual review, online feedback item, or failed tuning sample can be converted into a new evaluation sample, gradually filling in the system boundaries.

From Evaluation Set to Metric Evaluation and Iterative Monitoring

A RAG evaluation set is not a one-time spreadsheet, but a long-term asset in the evaluation process. The complete process usually includes preparing data, building the evaluation dataset, importing evaluation metrics, and evaluating the RAG application.

Prepare and Build the Dataset

First determine the domain topics, RAG tasks, and sample fields, then organize the questions, ground truth answers, and evidence snippets. At this stage, special attention should be paid to the quality of the reference content, because building and maintaining the evaluation engine, reference answers, and evidence materials requires continuous investment.

Import Metrics and Run Evaluation

During evaluation, metrics can be organized around checkpoints such as whether retrieval hits relevant documents, whether the answer matches the ground truth answer, and whether the response is supported by evidence. Automated evaluation frameworks can help record metric changes after each iteration and assist in determining whether optimizations have truly taken effect.

Use Continuously with Iteration and Monitoring

In the experimentation stage, the evaluation set can be used to compare the effects of different retrieval strategies, prompts, or model configurations; after launch, it can be used to monitor whether system quality fluctuates as the knowledge base, model, or user questions change.

StageRole of the evaluation setOutput result
Experimental tuningCompare different solutionsFind more stable retrieval and generation configurations
Regression testingCheck whether changes break existing capabilitiesDiscover degraded samples
Post-launch monitoringContinuously observe system performanceTrack quality fluctuations and risk samples
Sample accumulationAdd failure cases as new samplesExpand evaluation set coverage

Some automated evaluation mechanisms encapsulate the checking logic in a form similar to feedback functions, which are used to evaluate RAG outputs in batches and track metric changes after iterations. But regardless of whether tools are used, the evaluation set itself needs continuous maintenance: questions should represent real intent, ground truth answers should remain verifiable, and evidence snippets should be able to support judgments.