RAG Evaluation Set Building Tutorial: Questions, Answers, Evidence, and Hard Example Design
A RAG evaluation set is used to validate the effectiveness of retrieval-augmented generation. Samples are organized around questions, ground truth answers, and evidence snippet annotations. Through hard example design and metric evaluation, it helps locate problems such as inaccurate retrieval and answers lacking supporting evidence.
What Is a RAG Evaluation Set?
A RAG evaluation set is a collection of sample data used to evaluate the quality of a retrieval-augmented generation system. It is usually designed around questions, corresponding ground truth answers, and whether the answers can be obtained from retrieved documents. It does not merely judge whether one model response “looks reasonable”; instead, it helps teams verify whether retrieval, evidence support, and generated results are consistent with one another.
In RAG applications, evaluation results can be used both in the experimentation and tuning stages and for continuous monitoring after launch. As knowledge base content, prompts, retrieval strategies, or model versions change, the evaluation set can serve as a stable reference for observing whether system quality fluctuates.
Key point: A usable RAG evaluation sample should be able to answer at least three questions: what the user asked, what the ground truth answer is, and whether the retrieved documents support this answer.
Minimum Field Design for an Evaluation Set
When building a RAG evaluation set, it is not recommended to store only the two columns “question” and “answer.” To evaluate the retrieval pipeline, evidence snippets, domain topics, and task types need to be included in the sample structure, making it easier to later import evaluation metrics, conduct manual review, and perform grouped analysis.
| Field | Purpose | Design recommendation |
|---|---|---|
| Sample ID | Uniquely locate the sample | Use stable identifiers to facilitate retesting, troubleshooting, and version management |
| Question | Represent the user query | Try to derive it from real intent, not merely copy document titles |
| Ground truth answer | Serve as the reference answer | It should correspond to the question and be clear and verifiable |
| Evidence snippet | Support the ground truth answer | Record the document content or snippet location that can lead to the answer |
| Domain topic | Mark the knowledge scope | Such as product documentation, internal policies, technical knowledge bases, financial knowledge, etc. |
| RAG task | Mark the task type | Such as factual Q&A, summary extraction, intent recognition, document-based answer generation |
| Evaluation metric or checkpoint | Connect to the evaluation process | Can record checks such as answer correctness, evidence support, and intent match |
Among these, the question and corresponding ground truth answer are the basic fields; the evidence snippet is used to determine whether the answer comes from retrievable documents rather than “plausible but incorrect” content generated by the model based on language ability.
Split the Sample Scope by Domain Topic and RAG Task
An evaluation set must first define its coverage scope. A relatively sound approach is to split it from two dimensions: one is the domain topic, and the other is the RAG task. This prevents the evaluation set from covering only high-frequency simple questions, which would make it impossible to discover poor system performance in boundary scenarios.
| Dimension | Question to answer | Examples |
|---|---|---|
| Domain topic | What content does the knowledge base cover? | Product documentation, financial knowledge, internal policies, technical support documents |
| RAG task | What task does the user want the system to complete? | Find facts, extract summaries, recognize intent, generate answers based on documents |
| Sample distribution | Which topic and task combinations need key coverage? | High-frequency questions, easily confused topics, questions with scattered evidence |
A simple coverage matrix can be used to check whether the samples are balanced:
| Domain topic / RAG task | Factual Q&A | Summary extraction | Intent recognition | Document answer generation |
|---|---|---|---|---|
| Product documentation | Coverage needed | Optional coverage | Coverage needed | Coverage needed |
| Internal policies | Coverage needed | Optional coverage | Coverage needed | Coverage needed |
| Technical knowledge base | Coverage needed | Coverage needed | Optional coverage | Coverage needed |
The matrix does not require every cell to have the same number of samples, but it should reflect real business needs and system risk points. For enterprise knowledge base Q&A, in addition to high-frequency questions, attention should also be paid to samples that are easily affected by similar documents, similar concepts, or incomplete evidence.
Start Question Design from Real Intent
The core of question design is not to “make the question match a certain document title,” but to make the sample represent the user’s real query intent. A good question should reflect the task the user wants to solve and make it easy to judge whether the system has understood the intent.
During the evaluation preparation stage, a language model can be used to rephrase, summarize, or simplify questions in order to check whether the question intent is clear. For the same intent, different expressions can also be designed to observe whether the system remains stable when facing rewritten questions, colloquial questions, and simplified questions.
| Design method | Purpose | Key check |
|---|---|---|
| Original question | Preserve the real user expression | Whether the system can understand natural queries |
| Rewritten question | Change the wording while preserving the intent | Whether the retrieval results are stable |
| Simplified question | Remove redundant information | Whether the intent can still be recognized |
| Summarized question | Compress a long question into the core request | Whether the system captures the main task |
After question design is completed, it is recommended to check each item one by one:
- Whether the question corresponds to a clear intent;
- Whether the question can find supporting evidence within the knowledge base scope;
- Whether the question avoids ambiguity unrelated to the ground truth answer;
- Whether different expressions of the same intent are all included in the evaluation samples;
- Whether the question can support subsequent answer and evidence alignment checks.
How to Align Ground Truth Answers with Evidence Snippets
A ground truth answer is the reference content used to evaluate whether a model response is correct, while an evidence snippet is supporting material used to determine whether the answer has a documentary basis. The two need to be aligned; otherwise, the evaluation results may only reflect the fluency of generated text and cannot determine whether the RAG system truly used the retrieval results.
The ground truth answer should satisfy three points as much as possible: it should be verifiable, reproducible, and directly relevant to the question. The evidence snippet should be able to support the key conclusions in the ground truth answer, especially conditions, scope, limitations, and conclusive statements.
| Check item | Judgment question | Risk |
|---|---|---|
| Whether the answer can be derived from the evidence | Can the conclusions in the ground truth answer be obtained from the evidence snippet? | The answer is correct but lacks retrieval evidence |
| Whether the evidence covers key conditions | Are time, scope, objects, and limitations complete? | The model omits constraints |
| Whether the evidence is ambiguous | Could the same snippet support multiple interpretations? | It is difficult to judge right or wrong during evaluation |
| Whether the answer is overextended | Does it add information outside the evidence? | The generated content looks reasonable but is unverifiable |
| Whether the question and answer correspond one-to-one | Does the answer truly answer this question? | Metric scores are inconsistent with the actual experience |
Key point: In RAG evaluation, evidence snippets are not auxiliary information. They are the key material for determining “whether the retrieved documents contain the intent or the basis for the answer.”
Hard Example Design Exposes Retrieval and Generation Weaknesses
The goal of hard examples is not to artificially increase complexity, but to expose weaknesses in the system. You can select samples that are more easily confused at the intersections of domain topics and RAG tasks, and then design checkpoints around situations such as intent rewriting, closely related topics, and insufficient evidence support.
| Hard example direction | Design idea | Main observation point |
|---|---|---|
| Intent rewriting | Use different expressions for the same intent | Whether the system can still recognize the correct intent |
| Closely related topics | The question involves similar concepts or adjacent knowledge scopes | Whether retrieval hits the correct document |
| Insufficient evidence support | The document contains only partial information or cannot directly lead to the answer | Whether the model generates a response that lacks supporting evidence |
| Task switching | Mix tasks such as Q&A, summarization, and intent recognition under the same topic | Whether the system answers according to task requirements |
When designing hard examples, “whether the answer can be obtained from retrieved documents” can be used as the core checkpoint. If the evidence is insufficient, the system should not give an answer that appears certain but lacks support; if the evidence exists but is distributed across different snippets, the evaluation should focus on whether the system has found enough material to support the answer.
The number of hard examples does not need to be large at the beginning, but they should be accumulated continuously. Every manual review, online feedback item, or failed tuning sample can be converted into a new evaluation sample, gradually filling in the system boundaries.
From Evaluation Set to Metric Evaluation and Iterative Monitoring
A RAG evaluation set is not a one-time spreadsheet, but a long-term asset in the evaluation process. The complete process usually includes preparing data, building the evaluation dataset, importing evaluation metrics, and evaluating the RAG application.
Prepare and Build the Dataset
First determine the domain topics, RAG tasks, and sample fields, then organize the questions, ground truth answers, and evidence snippets. At this stage, special attention should be paid to the quality of the reference content, because building and maintaining the evaluation engine, reference answers, and evidence materials requires continuous investment.
Import Metrics and Run Evaluation
During evaluation, metrics can be organized around checkpoints such as whether retrieval hits relevant documents, whether the answer matches the ground truth answer, and whether the response is supported by evidence. Automated evaluation frameworks can help record metric changes after each iteration and assist in determining whether optimizations have truly taken effect.
Use Continuously with Iteration and Monitoring
In the experimentation stage, the evaluation set can be used to compare the effects of different retrieval strategies, prompts, or model configurations; after launch, it can be used to monitor whether system quality fluctuates as the knowledge base, model, or user questions change.
| Stage | Role of the evaluation set | Output result |
|---|---|---|
| Experimental tuning | Compare different solutions | Find more stable retrieval and generation configurations |
| Regression testing | Check whether changes break existing capabilities | Discover degraded samples |
| Post-launch monitoring | Continuously observe system performance | Track quality fluctuations and risk samples |
| Sample accumulation | Add failure cases as new samples | Expand evaluation set coverage |
Some automated evaluation mechanisms encapsulate the checking logic in a form similar to feedback functions, which are used to evaluate RAG outputs in batches and track metric changes after iterations. But regardless of whether tools are used, the evaluation set itself needs continuous maintenance: questions should represent real intent, ground truth answers should remain verifiable, and evidence snippets should be able to support judgments.