Guide to Building an AI Agent Evaluation Dataset: Task Success Rate, Trajectory Quality, and Tool Accuracy

An AI Agent evaluation dataset is used to measure agent task performance consistently. By organizing samples and Traces around task success rate, trajectory quality, and tool accuracy, it helps teams build continuous evaluation from a Golden Set to an online canary feedback loop.

What Problems Does an AI Agent Evaluation Dataset Solve?

An AI Agent evaluation dataset is a set of repeatedly executable test samples used to observe an agent’s performance in task execution, decision-making processes, and tool use. It does not only judge whether the final answer is correct; it also focuses on whether the Agent completes the task through a reasonable path, whether it selects the right tool, and whether the tool parameters match the context.

Compared with ordinary Q&A evaluation, an AI Agent evaluation dataset is closer to a “task acceptance checklist.” Ordinary Q&A evaluation usually checks whether the answer is correct, while Agent evaluation needs to check whether the result, process, and actions are reliable at the same time.

The purposes of building an evaluation dataset usually include three categories:

  • Technical quality: Identify issues such as task failure, tool misuse, trajectory drift, and insufficient error recovery.
  • Business outcomes: Determine whether the Agent can consistently complete high-frequency, critical, or high-value tasks.
  • Risk control: Continuously observe safety, compliance, and abnormal behavior before and after launch.

Key point: A single metric cannot summarize agent performance. An Agent may ultimately answer a question correctly, but the process may consume too much, call the wrong tool, or perform inconsistently across multiple runs. Therefore, the evaluation dataset should adopt a multidimensional framework rather than looking only at one total score.

How Should the Three Core Metrics Be Divided?

When building an AI Agent evaluation dataset, you can first organize it around three core metrics: task success rate, trajectory quality, and tool accuracy. They respectively answer “whether the task was completed,” “whether the process was reasonable,” and “whether the tool was used correctly.”

MetricMain question answeredTypical observation pointsProblems it is suitable for discovering
Task success rateWhether the Agent completed the user’s goalTask completion rate, success rate, decision accuracy, whether the final result meets acceptance criteriaIncomplete final answers, goals not achieved, key steps omitted
Trajectory qualityWhether the Agent’s process for completing the task is reasonableTrajectory accuracy, number of steps, Token consumption, latency, error recovery rate, consistency across multiple runsInefficient paths, unstable processes, intermediate decision drift, inability to recover from errors
Tool accuracyWhether the Agent uses tools correctlyTool selection quality, tool parameter correctness, tool invocation metricsWrong tool selected, tool call omitted, parameter construction errors, context dependency errors

These three types of metrics do not replace one another; they complement one another:

  • Task success rate is suitable as result-level acceptance, determining whether the user’s goal has been achieved.
  • Trajectory quality is suitable as process-level diagnosis, locating inefficient, unstable, or uncontrollable execution paths.
  • Tool accuracy is suitable as action-level diagnosis, especially for Agents that rely on plugins, APIs, database queries, or external systems.

What Information Should Evaluation Samples Include?

Each evaluation sample should be built around a clear task. Simple tasks can include only the user query, reference response, and scoring criteria; tasks involving tool calls or multi-step execution need to record expected tool usage and intermediate processes.

An evaluation can consist of one or more turns, so evaluation samples can cover not only single-turn Q&A, but also multi-turn dialogue, step-by-step operations, and tasks that require context carryover.

FieldPurposeApplicable situation
User queryDescribes the user input and task objectiveAll samples should include it
Reference responseProvides the expected final output or the range of acceptable answersUsed to determine whether the final result meets the goal
Expected tool usageSpecifies which tools should be called, or which tools should not be calledSuitable for tool-based Agents
Expected intermediate agent responseRecords key intermediate steps, plans, confirmation information, or staged outputsSuitable for multi-step tasks and multi-turn tasks
Acceptance criteriaBreaks “success” into judgeable conditionsSuitable for complex tasks or open-ended tasks
Trace recordsSaves the actual execution trajectory, including call inputs, outputs, and contextUsed for automated evaluation and issue localization

In actual construction, it is recommended to first consolidate high-frequency, critical, and error-prone tasks into a Golden Set. The Golden Set does not need to be large at the beginning, but it should be stable enough, reusable, and suitable for regression evaluation after changes to models, prompts, toolchains, or business rules.

How to Design Task Success Rate?

Task success rate focuses on whether the final objective has been achieved. When designing it, do not look only at whether the answers appear similar; instead, break the task down into judgeable completion conditions.

Break Task Objectives into Acceptance Conditions

For example, an Agent task to “query order status and provide handling suggestions” can be broken down into the following conditions:

  • Whether the correct order was identified;
  • Whether the current order status was queried;
  • Whether a reasonable explanation was given based on the status;
  • Whether the next action was prompted when necessary;
  • Whether fabricating nonexistent information was avoided.

These conditions are easier to review than “whether the answer is good,” and they are more suitable for subsequent automated or semi-automated evaluation.

Set Reference Responses for Different Task Types

Different tasks are not necessarily suitable for the same type of reference answer. Information query tasks can provide relatively clear reference responses; planning or multi-step tasks are more suitable for acceptance criteria that specify which information must appear and which behaviors must not appear.

Task typeMore suitable evaluation method
Information queryReference response, key fact checking
Tool operationOperation completion conditions, tool call results
Multi-step planningStaged goals, key node checks
Exception handlingError recovery rate, whether an executable alternative is provided

Do Not Equate Task Success Rate with User Satisfaction

Task success rate is more suitable for measuring whether a specific goal has been completed, while user satisfaction may also be affected by tone, speed, interaction experience, business expectations, and other factors. For complex Agents, task success rate can be one of the core metrics, but it should not be the only metric.

How to Design Trajectory Quality?

Trajectory quality focuses on the execution process between the user request and the final result, including the reasoning path, tool call sequence, intermediate responses, and error recovery performance.

An Agent with a correct result may still have trajectory risks. For example, it may take many irrelevant steps, consume too many Tokens, call unnecessary tools, or produce very different execution paths across multiple runs. These problems may not directly lead to failure in small-scale testing, but after launch they may be amplified into cost, stability, or security risks.

Observable Trajectory Dimensions

DimensionFocus
Trajectory accuracyWhether the execution process conforms to the expected path or key nodes
Number of stepsWhether there are obvious redundant steps or skipped steps
Token consumptionWhether extra cost is caused by ineffective reasoning or repeated calls
LatencyWhether task response efficiency requirements are met
Error recovery rateWhether problems can be identified and the path corrected after an error occurs
Consistency across multiple runsWhether stable performance is maintained when the same task is executed multiple times

Record Key Intermediate Nodes for Multi-Step Tasks

For multi-step tasks, it is not recommended to record only the final answer. A more robust approach is to write key intermediate nodes into the evaluation sample, for example:

  • Whether the Agent first clarified necessary conditions;
  • Whether tools were called in a reasonable order;
  • Whether the judgment was updated after receiving tool results;
  • Whether a recovery strategy was adopted when a tool failed;
  • Whether repeating the same ineffective action was avoided.

This can break down “it seems to have answered correctly” into “whether the process is reliable.” When failures occur online, it also becomes easier to determine whether the problem comes from model understanding, prompts, tool returns, parameter construction, or business rules.

How to Design Tool Accuracy?

Tool accuracy is suitable for evaluating AI Agents that depend on toolchains, such as scenarios requiring calls to plugins, retrieval systems, business APIs, databases, or external services. It can be divided into two levels: whether the appropriate tool was selected, and whether the tool call parameters were correct.

LevelEvaluation questionExample observation points
Tool selection qualityWhether the Agent selected the right toolWhether the tool that should be called was called; whether irrelevant tools were avoided; whether tools were selected according to the task stage
Tool parameter correctnessWhether the Agent constructed correct parameters based on the contextWhether parameter fields are complete; whether values come from the correct context; whether necessary constraints are omitted

Trace-based automated evaluation is suitable for checking the tool usage process. A Trace can record the call parameters, context, tool returns, and subsequent responses after the model selects a tool, thereby helping locate whether the error occurred in “selecting the tool” or “filling in parameters.”

For Agents with longer toolchains, it is recommended to record the following for each call:

  • Tool name or tool type;
  • Call input and key parameters;
  • Tool output summary;
  • Context dependencies before and after the call;
  • Recovery actions when the call fails.

Key point: Tool accuracy should not only look at “whether a tool was called.” If the tool is selected correctly but the parameters are wrong, the final result may still be wrong; if the parameters are correct but the call order is unreasonable, it may also affect the quality of task completion.

How to Move from Offline Evaluation to an Online Feedback Loop?

An AI Agent evaluation dataset should not be used only for one-time acceptance testing before launch. A more robust approach is to connect the offline Golden Set, Trace automated evaluation, and online canary feedback to form a continuous improvement loop.

Step 1: Use the Golden Set for Offline Regression

First select core tasks, boundary scenarios, and high-risk tool calls to build a stable Golden Set. After each modification to the model, prompts, tool descriptions, business rules, or workflows, this batch of samples can be used for regression evaluation to observe whether task success rate, trajectory quality, and tool accuracy have degraded.

Step 2: Use Trace Automated Evaluation to Locate Process Issues

After offline evaluation finds failed samples, do not look only at the final answer. You should further review the Trace and analyze tool selection, parameter construction, intermediate responses, and trajectory deviations. For tool-based Agents, tool selection quality and tool parameter correctness can be checked first to reduce the burden of manual item-by-item troubleshooting.

Step 3: Feed Online Canary Feedback Back into the Evaluation Dataset

After launch, real failure samples, high-frequency user issues, and abnormal tool calls during the canary process should be organized into new evaluation cases. In this way, the evaluation dataset will gradually evolve from static samples into a continuously evolving quality asset.

Evaluation Dataset Construction Checklist

  • Do the samples cover core tasks, boundary scenarios, and high-risk tool calls?
  • Does each sample have a clear user query, reference response, or acceptance criteria?
  • Do samples involving tool calls record expected tool usage and key parameters?
  • Do multi-step tasks record key intermediate nodes, not just the final answer?
  • Do the metrics cover task success rate, trajectory quality, and tool accuracy at the same time?
  • Are real online failure samples fed back into the Golden Set?
  • Is regression evaluation continuously performed after changes to models, prompts, and toolchains?

Through this method, an AI Agent evaluation dataset can be upgraded from simple answer checking to a quality system that covers results, processes, and actions. It can help teams determine whether the Agent has completed the task, and also help locate why it failed and how to continuously improve after launch.