跳到主要内容
非官方简体中文翻译，内容版权归 Mastra 所有，遵循 Apache-2.0 许可。

runEvals

The runEvals function enables batch evaluation of agents and workflows by running multiple test cases against scorers concurrently. This is essential for systematic testing, performance analysis, and validation of AI systems.

Usage example
Usage example的直接链接

import { runEvals } from '@mastra/core/evals'
import { mastra } from './mastra'

const result = await runEvals({
  target: mastra.getAgent('myAgent'),
  data: [
    { input: 'What is machine learning?' },
    { input: 'Explain neural networks' },
    { input: 'How does AI work?' },
  ],
  scorers: [mastra.getScorer('myScorer1'), mastra.getScorer('myScorer2')],
  targetOptions: { maxSteps: 5 },
  concurrency: 2,
  onItemComplete: ({ item, targetResult, scorerResults }) => {
    console.log(`Completed: ${item.input}`)
    console.log(`Scores:`, scorerResults)
  },
})

console.log(`Average scores:`, result.scores)
console.log(`Processed ${result.summary.totalItems} items`)

Resolve the target from the Mastra instance with mastra.getAgent() or mastra.getWorkflow() rather than importing the agent or workflow directly. A directly imported target has no Mastra instance attached, so registry lookups made during execution (a step calling mastra.getAgent(), a child workflow, an .agent('agent-id') reference, a tool calling mastra.getWorkflow()) fail, scores aren't persisted, and trace-based trajectory extraction is unavailable. runEvals logs a warning when the target isn't registered.

Multi-turn evaluation
Multi-turn evaluation的直接链接

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { mastra } from './mastra'

const result = await runEvals({
  target: mastra.getAgent('weatherAgent'),
  data: [
    {
      inputs: [
        'What is the weather in Brooklyn?',
        'What about tomorrow?',
        'Compare the two forecasts.',
      ],
    },
  ],
  scorers: [checks.calledTool('get_weather', { times: 2 }), checks.includes('Brooklyn')],
})

With gates and thresholds
With gates and thresholds的直接链接

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { mastra } from './mastra'

const result = await runEvals({
  target: mastra.getAgent('weatherAgent'),
  data: [{ input: 'What is the weather in Brooklyn?' }],
  gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
  scorers: [
    { scorer: mastra.getScorer('faithfulnessScorer'), threshold: 0.7 },
    checks.includes('Brooklyn'),
  ],
})

result.verdict // 'passed' | 'scored' | 'failed'
result.gateResults // [{ id, passed, score }]
result.thresholdResults // [{ id, passed, averageScore, threshold }]

Parameters
Parameters的直接链接

target:

Agent | Workflow
The agent or workflow to evaluate.

data:

RunEvalsDataItem[]
Array of test cases with input data and optional ground truth.

scorers?:

ScorerEntry[] | AgentScorerConfig | WorkflowScorerConfig
Scorers to use. Each entry is either a bare MastraScorer or { scorer, threshold } for threshold tracking. An AgentScorerConfig object separates agent-level and trajectory scorers. A WorkflowScorerConfig object specifies scorers for the workflow, individual steps, and trajectory. Optional when at least one gate is provided (gate-only runs).

gates?:

MastraScorer[]
Scorers that must score 1.0 for the run to pass. If any gate averages below 1.0 across data items, the verdict is failed. Gates run before regular scorers on each data item. When provided, scorers may be omitted.

targetOptions?:

AgentExecutionOptions | WorkflowRunOptions
Options forwarded to the target during execution. For agents: options passed to agent.generate() (e.g. maxSteps, modelSettings, instructions). For workflows: options passed to run.start() (e.g. perStep, outputOptions, initialState). For multi-turn agent runs (inputs/turns), runEvals generates and injects the shared thread and a resource, so memory.thread is optional; provide memory.resource to reuse a specific resource.

concurrency?:

number
= 1
Number of test cases to run concurrently.

onItemComplete?:

function
Callback function called after each test case completes. Receives item, target result, and scorer results.

Data item structure
Data item structure的直接链接

input?:

string | string[] | CoreMessage[] | any
Input data for the target. For agents: messages or strings. For workflows: workflow input data. Optional when inputs is provided.

inputs?:

(string | string[] | CoreMessage[] | any)[]
Multi-turn inputs. Each entry is one turn (same shape as input) sent sequentially to the agent on the same thread. Scorers see the accumulated output from all turns. Only supported for Agent targets. When provided, input can be omitted. Mutually exclusive with turns.

turns?:

EvalTurn[]
Multi-turn conversation with per-turn assertions. Each turn is an object { input, gates?, scorers? } sent sequentially on the same thread; its gates/scorers evaluate only that turn's input and output. Per-turn outcomes are reported in turnResults and folded into the overall verdict. Only supported for Agent targets. Mutually exclusive with input and inputs.

groundTruth?:

any
Expected or reference output for comparison during scoring.

expectedTrajectory?:

TrajectoryExpectation
Expected trajectory configuration for trajectory scoring. Includes expected steps, ordering, efficiency budgets, blacklists, and tool failure tolerance. Passed to trajectory scorers as run.expectedTrajectory. Overrides the static defaults in scorer constructors.

requestContext?:

RequestContext
Request Context to pass to the target during execution.

tracingContext?:

TracingContext
Tracing context for observability and debugging.

startOptions?:

WorkflowRunOptions
Per-item workflow run options (e.g. initialState, perStep, outputOptions). Merged on top of targetOptions, so per-item values take precedence. Only applicable when the target is a workflow.

Agent scorer configuration
Agent scorer configuration的直接链接

For agents, use AgentScorerConfig to separate agent-level scorers from trajectory scorers:

agent?:

MastraScorer[]
Scorers that receive the raw agent output (MastraDBMessage[]). Use for evaluating response quality, content, etc.

trajectory?:

MastraScorer[]
Scorers that receive a pre-extracted Trajectory object. When storage is configured, the pipeline extracts a hierarchical trajectory from observability traces (including nested tool calls and model generations). Otherwise, it falls back to extracting tool calls from agent messages.

Workflow scorer configuration
Workflow scorer configuration的直接链接

For workflows, use WorkflowScorerConfig to specify scorers at different levels:

workflow?:

MastraScorer[]
Scorers to evaluate the entire workflow output.

steps?:

Record<string, MastraScorer[]>
Object mapping step IDs to arrays of scorers for evaluating individual step outputs.

trajectory?:

MastraScorer[]
Scorers that receive a pre-extracted Trajectory from the workflow execution. When storage is configured, the pipeline extracts a hierarchical trajectory from observability traces (including nested agent runs and tool calls within workflow steps). Otherwise, it falls back to extracting step results from the workflow output.

Returns
Returns的直接链接

scores:

Record<string, any>
Average scores across all test cases, organized by scorer name.

summary:

object
Summary information about the experiment execution.

summary.totalItems:

number
Total number of test cases processed.

summary.notScorable?:

Record<string, number>
Number of runs each scorer or gate declared not scorable via notScorable(), keyed by id. Those runs are left out of scores, gate and threshold averages, and the verdict. Present only when at least one run was not scorable.

verdict?:

'passed' | 'scored' | 'failed'
Present when at least one configured gate or threshold (top-level or per-turn) produces a numeric score. Omitted when none do, including when every assertion returned notScorable(). passed = all gates and thresholds met. scored = gates passed but a threshold was missed. failed = at least one gate did not score 1.0.

gateResults?:

GateResult[]
Per-gate results averaged across all data items. Each entry has id, passed (boolean), and score (0–1).

thresholdResults?:

ThresholdResult[]
Per-threshold-scorer results averaged across all data items. Each entry has id, passed, averageScore, and threshold.

turnResults?:

TurnResult[]
Present when any data item uses turns. Each entry has index (zero-based turn), optional gateResults, thresholdResults, and scores (bare-scorer averages keyed by scorer id), aggregated by turn index across data items.

EvalTurn
EvalTurn的直接链接

A single turn in a turns array. Its gates/scorers evaluate only that turn's input and output:

input:

string | string[] | CoreMessage[] | any
The input sent to the agent for this turn.

gates?:

MastraScorer[]
Gates that must score 1.0 for this turn. A failing turn gate makes the overall verdict failed.

scorers?:

ScorerEntry[]
Scorers (optionally with thresholds) evaluated against this turn only. A missed per-turn threshold (with gates passing) makes the verdict scored.

ScorerEntry
ScorerEntry的直接链接

A scorer entry in the scorers array can be either a bare scorer or a scorer with a threshold:

scorer:

MastraScorer
The scorer instance.

threshold:

number | { min?: number; max?: number }
A number implies minimum threshold (score at or above passes). Use { min, max } for range-based checks — e.g. { max: 0.3 } for scorers like hallucination where a high score is bad. Both min and max must be between 0 and 1.

Examples
Examples的直接链接

Gates and verdict
Gates and verdict的直接链接

Use gates for hard pass/fail requirements and { scorer, threshold } for tracked quality metrics:

import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'

const result = await runEvals({
  target: mastra.getAgent('weatherAgent'),
  data: [{ input: 'What is the weather in Brooklyn?' }],
  gates: [checks.calledTool('get_weather'), checks.noToolErrors()],
  scorers: [
    { scorer: faithfulnessScorer, threshold: 0.7 }, // min threshold (number shorthand)
    { scorer: hallucinationScorer, threshold: { max: 0.3 } }, // max threshold (high = bad)
    { scorer: toneScorer, threshold: { min: 0.5, max: 0.9 } }, // range threshold
    checks.includes('Brooklyn'), // bare scorer, no threshold
  ],
})

if (result.verdict === 'failed') {
  console.log(
    'Gate failures:',
    result.gateResults?.filter(g => !g.passed),
  )
} else if (result.verdict === 'scored') {
  console.log(
    'Threshold misses:',
    result.thresholdResults?.filter(t => !t.passed),
  )
}

Agent Evaluation
Agent Evaluation的直接链接

import { createScorer, runEvals } from '@mastra/core/evals'

const myScorer = createScorer({
  id: 'my-scorer',
  description: "Check if Agent's response contains ground truth",
  type: 'agent',
}).generateScore(({ run }) => {
  const response = run.output[0]?.content || ''
  const expectedResponse = run.groundTruth
  return response.includes(expectedResponse) ? 1 : 0
})

const result = await runEvals({
  target: mastra.getAgent('chatAgent'),
  data: [
    {
      input: 'What is AI?',
      groundTruth: 'AI is a field of computer science that creates intelligent machines.',
    },
    {
      input: 'How does machine learning work?',
      groundTruth: 'Machine learning uses algorithms to learn patterns from data.',
    },
  ],
  scorers: [relevancyScorer],
  concurrency: 3,
})

Agent trajectory evaluation
Agent trajectory evaluation的直接链接

Use AgentScorerConfig to evaluate both the agent response and its tool-calling trajectory:

import { runEvals } from '@mastra/core/evals'
import { createTrajectoryAccuracyScorerCode } from '@mastra/evals/scorers/prebuilt'

const trajectoryScorer = createTrajectoryAccuracyScorerCode()

const result = await runEvals({
  target: mastra.getAgent('chatAgent'),
  data: [
    {
      input: 'What is the weather in London?',
      expectedTrajectory: {
        steps: [{ stepType: 'tool_call', name: 'weatherTool' }],
      },
    },
  ],
  scorers: {
    // agent: [responseQualityScorer], // Optional: add agent-level scorers
    trajectory: [trajectoryScorer],
  },
})

// result.scores.agent — average agent-level scores
// result.scores.trajectory — average trajectory scores

Agent with targetOptions
agent-with-targetoptions的直接链接

Pass execution options like maxSteps or modelSettings to customize agent behavior during evaluation:

const result = await runEvals({
  target: mastra.getAgent('chatAgent'),
  data: [{ input: 'Summarize this article' }, { input: 'Translate to French' }],
  scorers: [relevancyScorer],
  targetOptions: {
    maxSteps: 5,
    modelSettings: { temperature: 0 },
  },
})

Workflow Evaluation
Workflow Evaluation的直接链接

Resolve the workflow with mastra.getWorkflow() so steps that call mastra.getAgent() or run child workflows can reach the registry:

const workflowResult = await runEvals({
  target: mastra.getWorkflow('myWorkflow'),
  data: [
    { input: { query: 'Process this data', priority: 'high' } },
    { input: { query: 'Another task', priority: 'low' } },
  ],
  scorers: {
    workflow: [outputQualityScorer],
    steps: {
      'validation-step': [validationScorer],
      'processing-step': [processingScorer],
    },
  },
  onItemComplete: ({ item, targetResult, scorerResults }) => {
    console.log(`Workflow completed for: ${item.inputData.query}`)
    if (scorerResults.workflow) {
      console.log('Workflow scores:', scorerResults.workflow)
    }
    if (scorerResults.steps) {
      console.log('Step scores:', scorerResults.steps)
    }
  },
})

Workflow trajectory evaluation
Workflow trajectory evaluation的直接链接

Add trajectory scoring to workflow evaluations to validate step execution order:

const workflowResult = await runEvals({
  target: mastra.getWorkflow('myWorkflow'),
  data: [
    {
      input: { query: 'Process this data' },
      expectedTrajectory: {
        steps: [
          { stepType: 'workflow_step', name: 'validate' },
          { stepType: 'workflow_step', name: 'process' },
          { stepType: 'workflow_step', name: 'output' },
        ],
      },
    },
  ],
  scorers: {
    workflow: [outputQualityScorer],
    steps: {
      validate: [validationScorer],
    },
    trajectory: [trajectoryScorer],
  },
})

// result.scores.trajectory — workflow trajectory scores

Workflow with per-item startOptions
workflow-with-per-item-startoptions的直接链接

Use startOptions on individual data items to customize each workflow run. Per-item values take precedence over targetOptions:

const result = await runEvals({
  target: mastra.getWorkflow('myWorkflow'),
  data: [
    {
      input: { query: 'hello' },
      startOptions: { initialState: { counter: 1 } },
    },
    {
      input: { query: 'world' },
      startOptions: { initialState: { counter: 2 } },
    },
  ],
  scorers: [outputQualityScorer],
  targetOptions: { perStep: true },
})

Multi-turn conversation evaluation
Multi-turn conversation evaluation的直接链接

Use inputs to send sequential turns on a shared thread. Scorers see the accumulated output from all turns:

const result = await runEvals({
  target: mastra.getAgent('chatAgent'),
  data: [
    {
      inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
    },
  ],
  gates: [checks.calledTool('get_weather')],
  scorers: [{ scorer: checks.similarity('Brooklyn weather forecast'), threshold: 0.5 }],
})

// result.verdict: 'passed' | 'scored' | 'failed'

Each turn runs agent.generate() with the same threadId, so the agent sees the full conversation history. runEvals also injects a resourceId (Mastra memory scopes messages by resource + thread), defaulting it to the generated thread. Pass targetOptions.memory.resource to pin a specific one. Cross-turn recall requires the agent to have a memory store configured. Otherwise turns run in isolation. Mix single-turn (input) and multi-turn (inputs) items in the same data array. When using inputs, input can be omitted.

Scoring uses the accumulated output from all turns as run.output, but only the first turn as run.input. Prefer output-based scorers (checks.includes, checks.calledTool, checks.similarity) for multi-turn. Input-relative scorers (e.g. faithfulness) only see the first turn's input. Trajectory scorers that read from the trace (AgentScorerConfig.trajectory) resolve against the last turn's span. Tool-call checks that read run.output (like checks.calledTool) still see every turn.

Per-turn assertions
Per-turn assertions的直接链接

Use turns to attach gates/scorers to individual turns. Each per-turn assertion sees only that turn's input and output, so a later-turn regression can't be hidden by an earlier turn:

const result = await runEvals({
  target: mastra.getAgent('chatAgent'),
  data: [
    {
      turns: [
        {
          input: 'What is the weather in Brooklyn?',
          gates: [checks.calledTool('get_weather')],
        },
        {
          input: 'What about tomorrow?',
          gates: [checks.calledTool('get_weather')], // must call again this turn
          scorers: [{ scorer: checks.similarity('tomorrow forecast'), threshold: 0.5 }],
        },
      ],
    },
  ],
})

result.verdict // folds in per-turn gate/threshold outcomes
result.turnResults // [{ index, gateResults, thresholdResults, scores }]

Per-turn gates/scorers evaluate only that turn (run.input/run.output are that turn's). A failing turn gate makes the verdict failed. A missed turn threshold (gates passing) makes it scored. Top-level scorers/gates still score the accumulated conversation as a whole. turns is Agent-only and can't be combined with input or inputs.

On this page