Running scorers in CI

在 CI 流水线中运行 scorer 可以提供可量化的指标，用于长期衡量 agent 质量。runEvals 函数会将多个测试用例通过你的 agent 或 workflow 运行，并返回聚合得分。

基本设置

你可以使用任何支持 ESM 模块的测试框架，例如 Vitest、Jest 或 Mocha。

创建测试用例

使用 runEvals 针对多个测试用例评估你的 agent。该函数接受一个数据项数组，每个数据项包含 input 和可选的 groundTruth，供 scorer 校验使用。

请通过 mastra.getAgent() 或 mastra.getWorkflow() 解析 target，而不是直接导入它。直接导入的 agent 或 workflow 没有附加 Mastra 实例，因此步骤或工具内部的注册表查找（例如 mastra.getAgent('weatherAgent')）会失败，得分不会被持久化，基于轨迹的轨迹评分也不可用。

tests/weather-agent.test.ts

import { describe, it, expect } from 'vitest'
import { runEvals } from '@mastra/core/evals'
import { mastra } from '../src/mastra'
import { locationScorer } from '../src/mastra/scorers/location-scorer'

describe('Weather Agent Tests', () => {
  it('should correctly extract locations from queries', async () => {
    const result = await runEvals({
      data: [
        {
          input: 'weather in Berlin',
          groundTruth: { expectedLocation: 'Berlin', expectedCountry: 'DE' },
        },
        {
          input: 'weather in Berlin, Maryland',
          groundTruth: { expectedLocation: 'Berlin', expectedCountry: 'US' },
        },
        {
          input: 'weather in Berlin, Russia',
          groundTruth: { expectedLocation: 'Berlin', expectedCountry: 'RU' },
        },
      ],
      target: mastra.getAgent('weatherAgent'),
      scorers: [locationScorer],
    })

    // Assert aggregate score meets threshold
    expect(result.scores['location-accuracy']).toBe(1)
    expect(result.summary.totalItems).toBe(3)
  })
})

理解结果

runEvals 函数返回一个对象，包含：

• scores：每个 scorer 在所有测试用例上的平均得分
• summary.totalItems：处理的测试用例总数

{
  scores: {
    'location-accuracy': 1.0,  // Average score across all items
    'another-scorer': 0.85
  },
  summary: {
    totalItems: 3
  }
}

多个测试场景

为不同的评估场景创建单独的测试用例：

tests/weather-agent.test.ts

describe('Weather Agent Tests', () => {
  const weatherAgent = mastra.getAgent('weatherAgent')

  it('should handle location disambiguation', async () => {
    const result = await runEvals({
      data: [
        {
          input: 'weather in Berlin',
          groundTruth: {/* ... */},
        },
        {
          input: 'weather in Berlin, Maryland',
          groundTruth: {/* ... */},
        },
      ],
      target: weatherAgent,
      scorers: [locationScorer],
    })

    expect(result.scores['location-accuracy']).toBe(1)
  })

  it('should handle typos and misspellings', async () => {
    const result = await runEvals({
      data: [
        {
          input: 'weather in Berln',
          groundTruth: { expectedLocation: 'Berlin', expectedCountry: 'DE' },
        },
        {
          input: 'weather in Parris',
          groundTruth: { expectedLocation: 'Paris', expectedCountry: 'FR' },
        },
      ],
      target: weatherAgent,
      scorers: [locationScorer],
    })

    expect(result.scores['location-accuracy']).toBe(1)
  })
})

后续步骤

• 了解创建自定义 scorer
• 探索内置 scorer
• 针对启用了内存的 agents运行 scorer
• 阅读 runEvals API 参考