跳到主要内容
非官方简体中文翻译，内容版权归 Mastra 所有，遵循 Apache-2.0 许可。

DatasetsManager.compareExperiments()

Added in: @mastra/core@1.4.0

Compares two or more experiments, producing per-item and per-scorer comparisons. Requires at least two experiment IDs.

Usage example
Usage example的直接链接

import { Mastra } from '@mastra/core'

const mastra = new Mastra({/* storage config */})

const comparison = await mastra.datasets.compareExperiments({
  experimentIds: ['exp-baseline', 'exp-prompt-v2', 'exp-new-model'],
  baselineId: 'exp-baseline',
})

console.log(`Baseline: ${comparison.baselineId}`)

for (const item of comparison.items) {
  console.log(`Item ${item.itemId}:`)
  console.log(`  Input: ${JSON.stringify(item.input)}`)

  for (const [expId, result] of Object.entries(item.results)) {
    if (!result) {
      console.log(`  ${expId}: did not run this item`)
      continue
    }
    console.log(
      `  ${expId}: output=${JSON.stringify(result.output)}, scores=${JSON.stringify(result.scores)}`,
    )
  }
}

Parameters
Parameters的直接链接

experimentIds:

string[]
Array of experiment IDs to compare. Duplicate IDs are removed before the 2-ID minimum is checked, so the list must contain at least 2 distinct IDs.

baselineId?:

string
ID of the baseline experiment. Must be one of experimentIds. Defaults to the first ID in experimentIds.

Returns
Returns的直接链接

Throws a MastraError with ID COMPARE_INVALID_INPUT when:

  • experimentIds contains fewer than 2 distinct IDs.
  • baselineId isn't one of experimentIds. Earlier versions silently ignored an unknown baselineId.

result:

Promise<object>
Comparison results.
object

baselineId:

string
ID of the baseline experiment used for comparison.

items:

Array<object>
Per-item comparison data.
object

itemId:

string
ID of the dataset item.

input:

unknown | null
Input data for the item.

groundTruth:

unknown | null
Ground truth for the item.

results:

Record<string, { output: unknown; scores: Record<string, number | null> } | null>
Results keyed by experiment ID, with an entry for every experiment in experimentIds. Each entry contains the output and scorer results for that experiment, or null when that experiment did not run the item.

HTTP route
HTTP route的直接链接

POST /api/datasets/:datasetId/compare calls this method. The route accepts between 2 and 10 IDs in experimentIds. It also still accepts the deprecated { experimentIdA, experimentIdB } body, which compares the two experiments with experimentIdA as the baseline.

On this page