Skip to main content

DatasetsManager.compareExperiments()

Added in: @mastra/core@1.4.0

Compares two or more experiments, producing per-item and per-scorer comparisons. Requires at least two experiment IDs.

Usage example
Direct link to Usage example

import { Mastra } from '@mastra/core'

const mastra = new Mastra({/* storage config */})

const comparison = await mastra.datasets.compareExperiments({
  experimentIds: ['exp-baseline', 'exp-prompt-v2', 'exp-new-model'],
  baselineId: 'exp-baseline',
})

console.log(`Baseline: ${comparison.baselineId}`)

for (const item of comparison.items) {
  console.log(`Item ${item.itemId}:`)
  console.log(`  Input: ${JSON.stringify(item.input)}`)

  for (const [expId, result] of Object.entries(item.results)) {
    if (!result) {
      console.log(`  ${expId}: did not run this item`)
      continue
    }
    console.log(
      `  ${expId}: output=${JSON.stringify(result.output)}, scores=${JSON.stringify(result.scores)}`,
    )
  }
}

Parameters
Direct link to Parameters

experimentIds:

string[]
Array of experiment IDs to compare. Duplicate IDs are removed before the 2-ID minimum is checked, so the list must contain at least 2 distinct IDs.

baselineId?:

string
ID of the baseline experiment. Must be one of experimentIds. Defaults to the first ID in experimentIds.

Returns
Direct link to Returns

Throws a MastraError with ID COMPARE_INVALID_INPUT when:

  • experimentIds contains fewer than 2 distinct IDs.
  • baselineId isn't one of experimentIds. Earlier versions silently ignored an unknown baselineId.

result:

Promise<object>
Comparison results.
object

baselineId:

string
ID of the baseline experiment used for comparison.

items:

Array<object>
Per-item comparison data.
object

itemId:

string
ID of the dataset item.

input:

unknown | null
Input data for the item.

groundTruth:

unknown | null
Ground truth for the item.

results:

Record<string, { output: unknown; scores: Record<string, number | null> } | null>
Results keyed by experiment ID, with an entry for every experiment in experimentIds. Each entry contains the output and scorer results for that experiment, or null when that experiment did not run the item.

HTTP route
Direct link to HTTP route

POST /api/datasets/:datasetId/compare calls this method. The route accepts between 2 and 10 IDs in experimentIds. It also still accepts the deprecated { experimentIdA, experimentIdB } body, which compares the two experiments with experimentIdA as the baseline.

On this page