Faithfulness scorer

The createFaithfulnessScorer() function evaluates how factually accurate an LLM's output is compared to the provided context. It extracts claims from the output and verifies them against the context. Use it to check whether a RAG response is supported by the retrieved information.

Parameters

The createFaithfulnessScorer() function accepts a single options object with the following properties:

model:

LanguageModel

Configuration for the model used to evaluate faithfulness.

options:

{ context?: string[]; scale?: number }

Configuration options for the scorer.

{ context?: string[]; scale?: number }

context:

string[]

Array of context chunks against which the output's claims will be verified. When omitted, the scorer uses the results of tool calls in the run output.

scale:

number

The maximum score value. The final score will be normalized to this scale.

This function returns an instance of the MastraScorer class. The .run() method accepts the same input as other scorers (see the MastraScorer reference), but the return value includes LLM-specific fields as documented below.

.run() returns

runId:

string

The id of the run (optional).

preprocessStepResult:

object

Object with the claims extracted from the output: { claims: string[] }

preprocessPrompt:

string

The prompt sent to the LLM for the preprocess step (optional).

analyzeStepResult:

object

Object with verdicts: { verdicts: Array<{ verdict: 'yes' | 'no' | 'unsure', reason: string }> }

analyzePrompt:

string

The prompt sent to the LLM for the analyze step (optional).

score:

number

A score between 0 and the configured scale, representing the proportion of claims that are supported by the context.

reason:

string

A detailed explanation of the score, including which claims were supported, contradicted, or marked as unsure.

generateReasonPrompt:

string

The prompt sent to the LLM for the generateReason step (optional).

Scoring details

The scorer evaluates faithfulness through claim verification against provided context.

Scoring Process

1. Analyzes claims and context:
   • Extracts all claims (factual and speculative)
   • Verifies each claim against context
   • Assigns one of three verdicts:
     • "yes" - claim supported by context
     • "no" - claim contradicts context
     • "unsure" - claim unverifiable
2. Calculates faithfulness score:
   • Counts supported claims
   • Divides by total claims
   • Scales to configured range

Final score: (supported_claims / total_claims) * scale

Score interpretation

A faithfulness score between 0 and the configured scale (default 1). With the default scale:

• 1.0: All claims are accurate and directly supported by the context.
• 0.7 to 0.9: Most claims are correct, with minor additions or omissions.
• 0.4 to 0.6: Some claims are supported, but others are unverifiable.
• 0.1 to 0.3: Most of the content is inaccurate or unsupported.
• 0.0: All claims are false or contradict the context.

Example

Evaluate agent responses for faithfulness to provided context:

src/example-faithfulness.ts

import { runEvals } from '@mastra/core/evals'
import { createFaithfulnessScorer } from '@mastra/evals/scorers/prebuilt'
import { myAgent } from './agent'

// Context is typically populated from agent tool calls or RAG retrieval
const scorer = createFaithfulnessScorer({
  model: 'openai/gpt-5.6-sol',
})

const result = await runEvals({
  data: [
    {
      input: 'Tell me about the Tesla Model 3.',
    },
    {
      input: 'What are the key features of this electric vehicle?',
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.id].score,
      reason: scorerResults[scorer.id].reason,
    })
  },
})

console.log(result.scores)

For more details on runEvals, see the runEvals reference.

To add this scorer to an agent, see the Scorers overview guide.

Related

• Answer Relevancy Scorer
• Hallucination Scorer