Context precision scorer

The createContextPrecisionScorer() function creates a scorer that evaluates how relevant and well-positioned retrieved context pieces are for generating expected outputs. It uses Mean Average Precision (MAP) to reward systems that place relevant context earlier in the sequence.

It's especially useful for these use cases:

RAG system evaluation

Ideal for evaluating retrieved context in RAG pipelines where:

• Context ordering matters for model performance
• You need to measure retrieval quality beyond basic relevance
• Early relevant context is more valuable than later relevant context

Context window optimization

Use when optimizing context selection for:

• Limited context windows
• Token budget constraints
• Multi-step reasoning tasks

Parameters

model:

MastraModelConfig

The language model to use for evaluating context relevance

options:

ContextPrecisionMetricOptions

Configuration options for the scorer

Either context or contextExtractor must be provided. If both are provided, contextExtractor takes precedence.

.run() returns

score:

number

Mean Average Precision score between 0 and scale (default 0-1). Assumes a positive scale.

reason:

string

Human-readable explanation of the context precision evaluation

Scoring details

Mean Average Precision (MAP)

Context Precision uses Mean Average Precision to evaluate both relevance and positioning:

1. Context Evaluation: Each context piece is classified as relevant or irrelevant for generating the expected output
2. Precision Calculation: For each relevant context at position i, precision = relevant_items_so_far / (i + 1)
3. Average Precision: Sum all precision values and divide by total relevant items
4. Final Score: Multiply by scale factor and round to 2 decimals

Scoring Formula

MAP = (Σ Precision@k) / R

Where:
- Precision@k = (relevant items in positions 1...k) / k
- R = total number of relevant items
- Only calculated at positions where relevant items appear

Score Interpretation

• 0.9-1.0: Excellent precision - all relevant context early in sequence
• 0.7-0.8: Good precision - most relevant context well-positioned
• 0.4-0.6: Moderate precision - relevant context mixed with irrelevant
• 0.1-0.3: Poor precision - little relevant context or poorly positioned
• 0.0: No relevant context found

Reason analysis

The reason field explains:

• Which context pieces were deemed relevant/irrelevant
• How positioning affected the MAP calculation
• Specific relevance criteria used in evaluation

Optimization insights

Use results to:

• Improve retrieval: Filter out irrelevant context before ranking
• Optimize ranking: Ensure relevant context surfaces early
• Tune chunk size: Balance context detail vs. relevance precision
• Evaluate embeddings: Test different embedding models for better retrieval

Example Calculation

context: [relevant, irrelevant, relevant, irrelevant]

• Position 0: Relevant → Precision = 1/1 = 1.0
• Position 1: Skip (irrelevant)
• Position 2: Relevant → Precision = 2/3 = 0.67
• Position 3: Skip (irrelevant)

MAP = (1.0 + 0.67) / 2 = 0.835 ≈ 0.83

Scorer configuration

Dynamic context extraction

const scorer = createContextPrecisionScorer({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      // Extract context dynamically based on the query
      const query = input?.inputMessages?.[0]?.content || ''

      // Example: Retrieve from a vector database
      const searchResults = vectorDB.search(query, { limit: 10 })
      return searchResults.map(result => result.content)
    },
    scale: 1,
  },
})

Large context evaluation

const scorer = createContextPrecisionScorer({
  model: 'openai/gpt-5.6-sol',
  options: {
    context: [
      // Simulate retrieved documents from vector database
      'Document 1: Highly relevant content...',
      'Document 2: Somewhat related content...',
      'Document 3: Tangentially related...',
      'Document 4: Not relevant...',
      'Document 5: Highly relevant content...',
      // ... up to dozens of context pieces
    ],
  },
})

Example

Evaluate RAG system context retrieval precision for different queries:

src/example-context-precision.ts

import { runEvals } from '@mastra/core/evals'
import { createContextPrecisionScorer } from '@mastra/evals/scorers/prebuilt'
import { myAgent } from './agent'

const scorer = createContextPrecisionScorer({
  model: 'openai/gpt-5.6-sol',
  options: {
    contextExtractor: (input, output) => {
      // Extract context from agent's retrieved documents
      return output.metadata?.retrievedContext || []
    },
  },
})

const result = await runEvals({
  data: [
    {
      input: 'How does photosynthesis work in plants?',
    },
    {
      input: 'What are the mental and physical benefits of exercise?',
    },
  ],
  scorers: [scorer],
  target: myAgent,
  onItemComplete: ({ scorerResults }) => {
    console.log({
      score: scorerResults[scorer.id].score,
      reason: scorerResults[scorer.id].reason,
    })
  },
})

console.log(result.scores)

For more details on runEvals, see the runEvals reference.

To add this scorer to an agent, see the Scorers overview guide.

Comparison with context relevance

Choose the right scorer for your needs:

Use Case	Context Relevance	Context Precision
RAG evaluation	When usage matters	When ranking matters
Context quality	Nuanced levels	Binary relevance
Missing detection	✓ Identifies gaps	✗ Not evaluated
Usage tracking	✓ Tracks utilization	✗ Not considered
Position sensitivity	✗ Position agnostic	✓ Rewards early placement

Related

• Answer Relevancy Scorer: Evaluates if answers address the question
• Faithfulness Scorer: Measures answer groundedness in context
• Custom Scorers: Creating your own evaluation metrics