Observability API

The Observability API provides methods to retrieve traces, monitor application performance, score traces for evaluation, and record feedback linked to traces and spans.

Getting a specific trace

Retrieve a specific trace by its ID, including all its spans and details:

const trace = await mastraClient.getTrace('trace-id-123')

Getting traces with filtering

Retrieve a paginated list of trace root spans with optional filtering:

const traces = await mastraClient.getTraces({
  pagination: {
    page: 1,
    perPage: 20,
    dateRange: {
      start: new Date('2024-01-01'),
      end: new Date('2024-01-31'),
    },
  },
  filters: {
    name: 'weather-agent', // Filter by trace name
    spanType: 'agent', // Filter by span type
    entityId: 'weather-agent-id', // Filter by entity ID
    entityType: 'agent', // Filter by entity type
  },
})

console.log(`Found ${traces.spans.length} root spans`)
console.log(`Total pages: ${traces.pagination.totalPages}`)

// To get the complete trace with all spans, use getTrace
const completeTrace = await mastraClient.getTrace(traces.spans[0].traceId)

Listing traces for a list view

listTracesLight() returns the same rows as a trace list, without the input, output and attributes payloads. Each row carries short inputPreview, outputPreview and errorPreview strings instead (at most 100 characters, ending with … when cut), so a list can render preview columns without transferring the whole payload, plus threadId and resourceId for the memory context columns.

Prefer it whenever you are rendering many traces at once, and fetch the full record only when a row is opened:

const list = await mastraClient.listTracesLight({
  pagination: { page: 0, perPage: 25 },
  filters: { entityType: 'agent' },
})

for (const span of list.spans) {
  console.log(span.name, span.inputPreview)
}

// Fetch the full payload only for the trace the user selects
const selected = await mastraClient.getTrace(list.spans[0].traceId)

It accepts the same filtering, ordering and delta-polling arguments as listTraces(). Use listTraces() when you actually need the full span payloads.

Querying spans

querySpans() returns completed span summaries across traces. It requires a positive ISO timestamp range over span start time and supports cursor pagination:

const result = await mastraClient.querySpans({
  timeRange: {
    from: '2026-10-01T00:00:00.000Z',
    to: '2026-10-02T00:00:00.000Z',
  },
  where: {
    op: 'and',
    args: [
      { op: 'eq', left: { path: 'threadId' }, right: { literal: 'thread-123' } },
      { op: 'gte', left: { path: 'durationMs' }, right: { literal: 1000 } },
    ],
  },
  orderBy: [{ field: 'startedAt', direction: 'desc' }],
  page: { limit: 100 },
})

// Pass result.page.next as page.after to continue with the same filters and ordering.
console.log(result.spans)

Pages default to 100 rows and allow at most 1,000. The response is { spans, page: { next } }, without a total count. Ordering accepts exactly one of startedAt or endedAt, ascending or descending, and defaults to descending start time.

Fields	Operators
traceId, spanId, parentSpanId, threadId	eq, ne, in, notIn, exists, notExists
organizationId, resourceId, runId, sessionId, userId, requestId, serviceName, experimentId, environment, executionSource	Same string operators
Entity, parent entity and root entity Type, Id, Name, VersionId fields, such as parentEntityId	Same string operators
name, spanType, status	Same string operators; name also accepts matches / notMatches for normalized whole words and phrases
startedAt, endedAt, durationMs	Comparisons, membership and presence
error	exists, notExists

Combine predicates with and, or and not. Membership uses { op: 'in', value: { path: 'spanType' }, set: ['agent_run', 'tool_call'] }. A span's status is error when it has an error and success otherwise.

Each summary includes identity, context, entity lineage, status, timestamps, durationMs, model/provider and input/output/error previews. Model/provider are available in returned rows but aren't filterable.

The additional identity, context and parent/root entity filters require an updated storage adapter. With an older adapter, requests using these filters return a structured 422 error.

DuckDB returns null for parent/root entity type, ID and name, and for executionSource. Equality filters on these fields match no spans. Parent/root entity version IDs remain available.

Querying threads

queryTraceThreads() returns thread summaries from completed traces through POST /observability/threads/query. Each row includes activity timestamps, trace counts, context and previews. Traces without a thread ID aren't included.

const result = await mastraClient.queryTraceThreads({
  timeRange: {
    traces: {
      from: '2026-10-01T00:00:00.000Z',
      to: '2026-10-02T00:00:00.000Z',
    },
  },
  where: { op: 'eq', left: { path: 'status' }, right: { literal: 'error' } },
  orderBy: [{ field: 'endedAt', direction: 'desc' }],
  page: { limit: 100 },
})

for (const thread of result.threads) {
  console.log(thread.firstTraceId, thread.lastTraceId, thread.traceCount, thread.outputPreview)
}

Thread time ranges

Choose exactly one time selector. Each range requires ISO timestamps with from earlier than to. The range includes from and excludes to.

Selector	Matches threads with
timeRange.traces: { from, to }	At least one trace that started in the window
timeRange.threads: { field: 'startedAt', from, to }	Their first activity in the window
timeRange.threads: { field: 'endedAt', from, to }	Their last activity in the window

For example, find threads whose most recent activity falls on October 1:

const result = await mastraClient.queryTraceThreads({
  timeRange: {
    threads: {
      field: 'endedAt',
      from: '2026-10-01T00:00:00.000Z',
      to: '2026-10-02T00:00:00.000Z',
    },
  },
})

Rows include the full retained history within the authorized scope, including traces before and after the window. Filters apply to these complete rows. An environment filter checks the first trace's environment.

Thread filters

Use where to filter thread rows. Combine predicates with and, or and not.

Fields	Operators
threadId, organizationId, firstTraceId, lastTraceId	eq, ne, in, notIn, exists, notExists
resourceId, userId, runId, sessionId, requestId, serviceName, experimentId, environment, executionSource	Same string operators
entityType, entityId, entityName, entityVersionId	Same string operators
status	eq, ne, in, notIn, with values success or error
startedAt, endedAt, timeToFirstChunkMs	eq, ne, lt, lte, gt, gte, in, notIn, exists, notExists
traceCount	eq, ne, lt, lte, gt, gte

For example, filter for threads from selected agents with at least three traces:

const result = await mastraClient.queryTraceThreads({
  timeRange: {
    traces: {
      from: '2026-10-01T00:00:00.000Z',
      to: '2026-10-02T00:00:00.000Z',
    },
  },
  where: {
    op: 'and',
    args: [
      { op: 'in', value: { path: 'entityId' }, set: ['support-agent', 'sales-agent'] },
      { op: 'gte', left: { path: 'traceCount' }, right: { literal: 3 } },
    ],
  },
})

Presence filters use { op: 'exists', path: 'userId' } to check for a non-null value. Model/provider and previews are available in returned rows but aren't filterable.

Thread response

The response is { threads, page: { next } }, without a total count. Each entry in threads contains these fields:

Returned fields	Meaning
threadId, organizationId	Thread identity. Equal thread IDs in different organizations remain separate.
firstTraceId, lastTraceId	Earliest and latest trace by start time. Equal start times use trace ID as a deterministic tie-breaker.
resourceId, userId, runId, sessionId, requestId, serviceName, experimentId, environment, executionSource	First-trace context, or null when missing.
entityType, entityId, entityName, entityVersionId	First-trace entity.
startedAt, endedAt	Earliest start and maximum end time across the thread.
status	error when the last-started trace's root span has an error, otherwise success.
traceCount	Number of logical traces in the thread.
inputPreview	First trace's input preview.
outputPreview, errorPreview	Last-started trace's previews.
timeToFirstChunkMs	First trace's root agent time to first chunk, in milliseconds. null when unavailable or the root isn't an agent.
model, provider	Model/provider from the first completed model generation directly under the first trace's root agent. null when unavailable.

An earlier-started trace can finish last. In that case, it supplies endedAt, while the last-started trace still supplies status and output/error previews. Missing first-trace values aren't replaced with values from a later trace.

Thread ordering and pagination

Ordering accepts exactly one of startedAt or endedAt, with direction asc or desc. The default is endedAt descending. Pages default to 100 rows and allow at most 1,000. Pass result.page.next as page.after with the same time selector, filters, scope and ordering. A null cursor means no more pages are available. Pagination reflects live data, so new traces can move a thread between pages.

To load the traces within a thread, use queryTraces() with a threadId filter and a time range covering the thread's activity.

Thread query support

Check capabilities.threadQueryRows from getObservabilityCapabilities() before using this contract. A missing or false value means full thread rows aren't available. Older core or storage packages return a structured 501 for these requests. Legacy identity requests remain supported for mixed package versions.

ClickHouse rejects broad thread queries when the authorized history exceeds 250,000 stored trace-root rows, including duplicate copies. A narrower time range can still succeed by selecting fewer candidate threads. Rejected queries return 503 TRACE_QUERY_RESOURCE_LIMIT without partial results.

Querying traces

queryTraces() finds completed logical traces using trace fields and conditions over related spans. Every query requires an ISO timestamp range.

const result = await mastraClient.queryTraces({
  timeRange: {
    from: '2026-08-01T00:00:00.000Z',
    to: '2026-08-08T00:00:00.000Z',
  },
  where: {
    spans: {
      some: {
        op: 'and',
        args: [
          { op: 'eq', left: { path: 'spanType' }, right: { literal: 'tool_call' } },
          { op: 'exists', path: 'error' },
        ],
      },
    },
  },
})

See Query traces for request fields, predicates, limits, response shapes, and errors.

Load a page and poll for traces

queryTraces() supports keyset traversal with page, numbered pages with pagination, and delta polling with mode: 'delta'. Use one mode per request. To migrate from listTracesLight(), read traces instead of spans and retain the numbered page's deltaCursor:

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}
const initial = await mastraClient.queryTraces({
  timeRange,
  pagination: { page: 0, perPage: 100 },
})
if (!initial.deltaCursor) throw new Error('Delta polling is unavailable')

const changes = await mastraClient.queryTraces({
  timeRange,
  mode: 'delta',
  after: initial.deltaCursor,
  limit: 100,
})

Merge changes.traces by traceId, retain changes.deltaCursor, and continue while changes.delta.hasMore is true. Keep the same time range and predicate across polls. Only completed traces are returned; polling doesn't guarantee notifications for related-record updates, deletions, or traces that stop matching.

See Delta polling for bootstrap behavior, cursor lifetime, retention, and a complete polling loop.

Aggregate traces

aggregateTraces() groups the traces selected by a time range and optional where predicate, and returns one row per group. It accepts the same predicates as queryTraces().

const { capabilities } = await mastraClient.getObservabilityCapabilities()

if (capabilities.traceAggregate) {
  const { rows, truncated } = await mastraClient.aggregateTraces({
    timeRange: {
      from: '2026-08-01T00:00:00.000Z',
      to: '2026-08-08T00:00:00.000Z',
    },
    where: { op: 'eq', left: { path: 'environment' }, right: { literal: 'production' } },
    groupBy: ['entityName'],
    interval: '1d',
    measures: ['count', 'errorRate', 'duration.p95', 'cost.sum'],
    orderBy: { field: 'count', direction: 'desc' },
    limit: 10,
  })
}

Each row has dimensions keyed by groupBy field, measures keyed by measure name, a bucket timestamp when interval is set, and cost coverage when a cost measure is requested. truncated is true when more groups matched than limit. Check capabilities.traceAggregate before calling. The client retries server errors like queryTraces() but doesn't retry 422 validation errors.

See Aggregate traces for dimensions, measures, token and cost semantics, time buckets, limits, and errors.

Discover trace-query fields and values

getTraceQueryFields() returns canonical query fields and observed top-level string metadata fields for one predicate scope. The response includes each field's value kind, supported operators, and whether value suggestions are available.

const timeRange = {
  from: '2026-08-01T00:00:00.000Z',
  to: '2026-08-08T00:00:00.000Z',
}

const fields = await mastraClient.getTraceQueryFields({
  timeRange,
  predicateScope: 'spans',
  search: 'model',
  limit: 25,
})

For the trace scope on a server that supports aggregation, the response also includes aggregate: the dimensions you can pass to groupBy and the measures you can request. See Discover fields and values.

Use the exact local path from the response with the same scope. For example, model belongs inside a spans.some or spans.none predicate. A field picker can fetch the trace and spans scopes in parallel.

Call getTraceQueryValues() only after selecting a field with valueSuggestions: true:

const controller = new AbortController()

const values = await mastraClient.getTraceQueryValues(
  {
    timeRange,
    predicateScope: 'spans',
    path: 'model',
    search: 'claude',
    limit: 25,
  },
  { signal: controller.signal },
)

// Cancel this autocomplete request when the search text changes.
controller.abort()

Both methods use case-insensitive literal substring search and accept an empty search. Limits default to 25 and can't exceed 100. Results are ordered by occurrence count, then deterministically by path or value. observedFieldsTruncated and valuesTruncated indicate that the caller should refine search. Discovery has no cursor.

Suggestions are bounded and advisory. Manual values remain valid when suggestions are unavailable, empty, or truncated. Discovery doesn't accept a draft query predicate. The client doesn't retry either request, and a per-call AbortSignal takes precedence over the signal configured on MastraClient.

A discovery timeout rejects with 504 TRACE_QUERY_EXECUTION_TIMEOUT. Backend memory or resource exhaustion rejects with 503 TRACE_QUERY_RESOURCE_LIMIT. Neither error contains partial suggestions; truncation flags are only present on successful, completely ranked responses.

Deleting traces

Delete traces and their associated spans, metrics, logs, scores, and feedback:

const result = await mastraClient.deleteTraces({
  traceIds: ['trace-1', 'trace-2'],
})

console.log(result.success)

Each request accepts up to 1,000 trace IDs and no tenant scope fields. Signals that aren't linked to a trace are preserved. Traces created by experiments are deleted like any other trace.

Scoring traces

Score specific traces using registered scorers for evaluation:

const result = await mastraClient.score({
  scorerName: 'answer-relevancy',
  targets: [
    { traceId: 'trace-1', spanId: 'span-1' }, // Score specific span
    { traceId: 'trace-2' }, // Score specific span which defaults to the parent span
  ],
})

Getting scores by span

Retrieve scores for a specific span within a trace:

const scores = await mastraClient.listScoresBySpan({
  traceId: 'trace-123',
  spanId: 'span-456',
  page: 1,
  perPage: 20,
})

Feedback

Feedback methods create, list, and query human-in-the-loop signals such as ratings, thumbs, comments, and corrections through the target Mastra runtime and its configured observability storage. They don't call the hosted Mastra Platform Feedback API. See the feedback guide for examples and the feedback reference for full schemas.

Creating feedback

Create a feedback record linked to a trace or span:

const response = await mastraClient.createFeedback({
  feedback: {
    traceId: 'trace-123',
    spanId: 'span-456',
    feedbackSource: 'user',
    feedbackType: 'rating',
    value: 1,
    comment: 'Helpful answer.',
  },
})

Deleting feedback and scores

Delete feedback or score records by id. Deletion is idempotent, so missing ids are ignored. Each request accepts at most 1,000 feedbackIds or 1,000 scoreIds. Optional organizationId and resourceId fields restrict each deletion to records with matching scope fields:

await mastraClient.deleteFeedback({
  feedbackIds: ['feedback-1', 'feedback-2'],
  organizationId: 'org-1',
  resourceId: 'resource-1',
})

await mastraClient.deleteScores({
  scoreIds: ['score-1', 'score-2'],
  organizationId: 'org-1',
  resourceId: 'resource-1',
})

Listing feedback

Retrieve paginated feedback records with optional filters:

const feedback = await mastraClient.listFeedback({
  filters: {
    feedbackType: 'rating',
    feedbackSource: 'studio',
  },
  pagination: { page: 0, perPage: 20 },
  orderBy: { field: 'timestamp', direction: 'DESC' },
})

filters accepts every FeedbackFilter field. The client sends them as query parameters on GET /api/observability/feedback. See list query parameters.

Aggregating feedback

Aggregate numeric feedback values, such as ratings or thumbs encoded as 1 and -1:

const averageRating = await mastraClient.getFeedbackAggregate({
  feedbackType: 'rating',
  aggregation: 'avg',
  comparePeriod: 'previous_day',
})

Grouping feedback

Group numeric feedback by observability dimensions:

const ratingsByAgent = await mastraClient.getFeedbackBreakdown({
  feedbackType: 'rating',
  groupBy: ['entityName'],
  aggregation: 'avg',
})

Querying feedback over time

Bucket numeric feedback by interval:

const ratingsOverTime = await mastraClient.getFeedbackTimeSeries({
  feedbackType: 'rating',
  interval: '1h',
  aggregation: 'avg',
  groupBy: ['feedbackSource'],
})

Querying feedback percentiles

Return percentile series for numeric feedback values:

const ratingPercentiles = await mastraClient.getFeedbackPercentiles({
  feedbackType: 'rating',
  percentiles: [0.5, 0.95],
  interval: '1d',
})

Related

• Advanced trace queries: Query and aggregate completed traces with recursive predicates
• Feedback guide: Learn how to create, list, and query feedback
• Feedback reference: Review feedback schemas and HTTP routes
• Agents API: Learn about agent interactions that generate traces
• Workflows API: Understand workflow execution monitoring