Datadog

Datadog is a complete monitoring platform with dedicated LLM Observability features. Mastra supports Datadog through an exporter for sending completed traces and a bridge for integrating with dd-trace context in real time.

Exporter

The Datadog exporter sends your traces to Datadog's LLM Observability product, providing insights into model performance and token usage, plus conversation flows.

Installation

npm

npm install @mastra/datadog@latest

pnpm

pnpm add @mastra/datadog@latest

Yarn

yarn add @mastra/datadog@latest

Bun

bun add @mastra/datadog@latest

Configuration

Prerequisites

1. Datadog Account: Sign up at datadoghq.com with LLM Observability enabled
2. API Key: Get your API key from Datadog Organization Settings → API Keys
3. Environment Variables: Set your credentials

.env

DD_API_KEY=your-datadog-api-key
DD_LLMOBS_ML_APP=my-llm-app
DD_SITE=datadoghq.com  # Optional: defaults to datadoghq.com
DD_ENV=production      # Optional: environment name

Zero-Config Setup

With environment variables set, use the exporter with no configuration:

src/mastra/index.ts

import { Mastra } from '@mastra/core'
import { Observability } from '@mastra/observability'
import { DatadogExporter } from '@mastra/datadog'

export const mastra = new Mastra({
  observability: new Observability({
    configs: {
      datadog: {
        serviceName: 'my-service',
        exporters: [new DatadogExporter()],
      },
    },
  }),
})

Explicit Configuration

You can also pass credentials directly (takes precedence over environment variables):

src/mastra/index.ts

import { Mastra } from '@mastra/core'
import { Observability } from '@mastra/observability'
import { DatadogExporter } from '@mastra/datadog'

export const mastra = new Mastra({
  observability: new Observability({
    configs: {
      datadog: {
        serviceName: 'my-service',
        exporters: [
          new DatadogExporter({
            mlApp: process.env.DD_LLMOBS_ML_APP!,
            apiKey: process.env.DD_API_KEY!,
          }),
        ],
      },
    },
  }),
})

Configuration options

Complete Configuration

new DatadogExporter({
  // Required settings
  mlApp: process.env.DD_LLMOBS_ML_APP!, // Groups traces under this ML app name
  apiKey: process.env.DD_API_KEY!, // Required for agentless mode (default)

  // Optional settings
  site: 'datadoghq.com', // Datadog site (datadoghq.eu, us3.datadoghq.com, etc.)
  service: 'my-service', // Service name (defaults to mlApp)
  env: 'production', // Environment name
  agentless: true, // true = direct HTTPS, false = local Datadog Agent

  // Advanced settings
  integrationsEnabled: false, // Enable dd-trace auto-instrumentation

  // Diagnostic logging
  logLevel: 'info', // debug | info | warn | error
})

With Local Datadog Agent

If you have a Datadog Agent running locally, you can route traces through it instead of direct HTTPS:

new DatadogExporter({
  mlApp: process.env.DD_LLMOBS_ML_APP!,
  agentless: false, // Use local Datadog Agent
  env: 'production',
})

Note: When using agent mode, the API key is read from the local agent's configuration.

Span type mapping

Mastra span types are automatically mapped to Datadog LLM Observability span kinds:

Mastra SpanType	Datadog Kind
AGENT_RUN	agent
MODEL_GENERATION	workflow
MODEL_STEP	llm
TOOL_CALL	tool
MCP_TOOL_CALL	tool
WORKFLOW_RUN	workflow
Other workflow types	task
GENERIC	task

Other/future Mastra span types will default to 'task' when mapped unless specified.

Application Performance Monitoring

The sections above cover Mastra's LLM Observability integration. To trace your Mastra HTTP server routes (request latency, error tracking, service maps), use dd-trace directly for Datadog Application Performance Monitoring (APM).

APM prerequisites

1. Datadog Agent: Install a Datadog Agent on the same host or accessible via network. The agent receives traces from dd-trace on localhost:8126 and forwards them to Datadog. Follow the agent installation guide to set it up.

2. dd-trace package: Install the tracing library in your project:

   npm

   npm install dd-trace

   pnpm

   pnpm add dd-trace

   Yarn

   yarn add dd-trace

   Bun

   bun add dd-trace

备注

APM traces always route through the Datadog Agent. This is different from LLM Observability, which supports agentless mode (direct HTTPS to Datadog).

APM only

Import and initialize dd-trace at the top of your entry file, before any other imports:

src/mastra/index.ts

import tracer from 'dd-trace'

tracer.init({
  service: process.env.DD_SERVICE || 'my-mastra-app',
  env: process.env.DD_ENV || 'production',
  version: process.env.DD_VERSION,
})

import { Mastra } from '@mastra/core'

export const mastra = new Mastra({
  bundler: {
    externals: [
      'dd-trace',
      '@datadog/native-metrics',
      '@datadog/native-appsec',
      '@datadog/native-iast-taint-tracking',
      '@datadog/pprof',
    ],
  },
})

Set the tracer metadata environment variables:

.env

DD_SERVICE=my-mastra-app
DD_ENV=production
DD_VERSION=1.0.0

dd-trace auto-instruments popular HTTP frameworks, including those supported by Mastra's server adapters. Inbound requests, outbound HTTP calls, and database queries appear as APM traces in Datadog.

APM and LLM Observability

Import and initialize dd-trace before creating the Mastra instance. The DatadogExporter detects the existing tracer and skips re-initialization, adding LLM Observability on top of your APM setup:

src/mastra/index.ts

import tracer from 'dd-trace'

tracer.init({
  service: process.env.DD_SERVICE || 'my-mastra-app',
  env: process.env.DD_ENV || 'production',
  version: process.env.DD_VERSION,
})

import { Mastra } from '@mastra/core'
import { Observability } from '@mastra/observability'
import { DatadogExporter } from '@mastra/datadog'

export const mastra = new Mastra({
  observability: new Observability({
    configs: {
      datadog: {
        serviceName: 'my-mastra-app',
        exporters: [
          new DatadogExporter({
            mlApp: process.env.DD_LLMOBS_ML_APP!,
            apiKey: process.env.DD_API_KEY!,
          }),
        ],
      },
    },
  }),
  bundler: {
    externals: [
      'dd-trace',
      '@datadog/native-metrics',
      '@datadog/native-appsec',
      '@datadog/native-iast-taint-tracking',
      '@datadog/pprof',
    ],
  },
})

.env

DD_SERVICE=my-mastra-app
DD_ENV=production
DD_VERSION=1.0.0
DD_API_KEY=your-datadog-api-key
DD_LLMOBS_ML_APP=my-llm-app

Server routes appear as APM traces and LLM calls appear as LLM Observability spans, all under the same service in Datadog.

备注

Import and initialize dd-trace before all other modules. This allows its auto-instrumentation to patch HTTP, database, and framework libraries at load time.

Troubleshooting

Native module ABI mismatch

If you see errors like:

Error: No native build was found for runtime=node abi=137 platform=linuxglibc arch=x64

This indicates a Node.js version compatibility issue with dd-trace's native modules. These native modules are optional and provide performance monitoring features.

Solutions:

1. Use Node.js 22.x: Native modules have the best compatibility with Node.js 22.x.

2. Ignore native module warnings: The native modules (@datadog/native-metrics, @datadog/native-appsec, etc.) are optional. If they fail to load, core tracing functionality still works.

Bundler externals configuration

When using bundlers like esbuild, webpack, or the Mastra CLI bundler, you may need to mark dd-trace and its dependencies as external:

src/mastra/index.ts

export const mastra = new Mastra({
  bundler: {
    externals: [
      'dd-trace',
      '@datadog/native-metrics',
      '@datadog/native-appsec',
      '@datadog/native-iast-taint-tracking',
      '@datadog/pprof',
    ],
  },
})

Related

• Tracing overview
• Datadog LLM Observability documentation
• Datadog APM documentation

Bridge

注意

The Datadog Bridge is currently experimental. APIs and configuration options may change in future releases.

The Datadog Bridge enables bidirectional integration between Mastra's tracing system and Datadog. Unlike exporters that send trace data after execution completes, the bridge creates native dd-trace spans in real time so that auto-instrumented APM operations (HTTP calls, database queries, etc.) inside your tools and processors are correctly nested under their parent Mastra spans.

When to use the bridge

Use the DatadogBridge when you:

• Use dd-trace auto-instrumentation in your application (HTTP servers, database clients, etc.)
• Want APM service calls made by tools, MCP tools, or output processors to appear under their parent Mastra span instead of the request handler
• Need both APM traces and LLM Observability data to share a consistent trace topology
• Are building a distributed system where Datadog trace context must propagate across services

How it works

The DatadogBridge participates in two parts of the dd-trace pipeline:

APM context propagation (real time):

• Creates a dd-trace APM span via tracer.startSpan() when each Mastra span is created
• Activates the APM span in dd-trace's scope via tracer.scope().activate() during execution
• Auto-instrumented operations inside the active scope are parented to the correct Mastra span
• Inherits the active dd-trace context (e.g., an incoming request span) when no explicit Mastra parent exists

LLM Observability emission (on span end):

• Emits annotations (model info, token usage, input/output, errors) through dd-trace's LLM Observability pipeline
• Maintains parent-child relationships in Datadog LLM Observability using nested llmobs.trace() calls
• Reuses the same data shape and span-kind mapping as the Datadog Exporter

Trace and log correlation

Without the bridge, the Datadog Exporter only creates LLM Observability spans after a trace completes. During execution, no dd-trace span is active in scope, so any HTTP or database call made by a tool falls back to whatever dd-trace span is active at the time, typically the incoming request handler. The result is that service calls from MCP tools or output processors appear as children of the request span instead of the agent or processor span that actually made them.

The bridge fixes this by creating real dd-trace spans up front, so the scope is correct when auto-instrumentation runs.

Installation

npm

npm install @mastra/datadog dd-trace

pnpm

pnpm add @mastra/datadog dd-trace

Yarn

yarn add @mastra/datadog dd-trace

Bun

bun add @mastra/datadog dd-trace

The bridge requires dd-trace to be installed and a local Datadog Agent (or compatible OTLP receiver) to receive APM data. See the APM prerequisites for agent setup details.

Configuration

Using the DatadogBridge requires two steps:

1. Initialize dd-trace so its auto-instrumentation patches HTTP, database, and framework libraries
2. Add the DatadogBridge to your Mastra observability config

Step 1: Initialize dd-trace

dd-trace must be initialized before any other imports so its auto-instrumentation can patch libraries at load time. The bridge will detect an already-initialized tracer and reuse it.

src/mastra/index.ts

import tracer from 'dd-trace'

tracer.init({
  service: process.env.DD_SERVICE || 'my-mastra-app',
  env: process.env.DD_ENV || 'production',
  version: process.env.DD_VERSION,
})

import { Mastra } from '@mastra/core'
import { Observability } from '@mastra/observability'
import { DatadogBridge } from '@mastra/datadog'

// ...

备注

Import and initialize dd-trace at the very top of your application's entry file, before any other imports.

Step 2: Mastra Configuration

Add the DatadogBridge to your Mastra observability config:

src/mastra/index.ts

export const mastra = new Mastra({
  observability: new Observability({
    configs: {
      default: {
        serviceName: 'my-mastra-app',
        bridge: new DatadogBridge({
          mlApp: process.env.DD_LLMOBS_ML_APP!,
        }),
      },
    },
  }),
  bundler: {
    externals: [
      'dd-trace',
      '@datadog/native-metrics',
      '@datadog/native-appsec',
      '@datadog/native-iast-taint-tracking',
      '@datadog/pprof',
    ],
  },
})

.env

DD_SERVICE=my-mastra-app
DD_ENV=production
DD_VERSION=1.0.0
DD_LLMOBS_ML_APP=my-llm-app

When dd-trace is initialized, it routes APM data to your local Datadog Agent on localhost:8126. The bridge enables LLM Observability on top of the same tracer, so both sets of data appear under the same service in Datadog.

No Mastra exporters are required when using the bridge. Both APM and LLM Observability data flow through dd-trace. You can still add Mastra exporters if you want to send traces to additional destinations.

Agent vs. agentless mode

The bridge defaults to agent mode (agentless: false). This assumes a local Datadog Agent is running on localhost:8126 to receive both APM and LLM Observability data. This is the typical setup when using dd-trace auto-instrumentation, since APM data always routes through the agent.

If you don't have a local Datadog Agent and only need LLM Observability data (no APM auto-instrumentation), you can enable agentless mode to send data directly to Datadog. In this case, you must provide an API key.

new DatadogBridge({
  mlApp: process.env.DD_LLMOBS_ML_APP!,
  apiKey: process.env.DD_API_KEY!,
  agentless: true,
})

备注

For most bridge users, agent mode is the right choice. APM data can't be sent in agentless mode, so enabling agentless splits LLM Observability traffic away from APM traffic. If you want LLM Observability only without an agent, use the Datadog Exporter instead.

Trace hierarchy

With the DatadogBridge, your traces maintain proper hierarchy across dd-trace and Mastra boundaries. Service calls made by tools and processors appear under the correct Mastra span:

HTTP POST /api/chat (from web framework instrumentation)
└── agent.orchestrator (from Mastra via DatadogBridge)
    ├── chat gpt-5.4 (LLM call)
    ├── tool.execute search (tool execution)
    │   └── HTTP GET api.example.com (auto-instrumented from inside the tool)
    └── processor.guardrail (output processor)
        └── HTTP POST guardrail-service/check (auto-instrumented from inside the processor)

In Datadog, the APM trace shows this full topology, and the LLM Observability product shows the agent and LLM-specific spans with their inputs, outputs, and token metrics.

Span type mapping

The bridge uses the same span-kind mapping as the Datadog Exporter for LLM Observability. See span type mapping in the exporter section.

Using tags

Tags help you categorize and filter traces in Datadog. Add tags when executing agents or workflows:

const result = await agent.generate('Hello', {
  tracingOptions: {
    tags: ['production', 'experiment-v2', 'user-request'],
  },
})

Tags formatted as key:value (e.g., instance_name:career-scout-api) are split into structured tag entries. Tags without a colon are set with a true value.

Promoting context keys to flat tags

Use requestContextKeys to promote specific keys from the request context or span attributes into flat, indexable LLM Observability tags. This makes them filterable in the Datadog UI:

new DatadogBridge({
  mlApp: process.env.DD_LLMOBS_ML_APP!,
  requestContextKeys: ['tenantId', 'agentId'],
})

Promoted keys are removed from annotations.metadata and added as flat tags on each LLM Observability span.

Troubleshooting

If APM spans aren't connecting to Mastra spans as expected:

• Verify dd-trace is initialized before any other imports (it patches libraries at load time)
• Verify a local Datadog Agent is running and reachable at localhost:8126
• Ensure the DatadogBridge is set as bridge (not as an entry in exporters) in your observability config
• Confirm you haven't also added the DatadogExporter to exporters: using both will double-emit LLM Observability data

For native-module compatibility issues with dd-trace and bundler externals, see the Datadog exporter troubleshooting section.

Related

• Tracing overview
• DatadogBridge reference
• Datadog APM documentation
• Datadog LLM Observability documentation