Skip to main content

Datadog Agent Observability

Datadog Agent Observability helps AI engineers, data scientists, and application developers quickly develop, evaluate, and monitor agentic applications. Confidently improve output quality, performance, costs, and overall risk with structured experiments, end-to-end tracing across AI agents, and evaluations.

Datadog Agent Observability fully supports:

  • Cost tracking
  • Prompt tracking and management
  • Online and offline evaluations
  • Experiments

Setup

1. Account setup and API key

Create a Datadog account if you do not have one, and provision an API key.

2. Choose a way to send traces

  • Use OpenTelemetry to send the AI SDK's GenAI spans to Datadog through an OTLP exporter.
  • Use dd-trace, the Datadog Agent Observability SDK to send directly to Datadog Agent Observability.

On top of these setup steps, there are no changes necessary when using the ai SDK.

OpenTelemetry

Datadog accepts spans that follow the OpenTelemetry GenAI semantic conventions. Use the AI SDK's OpenTelemetry integration from @ai-sdk/otel.

Next.jsNode.js

Install the OpenTelemetry packages:

npm install @ai-sdk/otel @opentelemetry/api @vercel/otel

Register the OpenTelemetry provider before the AI SDK OpenTelemetry integration:

import { OpenTelemetry } from '@ai-sdk/otel';
import { registerTelemetry } from 'ai';
import { OTLPHttpProtoTraceExporter, registerOTel } from '@vercel/otel';

export function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const ddSite = process.env.DD_SITE || 'datadoghq.com';

    registerOTel({
      serviceName: 'my-ai-app',
      traceExporter: new OTLPHttpProtoTraceExporter({
        url: `https://otlp.${ddSite}/v1/traces`,
        headers: {
          'dd-api-key': process.env.DD_API_KEY!,
          'dd-otlp-source': 'llmobs',
        },
      }),
    });

    registerTelemetry(new OpenTelemetry());
  }
}

Set DD_SITE to the datacenter configured for your Datadog account.

The serviceName identifies the application in OpenTelemetry. Datadog uses the root span's service to set the ml_app value in Agent Observability.

Datadog Agent Observability SDK (dd-trace)

Note

The tracer does not support Edge runtime routes; use the Node.js runtime for routes that make AI SDK calls.

npm install dd-trace

Next.jsNode.js

Add dd-trace and ai to serverExternalPackages in next.config.ts so they can be properly auto-instrumented.

import type { NextConfig } from 'next';

const nextConfig: NextConfig = {
  serverExternalPackages: ['dd-trace', 'ai'],
};

export default nextConfig;

Set these variables before the Next.js process starts:

DD_LLMOBS_ENABLED=1
DD_LLMOBS_ML_APP=my-ai-app
DD_SITE=<YOUR_DATADOG_SITE>
DD_API_KEY=<YOUR_DATADOG_API_KEY>

Create instrumentation.ts alongside the application's app or pages directory:

export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const initializeImportName = 'dd-trace/initialize.mjs';
    await import(initializeImportName as string);
  }
}

If you can set NODE_OPTIONS in your environment, it is recommended to use it instead of the instrumentation.ts setup:

NODE_OPTIONS="--import dd-trace/initialize.mjs"

Set this option and the other Datadog variables in the same environment before starting the Next.js process. This loads dd-trace early enough to instrument server-side calls.

Troubleshooting

Not seeing traces? Try one of the following options to help debug:

  • If using dd-trace, enable debug logs (DD_TRACE_DEBUG=true) to see detailed tracer execution logs
  • Double-check your API and Datadog site are set correctly
  • Ensure all environment variables are set before the Datadog Agent Observability or OpenTelemetry SDK is initialized

Open a GitHub issue or a support ticket for more assistance.

Resources

Edit on GitHub