Datadog LLM Observability: Monitor & Trace AI in Production

About this course

Are your LLM applications running blind in production?You've deployed an AI agent, a RAG pipeline, or an LLM-powered chatbot.  But can you answer these questions?How much did that runaway agent loop cost before someone noticed?Why did hallucination rates spike last Tuesday?Which step in your RAG pipeline is returning irrelevant documents? How do you prove to compliance that you're protecting customer PII in LLM conversations?  If you can't answer these questions with data, you have a production problem.Traditional APM tools see your LLM as a black box.  They measure latency and error rates, but they can't show you token flows, prompt effectiveness, or quality degradation. LLMs are fundamentally different—non-deterministic, multi-step, token-priced, and quality-sensitive.  You need LLM-native observability.Introducing Datadog LLM ObservabilityThis course is the definitive guide to Datadog's LLM Observability platform for enterprise teams.  If you're already using Datadog for APM, infrastructure, or security, this integrates directly into your existing stack—no new tools to learn, no separate dashboards to monitor.  What you'll build:  Throughout this course, you'll instrument a production-grade Customer Support AI Agent with:Multi-turn conversation tracingTool integration (order lookup, refund processing) Custom quality evaluations  Cost monitoring dashboard   PII scrubbing compliance  This isn't a toy example—it's the architecture real enterprise teams deploy.

Course objectives

• understand LLM-specific observability needs
• utilize Datadog for monitoring AI applications
• improve quality evaluation of AI outputs