agent-observability-eval-pipeline
Turn Datadog traces into tested LLM evaluations.
What it does
Guides users from production LLM traces to failure diagnoses, evaluators, published datasets, and analyzed experiments.
When to use it
Use when an instrumented LLM application already has production traces and the user wants a guided path to evaluations and experiments.
How to use it
Give it an instrumented application's name and project; it guides you through evaluations, published datasets, and experiments, pausing between phases.
What you provide
- the ml_app name
- An existing project
Uses
Good to know
- cannot be undone: the app's LLM provider
What you need · 9
Requires an already-instrumented LLM application with traces flowing into Datadog; it does not set up instrumentation.
Requires a Datadog account with API and application credentials for accessing traces and publishing datasets and experiments.
Access · 4
This skill
the app's LLM provider
Spends provider tokens on experiments (irreversible)
Datadog
Publishes datasets and experiment results