agent-observability-eval-bootstrap
datadog-labs514 installs
Turn production AI traces into evaluator drafts.
What it does
Generates evaluator suites from production LLM traces and, after confirmation, saves online judges as disabled Datadog drafts.
When to use it
Use when asked to bootstrap evaluators, build an evaluation suite, or generate judge configurations from production LLM traces.
How to use it
Give it an ml_app name; it samples production traces, asks you to curate the evaluator suite, and saves confirmed online judges as disabled Datadog drafts.
What you provide
- the ml_app name
Uses
Good to know
- cannot be undone: Datadog
Latest review
What you need · 1
Requires authenticated Datadog access through connected MCP tools or an executable pup CLI; pup authentication errors require the user to log in before continuing.
Access · 1
This skill
Datadog
Write
Saves evaluator drafts to Datadog (irreversible)