eval-audit
Find weak spots in your LLM evaluation pipeline.
What it does
Audits an LLM evaluation pipeline to identify its most consequential weaknesses and recommend concrete fixes.
When to use it
Use when inheriting an evaluation system, questioning whether evaluations are trustworthy, or starting without evaluation infrastructure; not to build an evaluator from scratch.
How to use it
Give it available evaluation artifacts; it returns an impact-ranked findings report with concrete fixes, or directs the starting work when no artifacts exist.
What you provide
- Your files
What you need · 2
When no evaluation infrastructure exists, directs error analysis through error-discovery on real or generated traces; installation requirements are unstated.
When no production data exists, directs generate-synthetic-data to create test inputs before running the pipeline and analyzing its traces; installation requirements are unstated.
Access · 1
This skill
observability MCP server
Reads your evaluation service