validate-evaluator
Check your AI judge against human ratings.
What it does
Calibrates an LLM judge against human labels to measure how reliably it distinguishes passing and failing examples.
When to use it
Use after writing an LLM judge prompt to verify alignment before trusting its outputs, not for deterministic code-based evaluators.
How to use it
Give it a judge prompt and human-labeled traces; it calibrates the judge and returns held-out measurements of passing and failing accuracy.
What you provide
- the judge prompt
- Your files
What you need · 3
The supplied data-splitting and measurement implementations use Python; no runtime version is specified.
The supplied implementation imports train_test_split for data splitting and confusion_matrix for accuracy measurements.
The supplied bootstrap confidence-interval implementation imports numpy for resampling and numerical calculations.
Access · 0
No additional actions listed.