Check your AI judge against human-labeled examples.
What it does
Calibrates an LLM judge against human labels to produce held-out accuracy measurements and bias-corrected success estimates.
When to use it
Use after writing an LLM judge prompt to verify alignment before trusting its outputs, not for deterministic code-based evaluators.
How to use it
Give it a judge prompt and human-labeled examples; it returns validation measurements and, when needed, a corrected success estimate.
What you provide
- Your files
What you need · 3
The supplied data-splitting, metric, and bootstrap code uses Python; no version is specified.
The supplied code imports sklearn for stratified data splitting and confusion-matrix measurement.
The supplied bootstrap confidence-interval implementation imports numpy.
Access · 0
No additional actions listed.