Lightlines
By ai-evals-course

write-judge-prompt

ai-evals-course959 installs

Judge prompts for failures code cannot catch.

What it does

Creates a binary LLM judge prompt that evaluates one specific failure mode.

When to use it

Use when a failure requires interpretation and code-based checks cannot evaluate it.

How to use it

Give it an identified failure mode and human-labeled traces with at least 20 Pass and 20 Fail examples; it returns a binary judge prompt.

What you provide

  • the failure mode
  • Your files

Latest review

What you need · 0

No setup requirements listed.


Access · 0

No additional actions listed.