create-task
Build a Harbor benchmark task you can verify.
What it does
Guides users through creating a Harbor evaluation task with instructions, a container environment, a verifier, and a reference solution.
When to use it
Use when the user wants to scaffold, build, or design a new evaluation task or benchmark problem.
How to use it
Give it a request for a new evaluation task; it walks through design decisions with you and creates the task files.
Uses
What you need · 4
Requires the Harbor CLI to scaffold the task and run Oracle verification.
Requires Docker for the instructed interactive task-environment check.
Invokes rewardkit to design grading criteria when the user chooses the Reward Kit verifier path; it is not required for the other verifier paths.
For Reward Kit judges needing this API key, forwards ANTHROPIC_API_KEY from the environment into the verifier environment.
Access · 0
No additional actions listed.