nemo-evaluator-sdk

Benchmarks language models and produces reproducible results for comparing their capabilities across tasks.

When to use it

Use for broad benchmark coverage, reproducible containerized evaluations, or evaluations on Slurm clusters and cloud infrastructure.

Give it a model endpoint and benchmarks to run; it executes evaluations and saves results locally.