nemo-curator
orchestra-research890 installs
Clean training data with GPU-powered filters.
What it does
Prepares cleaner training datasets by filtering content, removing duplicates, and redacting personal information with GPU acceleration.
When to use it
Use when preparing training data from web scrapes, deduplicating large datasets, filtering unwanted content, or curating multiple media types.
How to use it
Give it training data files; it saves a curated dataset after filtering, deduplication, and personal-information redaction.
What you provide
- Your files
Latest review
What you need · 7
Requires the NeMo Curator package; the GPU text-curation setup installs its text_cuda12 extra.
Lists cudf as a dependency.
Lists dask as a dependency.
Access · 1
This skill
nemo-curator
Execute
Installs NeMo Curator