quantizing-models-bitsandbytes
orchestra-research905 installs
Fit larger language models into your GPU memory.
What it does
Guides users to load large language models with reduced GPU memory requirements.
When to use it
Use when GPU memory is limited, for QLoRA fine-tuning or memory-efficient inference with HuggingFace Transformers, and some accuracy degradation is acceptable.
How to use it
Give it a model-loading or fine-tuning task; it produces quantization and training code for reduced memory use.
Latest review
What you need · 3
The primary GPU path specifies NVIDIA compute capability 7.0+; required VRAM depends on the model and quantization.
The NVIDIA path requires CUDA 11.1+ and recommends 12.0+.
Requires PyTorch 2.0+.
Access · 1
This skill
pip install bitsandbytes transformers accelerate
Execute
Installs model quantization packages