gptq
orchestra-research889 installs
Fit large language models into less GPU memory.
What it does
Guides agents in compressing language models to four-bit weights and running them with less GPU memory.
When to use it
Use when large language models need to fit limited GPU memory or run faster than FP16 inference.
How to use it
It changes how the agent implements model quantization, choosing calibration data and checking quality before deployment.
What you provide
- An existing project
Uses
Latest review
What you need · 2
The model-loading, quantization, and inference examples run as Python code.
The main model-loading and inference examples use a CUDA GPU.
Access · 3
This skill
HuggingFace
Write
Uploads models to HuggingFace
c4
Read
Streams calibration text from c4