awq-quantization

Guides the creation and use of four-bit language models that require less memory for inference.

When to use it

Use when deploying large models on limited GPU memory, seeking faster inference, or working with instruction-tuned and multimodal models.

It changes how the agent implements language-model quantization and inference, providing configurations and examples for loading, compressing, and saving models.