turboquant-pytorch
reason-machines814 installs
Smaller LLM memory caches with accurate attention.
What it does
Guides implementation of compressed LLM key-value caches in PyTorch while preserving attention inner products.
When to use it
Use when implementing TurboQuant compression or reducing the memory used by an LLM's key-value cache.
How to use it
Give it a project needing compressed LLM caches; it provides code for compressing keys and values and computing attention.
What you provide
- An existing project
Latest review
What you need · 2
Requires Python to run the implementation's validation commands; no Python version is specified.
The implementation uses PyTorch and lists torch version 2.0 or newer in its requirements.
Access · 1
This skill
pip install -r requirements.txt
Execute
Installs quantization dependencies