tensorrt-llm
orchestra-research980 installs
Faster language model serving on NVIDIA GPUs.
What it does
Guides the deployment and tuning of language model inference on NVIDIA GPUs with TensorRT-LLM.
When to use it
Use for NVIDIA GPU deployments requiring high throughput, low latency, quantized models, or multi-GPU scaling.
How to use it
It changes how the agent configures, tunes, and deploys language model inference in the project.
What you provide
- An existing project
Uses
TL
Latest review
What you need · 7
Requires NVIDIA GPUs; the guide names A100, H100, and GB200 deployments.
The installation instructions require Python 3.10-3.12.
Access · 4
This skill
trtllm-serve
Execute
Starts a local inference server
Write
Sends prompts to local inference
meta-llama/Meta-Llama-3-8B
Read
Downloads language model weights