Faster token generation on your Jetson server.
What it does
Tunes an existing Jetson vLLM server for faster token generation and checks whether speculative decoding improves its speed.
When to use it
Use when per-token latency is the bottleneck on a Thor or AGX Orin server serving one or at most two concurrent streams.
How to use it
Give it a request to tune an existing Jetson vLLM deployment; it returns serving commands and configuration instructions for speculative decoding.
Uses
What you need · 4
Requires Thor or AGX Orin hardware with enough memory for the target model and an additional draft model or EAGLE-3 head.
Requires a working server recipe. Keep its existing runtime path: upstream or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+, or a JetPack-matched NVIDIA-AI-IOT vLLM image on older Orin.
Requires a working serving recipe from this skill and appends speculative configuration to its serving command.
Requires a baseline from this skill and uses it before and after enabling speculation to verify the change.
Access · 0
No additional actions listed.