Speculative Speculative Decoding hits 5x speedups
Speculative Speculative Decoding is a new inference method from Tanishq Kumar, Tri Dao, and Avner May that overlaps drafting and verification on separate hardware, letting the draft model precompute likely verification branches instead of waiting for each verifier pass to finish. The accompanying Saguaro implementation and open-source `ssd` engine report up to 2x faster decoding than optimized speculative decoding and up to 5x faster than autoregressive baselines on Llama and Qwen model setups.
This is the kind of inference paper developers should pay attention to: it does not change the model, it changes the serving loop, and that is often where real-world latency wins come from.
- –The core idea is clever: while the target model verifies one speculation, the draft model predicts likely verification outcomes and prepares the next branches in advance
- –Unlike many speedup tricks, SSD is presented as exact rather than approximate, so the pitch is lower latency without changing the sampled distribution
- –The paper’s strongest practical signal is the released GitHub implementation, which bundles SSD with optimized speculative decoding, autoregressive baselines, and support for Llama 3 and Qwen 3 families
- –The catch is systems complexity: the reported wins rely on separate hardware for the draft model, custom cache logic, NCCL communication, and H100-class GPU setups
- –If these results hold up beyond the authors’ engine, SSD could become a meaningful new layer in open-source inference stacks rather than just a one-off research curiosity
DISCOVERED
37d ago
2026-03-06
PUBLISHED
37d ago
2026-03-06
RELEVANCE
AUTHOR
callmeteji