How Speculative Decoding Works (and When It Doesn't)

Speculative decoding speeds up LLM inference by letting a small draft model propose multiple tokens that a larger model verifies in a single forward pass. This post explains the GPU bottlenecks it exploits, why the algorithm preserves the target model’s output exactly, and when the technique delivers the greatest speedups.