I write about technology here—LLMs, compilers, cloud infrastructure, performance engineering, and more. For everything else, check out my personal blog.
Subscribe via RSSThis post compares trained drafter architectures for speculative decoding, from separate models to shared heads, feature predictors, and parallel drafters. It explains what each method changes and how it trades draft cost, acceptance length, reuse, and serving complexity.
Speculative decoding speeds up LLM inference by letting a small draft model propose multiple tokens that a larger model verifies in a single forward pass. This post explains where the speedup comes from, why the algorithm is mathematically lossless, and when it helps or hurts in practice.