Speculative Decoding: Accelerating LLM Inference with Draft Models

Large Language Model (LLM) generation is slow because it is autoregressive: the model generates tokens one by one, requiring a full forward pass through the entire network for each token.

• For a 70B parameter model, this requires reading 140 GB of weights from memory for every single token generated.
• This makes generation memory-bound rather than compute-bound.

Speculative Decoding resolves this memory bottleneck by generating candidate sequences using a smaller, faster model (the "Draft Model") and then validating them in parallel with the larger model (the "Target Model").

Speculative Decoding Inference Pipeline

The Mechanics of Speculative Decoding

The speculative decoding workflow involves:

1. Draft Generation: The Draft Model (e.g. Llama-3-8B) generates a sequence of K candidate tokens:

T_{cand} = \{t_1, t_2, \dots, t_K\}

2. Parallel Verification: The Target Model (e.g. Llama-3-70B) runs a single forward pass over the concatenated sequence. Because the target model processes all K tokens simultaneously in a batch, this pass takes almost the same time as generating a single token.
3. Acceptance Check: The target model evaluates the probability of each candidate token. If a token is accepted, we keep it. If a token is rejected, we discard it and all subsequent tokens in the draft, and the target model generates the next correct token.

Mathematical Acceptance Criteria

To ensure the output distribution remains identical to the target model, we use the following acceptance probability for candidate token x:

P_{\text{accept}}(x) = \min\left(1, \frac{P_{\text{target}}(x | \text{context})}{P_{\text{draft}}(x | \text{context})}\right)

If the token is rejected, we sample a new token from the difference distribution:

P_{\text{diff}}(x) = \max\left(0, P_{\text{target}}(x | \text{context}) - P_{\text{draft}}(x | \text{context})\right)

This ensures that speculative decoding accelerates inference without degrading output quality.

Code Sample: Configuring Speculative Decoding in vLLM

Using frameworks like vLLM, speculative decoding can be enabled with a simple configuration flag:

# Serving with speculative decoding using vLLM
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-70B-Instruct",
    speculative_model="meta-llama/Meta-Llama-3-8B-Instruct",
    num_speculative_tokens=5,   # K: draft tokens proposed per verification pass
    tensor_parallel_size=4,
    gpu_memory_utilization=0.90,
)

sampling_params = SamplingParams(temperature=0.0, max_tokens=256)

prompts = [
    "Explain why autoregressive decoding is memory-bound.",
    "Write a Python function that reverses a linked list.",
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"--- Prompt: {output.prompt}")
    print(output.outputs[0].text)

The num_speculative_tokens value is the K from the equations above. Raising it increases the potential speedup per verification pass but also the amount of work thrown away whenever an early token is rejected.

Performance Benefits and Tradeoffs

• Speedup: Speculative decoding typically yields a 2x to 3x increase in generation speed, depending on the alignment between the draft and target models.
• Draft Alignment: If the draft model's predictions diverge significantly from the target model, the acceptance rate drops, reducing the speed benefit.
• Memory Cost: Both models must be loaded into memory, which requires additional VRAM.

The Case Against Turning It On

Speculative decoding optimises the wrong axis for a lot of real deployments, and it's worth knowing whether yours is one of them before you spend a week on it.

The technique converts spare compute into reduced latency. That trade only exists when you have spare compute — which is to say, at low batch size. A server handling one request at a time is badly memory-bound and speculation is close to free. A server running continuous batching at batch 64 is already compute-saturated: the verification pass over K draft tokens now competes with real work from other requests, and speculation can reduce your aggregate throughput even while each individual request finishes sooner. If your bottleneck is tokens-per-second across a busy fleet rather than time-to-last-token for a single user, this may cost you money rather than save it.

Acceptance rate is the number that decides everything else, and it is workload-dependent in ways that are hard to predict from the model pair alone. Boilerplate code, structured output, and formulaic prose accept at high rates because the draft model finds them just as obvious as the target does. Open-ended creative generation at high temperature accepts poorly — the distributions genuinely diverge, which is exactly when the rejection sampler earns its keep and exactly when you get no speedup. Measure acceptance on your own traffic; if it lands below roughly 60%, the wasted draft passes eat the benefit and a larger K makes it worse, not better.

The most common tuning mistake is treating K as a dial to turn up. Rejection discards every token after the first failure, so expected accepted length grows sub-linearly in K while draft cost grows linearly. Most pairs peak somewhere around K = 3 to 5. Start there, and be willing to conclude the answer is to leave it off.