Speculative Decoding: Accelerating LLM Inference with Draft Models

Large Language Model (LLM) generation is slow because it is autoregressive: the model generates tokens one by one, requiring a full forward pass through the entire network for each token.

• For a 70B parameter model, this requires reading 140 GB of weights from memory for every single token generated.
• This makes generation memory-bound rather than compute-bound.

Speculative Decoding resolves this memory bottleneck by generating candidate sequences using a smaller, faster model (the "Draft Model") and then validating them in parallel with the larger model (the "Target Model").

Speculative Decoding Inference Pipeline

The Mechanics of Speculative Decoding

The speculative decoding workflow involves:

1. Draft Generation: The Draft Model (e.g. Llama-3-8B) generates a sequence of K candidate tokens:

T_{cand} = \{t_1, t_2, \dots, t_K\}

2. Parallel Verification: The Target Model (e.g. Llama-3-70B) runs a single forward pass over the concatenated sequence. Because the target model processes all K tokens simultaneously in a batch, this pass takes almost the same time as generating a single token.
3. Acceptance Check: The target model evaluates the probability of each candidate token. If a token is accepted, we keep it. If a token is rejected, we discard it and all subsequent tokens in the draft, and the target model generates the next correct token.

Mathematical Acceptance Criteria

To ensure the output distribution remains identical to the target model, we use the following acceptance probability for candidate token x:

P_{\text{accept}}(x) = \min\left(1, \frac{P_{\text{target}}(x | \text{context})}{P_{\text{draft}}(x | \text{context})}\right)

If the token is rejected, we sample a new token from the difference distribution:

P_{\text{diff}}(x) = \max\left(0, P_{\text{target}}(x | \text{context}) - P_{\text{draft}}(x | \text{context})\right)

This ensures that speculative decoding accelerates inference without degrading output quality.

Code Sample: Configuring Speculative Decoding in vLLM

Using frameworks like vLLM, speculative decoding can be enabled with a simple configuration flag:

# Serving with speculative decoding using vLLM
# from vllm import LLM
# 
# llm = LLM(
#     model="meta-llama/Meta-Llama-3-70B-Instruct",
#     speculative_model="meta-llama/Meta-Llama-3-8B-Instruct",
#     num_speculative_tokens=5
# )

Performance Benefits and Tradeoffs

• Speedup: Speculative decoding typically yields a 2x to 3x increase in generation speed, depending on the alignment between the draft and target models.
• Draft Alignment: If the draft model's predictions diverge significantly from the target model, the acceptance rate drops, reducing the speed benefit.
• Memory Cost: Both models must be loaded into memory, which requires additional VRAM.

Conclusion

Speculative decoding is a powerful technique for accelerating LLM inference. By leveraging a fast draft model to pre-generate tokens and using the target model for parallel verification, it bypasses memory constraints to deliver faster generation speeds.