Speculative Decoding Explained: How AI Generates Responses Faster

Large language models have become increasingly capable, but generating text with them can still be computationally expensive.

When an AI model generates a response, it typically produces tokens one at a time. For every new token, the large model performs another forward pass before generating the next token.

For long responses, this repeated computation can create significant latency and inference costs.

Speculative decoding is an inference technique designed to make this process faster.

Instead of asking a large model to generate every token itself, speculative decoding uses a small, fast draft model to propose several tokens and a larger target model to verify them efficiently.

The basic idea is:

Let a smaller model guess several tokens, then let the larger model quickly verify those guesses.

When the draft model's predictions are good, multiple tokens can be accepted in a single verification step, potentially reducing the number of expensive target-model computations.

What Is Speculative Decoding?

Speculative decoding is a technique for accelerating autoregressive language model inference.

It uses two models:

Draft Model

A smaller and faster model that generates candidate tokens.

Target Model

The larger, more capable model that verifies those candidate tokens.

A simplified workflow looks like this:

User Prompt
     ↓
Draft Model
     ↓
Candidate Tokens
     ↓
Target Model
     ↓
Verify Candidates
     ↓
Accepted Tokens
     ↓
Continue Generation

The target model remains responsible for the final generation behavior, while the draft model helps reduce the amount of sequential work required from it.

Why Is Text Generation Slow?

To understand speculative decoding, first consider how traditional language-model generation works.

Suppose the model needs to generate:

AI is transforming the world

An autoregressive model might generate:

AI

Then:

AI is

Then:

AI is transforming

Then:

AI is transforming the

Then:

AI is transforming the world

Conceptually:

Prompt
  ↓
Token 1
  ↓
Token 2
  ↓
Token 3
  ↓
Token 4
  ↓
Token 5

Each generation step depends on the previous tokens.

This sequential dependency makes autoregressive generation difficult to fully parallelize.

The Problem With Large Language Models

Large language models can contain billions or even hundreds of billions of parameters.

When generating a response, repeatedly running the large model can require significant computation.

For example:

Large Model
     ↓
Generate Token 1
     ↓
Large Model
     ↓
Generate Token 2
     ↓
Large Model
     ↓
Generate Token 3
     ↓
Large Model
     ↓
Generate Token 4

The model repeatedly performs expensive computation.

Speculative decoding tries to reduce how many times the large target model needs to perform this sequential generation process.

The Core Idea Behind Speculative Decoding

Imagine you have a very fast assistant and a very accurate expert.

Instead of asking the expert to produce every word:

Expert → Word 1
Expert → Word 2
Expert → Word 3
Expert → Word 4

you let the fast assistant propose:

Assistant → Word 1, Word 2, Word 3, Word 4

Then the expert checks them.

Assistant:
"AI is transforming the world"

Expert:
✓ AI
✓ is
✓ transforming
✓ the
✗ world

The accepted tokens can be reused, while the target model handles the point where the draft prediction does not match.

This is the intuition behind speculative decoding.

How Speculative Decoding Works

A simplified process contains several stages.

Step 1: Start With the Prompt

Suppose the user asks:

Explain artificial intelligence in simple words.

The prompt is given to the system.

Step 2: Draft Model Generates Multiple Tokens

The smaller draft model predicts several tokens.

For example:

AI → is → a → technology → that

These are candidate tokens.

The draft model can generate them quickly because it is smaller.

Step 3: Target Model Verifies the Candidates

The candidate sequence is passed to the larger target model.

Instead of generating each candidate independently, the target model can evaluate the proposed sequence in a more parallel manner.

Conceptually:

Draft:

AI | is | a | technology | that

Target:

✓  | ✓  | ✓ | ✓          | ✗

The target model accepts the valid prefix.

Step 4: Continue From the Accepted Tokens

If the first four tokens are accepted:

AI is a technology

generation continues from that point.

The draft model proposes another group of tokens.

Accepted tokens
       ↓
Draft model
       ↓
New candidates
       ↓
Target model
       ↓
Verification

This process repeats until the response is complete.

Traditional Decoding vs Speculative Decoding

Traditional Decoding

Prompt
  ↓
Large Model
  ↓
Token 1
  ↓
Large Model
  ↓
Token 2
  ↓
Large Model
  ↓
Token 3
  ↓
Large Model
  ↓
Token 4

The large model is repeatedly used to generate individual tokens.

Speculative Decoding

Prompt
  ↓
Small Draft Model
  ↓
Token 1 + Token 2 + Token 3 + Token 4
  ↓
Large Target Model
  ↓
Verify Candidates
  ↓
Accept Multiple Tokens

The target model can verify several candidate tokens during a single verification step.

Why Does Speculative Decoding Work?

The key assumption is that a smaller model can often predict the next few tokens reasonably well.

For example:

The capital of France is

A small model might quickly predict:

Paris

If the target model agrees, there is no need for the target model to independently generate that token through another sequential step.

For predictable text, the draft model can produce candidates with a high acceptance rate.

This is where speculative decoding can provide significant speed improvements.

Draft Model and Target Model

The two models have different responsibilities.

Draft Model

The draft model should generally be:

• Smaller
• Faster
• Less computationally expensive
• Good enough to make useful predictions

Its primary purpose is to generate candidate tokens quickly.

Target Model

The target model is:

• Larger
• More capable
• The model whose output behavior is being preserved

It verifies the draft model's predictions.

A simplified architecture is:

             Prompt
                ↓
        ┌───────────────┐
        │ Draft Model   │
        │ Small & Fast  │
        └───────┬───────┘
                ↓
        Candidate Tokens
                ↓
        ┌───────────────┐
        │ Target Model  │
        │ Large & Strong│
        └───────┬───────┘
                ↓
       Accepted Tokens
                ↓
             Output

What Happens When a Token Is Rejected?

Not every prediction from the draft model will be correct.

For example:

Draft:

AI can improve healthcare outcomes

The target model may accept:

AI
can
improve
healthcare

but reject:

outcomes

The generation process then continues using the target model's appropriate probability distribution from that point.

This means speculative decoding does not blindly trust the draft model.

The target model remains the authority.

Acceptance Rate

One of the most important factors in speculative decoding is the acceptance rate.

The acceptance rate describes how often the target model accepts tokens proposed by the draft model.

For example:

Draft tokens:
10

Accepted:
8

Acceptance rate:
80%

A high acceptance rate generally makes speculative decoding more useful.

A low acceptance rate means the target model frequently rejects draft predictions, reducing the potential speedup.

What Determines the Acceptance Rate?

Several factors can influence acceptance.

Draft Model Quality

A better draft model may produce more accurate candidates.

Similarity Between Models

If the draft and target models have similar behavior, candidate acceptance can improve.

Type of Text

Predictable text may be easier to speculate about than highly unusual or creative text.

Sampling Settings

Generation parameters can affect token distributions and acceptance.

Number of Draft Tokens

Generating too many candidate tokens can increase the probability that at least one will be rejected.

A Simple Example

Suppose the draft model generates five tokens:

AI is transforming how businesses

The target model evaluates them:

AI          ✓
is          ✓
transforming ✓
how         ✓
businesses  ✓

All five are accepted.

Instead of requiring five sequential target-model generation steps, the system may have verified the sequence efficiently in a single target-model pass.

Now imagine:

AI          ✓
is          ✓
transforming ✓
how         ✗

The accepted portion is reused, and generation continues from the appropriate point.

Why Speculative Decoding Can Be Faster

The main benefit comes from reducing the number of expensive sequential target-model steps.

Without speculation:

Target → Token 1
Target → Token 2
Target → Token 3
Target → Token 4
Target → Token 5

With successful speculation:

Draft → Token 1, 2, 3, 4, 5
                  ↓
              Target
                  ↓
        Verify multiple tokens

The exact speedup depends on the model, hardware, implementation, acceptance rate, sequence length, and workload.

Speculative decoding is therefore an optimization technique rather than a guaranteed fixed-speed improvement.

Speculative Decoding and Model Quality

One important feature of correctly implemented speculative decoding is that the target model can remain the source of the final sampling decision.

The draft model is proposing candidates, not replacing the target model.

This distinction matters.

The goal is:

Smaller Model
      ↓
Fast Suggestions
      ↓
Larger Model
      ↓
Verification
      ↓
Target Model Behavior

Rather than:

Smaller Model
      ↓
Final Answer

This allows speculative decoding to accelerate generation while preserving the target model's intended output distribution under appropriate algorithms.

Speculative Decoding With Greedy Generation

The idea is particularly easy to understand with greedy decoding.

Suppose the target model normally chooses the most probable token at each step.

The draft model proposes several tokens.

The target model checks whether those tokens match its own preferred sequence.

If they do:

Draft:
A → B → C → D

Target:
A → B → C → D

The entire candidate sequence can be accepted.

If they differ:

Draft:
A → B → C → D

Target:
A → B → X

the system can accept the matching prefix and continue from the divergence point.

Speculative Decoding With Sampling

Modern language models often use probabilistic sampling rather than always choosing the highest-probability token.

Speculative decoding can also be adapted to sampling.

The target model uses a verification or correction procedure so that the final output distribution remains consistent with the target model.

A simplified conceptual process is:

Draft Distribution
        ↓
Candidate Tokens
        ↓
Target Distribution
        ↓
Acceptance / Rejection
        ↓
Corrected Sampling
        ↓
Final Output

This is more mathematically involved than simple greedy verification.

The Mathematics Behind Acceptance

A simplified acceptance rule used in speculative sampling can be represented as:

r=Ptarget(t)Pdraft(t)r = \frac{P_{\text{target}}(t)}{P_{\text{draft}}(t)}

The proposed token can be accepted according to a probability related to:

min⁡(1,Ptarget(t)Pdraft(t))\min\left(1,\frac{P_{\text{target}}(t)}{P_{\text{draft}}(t)}\right)

where:

• Ptarget(t)P_{\text{target}}(t) is the target model's probability for token tt
• Pdraft(t)P_{\text{draft}}(t) is the draft model's probability for token tt

The exact algorithm includes additional steps for rejected tokens and maintaining the desired output distribution.

The important intuition is:

The target model determines whether the draft model's proposal is acceptable.

Speculative Decoding Architecture

A practical architecture may look like this:

                    User Prompt
                         ↓
                ┌────────────────┐
                │  Draft Model   │
                └───────┬────────┘
                        ↓
                Candidate Tokens
                [t1][t2][t3][t4]
                        ↓
                ┌────────────────┐
                │ Target Model   │
                └───────┬────────┘
                        ↓
                Parallel Verification
                        ↓
             ┌──────────┴──────────┐
             ↓                     ↓
        Accepted Tokens       Rejected Token
             ↓                     ↓
             └──────────┬──────────┘
                        ↓
                 Continue Loop

This loop continues until the model finishes generating the response.

Speculative Decoding vs Traditional Generation

FeatureTraditional DecodingSpeculative DecodingMain modelTarget modelDraft + targetCandidate generationOne token at a timeMultiple candidate tokensVerificationNot separateTarget model verifiesSequential target stepsMorePotentially fewerArchitectureSimplerMore complexPotential latencyHigherLower in suitable workloadsExtra modelNot requiredDraft model requiredQualityTarget modelCan preserve target distribution with correct implementationBenefits of Speculative Decoding

1. Lower Generation Latency

The primary goal is to reduce the time required to generate responses.

This can be valuable for interactive applications.

2. Higher Token Throughput

If multiple draft tokens are accepted per verification step, the system can potentially generate more output tokens per unit of time.

3. Better User Experience

Faster responses can improve applications such as:

• AI chatbots
• Coding assistants
• Voice assistants
• AI search
• Real-time applications

4. Efficient Use of Large Models

The target model remains responsible for high-quality generation while the smaller model assists with candidate generation.

5. Can Work Without Retraining the Target Model

Some speculative decoding methods can be applied to existing models without changing the target model's learned parameters.

However, compatibility between draft and target models and implementation details matter.

Challenges of Speculative Decoding

Speculative decoding is not universally beneficial.

1. Draft Model Overhead

The draft model itself requires computation.

If the draft model is too slow, the potential benefit can disappear.

2. Low Acceptance Rate

If the target model frequently rejects draft tokens, speculation provides less acceleration.

Low Acceptance
      ↓
Many Rejections
      ↓
Less Speedup

3. Model Compatibility

The draft and target models need to be compatible with the speculative decoding approach.

Using completely unrelated models may not produce useful acceptance rates.

4. Implementation Complexity

A production implementation needs to manage:

• Candidate generation
• Verification
• Sampling
• KV cache handling
• Rejected tokens
• Batch processing
• GPU execution

This is more complicated than ordinary autoregressive generation.

5. Memory Considerations

Running two models can increase memory requirements.

The draft model needs to be loaded alongside the target model or otherwise made available to the inference system.

6. Hardware Matters

The performance benefit depends on the hardware and serving architecture.

A technique that works well on one GPU configuration may provide a smaller improvement on another.

Speculative Decoding and KV Cache

Modern Transformer inference often uses a Key-Value (KV) cache to avoid recomputing information from previous tokens.

Speculative decoding also needs to manage KV caches carefully.

Conceptually:

Previous Context
      ↓
KV Cache
      ↓
Draft Generation
      ↓
Candidate Tokens
      ↓
Target Verification
      ↓
Update Cache

Efficient KV-cache management is important for achieving good inference performance.

Speculative Decoding and LLM Serving

Large language model serving systems need to balance:

• Latency
• Throughput
• GPU memory
• Cost
• Batch size
• Number of concurrent users

Speculative decoding can be one optimization within this larger serving stack.

A production inference system might combine it with:

• Quantization
• KV caching
• Continuous batching
• Tensor parallelism
• Model parallelism
• Efficient attention
• Optimized GPU kernels

These techniques solve different bottlenecks and can sometimes be used together.

Speculative Decoding and Quantization

These two techniques address different problems.

Speculative Decoding

Focuses primarily on reducing sequential generation work.

Draft
 ↓
Multiple Candidates
 ↓
Target Verification

Quantization

Reduces numerical precision.

FP32
 ↓
FP16
 ↓
INT8
 ↓
INT4

Quantization can reduce memory usage and computational requirements.

Speculative decoding can reduce generation latency.

They can potentially be combined.

Speculative Decoding and Knowledge Distillation

Knowledge distillation can be useful when creating a smaller draft model.

For example:

Large Teacher
      ↓
Knowledge Distillation
      ↓
Small Draft Model
      ↓
Speculative Decoding
      ↓
Large Target Model

A good draft model should be fast while remaining sufficiently aligned with the target model.

Distillation can therefore be one possible strategy for building or improving draft models.

Where Is Speculative Decoding Useful?

AI Chatbots

Chatbots need low latency because users expect interactive responses.

Coding Assistants

Developers benefit from faster code completion and generation.

AI Search

Search systems may need to generate concise summaries quickly.

Voice Assistants

Lower latency is particularly important when users expect natural conversations.

Real-Time AI

Applications involving interactive generation can benefit from reduced response time.

Model Serving

Organizations serving large numbers of users can explore speculative decoding as one way to improve inference efficiency.

A Practical Example

Suppose a target model needs to generate 100 tokens.

Without speculative decoding:

Target Model
100 sequential generation steps

Now suppose a draft model proposes 5 tokens at a time.

If the target model frequently accepts most of the proposals, the target may process the sequence in fewer effective generation rounds.

For example:

Draft:
5 tokens
Target:
Accept 4

Draft:
5 tokens
Target:
Accept 5

Draft:
5 tokens
Target:
Accept 4

The exact number of target-model computations depends on the implementation and acceptance pattern.

The important point is that multiple tokens can potentially be verified during each target-model step.

When Speculative Decoding May Not Help

Speculative decoding is not always the right optimization.

It may provide limited benefits when:

• The draft model is too slow
• The draft model predicts poorly
• Acceptance rates are low
• The target model is already very small
• Hardware communication dominates runtime
• The workload has unusual generation behavior
• The implementation has significant overhead

Therefore, performance should be measured rather than assumed.

How to Evaluate Speculative Decoding

A production team should compare the system before and after optimization.

Important metrics include:

Time to First Token

How quickly the first generated token appears.

Inter-Token Latency

How much time passes between generated tokens.

Tokens Per Second

How quickly the system produces output.

Acceptance Rate

How many proposed tokens are accepted.

GPU Utilization

How effectively the hardware is being used.

Memory Usage

How much GPU memory the draft and target models consume.

Cost Per Request

How much infrastructure is required to serve each request.

A useful benchmark might look like:

                    Before       After
Latency             2.4 sec      1.4 sec
Tokens/sec          42           71
Acceptance Rate     —            78%
GPU Memory          18 GB        22 GB

These numbers are illustrative only. Real performance depends on the model and infrastructure.

The Importance of the Draft Model

The draft model is one of the most important components in speculative decoding.

A good draft model needs to balance:

Speed
  +
Prediction Quality
  +
Compatibility

A larger draft model may predict better but take longer.

A smaller draft model may be extremely fast but produce fewer accepted tokens.

The ideal point depends on the target model and workload.

Draft Model Selection

Possible strategies include:

Smaller Version of the Same Model Family

A smaller model from the same family may have similar token probabilities.

Distilled Model

A smaller model can be trained to approximate a larger target model.

Specialized Draft Model

A model optimized for a particular domain or generation task can potentially be used.

Adaptive Drafting

Future systems may dynamically adjust draft length based on the observed acceptance rate.

Dynamic Speculation

Instead of always asking the draft model to generate the same number of tokens, a system could adapt dynamically.

For example:

High Acceptance Rate
        ↓
Generate more draft tokens

Low Acceptance Rate
        ↓
Generate fewer draft tokens

This can help balance speculation overhead and verification efficiency.

Speculative Decoding for AI Agents

AI agents often generate many intermediate outputs.

For example:

User Request
     ↓
Agent Reasoning
     ↓
Tool Selection
     ↓
Tool Call
     ↓
Observation
     ↓
Next Action

Reducing generation latency can potentially make agent workflows more responsive.

However, agent performance depends on many other factors, including tool execution time and external API latency.

Speculative Decoding and Multimodal AI

As AI systems increasingly combine:

• Text
• Images
• Audio
• Video
• Code

efficient generation becomes more important.

Speculative decoding is primarily associated with autoregressive generation, but similar ideas around candidate generation and verification may become useful in broader multimodal systems.

The exact implementation depends on the model architecture.

Future of Speculative Decoding

Research into speculative decoding continues to explore ways to make inference faster and more efficient.

Potential directions include:

Better Draft Models

Smaller models that closely approximate target-model behavior.

Adaptive Speculation

Automatically adjusting the number of proposed tokens.

Multi-Level Speculation

Using multiple draft models with different sizes.

Tiny Draft
    ↓
Small Draft
    ↓
Large Target

Better Hardware Support

Specialized inference systems could make verification more efficient.

Integration With Other Optimizations

Speculative decoding can potentially work alongside:

• Quantization
• KV caching
• Efficient attention
• Continuous batching
• Model parallelism
• Optimized inference engines

Multimodal Generation

Future research may explore speculative methods for more complex generative architectures.

Speculative Decoding in Simple Terms

The easiest way to understand speculative decoding is to imagine two writers.

Writer 1: Fast Assistant

Writes several possible words quickly.

Writer 2: Expert Editor

Checks the proposed words.

Instead of:

Expert:
Write word 1
Write word 2
Write word 3
Write word 4
Write word 5

we use:

Assistant:
Here are five likely words.

Expert:
✓ ✓ ✓ ✓ ✗

The accepted words can be used immediately.

Then the process repeats.

Fast Draft
    ↓
Candidate Tokens
    ↓
Expert Verification
    ↓
Accepted Tokens
    ↓
Fast Draft Again

That is the core idea behind speculative decoding.

Frequently Asked Questions

What is speculative decoding?

Speculative decoding is an LLM inference technique that uses a smaller draft model to propose multiple tokens and a larger target model to verify them.

Why does speculative decoding make AI faster?

It can reduce the number of sequential target-model generation steps by allowing multiple candidate tokens to be verified together.

Does speculative decoding change the target model?

The target model's parameters do not necessarily need to be changed. Speculative decoding can operate as an inference-time optimization.

What is a draft model?

A draft model is a smaller and faster model that generates candidate tokens for the target model to verify.

What is a target model?

The target model is the larger model whose output distribution or generation behavior the system aims to preserve.

What happens when the target model rejects a token?

The accepted prefix is retained, while generation continues from the rejection point using the appropriate target-model sampling process.

Is speculative decoding always faster?

No. Its effectiveness depends on factors such as draft-model speed, acceptance rate, hardware, workload, and implementation overhead.

Does speculative decoding reduce AI quality?

A correctly implemented speculative sampling method can preserve the target model's intended output distribution. The draft model primarily serves as a proposal mechanism.

What is acceptance rate?

Acceptance rate measures how frequently candidate tokens proposed by the draft model are accepted by the target model.

Can speculative decoding work with quantization?

Yes. Speculative decoding and quantization address different aspects of inference and can potentially be combined.

Does speculative decoding require two models?

The standard approach uses a draft model and a target model. Other acceleration approaches exist, but they are not necessarily standard speculative decoding.

Is speculative decoding useful for chatbots?

It can be useful for reducing generation latency in interactive applications, provided the workload and model pair produce sufficient acceptance rates.

Key Takeaways

• Speculative decoding is an inference optimization technique for accelerating autoregressive AI generation.
• It uses a small draft model to propose multiple tokens.
• A large target model verifies those candidate tokens.
• Multiple tokens can potentially be accepted during a single target-model verification step.
• A high draft-token acceptance rate generally improves the potential speedup.
• The draft model should be fast and sufficiently aligned with the target model.
• Speculative decoding can reduce generation latency and improve token throughput in suitable workloads.
• It introduces additional complexity, memory requirements, and model-serving considerations.
• It can potentially be combined with quantization, KV caching, efficient attention, and other inference optimizations.
• Performance should always be benchmarked on the actual model, hardware, and workload.
• The technique is especially relevant for LLMs, chatbots, coding assistants, AI search, voice systems, and large-scale model serving.

Conclusion

Speculative decoding provides an elegant solution to one of the fundamental challenges of large language model inference: autoregressive generation is sequential, while modern hardware is highly optimized for parallel computation.

Instead of forcing the large target model to generate every token individually, speculative decoding allows a smaller model to make several predictions ahead of time.

The target model then verifies those predictions and accepts the tokens that are consistent with its own generation process.

The workflow can be summarized as:

User Prompt
     ↓
Fast Draft Model
     ↓
Multiple Candidate Tokens
     ↓
Large Target Model
     ↓
Parallel Verification
     ↓
Accepted Tokens
     ↓
Continue Generation

The technique does not eliminate the computational requirements of large models, and it does not guarantee a fixed speed improvement. Its effectiveness depends on the relationship between the draft and target models, acceptance rates, hardware, memory management, and the inference implementation.

As AI models continue to grow, techniques such as speculative decoding, quantization, knowledge distillation, efficient attention, and better model serving will play an increasingly important role in making advanced AI faster, more responsive, and more practical to deploy.