Speculative Decoding Explained for Laymen

Abstract AI-generated visualization of token generation in a language model

Recently, Google updated its Gemma 4 models.
The announcement can be read in this blog post.

When I first read it, I didn’t really get it.
But along with the blog, they also published a more technical post on X.

But my curiosity was sparked, and I really wanted to figure out what they actually changed. So, to get my head around it, I decided to write this blog to understand what the change — speculative decoding — really is. How it works, what benefits it has, and why it might not be an improvement for your specific task anyway.

Let’s explore this together from a bird’s-eye view so that even laymen understand it. So, let’s go!

Large Language Models (LLMs), by default, generate one token at a time. I won’t go into detail about what a token really is. Just assume it is a word or maybe even just parts of a word.

Take an example. The input is:

Why is the sky

When an LLM runs inference, the part where it generates the next token, it spits out blue.

LLMs are autoregressive, meaning they run “in a loop” — again and again — until a special “End of Sequence” token is generated. So after this first inference run, they take Why is the sky blue as input and, based on this, generate the next token. This time it generated ?. Running a third time, and it generates The. We just assume that the final output will be:

Why is the sky blue? The sky appears blue because […]

While Why is the sky is the original user input.
All the following tokens are generated by the LLM.

This is, in a nutshell, how a Language Model works.

What is the enhancement in Gemma 4, which this blog post is all about?

In simple terms, they put a smaller model (named “drafter”) before each inference run to predict the next tokens.

Taking the example from above. Your input is Why is the sky. The drafter now puts, let’s say, three additional tokens to the input:
blue, ?, is.
The last prediction is “wrong”, right?
Keep this in mind! We will get back to this later.

The larger LLM doesn’t receive only Why is the sky (your user input), but Why is the sky blue? is.

Now you need to know that LLMs do not generate the last token only when running inference. Instead, they generate the next token for each token sequence — in parallel. So, on a single inference run, it will predict the next word for

• Why is the sky
• Why is the sky blue
• Why is the sky blue?
• Why is the sky blue? is

Again, all of this in parallel. In a single run.

After this single inference run, the model only has to check (verify) each generated output if it matches the predicted output from the drafter:

• Is blue correct? -> Yes -> Keep it
• Is ? correct? -> Yes -> Keep it
• Is is correct? -> No -> Discard it and replace with The

Done

Let us compare the inference run count:
In the world without a drafter, it has to run inference three times to get

Why is the sky blue? The

For blue, ?, and The.

In the world with a drafter, it has to run inference only once to get the exact same output!

In short, we save two inference runs.
This not only saves time but also real money.

This whole technique I described here is called Speculative Decoding. Basically, the smaller model (the drafter) tries to predict a few more tokens, based on the user input.

The drafter needs to be small enough to be fast enough to produce a few tokens before the larger LLM starts generating the first word, but at the same time smart enough to predict the correct words!

You might wonder why we need a drafter at all. Couldn’t we just add random tokens to any input? The random tokens can then be just “replaced” by the LLM. The idea behind it: We generate more tokens with one inference run. Sounds smart, right?

Hold on, Sherlock.

The problem with that approach is that the LLM would run inference, generating the next token, on a false assumption and thus, generate false output as well!

Take the example from above, but change the drafter prediction:

Input: Why is the sky
Drafter: Keyboard, Star, Dog

The LLM would run inference with these inputs:

• Why is the sky
• Why is the sky Keyboard
• Why is the sky Keyboard Star
• Why is the sky Keyboard Star Dog

It will generate the next token for all of these inputs. Since Keyboard is already wrong, the inference run on Why is the Keyboard (to predict the next token for that input) builds already on a wrong foundation! So if the input is already wrong, the output is probably wrong as well.

How to avoid this, then?
As soon as the model detects a wrong drafter prediction token, it will “discard” the wrong token as well as all following tokens.

Therefore, in our example here, it will just output Why is the sky blue. Not more. The generated tokens from the [..] Keyboard, [..] Keyboard Star, and [..] Keyboard Star Dog will be just discarded. So, we get the same output for a single inference run as if we were running without a drafter at all.

With that information, we can conclude that this technique is only as good as the drafter model is.

I hope I can write this down so that it is understandable even for a layman like me. I at least had fun exploring this topic. I find this technique very clever and fascinating.