How Google made Gemma 4 3x faster (without changing the model)

LLMs generate text one token at a time. A token is roughly a word fragment -- "unbelievable" might be three tokens: "un", "believ", "able". Each time the model produces a token, it reads that token back as input and runs another full forward pass through all its layers to produce the next one. This is called autoregressiveGenerates each token after seeing the previous tokens. decoding: each output depends on everything before it, so you cannot skip ahead or parallelize generation. You are always waiting for token N before you can start token N+1.

LLM inference has two distinct phases with very different performance characteristics.

PrefillReads the full prompt before writing the answer. is when the model processes your input prompt. All input tokens are available upfront, so the model can process them all in parallel using large matrix-matrix multiplications. This is compute-bound -- the GPU is doing heavy arithmetic and is well-utilized. A long prompt takes more time than a short one, but it completes in one shot.

DecodeWrites the answer one token at a time. is when the model generates the response, one token at a time. At this point, the model is doing matrix-vector multiplications: one forward pass per token, loading all the model's weights from VRAM (the GPU's dedicated memory, separate from your laptop's regular RAM -- think of it as the GPU's working memory, with much higher bandwidth but typically 24-80GB of capacity) to the GPU compute units each time. For a 31B parameter model stored in 16-bit floats, that is roughly 62GB of data moving across the memory bus -- per token.

The problem is arithmetic intensityHow much math happens per byte read from memory.: the ratio of math operations performed to bytes read. Matrix-vector products at batch size 1 have very low arithmetic intensityHow much math happens per byte read from memory.. The GPU finishes the math almost instantly and then waits for the next batch of weights to arrive from memory. On modern hardware, the GPU is often less than 10% utilized during decodeWrites the answer one token at a time. at batch size 1. The bottleneck is memory bandwidth, not compute.

This distinction matters: the claim that "LLM inference is memory-bandwidthHow fast data moves from memory to compute. bound" is specifically true for the decodeWrites the answer one token at a time. phase at small batch sizes. During prefillReads the full prompt before writing the answer., or when serving many users simultaneously with large batch sizes, inference can shift toward being compute-bound.

Here is the asymmetry that makes speculative decoding possible: generating a token from scratch is expensive, but checking whether a given token is correct is cheap.

Think of it like a proofreader and a fast drafter. The drafter writes several sentences quickly -- they might be right, might need a small fix. The proofreader reads the whole draft in one pass and either stamps it approved or marks the first mistake. If the draft was right, you just got four sentences for the price of one review. If the drafter made a mistake at sentence two, you discard sentences two, three, and four -- but you kept sentence one, which you would have had to generate anyway.

In speculative decoding, the drafter is a small, fast model (Gemma 4's MTP drafter is far smaller than the 31B target). It generates K tokens quickly -- each pass is cheap because the model is tiny. The large target model then runs one forward pass over all K draft tokens simultaneously and produces a probability distribution over what each token should have been.

For each position, the target model either accepts the draft token (it agrees the token is plausible given the context) or rejects it. If all K tokens are accepted, the target model also produces one additional token of its own, giving you K+1 tokens for the cost of one target model forward pass. If the draft is wrong at position i, tokens after position i are discarded and inference continues normally from there.

Critically, this is not an approximation. With the proper rejection sampling scheme, the probability distribution of the output is mathematically identical to running the target model alone. The target model always has final authority. Speedup is free -- you are not trading quality for speed.

This question trips people up: why can you verify K tokens in parallel but not generate them in parallel?

During verification, you already have all K draft tokens. You can feed the entire sequence into the target model at once as a single forward pass (just like prefillReads the full prompt before writing the answer.). The model produces probability distributions for each position simultaneously. There is no sequential dependency between positions here -- you are just scoring a complete sequence that already exists.

During generation, you do not have the next token yet. To produce token N+1, the model must condition on token N. There is no way around this -- the whole architecture is designed to predict the next token given everything before it. You cannot run position N+1 until position N is finalized. The dependency is fundamental, not an implementation detail.

This asymmetry is precisely what speculative decoding exploits: the drafter fills in the dependency chain with guesses, then the target model evaluates the complete sequence in one cheap parallel pass.

Google introduced several architectural choices that go beyond basic speculative decoding.

KV cacheSaved attention calculations from earlier tokens. sharing is the most important. The KV cacheSaved attention calculations from earlier tokens. is a stored record of the Key and Value matrices the model computed for every token in the context so far. It is what prevents the model from recomputing context from scratch on each new token -- without it, a 1000-token conversation would require reprocessing all 1000 tokens on every generation step. In a naive speculative decoding setup, the drafter would maintain its own separate KV cacheSaved attention calculations from earlier tokens. and recompute context independently. Gemma 4's drafter instead reads directly from the target model's KV cacheSaved attention calculations from earlier tokens.. It does not duplicate work the large model already did. This is significant because KV cacheSaved attention calculations from earlier tokens. computation can be a non-trivial fraction of inference cost for long contexts.

For the E2B and E4B edge models (designed to run on phones and tablets), the bottleneck shifts: generating the final token logits (the raw probability scores over the entire vocabulary -- typically 250,000 tokens) becomes expensive because the vocabulary is large relative to the model size. Google added an efficient embedding clustering technique to speed this up, essentially grouping similar vocabulary entries so fewer computations are needed.

For the 26B Mixture-of-Experts (MoE) model on Apple Silicon, batch size matters more than it does for dense models. In a MoE architecture, each token is routed to only a small subset of specialized sub-networks ("experts"). At batch size 1, most experts sit idle. At batch sizes of 4-8, different tokens in the batch activate different experts in parallel, dramatically improving hardware utilization -- up to 2.2x speedup compared to batch size 1.

The speedup from speculative decoding is not constant. It depends on how accurately the drafter can predict what the target model would say -- the acceptance rate.

Speculative decoding works best on predictable text: structured code (the next token in `for i in range(` is almost always `0`), formulaic writing, repetitive patterns, and factual recall. In these cases, the drafter gets most tokens right and the acceptance rate is high. Google reports up to 3x speedup across tested hardware and frameworks.

It works less well on highly creative or unpredictable outputs where the target model might reasonably generate any of many different tokens. The drafter's guesses miss more often, sequences get discarded more frequently, and the net benefit shrinks. In the worst case, if the drafter is consistently wrong, you pay the overhead of running the drafter on top of the target model. In practice, speculative decoding is almost never slower than standard inference -- rejecting all draft tokens costs one wasted drafter pass, and you fall back to normal target model behavior.

The general principle here extends beyond Gemma 4: speculative decoding is now a standard technique in production LLM serving. Anthropic, OpenAI, and most serious inference frameworks (vLLM, SGLang, TensorRT-LLM) support it. When you see "fast inference mode" or "draft model" in an API or library, this is almost always what is happening under the hood.

A 3x speedup is not just an infrastructure metric. It changes how the product feels.

Imagine a chat response that would normally take 9 seconds to stream. At 3x faster, it can finish in about 3 seconds. That moves the experience from "I am waiting for a slow printer" toward "someone is typing back quickly." For coding assistants, that gap is even more obvious: completions that arrive after your train of thought has moved on feel laggy, while completions that arrive immediately feel like part of the editor.

This is why speculative decoding is especially valuable for interactive LLM products. It improves tokens-per-second during the decodeWrites the answer one token at a time. phase without changing the model output, so the same model can feel more responsive without retraining, shrinking, or accepting lower quality.