Google Details Gemma 4 Multi‑Token Prediction Workflow to Reduce Inference Latency

Google published technical details on pairing Gemma 4 with lightweight multi‑token prediction (MTP) drafters that use speculative decoding to propose multiple future tokens, letting the main model verify them in a single pass to cut inference latency.

Google published technical documentation and a visual thread on May 25, 2026, describing a workflow that pairs its Gemma 4 family with lightweight multi‑token prediction (MTP) drafters. The approach uses speculative decoding — where a compact drafter proposes several next tokens in parallel and the primary model verifies them in one pass-to reduce inference latency, a key constraint for responsive on‑device and edge deployments.

MTP drafters are auxiliary, compact models that run alongside the heavier Gemma 4 target during inference. While the primary model would normally compute each next token sequentially, the drafter uses otherwise idle compute to propose multiple candidate tokens in parallel; the main model then performs a verification pass over those suggestions. The design specifically targets the memory‑bandwidth bottleneck created when processors repeatedly stream billions of parameters from VRAM to compute units for every token.

Google frames the technique as especially useful when the main model spends similar compute on trivial and complex tokens: the drafter can cheaply handle the “obvious” next tokens, allowing the larger model to focus expensive compute on decisions that require deeper reasoning. Implementations pair heavy target weights — such as Gemma 4 31B-with much smaller drafters, while keeping the main model responsible for final output decisions so frontier‑class reasoning and accuracy are preserved.

The company highlighted deployments across a range of devices: consumer PCs and GPUs running Gemma 26B MoE and 31B dense variants, and mobile scenarios using Gemma E2B and E4B variants. Google also noted that MTP‑enabled Gemma 4 variants are already available on public platforms such as Hugging Face, Kaggle and Ollama, enabling local and edge experiments outside of Google’s own stacks.

Community reactions acknowledged the potential and tradeoffs. One commenter called the MTP‑enabled Gemma 4 “pretty impressive” but cautioned that local models still produce many mistakes; another pointed out a long‑standing drawback — MTP requires loading two models into memory. A remark on Hacker News emphasized that MTP yields the largest benefits in settings with relatively few users, where per‑user compute is abundant, rather than at very large API scale.

Google engineers said they implemented architectural changes and hardware‑specific optimizations to maximize MTP efficiency. A key implementation detail is sharing the target model’s kV cache between drafter and verifier to reduce overhead. The published visuals map the drafter‑verify loop directly to hardware execution, underscoring the potential to improve responsiveness on constrained hardware without changing final model outputs.