Packed Self‑Speculative Decoding Slashes LLM Generation Latency

What is Packed Self‑Speculative Decoding?

The paper One Call Ahead: Packed Self‑Speculative Decoding with Diffusion Drafting proposes a new schedule, called Packed‑SS, for generating text with autoregressive language models. Traditional autoregressive (AR) models emit one token at a time, which makes long‑form generation expensive. Speculative decoding tries to alleviate this by letting a lightweight “drafter” suggest several tokens that the main model later checks in parallel. Packed‑SS takes the idea further: it lets a diffusion‑based drafter produce an entire block of candidate tokens one step ahead of verification, allowing both verification of the current block and drafting of the next block to happen within a single model invocation.

According to the authors, Yuxiao Chen and Changxiao Cai, this schedule keeps the input size proportional to the speculative width (linear) instead of quadratic, while preserving the statistical properties of the target model.

Read the full paper on arXiv.

---

Why does this matter for language‑model generation?

Generation speed is a bottleneck for many real‑time applications—chat assistants, code completion, and interactive storytelling all suffer when the model must wait for a series of sequential token predictions. Speculative decoding already reduces latency by parallelising verification, but it typically requires two separate model calls per iteration: one to verify the current draft and another to generate the next draft. Existing one‑call tricks avoid the wait but inflate the input tensor, leading to quadratic growth in memory and compute as the speculative width expands.

Packed‑SS eliminates the extra call without blowing up the input size. By overlapping verification with a one‑step‑ahead draft, it achieves the same or better throughput while using only linear additional context. For developers, this means higher token‑per‑second rates and lower overall wall‑time, especially when deploying large models such as Nemotron 3B/8B or SetDLM 1.7B.

---

How does Packed‑SS differ from earlier speculative decoding methods?

Earlier approaches fall into two categories:

1. Two‑call pipelines – The model first checks a batch of speculative tokens, then a second call drafts the next batch. This guarantees correctness but doubles the number of forward passes.
2. One‑call schedules with quadratic input – By concatenating the entire speculative context into a single input, these methods avoid the second call but require the model to process a much larger matrix, which hurts memory efficiency and scales poorly.

Packed‑SS introduces a self‑speculative schedule. The diffusion drafter runs ahead of the verifier: while the verifier validates the current block, the same forward pass also constructs the next block from an older, already‑verified prefix. The unresolved suffix of the draft is left untouched until the next iteration, at which point it becomes the verifier’s input. Under a mild monotonicity assumption about the drafter’s behavior, the authors prove that this scheme does not increase the number of model calls compared with the classic two‑call baseline when using greedy decoding.

---

What experimental evidence supports the claim?

The authors evaluated Packed‑SS on two families of models: Nemotron (3 B and 8 B parameters) and SetDLM (1.7 B parameters). In greedy decoding setups, Packed‑SS consistently outperformed both the standard two‑call speculative decoding and the quadratic one‑call alternative. Metrics reported include higher throughput (more tokens generated per second) and lower mean wall‑time per generated sequence. While the paper does not disclose exact speed‑up percentages in the abstract, the qualitative result is that Packed‑SS delivers a measurable efficiency gain across model sizes, suggesting it scales well to larger architectures.

---

How can developers experiment with Packed‑SS using Grafy?

Grafy does not yet ship a built‑in speculative decoding engine, but the platform provides the tools needed to prototype the schedule yourself. You can:

• Use Grafy Code (/code) to write a Python script that loads a diffusion‑enabled LLM (for example, a Hugging Face checkpoint supporting both AR and diffusion modes) and implements the one‑call‑ahead loop described in the paper.
• Leverage Grafy Chat (/chat) to compare generation latency of a standard greedy decoder versus your Packed‑SS implementation, switching between models side‑by‑side.
• If you prefer not to code from scratch, the Grafy Reader (/reader) lets you upload the PDF of the paper and ask the AI to walk you through the algorithm step‑by‑step, extracting pseudocode that you can then paste into the Code studio.

By combining these apps, a developer can benchmark the new schedule on their own hardware or on Grafy’s cloud model endpoints (OpenRouter, Vertex AI, etc.) and verify the claimed linear‑size input and reduced wall‑time. While Grafy does not yet expose a one‑click “Packed‑SS” toggle, the ecosystem’s modularity makes it straightforward to experiment and potentially contribute a community implementation to the public gallery.

---

Frequently asked questions

How does the "one call ahead" mechanism avoid the quadratic input blow‑up?

The mechanism keeps only a single speculative block in the current forward pass. The verified prefix provides context, while the draft for the next block is generated from an older, already‑verified prefix. Because only one speculative block is concatenated, the input length grows linearly with the speculative width instead of squaring.

Does Packed‑SS work with sampling‑based decoding (e.g., temperature or nucleus sampling)?

The paper’s theoretical guarantees focus on greedy decoding, where the next token is deterministic. Extending the proof to stochastic sampling would require additional analysis, but the underlying schedule—drafting ahead while verifying—can still be applied. Practitioners may need to monitor distributional drift when using temperature or top‑p sampling.

Can I use Packed‑SS with any autoregressive model, or does it require a diffusion component?

Packed‑SS relies on a diffusion drafter that can produce an entire block in one pass. Models that only support standard AR decoding lack this capability, so the schedule is most effective with hybrid models that expose both AR and diffusion modes, such as the Nemotron and SetDLM families evaluated in the study.

---

Source: arXiv Machine Learning — One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting