Grafy News
Try Grafy free

VelServe tackles decode jitter to meet LLM latency SLOs

Researchers introduce VelServe, a tail‑calibrated scheduler that reduces decode jitter in large‑language‑model inference.

Illustration for the news story: VelServe tackles decode jitter to meet LLM latency SLOsSource: Grafy Research Lab

Interactive large‑language‑model (LLM) services must keep token delivery predictable for real‑time applications. A new study from the Grafy Research Lab, titled VelServe: Taming Decode Jitters in LLM Inference with Tail‑Calibrated Scheduling, proposes a scheduling technique that directly addresses unpredictable latency spikes, known as decode jitter.

What is decode jitter and why does it matter?

Decode jitter refers to the variability in the time it takes an LLM to generate each token after the initial prompt. Two latency metrics are especially important for streaming and soft‑real‑time use cases: the time to first token (TTFTTTFT) and the time between tokens (TBTTBT). According to the Grafy Research Lab pre‑print, jitter can delay every request in a batch, and when a response spans hundreds of steps, even a rare overrun can cause many service‑level‑objective (SLO) violations. In practice, this means chat interfaces, code assistants, or any interactive AI that streams output may feel sluggish or inconsistent, eroding user trust.

How does VelServe address jitter?

VelServe introduces a tail‑calibrated scheduling approach. In aggregated serving environments, the system predicts the latency of the next decoding step and allocates a prefill budget—the amount of computation time allowed—so that the predicted latency exactly meets the TBTTBT target. This tight budgeting leaves virtually no slack for mispredictions, which is why VelServe also incorporates a calibration phase that adjusts the budget based on observed tail behavior. By continuously aligning the budget with the actual latency tail, the scheduler prevents the cascade of delays that cause jitter, keeping token intervals within the prescribed SLO.

What are the implications for real‑time AI applications?

For developers building chatbots, collaborative coding assistants, or live‑translation tools, VelServe’s method promises more reliable streaming performance. Consistent TBTTBT means smoother user experiences, fewer hiccups during long generations, and lower risk of exceeding latency budgets that could trigger throttling or increased costs. Moreover, the research highlights that jitter is not merely a hardware issue; it emerges from the interaction of batching strategies and model inference pipelines, suggesting that software‑level scheduling can be as impactful as hardware upgrades.

How can developers experiment with VelServe concepts using Grafy?

Grafy does not yet provide a built‑in LLM inference engine, but creators can explore the scheduling ideas in several ways:

  • Grafy Chat – Use the multi‑model chat interface to benchmark TTFTTTFT and TBTTBT across different models. By measuring token‑arrival times, you can observe jitter patterns in real time. (Grafy Chat)
  • Grafy Code – Implement a prototype of VelServe’s tail‑calibrated scheduler in Python or JavaScript. The coding studio lets you run scripts that call external LLM APIs (e.g., OpenRouter) and experiment with dynamic pre‑fill budgets. (Grafy Code)
  • Grafy Reader – Load the full arXiv paper into the Reader app to search, annotate, and ask follow‑up questions about the methodology without leaving the workspace. (Grafy Reader)

A practical workflow might look like this: first, use Chat to record baseline latency traces for a chosen model. Next, switch to Code to write a wrapper that predicts the next step’s latency and adjusts the request payload accordingly, mimicking VelServe’s budgeting. Finally, return to Chat to compare the jitter‑reduced stream against the baseline. While Grafy does not host the inference backend, its integrated environment makes it straightforward to prototype, test, and document the scheduling logic.

Frequently asked questions

Does VelServe eliminate all latency jitter?

VelServe substantially reduces jitter by aligning the pre‑fill budget with the observed latency tail, but it cannot guarantee zero variability. Unexpected hardware contention or network spikes may still introduce occasional delays.

Can VelServe be applied to disaggregated serving setups?

The paper focuses on aggregated serving, where a single scheduler controls a batch of requests. The authors note that jitter also appears in disaggregated scenarios, but adapting the tail‑calibrated approach would require coordination across independent serving nodes, which is left for future work.

Is the VelServe scheduler compatible with any LLM model?

VelServe’s budgeting logic is model‑agnostic; it relies on latency predictions rather than model internals. However, accurate prediction may be easier for models with stable per‑token compute patterns, so performance can vary across architectures.


Source: Grafy Research Lab — VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling