GAUSS Introduces Workload‑Aware Simulation for LLM Inference Serving
A new paper presents GAUSS, a simulator that predicts full latency distributions for LLM inference.
GAUSS, a newly released analytical and Monte‑Carlo simulator, aims to predict the full latency distribution of large language model (LLM) inference serving, addressing the difficulty operators face when trying to meet strict latency service‑level objectives (SLOs). By modeling both the prefill and decode phases and the stochastic nature of request arrivals, GAUSS promises more reliable capacity planning than traditional throughput‑only metrics.
What is GAUSS and what problem does it solve?
According to the paper on arXiv (https://arxiv.org/abs/2610.37989), GAUSS was created because a single throughput number no longer answers the operational questions that arise when serving generative LLMs. Inference workloads consist of two distinct phases—prefill, where the model processes the initial prompt, and decode, where it generates subsequent tokens. These phases have very different latency characteristics, and continuous batching further reshapes the service process. Moreover, user‑visible tail latency can degrade well before overall throughput reaches its limit, making it hard for operators to guarantee SLOs. GAUSS tackles this by delivering full probability distributions for two key metrics: time‑to‑first‑token (TTFT) and time‑between‑tokens (TBT), as well as the sustainable request rate for any given combination of workload, engine, and accelerator.
How does GAUSS model LLM inference serving?
The authors describe GAUSS as combining calibrated, phase‑specific latency surfaces with a queue‑theoretic model. The latency surfaces are derived from empirical measurements of prefill and decode stages on specific hardware, providing a deterministic mapping from request size to base latency. GAUSS then overlays a Lindley‑recurrence formulation, which is a classic way to model waiting times in queues with stochastic arrivals. Rather than replaying a single trace of traffic, GAUSS runs an ensemble of Monte‑Carlo samples, each representing a possible realization of request arrivals and service times. This approach captures the stochastic variability of real‑world traffic and produces a full distribution of TTFT and TBT rather than a single average value.
What insights can GAUSS provide that existing simulators cannot?
Traditional trace‑driven simulators often under‑sample latency tails because they rely on a finite set of observed requests. GAUSS, by design, estimates performance through repeated random sampling, which inherently explores rare but critical tail events. The paper highlights three practical questions that GAUSS can answer more effectively than trace‑based tools:
- Operating‑point selection – Determining the optimal mix of batch size, hardware, and routing policy that meets a target latency percentile.
- Traffic segmentation – Understanding how different request classes (e.g., short prompts vs. long generation tasks) affect overall latency and capacity.
- Shortest‑remaining‑processing‑time (SRPT) scheduling – Evaluating how SRPT policies impact the sustainable request rate and tail latency, something trace replay struggles to quantify.
By delivering full distributions, GAUSS enables operators to see not just the median but the 95th or 99th percentile latencies that directly affect user experience.
How can operators use GAUSS for capacity planning?
GAUSS takes three inputs: a description of the workload (arrival rate, prompt length distribution, token generation length), the inference engine configuration, and the accelerator characteristics. With these, it outputs:
- TTFT distribution – How quickly the first token is expected to appear for a given request class.
- TBT distribution – The interval between subsequent tokens, crucial for streaming applications.
- Sustainable request rate – The maximum arrival rate that keeps the chosen latency percentile within the SLO.
Operators can experiment with "what‑if" scenarios without deploying new hardware. For example, they might compare a GPU‑based accelerator against a specialized inference ASIC, or test the impact of increasing the maximum batch size. Because GAUSS uses analytical latency surfaces, the simulation runs quickly, allowing rapid iteration over many configurations.
Applying GAUSS concepts with Grafy tools
While Grafy does not yet host a dedicated LLM inference simulator, creators, developers, and researchers can still explore GAUSS and its implications using existing Grafy apps:
- Grafy Reader – Upload the GAUSS paper PDF and ask the AI to summarise sections, extract definitions of TTFT and TBT, or compare GAUSS to other simulators. This mirrors the paper‑reading workflow many engineers follow before adopting a new tool. (link)
- Grafy Chat – Pose scenario‑based questions such as "What batch size would keep 99th‑percentile TTFT under 200 ms on a V100?" and let the multi‑model chat provide back‑of‑the‑envelope calculations or suggest how to build a custom Monte‑Carlo model. (link)
- Grafy Code – For teams ready to prototype a lightweight version of GAUSS, the Code studio can run Python notebooks that implement Lindley‑recurrence queueing and Monte‑Carlo sampling, leveraging the same cloud models Grafy supports for inference.
By combining Reader’s document‑level understanding with Chat’s interactive reasoning, users can quickly assess whether GAUSS’s methodology fits their own serving stacks, and even begin building prototype simulations within Grafy’s coding environment.
Frequently asked questions
How does GAUSS differ from a simple throughput calculator?
GAUSS goes beyond average throughput by modelling the two distinct inference phases (prefill and decode) and by generating full latency distributions for TTFT and TBT. This reveals tail behavior and sustainable request rates that a single throughput figure cannot capture.
Can GAUSS be used with any LLM or only specific models?
The simulator is workload‑aware: it requires calibrated latency surfaces for the specific engine and accelerator in use. In principle, any LLM can be supported as long as the necessary phase‑specific latency data are collected.
Does Grafy provide a ready‑made GAUSS implementation?
Grafy does not currently host a built‑in GAUSS simulator. However, users can read the paper with Grafy Reader, discuss its concepts in Grafy Chat, and prototype the analytical model in Grafy Code.
Source: arXiv AI — GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
More news
FilingLoop Benchmark Shows Self‑Evolving LLM Agents Improve SEC Filing Review
Researchers unveil FilingLoop, an interactive benchmark for LLM agents that review SEC filings.
New AI Method Estimates Multi-Agent Risk Propagation
A new arXiv study proposes IE-Prop, an interaction-conditioned model that estimates propagated risk in LLM multi-agent systems. Researchers report AUROC gains.
Seq@K Benchmark Highlights Need for Feedback‑Aware Model Evaluation
A new arXiv paper introduces Seq@K, an evaluation protocol that measures a language model’s ability to solve tasks within K sequential attempts using feedback.