A Decision Tree for Choosing LLM Inference Optimizations

MachineLearningMastery has published a guide that matches LLM inference optimization techniques to the specific bottleneck limiting a workload, rather than recommending a single technique for every deployment. The piece matters because inference becomes slower and more expensive as traffic grows, with requests queuing behind long generations, GPU memory filling with cached attention state, and hardware spending time waiting for memory reads, according to the article by Bala Priya C on November 3, 2026.

The central argument is that techniques for memory, compute and scheduling each rest on different assumptions about a workload. Choosing without checking those assumptions can add complexity without relieving the real constraint, the article says. Its decision tree is designed to surface the constraint first.

What does each inference optimization technique assume about a workload?

According to MachineLearningMastery, the first step is to understand the conditions a technique was designed for. Quantization, sparsity and distillation assume the model itself is too large for the available hardware or the latency budget, and that shrinking it is worth some risk to accuracy. Quantization lowers the precision of weights and activations, structured sparsity removes less important weights to cut computation, and distillation trains a smaller model to imitate a larger one.

Prefix caching assumes many requests share an identical opening sequence of tokens, so recomputing that prefix wastes work. The article gives chat applications resending full history, agent frameworks reusing a system prompt across many calls, and retrieval-augmented generation pipelines prepending the same instructions to different retrieved contexts as examples. It stresses that the shared prefix must match exactly, character for character, with stable content placed first and variable content at the end.

Continuous batching assumes requests arrive concurrently and produce outputs of unpredictable length, so waiting for the slowest member of a batch wastes GPU time. PagedAttention assumes the KV cache is fragmented or over-provisioned, limiting batch size below what the hardware could otherwise support. Speculative decoding assumes decoding is the bottleneck, tokens are produced one at a time, and a cheaper draft model can correctly guess several of them.

Prefill-decode disaggregation assumes prefill and decode contend for the same GPU, harming both time-to-first-token and inter-token latency. Tensor and pipeline parallelism assume a model does not fit on one device, or per-device compute caps throughput. KV cache offloading assumes useful cache entries are evicted too early because GPU memory cannot retain everything worth keeping. Offline batch inference assumes requests do not need an interactive response, so total throughput matters more than latency for an individual request.

How does the six-question decision tree choose a technique?

The article presents six branching questions, each ending with a direct recommendation if the condition holds or a move to the next question if it does not. The first asks whether the model fits comfortably on the target hardware. A model that barely fits leaves little room for the KV cache, which caps batch size regardless of scheduler quality. If the model is too large, or inference is compute-bound even at a batch size of one, the article points to model-level compression first, noting these methods are largely independent of the serving stack and determine how much capacity remains.

The second question asks whether requests share repeated prompt structure. If a meaningful share of traffic has an identical prefix, the article calls prefix caching the highest-leverage option available, because it avoids recomputing KV cache for already processed tokens and gains grow as conversations lengthen. If prompts are mostly unique, the reader moves on.

The third question asks whether requests arrive concurrently with variable output lengths. A short support answer and a long code generation rarely finish together, and under naive batching the whole batch waits for the slowest request. Continuous batching lets finished sequences leave immediately and new requests start without waiting for a batch boundary, and most serving engines enable it by default, according to the article.

The fourth question asks whether KV cache fragmentation limits batch size. Some engines reserve a fixed contiguous block per request sized for maximum sequence length, even when most requests are far shorter, leaving reserved space unused. PagedAttention addresses this by splitting cache into fixed-size pages allocated in smaller chunks, and the article advises checking whether modern serving engines already have it enabled before adding configuration.

The fifth question asks whether per-token decode latency is the focus, rather than overall throughput. If faster token streaming for each user is the goal, speculative decoding uses a smaller draft model to predict several tokens ahead for verification by the main model in a single pass, potentially completing multiple sequential decode steps together when predictions are correct. The article notes the extra model and verification overhead, and that the approach may not pay off without a suitable draft model or a reasonably high acceptance rate.

The sixth question asks whether prefill and decode contend for the same GPU at scale. The article frames this as mainly a production-scale concern because it changes serving architecture. Separating phases onto dedicated hardware, with prefill nodes passing KV cache to decode nodes, can help when long prefills delay decode requests, but it adds transfer and routing overhead. For smaller workloads or non-interactive requests, offline batch inference may be simpler. If none of the conditions apply, the article suggests starting with continuous batching and prefix caching.

How do the techniques compare on trade-offs?

MachineLearningMastery maps each route to a best use case and a cost. Quantization, sparsity and distillation suit models too large for target hardware or latency budgets, at the cost of some accuracy loss, with quantized activations harder to get right than quantized weights. Prefix caching suits chat, agents and RAG workloads with repeated prompt structure, but breaks on any character-level mismatch in the shared prefix.

Continuous batching suits concurrent traffic with variable output lengths and has minimal downside, largely depending on whether it is enabled. PagedAttention suits workloads bottlenecked by KV cache fragmentation, while adding a page-table layer to attention computation. Speculative decoding suits latency-sensitive single-user generation, but needs a draft model with a reasonably high acceptance rate to justify itself.

Prefill-decode disaggregation suits large-scale traffic where the two phases contend for the same GPU, at the cost of network expense moving KV cache between nodes plus routing complexity. Tensor and pipeline parallelism suit models that do not fit on a single device, with pipeline bubbles and inter-device communication overhead. KV cache offloading suits long-context or high-concurrency workloads running out of GPU memory for cache, but fetching offloaded cache back adds latency. Offline batch inference suits non-interactive workloads prioritizing total throughput, and cannot be used where a strict response-time requirement applies.

What symptoms show a chosen technique is not working?

The article warns that a technique expected to help can behave differently in practice, and lists signals for when that happens. One symptom it names is a prefix cache hit rate near zero despite repeated prompts, which points to a likely cause involving non-deterministic serialization. The practical lesson from the guide is to treat the symptom as a prompt to check the underlying assumption first, such as whether the shared prefix is truly identical, rather than adding another optimization on top.

That approach generalises across the tree. If batch size remains capped, the article directs attention back to whether the model fits or the cache is fragmented. If streaming latency does not improve, it directs attention to whether a draft model is accepted often enough. If both time-to-first-token and inter-token latency suffer at scale, it directs attention to contention between prefill and decode. In each case, the technique is only as good as the workload property it assumes.

How can you explore this topic with Grafy?

For readers who want to work through the ideas without running their own serving stack, Grafy Chat lets you chat with many AI models side by side, which makes it easier to compare how different models handle the same prompt, repeated context and long outputs in practice. You can test whether a prompt structure is stable and reusable, compare responses across models, and build an intuition for latency and output length before committing to an optimization.

For deeper reading, Grafy Reader can help you read, search and question PDFs and papers with AI, including the original article and related research on serving systems. Developers who want to prototype the scheduling side can use Grafy Code, an AI coding studio, to experiment with batching logic, cache behaviour and measurement scripts and to record the assumptions behind each test. Used together, these tools support the article's core workflow: identify the bottleneck, check the assumption, and only then choose the technique.

Frequently asked questions

Which technique should you consider first if the model does not fit comfortably on the target hardware?

Consider model-level compression first, according to MachineLearningMastery. Quantization reduces weight and activation precision, structured sparsity removes less important weights, and distillation trains a smaller model to mimic a larger one. These methods are mostly independent of the serving stack and determine how much hardware capacity remains for the KV cache and batching.

Does prefix caching work when the shared prefix differs by a single character?

No. The article states prefix caching only works when the shared prefix matches exactly, character for character. A single mismatch breaks the cache reuse. That is why it advises putting stable, reusable content first and variable content at the end of the prompt, so repeated structure is preserved across requests.

Is speculative decoding aimed at per-token decode latency or at higher overall throughput?

It is aimed at per-token decode latency for latency-sensitive single-user generation. A smaller draft model predicts several tokens ahead and the main model verifies them in one pass, reducing inter-token latency when predictions are correct. It is not presented as a general throughput technique, and it needs a draft model with a reasonably high acceptance rate.

Source: MachineLearningMastery — https://machinelearningmastery.com/choosing-an-llm-inference-optimization-technique-a-decision-tree-approach/

---

Source: MachineLearningMastery — Choosing an LLM Inference Optimization Technique: A Decision-Tree Approach