Intent‑Aware Caching Boosts LLM Serving Efficiency by 34% on Real Workloads

What problem does intent‑aware caching aim to solve?

Modern large‑language‑model (LLM) serving platforms rely on prefix caching – storing the key‑value (KV) states of previously processed tokens so that subsequent requests that share the same conversation prefix can skip the costly pre‑fill step. While this technique reduces the time‑to‑first‑token (TTFT) and improves throughput, real‑world chat workloads still achieve low cache‑hit rates because a cached prefix is only useful if the user returns quickly and continues the same thread. Conventional eviction policies such as LRU, LFU or even learned approaches like Learned Prefix Caching (LPC) consider only generic signals (recency, frequency, size) and ignore the underlying user intent that drives whether a conversation will continue. As the authors explain, “cache reuse depends on whether a conversation continues and how soon the next turn arrives,” a dependency that existing policies fail to capture (according to the arXiv paper). Figure 1 illustrates the basic idea of prefix caching and how the growing context length inflates serving cost.

How does the Intent‑Aware Caching (IAC) policy use user intent?

IAC introduces three tightly coupled components:

1. Intent Classifier – a lightweight model that tags each incoming request with one of several intent categories (e.g., code_generation, translation, chitchat). The classifier is trained offline using a two‑step pipeline: intent discovery with a large LLM followed by an MLP head on frozen embeddings (see Figure 5 in the appendix of the paper).
2. Statistics Estimation – online aggregation of recent logs to compute, per intent, two key metrics: the empirical continuation probability (how often a conversation with that intent generates a follow‑up turn) and the distribution of inter‑turn delays.
3. Cache Eviction Policy – an oracle‑inspired decision rule that combines the per‑intent statistics with object size, mimicking the size‑aware Threshold‑Belady (TB+S) algorithm. Each cached prefix receives a priority score derived from the expected reuse time; the lowest‑scoring entry is evicted when space is needed.

By grouping requests by intent, IAC can distinguish, for example, that code_generation sessions typically involve many turns with longer pauses, whereas translation queries are often one‑off. This granularity lets the system keep prefixes that are likely to be revisited soon while discarding those that are unlikely to see a follow‑up, all without costly retraining.

Figure 4 visualises the data flow: a request arrives, the classifier assigns an intent, the statistics module updates its counters, and the eviction engine computes a priority score before deciding which leaf node to drop.

What performance gains were reported?

The authors evaluated IAC on two publicly available conversational datasets – WildChat and ShareChat – which together contain over a million real user interactions with ChatGPT‑family models. Using the SGLang serving framework, they compared IAC against several baselines:

• LRU – 12.2% hit rate
• LFU – (not explicitly listed, but similar to LRU)
• Learned Prefix Caching (LPC) – 18.8% hit rate
• Oracle‑inspired Threshold‑Belady (TB+S) – the upper bound

IAC achieved a 34.6% cache hit rate, more than double the best prior online method. This translated into up to 46.1% reduction in average TTFT latency and a noticeable uplift in overall throughput (tokens processed per second). The paper’s ablation study shows that removing the intent component drops the hit rate back toward LPC levels, confirming that intent signals are the primary driver of the improvement.

How was IAC evaluated and against which baselines?

Evaluation followed a realistic replay setup: the authors replayed logged requests from WildChat and ShareChat through a simulated SGLang server equipped with a fixed cache size. They measured four key metrics:

1. Cache Hit Rate – proportion of input tokens whose KV states were served from cache.
2. TTFT Latency – time from request arrival to first token emitted.
3. Throughput – total tokens processed per second across the workload.
4. Tail Latency – 95th‑percentile TTFT, important for user‑perceived responsiveness.

The online policies (LRU, LPC, IAC) operated without knowledge of future arrivals, while the oracle policies (MIN, TB+S) were given the full future request schedule to compute an upper bound. IAC consistently narrowed the gap to the oracle: on ShareChat, the oracle hit rate was around 45%, while IAC reached 34.6%, a 77% relative improvement over LPC.

Additional experiments explored robustness to varying arrival rates, different delay distributions, and the overhead of intent classification (found to be under 1 ms per request, negligible compared with model inference). The authors also tested IAC in an online serving mode where statistics are refreshed every few seconds, confirming that the method remains stable under workload drift.

How can Graf‑based creators and developers explore or apply these ideas?

While Grafy does not currently expose a built‑in LLM serving stack, the research insights can be examined through two of our tools:

• Reader – upload the arXiv PDF and use AI‑assisted search to dive into specific sections (e.g., the intent classifier design or the eviction formula). This lets researchers reproduce the experiment pipeline or adapt the taxonomy to their own workloads.
• Code – prototype a simplified version of IAC using Grafy’s coding studio. You can import the open‑source implementation from the GitHub repository linked in the paper, experiment with different intent taxonomies, and benchmark cache‑hit rates on a local LLM server.

By combining Reader’s paper‑analysis capabilities with Code’s rapid prototyping, developers can test intent‑aware eviction strategies on custom chatbots, evaluate latency improvements, and potentially integrate a lightweight classifier into existing inference pipelines.

---

Source: K. H. Ibn Arif et al., “Intent‑Aware Caching for Efficient LLM Serving,” arXiv:2611.09715 [cs.AI], 9 Nov 2026. https://arxiv.org/abs/2611.09715

Frequently asked questions

Does Intent‑Aware Caching require a separate model for intent classification?

Yes. IAC uses a small, pre‑trained text embedder with an MLP head that outputs an intent label. The classifier is lightweight (≈1 ms per request) and can be trained once on a labeled dataset; it does not need continuous retraining during serving.

How does IAC compare to simple time‑based eviction policies?

Time‑based policies that evict entries after a fixed idle period ignore the differing continuation probabilities across intents. IAC’s per‑intent statistics allow it to keep long‑delay prefixes (e.g., code‑generation) while discarding short‑delay ones that are unlikely to return, yielding higher hit rates than any fixed timeout approach.

Can the intent taxonomy be customized for niche applications?

Absolutely. The authors provide a 30‑category taxonomy but also show reduced versions (20 or 10 categories). Developers can define new intents relevant to their domain, retrain the classifier on a few hundred labeled examples, and the IAC framework will automatically incorporate the new statistics.

---

Source: Arxiv AI Recent — Intent-Aware Caching for Efficient LLM Serving