KV‑LTM Introduces Persistent Per‑Session KV Caches for Personal Assistant Memory
Researchers propose KV‑LTM, a tier that stores raw session text and its KV cache to cut repeat pre‑fill work in AI assistants.
Source: Grafy Research LabA new research paper proposes KV‑LTM, a persistent per‑session key‑value cache that acts as a long‑term memory tier for personal assistants. The approach promises to keep the original conversation text while eliminating repeated pre‑fill computation, a bottleneck for current AI assistants.
What is KV‑LTM and what problem does it address?
According to the authors Ziv Kfir and Ido Kfir, KV‑LTM adds a memory layer that stores, for every interaction session, (i) the raw transcript and (ii) the KV cache generated by a fixed reader model. When a user asks a follow‑up question, the system restores the cached state and simply appends the new query, avoiding the costly step of re‑prefilling the entire session text. The paper argues that today’s long‑term memory stacks either transform content at write time (e.g., fact extraction, summarisation) or rely on retrieval indices that must re‑process text at read time, both of which can lose fidelity.
How does KV‑LTM differ from existing memory stacks?
The authors distinguish four memory modes:
- L1 – Live context only – no persistence beyond the current turn.
- L2 – Retrieval or derived memory – indexed verbatim archives, extracted facts, or graph structures.
- L3 – Full‑history text reading – the entire retained transcript is read for each query.
- L4 – KV‑LTM – restores the native KV state of each session, keeping the raw text as a fallback.
KV‑LTM (L4) is complementary rather than a replacement; it can sit beside retrieval systems, acting as a verifier that reads cached sessions instead of re‑prefilling their text. Unlike L2 approaches, it does not alter the original conversation at write time, and unlike L3 it does not require attention over the whole history for every question.
What are the storage and compute implications?
The paper provides an analytical sizing of the cache. For pinned BF16 full‑attention and latent‑cached MLA configurations, the per‑token KV state occupies roughly – bytes. Assuming a user generates 30 000 tokens per day, the annual storage requirement ranges from about 0.34 TB to 3.6 TB per user, compared with only ~49 MB of raw text for the same period. The authors model per‑query cost as a combination of data transfer, attention over the restored cache, decoding, and aggregation. When the cache is available, the dominant saving is the avoidance of repeated pre‑fill compute; if the cache is evicted or invalidated, the system degrades gracefully to reading the raw transcript.
How was KV‑LTM evaluated and what were the findings?
The authors report a system‑level comparison on the LongMemEval benchmark. Two commercial assistants that rely on existing memory pipelines scored 0.577 and 0.330 respectively, while the same model (GPT‑4o) reading the full history offline achieved 0.918, indicating a noticeable content loss between write and read stages. KV‑LTM itself has not yet been measured on the benchmark; the paper instead outlines an evaluation protocol that isolates cache reuse without requiring a full persistent‑cache implementation. The protocol includes a reference arm that emulates cache restoration and a minimal real‑cache replay arm to verify numerical equivalence and measure actual cost.
How can creators or developers experiment with KV‑LTM concepts using Grafy?
Grafy does not yet offer a built‑in KV‑LTM service, but the idea can be explored with existing tools. Using Grafy Chat, developers can simulate a per‑session KV cache by capturing the model’s hidden states after a conversation and storing them alongside the transcript. The stored transcript can be managed with the Grafy Reader app, which lets users search and query PDFs or plain‑text files – the same mechanism can serve as the raw‑text fallback. For more complex workflows, the Graph app can track the lineage of prompts, cached states, and responses, making it easier to experiment with cache‑restore‑and‑append logic. By combining these apps, creators can prototype the KV‑LTM read path and measure the compute saved when reusing cached activations.
Frequently asked questions
Does KV‑LTM replace retrieval‑based memory systems?
No. KV‑LTM is intended to complement retrieval. It provides a fast path for recent sessions by restoring their KV cache, while retrieval indices can still be used for older or unrelated sessions.
How much extra storage does KV‑LTM require compared to plain text archives?
The analysis in the paper estimates 0.34–3.6 TB of KV data per user‑year for a typical 30 k‑token‑per‑day usage, versus roughly 49 MB of raw text for the same period.
Can I use KV‑LTM today without custom engineering?
Grafy does not ship a dedicated KV‑LTM feature yet, but the concept can be prototyped using Grafy Chat to capture model states and Grafy Reader to keep the original transcripts. This allows developers to test the cache‑restore workflow on a small scale.


