Grafy News
Try Grafy free

Learned Multigranular Memory Retrieval Boosts Small LLM QA Performance

A new paper introduces LMMR, a learned multigranular memory retrieval method that trims input to under 400 tokens while raising F1 scores on HotpotQA and other.

Illustration for the news story: Learned Multigranular Memory Retrieval Boosts Small LLM QA PerformanceSource: Grafy Research Lab

What is LMMR and why it matters?

The paper Learned Multigranular Memory Retrieval for Small Language Model Agents (LMMR) proposes a new way for tiny language‑model agents to pull just the right pieces of information from a large memory store. Small answer models typically have a hard limit on how many tokens they can read at once, so when relevant facts are scattered across long documents, much of the input budget is wasted on background text. According to the authors Pengfeng Song, Yunkai Zhuang and Liye Shi, LMMR learns both which source to consult next and how much of that source to ingest, dramatically shrinking the context while preserving the needed reasoning steps.

How does the multigranular retrieval work?

LMMR builds a representation that keeps full paragraphs linked to their constituent sentence fragments. During training, the system creates pairs of views – a coarse view (e.g., a whole paragraph) and a fine view (e.g., a single sentence) – that supply the same missing support for a given anchor passage. By contrasting these pairs with distractor snippets and repeated facts, the reranker learns a conditional score that predicts how much new information a candidate adds to the current evidence.

At inference time, the process unfolds in two stages:

  1. Reranking – each candidate paragraph or fragment is scored based on the additional support it provides to the anchor.
  2. Selection – a budget‑aware selector picks the highest‑scoring candidates until a token limit (set to 400400 in the experiments) is reached.

This coarse‑to‑fine pipeline lets the system decide, for example, to keep an entire biography paragraph for one entity but only a single sentence for another, matching the exact granularity the answer requires.

Performance results on benchmark QA sets

The authors evaluated LMMR on three multi‑hop question‑answering benchmarks: HotpotQA, 2WikiMultihopQA and MuSiQue, covering a total of 22,39822{,}398 official validation questions. Using a frozen 2B2\text{B} answer model, LMMR achieved the following F1 scores under the 400400‑token budget:

  • HotpotQA: 61.4761.47
  • 2WikiMultihopQA: 43.2643.26
  • MuSiQue: 30.5830.58

These numbers represent improvements of 3.063.06, 2.142.14 and 4.554.55 points respectively over a strong query‑expansion baseline. Moreover, the memory input required by LMMR was 23.1%23.1\%–26.9%26.9\% smaller than that of the baseline, and 23.5%23.5\%–27.7%27.7\% fewer tokens than a coarse retriever trained on the same questions.

When paired with the LLMLingua‑2 compression model at comparable input lengths, LMMR still delivered F1 gains of up to 3.993.99 points on HotpotQA and 2WikiMultihopQA, with statistical significance confirmed by Holm correction. Across eight different small answer models evaluated on HotpotQA, every model outperformed both query expansion and the classic Maximal Marginal Relevance (MMR) approach in terms of F1 and exact‑match scores.

How does LMMR compare to existing retrieval techniques?

Traditional multi‑hop retrieval methods such as MDR, IRCoT or IIER focus on which documents to fetch next, but they usually retrieve whole passages without adjusting granularity. Maximal Marginal Relevance (MMR) tries to reduce overlap among selected chunks, yet it still treats each chunk as a fixed unit. LMMR differs by explicitly learning conditional granularity: the system decides whether a paragraph, a sentence, or even a phrase best satisfies the missing support for the current reasoning state.

In addition, LMMR’s training strategy leverages equivalent coarse and fine views that supply the same missing fact, providing a clear supervision signal that many prior works lack. The inclusion of relevant distractors and repeated information as contrasting examples further sharpens the reranker's ability to spot truly novel evidence.

Practical implications for developers and researchers

For creators building AI agents that must operate under strict token budgets – such as on‑device assistants, edge‑deployed chatbots, or real‑time robotics controllers – LMMR offers a concrete recipe to squeeze more reasoning power out of modest models. The method can be plugged into any pipeline that already performs retrieval‑augmented generation: replace the final retrieval step with the LMMR reranker/selector combo, and keep the downstream answer model unchanged.

Researchers interested in memory‑efficient reasoning can also use LMMR as a testbed for studying what granularity matters for different question types. The paper’s analysis (section 4.8) shows that questions requiring deeper relational chains benefit more from fine‑grained fragments, while those needing broader context still prefer paragraph‑level evidence. This insight can guide the design of curriculum data or adaptive token‑budget policies.

Exploring LMMR with Grafy

While Grafy does not yet host a built‑in LMMR module, the Reader app lets you load the arXiv PDF, ask follow‑up questions, and experiment with the retrieval ideas described in the paper. You can also use Chat to prototype a simple reranking workflow: feed a set of candidate paragraphs, ask the model to score the added information relative to an anchor, and manually enforce a token budget. This hands‑on approach mirrors LMMR’s two‑stage pipeline and helps you gauge its impact on your own small‑model agents.

For a visual representation of the retrieval chain, the Graph app can be used to map the sequence of selected sources and their granularity levels, turning the abstract algorithm into an interactive diagram.


Frequently asked questions

How does LMMR decide how many tokens to keep from each source?

LMMR uses a learned selector that adds candidates one by one, each time checking the cumulative token count. The process stops as soon as adding another candidate would exceed the predefined budget (e.g., 400400 tokens).

Does LMMR require fine‑tuning the answer model?

No. In the reported experiments the answer model was frozen (a 2B2\text{B} language model) and only the retrieval components were trained. This makes LMMR compatible with any existing small model.

Can LMMR be applied to domains other than multi‑hop QA?

The core idea—conditional selection of granularity under a token budget—generalizes to any task where a model must consume limited context, such as summarization, fact‑checking, or on‑device recommendation. Adapting the training pairs to the new task would be the primary step.


Source: Grafy Research Lab — LMMR: Learned Multigranular Memory Retrieval for Small Language Model Agents