Grafy News
Try Grafy free

CarryOn boosts long‑horizon LLM agents with selective context management

Researchers unveil CarryOn, a lightweight context manager that keeps only essential evidence for long‑horizon LLM agents.

Illustration for the news story: CarryOn boosts long‑horizon LLM agents with selective context managementSource: Grafy Research Lab

What is CarryOn and why does context matter?

Long‑horizon language‑model agents execute sequences of tool calls that can span dozens or hundreds of steps. As each step produces observations, tool outputs, and intermediate state, the total interaction history grows quickly. Feeding the entire history to the acting LLM inflates token costs and can slow down inference, while trimming the history risks losing the exact details an upcoming decision depends on. The new paper CarryOn: Learning What to Keep in Context for Long‑Horizon LLM Agents (arXiv, 2026) proposes a dedicated context manager that separates a compact execution state from a fine‑grained evidence store, allowing the agent to keep the right facts at hand without bloating the prompt. (according to the paper)

How does CarryOn select and retain evidence?

CarryOn introduces two core mechanisms. First, every observation is broken into evidence units that retain a reference to the original source (e.g., a tool response or a snippet of retrieved text). Second, a learned selector scores these units based on the current control state and predicts which ones are likely to be needed in the next action. An adaptive recent retention policy ensures that the most recent interactions stay in the prompt, while older but still relevant evidence can be re‑introduced on demand. Crucially, the selector operates without invoking an additional LLM call for context management, keeping the overhead low.

How was CarryOn evaluated?

The authors benchmarked CarryOn on three established suites that test multi‑step reasoning:

  • AppWorld – a collection of simulated app‑based tasks such as calendar management and payment processing.
  • OfficeBench – office‑automation scenarios involving spreadsheets, emails, and document editing.
  • Tau2 Telecom – customer‑service diagnosis tasks in a telecom setting.

Each benchmark was run with three different LLM “Actors” (the models that actually generate actions), allowing the authors to assess cross‑model generalisation. Baselines included a naïve Full History approach, several pruning and summarisation methods (e.g., SWE‑Pruner, LLMLingua‑2), and recent fine‑tuned memory‑augmented agents.

What were the quantitative results?

CarryOn achieved a macro‑averaged task success of 75.7% across all benchmarks and actors. This is 6.7 percentage points higher than the strongest baseline (SWE‑Pruner) and 13.3 points above the Full History configuration. The paper also reports two resource‑aware metrics:

  • SαS_{\alpha} – a success‑efficiency score that rewards high completion rates while penalising token and API‑call costs. CarryOn recorded the highest S0.8S_{0.8} among all methods.
  • NCSNCS – normalised cost per success, measuring the average resources spent per successful task. CarryOn obtained the lowest mean NCSNCS at 0.792.

These numbers indicate that CarryOn not only improves raw success but does so with a smaller computational footprint, a crucial factor for production‑scale agents.

How can Grafy users experiment with CarryOn concepts?

Grafy does not ship CarryOn as a built‑in feature, but the platform provides the tools needed to prototype its ideas. Researchers can use Grafy Reader to ingest the CarryOn paper, ask the model to extract evidence‑unit definitions, and experiment with selector prompts. The Grafy Graph app lets you visualise a branching workflow where a compact state node is linked to evidence nodes, mirroring CarryOn’s separation of state and evidence. Developers building their own agents can also leverage Grafy Code to implement a lightweight selector model and test adaptive retention policies within a sandboxed environment. By combining these apps, users can explore selective context management without leaving the Grafy workspace.


Source: https://arxiv.org/abs/2611.17808

Frequently asked questions

How does CarryOn avoid extra LLM calls for context management?

The selector model runs as a separate, lightweight component that scores evidence units based on the current state. Because its predictions are cached and only the selected units are added to the prompt, no additional generation calls are needed to trim or summarise the context.

Does CarryOn require fine‑tuning the acting LLM?

No. CarryOn is an external manager that works with any off‑the‑shelf LLM Actor. The paper emphasises that the approach does not modify the underlying model, making it compatible with a wide range of providers.

What metrics should I track when testing my own context manager?

The authors introduced two complementary measures: SαS_{\alpha}, which balances task success against token and call costs, and NCSNCS, the normalised cost per successful task. Monitoring both gives insight into effectiveness and efficiency.


Source: Grafy Research Lab — CarryOn: Learning What to Keep in Context for Long-Horizon LLM Agents