Adding or Removing Context Changes LLM Coding Agent Costs More Than Expected
A new arXiv study measured over 20,000 runs of open‑source coding agents and found that retrieved code cuts searches but raises total cost.
Source: Grafy Research LabWhat did the researchers set out to measure?
The paper titled Context Has a Price investigates whether giving a coding agent more context – such as retrieved code snippets – or trimming old tool outputs actually reduces the monetary cost of completing a software‑fix task. Using more than 20,000 runs on three benchmarks – SWE‑bench Verified, SWE‑bench, and Loc‑Bench – the authors tracked three token streams separately: uncached input, cached input, and output tokens. They then applied a simple cost model where
is the count of uncached input tokens, the cached input tokens, the output tokens, the price of a cached token relative to an uncached token, and the price of an output token relative to an uncached token. According to the authors, this accounting captures the real‑world billing practices of modern LLM serving platforms.
How does adding retrieved code affect search behavior and total spend?
In the "evidence‑package" experiments the agents were supplied with code snippets that were likely relevant to the current bug. The authors report that this addition reduced the number of read and search commands by 18 %–34 % across all three benchmarks and for every model tested, including Qwen. However, when the cached‑input price was set to 0.1 (i.e., cached tokens cost one‑tenth of fresh tokens), the overall cost rose by 33 %–42 %. The increase persisted even when the agents could execute the repository’s test suite, and the resolution rate – the proportion of tasks solved – showed no statistically significant improvement. The authors conclude that the token savings from fewer searches do not translate into a lower bill because each added token is re‑read at every later turn, and cache invalidation can force re‑payment of previously cached prefixes.
What happens when old observations are removed from the prompt?
The study also evaluated several masking and summarisation strategies that truncate or replace earlier tool outputs. When the cache discount was absent (), masking all but the ten most recent observations saved 19 %–25 % of the cost. In contrast, with a modest cache discount () the same masking strategy increased cost by 116 % for a 27B Coder model and by 87 % for another large model. The sign flip was reproduced in SWE‑agent and GLM‑4.7‑Flash, showing that the effect is not model‑specific. A "break‑even" masking rule that halves the premium and applies when cached tokens cost ≥ 0.25 of uncached tokens delivered savings of 10 %–26 %. Summarisation and reflection‑style removal methods showed mixed results, sometimes improving function accuracy but often raising cost.
What are the practical takeaways for developers of coding agents?
- Token count alone is not a reliable proxy for cost. Because cached prefixes are priced lower, adding a few useful tokens may be cheap, but if those tokens cause cache invalidation later, the net bill can rise sharply.
- Retrieval helps the agent’s search efficiency but does not guarantee cheaper runs. Teams should weigh the value of reduced search steps against the higher token‑reuse cost, especially when operating under a cache‑discount pricing model.
- Context‑trimming policies must be price‑aware. A blanket rule such as "keep only the last ten observations" can backfire if the serving backend heavily discounts cached input. Adaptive masks that consider the current cache price – like the break‑even rule described in the paper – tend to be more cost‑effective.
- Report token‑savings and cost‑savings separately. The authors argue that future benchmark papers should disclose both metrics, as they can move in opposite directions.
How can Grafy users explore these findings?
Grafy’s Code app lets you build and run autonomous coding agents in a sandboxed environment, mirroring the experimental setup of the study. You can:
- Use the built‑in prompt editor to prepend retrieved code snippets and observe how many search calls the agent makes.
- Apply the "masking" feature by programmatically deleting older tool outputs from the conversation history before sending the next request.
- Switch between models with different pricing tiers via the Chat app to emulate cached‑input discounts (e.g., choose a cheaper model for the first turn and a more capable one for later steps).
- Track token usage with the Graph app, which visualises the growth of the prompt over time and highlights cache‑hit prefixes. For a deeper dive into the paper itself, the Reader app can ingest the PDF and let you ask natural‑language questions about the methodology or results.
Source: https://arxiv.org/abs/2611.22238
Frequently asked questions
Does prefix caching always make LLM calls cheaper?
No. Prefix caching reduces the price of tokens that are reused, but if a change to the prompt invalidates a cached prefix, the subsequent tokens must be paid at the full uncached rate. The study shows that, depending on the cache‑discount factor, adding or removing context can actually increase total cost.
When should I add retrieved code to a coding agent’s prompt?
Add it when the extra code meaningfully narrows the search space and you are operating under a pricing regime where cached‑input discounts are small (e.g., ). If cached tokens are heavily discounted, the cost benefit may be outweighed by the extra token‑reuse charges.
How can I test context‑management strategies with Grafy?
Create a coding agent in the Code studio, run it on a benchmark repository, and experiment with different prompt‑editing hooks (retrieval, masking, summarisation). Use Graph to visualise token growth and Chat to swap models that simulate different cache pricing, then compare the total token‑based cost reported by Grafy’s usage metrics.


