Grafy News
Try Grafy free

Just‑Enough Exploration Tuning Cuts Tool Calls in Code‑Repo QA Agents

Researchers introduce JET, a fine‑tuning approach that trims unnecessary tool calls in repository‑level QA agents while keeping answer quality high.

Illustration for the news story: Just‑Enough Exploration Tuning Cuts Tool Calls in Code‑Repo QA AgentsSource: Grafy Research Lab

What is repository‑level question answering and why does tool overuse matter?

Repository‑level question answering (QA) asks an AI agent to answer natural‑language questions that require information spread across many files in a code base. The agent can call tools – such as a file‑searcher, a code‑reader, or a test runner – to collect snippets before formulating a final answer. In practice, each tool call adds the tool’s output to the prompt context, increasing token count and inference latency. When an agent continues to invoke tools after it already has enough evidence, the extra calls waste compute and raise costs without improving the answer.

How did the researchers identify over‑exploration?

The authors of the paper "JET: Just‑Enough Exploration Tuning for Efficient Tool‑Calling Agents in Repository‑Level Question Answering" observed that state‑of‑the‑art LLM agents often keep exploring long after the relevant file has been found. They demonstrated the issue on the SWE‑QA‑Pro benchmark by forcing the agent to answer after every step and scoring each intermediate answer. The score plateaued within a few steps, yet the agents kept calling tools for many more steps (see Figure 1 in the paper). The same pattern appeared in the expert trajectories used for supervised fine‑tuning (SFT): even the high‑quality demonstrations continued to call tools beyond the point where the answer quality stopped improving. This phenomenon, which they call over‑exploration, inflates inference cost and can hit token limits on long‑running sessions.

What is the JET fine‑tuning method?

JET (Just‑Enough Exploration Tuning) tackles over‑exploration directly during training. Its pipeline consists of three stages:

  1. Golden step labeling – For each training trajectory, JET finds the earliest step where the accumulated context is sufficient to produce a high‑scoring answer. This step is called the golden step.
  2. Rationale insertion – At the golden step, JET inserts a short rationale that explains why the current information is enough to answer. The rationale is phrased as a natural‑language comment that the model can learn to emit.
  3. Training with a halting weight – The agent is fine‑tuned on the truncated trajectory, but the loss on the rationale token sequence receives a higher weight. This encourages the model to recognise the halting signal and to stop invoking tools once the rationale is generated. By trimming the trajectory and emphasizing the decision to answer, JET teaches the model to stop earlier without needing an external budget prompt or a separate autorater at inference time.

How does JET perform compared to existing techniques?

The authors evaluated JET on two public repository‑level QA benchmarks using two open‑weight 8B models. Results can be summarised as follows:

  • On self‑generated trajectories (agents that collected their own data), JET reduced tool calls by 11 % to 33 % while preserving the benchmark score.
  • On expert trajectories (the SFT data), JET cut tool calls by 18 % to 28 % compared with standard SFT, and retained most of the performance gain that SFT provides over the base agent.
  • In every setting, JET lowered the overall inference cost because fewer tool outputs were appended to the context. The paper also compared JET against inference‑time baselines such as Budget Tracker and an autorater that decides sufficiency on the fly. Those methods either failed to reduce calls or added extra model invocations, making them less efficient for the long‑horizon, free‑form answering tasks typical of repository‑level QA.

Applying JET insights with Grafy

While JET itself is a research‑level fine‑tuning recipe, its core ideas can be experimented with using Grafy’s existing tools:

  • Grafy Code – Developers can prototype a repository‑level QA agent, plug in a file‑search tool, and customise the prompting logic. By logging each step, you can manually identify the golden step where the answer stabilises and add a rationale comment, mimicking JET’s truncation strategy.
  • Grafy Reader – The paper itself is available on arXiv; you can load the PDF into Reader to ask follow‑up questions about the methodology, explore the figures, or retrieve specific equations without leaving the workspace.
  • For teams that already have fine‑tuning pipelines, the JET loss‑weighting technique can be implemented by adjusting the training script to up‑weight tokens that belong to the rationale segment. Grafy’s Graph view can then visualise the original versus truncated trajectories, making it easier to audit the impact of the change. By combining these apps, creators and researchers can experiment with “just‑enough” stopping criteria in their own code‑assistant projects, potentially reproducing the cost savings reported in the study.

Source: https://arxiv.org/abs/2611.22539

Frequently asked questions

How does JET decide when the gathered information is sufficient?

JET scans each training trajectory to find the earliest step where the answer score stops improving – the "golden step." It then treats that step as the point where enough context has been collected to answer the question.

Does JET require a separate model at inference time?

No. JET embeds the stopping decision into the fine‑tuned agent itself by training it to emit a rationale and halt tool calls. Unlike Budget Tracker or autorater approaches, it adds no extra model calls during inference.

Can JET be used with models larger than 8B?

The paper demonstrates JET with open‑weight 8B models, but the method is model‑agnostic. Any LLM that can be fine‑tuned on trajectory data could apply the same truncation and halting‑weight strategy.


Source: Grafy Research Lab — JET: Just-Enough Exploration Tuning for Efficient Tool-Calling Agents in Repository-Level Question Answering