Grafy News
Try Grafy free

SkillBuffBench Reveals Real‑World Impact of Agent Skills Across Harnesses and Memory Modes

A new benchmark called SkillBuffBench measures how file‑based Agent Skills affect coding agents in deployment.

Illustration for the news story: SkillBuffBench Reveals Real‑World Impact of Agent Skills Across Harnesses and Memory ModesSource: Grafy Research Lab

SkillBuffBench, a benchmark introduced by researchers from Oxford, Imperial College London, Nanyang Technological University and BraneMatrix AI, quantifies how file‑based Agent Skills influence coding agents when they are deployed in realistic conditions. The work matters because today every major agent harness ships a library of Skills, yet developers have little empirical evidence about whether those Skills actually help when the agent must discover and load them on its own.

What is SkillBuffBench and why it matters?

SkillBuffBench is a paired evaluation protocol that treats Skill access as a three‑level factor – no Skill, a curated Skill supplied by a human, and automatic invocation from an installed library – and crosses this with two memory modes (off and on). According to the paper, the benchmark runs every configuration under four different agent harnesses and multiple large language models, including proprietary offerings from Anthropic, Google and OpenAI as well as open‑weight models from Chinese providers. By measuring performance on research‑oriented tasks, the benchmark aims to answer whether Skills provide a tangible benefit in the messy conditions of real deployments.

How does the benchmark evaluate agent Skills?

The authors designed a full factorial experiment. Each run combines one of the three Skill conditions with one of the two memory modes, then executes the pair on a given task under each of the four harnesses and each selected model. Every cell of this design is repeated three times, and a deterministic verifier checks the final output against versioned inputs. Scores, uncertainty (via paired bootstrap confidence intervals), wall‑clock time, and agent‑round counts are reported separately rather than being folded into a single metric. This rigorous protocol isolates the contribution of Skills from that of memory, harness implementation, and model capability.

What research questions does SkillBuffBench address?

The benchmark targets three open questions identified by the authors:

  1. Curated access vs. deployment – Prior work measured the upper bound where a human pre‑selects the appropriate Skill. SkillBuffBench instead evaluates automatic discovery from a full library, exposing the cost of finding the right Skill.
  2. Memory as a confounding factor – Modern harnesses persist notes, project instructions and prior results. By toggling memory on and off, the benchmark can determine whether a Skill still adds value when the agent already remembers a successful prior attempt.
  3. Model‑harness interaction – Since Skill loading is implemented by the harness, not the model, the benchmark runs the same models under different harnesses to separate model effects from harness effects.

What tasks and Skills are included?

The current suite contains four expert‑curated research problems, two from materials science and two from biology:

  • SIMS data analysis – extracting oxygen transport coefficients from isotope‑exchange depth profiles.
  • X‑ray diffraction (XRD) refinement – identifying crystalline phases and refining unit‑cell parameters.
  • Regulatory target prioritisation – ranking candidate gene targets for experimental follow‑up.
  • Spheroid imaging – reconstructing three‑dimensional cell distribution from microscopy stacks. Each task is paired with the specific Skill that domain experts use to solve it, such as specialized data‑processing scripts or analysis pipelines. The suite is designed to grow, with new tasks and Skills added on a rolling basis.

How can developers experiment with similar setups in Grafy?

Grafy Code provides an AI‑assisted coding studio where you can assemble your own agent harnesses, load custom Skill directories and control session memory. By reproducing a subset of the SkillBuffBench factorial design, you can:

  1. Create a Skill library – Place procedural scripts, templates and description files in a folder that Grafy Code can expose to the agent.
  2. Toggle memory – Use Grafy Code’s project‑level persistence to enable or disable memory between runs, mirroring the benchmark’s memory modes.
  3. Swap models – Grafy Chat lets you run the same prompt against multiple models side‑by‑side, making it easy to compare proprietary and open‑weight options.
  4. Automate verification – Write deterministic checks (e.g., unit‑tests or output diffs) that act as the benchmark’s verifier, ensuring reproducibility. While SkillBuffBench itself is still collecting results, Grafy Code offers a practical sandbox for developers to explore how Skills and memory affect agent performance in their own domains.

To explore this yourself, open Grafy Code: an AI coding studio.

Frequently asked questions

What distinguishes a curated Skill from an automatically invoked Skill?

A curated Skill is supplied by a human who has already identified the most relevant package for the task. An automatically invoked Skill is selected by the agent from the full installed library, requiring the agent to discover and load the correct package on its own.

Does enabling memory always improve an agent’s performance?

Not necessarily. SkillBuffBench measures performance with memory both on and off, because persistent notes can sometimes replace the need for a Skill, but they can also introduce stale information that harms the current task.

Can I run SkillBuffBench on Grafy Code without access to the proprietary models?

Yes. The benchmark’s design runs every model under every harness, so you can evaluate open‑weight models available through Grafy Chat while still exercising the same Skill and memory conditions.


Source: Grafy Research Lab — SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory