guardscope Framework Quantifies Runtime Control Coverage for LLM Agents

Large‑language model (LLM) agents can trigger actions on external systems, and enterprises rely on runtime controls—such as agent gateways—to enforce security policies. A new pre‑print titled guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems presents the first framework that quantifies exactly how much of an agent’s behavior these controls actually see.

What is guardscope and why it matters?

According to the arXiv paper (Toyoda et al., 2026), guardscope is a measurement framework that makes the coverage of runtime controls verifiable against a kernel‑level ground truth. The importance lies in the fact that a control’s placement determines which actions it can observe; without a systematic way to assess this, enterprises cannot reliably claim that their safeguards are comprehensive. The framework therefore fills a gap in AI security research by turning an opaque property—"coverage"—into a measurable metric.

How does guardscope measure control coverage?

Guardscope operates in two stages. The static stage parses a control’s source code to predict which of the 15 enumerated action paths the control can potentially inspect. The empirical stage then runs a fixed workload of 15 representative action instances on a deployment that implements one of eight topology designs. It records the control’s decisions and reconciles them with kernel‑level observations collected via an eBPF sensor (Tracee). Each instance is classified as checked, bypassed, or prevented, providing a concrete coverage score for that control‑design pair.

What are the 15 action paths and 8 topology designs?

The authors enumerate fifteen distinct routes an agent’s decision can travel from the harness runtime to its effect on external resources. These range from direct tool‑call traffic that passes through a network gateway, to local child‑process execution, to raw socket connections opened by the agent code itself. In parallel, they define eight topology designs that capture where a runtime control might be placed, including:

1. A gateway in front of the harness (network‑level interception).
2. In‑process enforcement inside the harness.
3. OS‑level enforcement via eBPF.
4. Hybrid designs that combine gateway and kernel monitoring. For each path‑design pairing, the paper derives whether the design can observe the action, producing a matrix that highlights coverage gaps—for example, a gateway cannot see actions launched by a local child process that bypasses the network stack.

What did the evaluation reveal about real controls?

The authors evaluated guardscope with two agent harnesses—Pi and OpenClaw—and two production‑grade runtime controls:

• Microsoft’s Agent Governance Toolkit (AGT), deployed as a gateway control.
• ActPlane, a kernel‑level control. A Tracee eBPF sensor supplied the ground‑truth record of every system call made by the agent processes. The experiments confirmed the theoretical matrix: the gateway control successfully inspected all tool‑call‑based paths but missed actions that originated from local processes or direct network connections. ActPlane, operating at the kernel level, captured a broader set of actions but lacked payload‑level details, limiting policy granularity. Across both harnesses, the coverage outcomes were stable, demonstrating that guardscope’s static predictions align with empirical results.

How can Grafy users apply these insights?

While Grafy does not currently ship a built‑in guardscope tool, the findings are directly relevant to creators, developers, and researchers who build or audit LLM agents on the platform. Users can:

• Inspect agent architectures with Grafy Reader (/reader) to load the full pre‑print and explore the detailed action‑path taxonomy.
• Prototype different control placements using Grafy Code (/code) to spin up lightweight harnesses and experiment with gateway‑style proxies or eBPF‑based monitors.
• Validate coverage by reproducing the static analysis step: copy the control’s source into a Grafy Code workspace, run a static‑analysis script, and compare the predicted paths against the 15‑path list from the paper.
• Iterate on policy design by leveraging Grafy Chat (/chat) to discuss trade‑offs between coverage and inspectable content with multiple model viewpoints. By integrating these steps, Grafy users can adopt a more security‑aware development workflow, ensuring that any runtime control they embed actually observes the actions they intend to protect.

Frequently asked questions

Does guardscope verify that a runtime control blocks unsafe actions?

Guardscope classifies each action instance as checked, bypassed, or prevented by reconciling the control’s decision log with kernel‑level observations. While it can confirm whether a control prevented an action, the framework does not evaluate the underlying policy’s correctness—only the coverage of the control’s enforcement point.

Can guardscope be used with proprietary agent platforms that are not open‑source?

The static analysis stage relies on access to the control’s source code to predict inspectable paths. For closed‑source controls, only the empirical stage is feasible: a workload can be run while the control logs its decisions, and the results can be compared to kernel‑level ground truth. However, without source visibility, the predicted‑vs‑actual gap cannot be fully quantified.

How does the number of action paths affect the complexity of measuring coverage?

Guardscope defines fifteen distinct action paths, each representing a different way an agent can affect external resources. Measuring coverage requires running a workload that exercises each path at least once, which can increase testing complexity. The authors mitigated this by designing a fixed workload of fifteen action instances—one per path—ensuring comprehensive yet manageable evaluation.

---

Source: Grafy Research Lab — guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems