Grafy News
Try Grafy free

Soteria introduces verified planning to keep LLM agents safe

Researchers unveiled Soteria, a framework that uses formally verified hierarchical planning and runtime enforcement to keep LLM agents both useful and safe.

Illustration for the news story: Soteria introduces verified planning to keep LLM agents safeSource: Grafy Research Lab

LLM agents that call external tools must balance finishing a user’s request with obeying safety policies. A new framework called Soteria aims to guarantee both goals by combining formal verification with runtime enforcement, according to the paper accepted as a NeurIPS 2026 oral presentation.

What problem does Soteria address?

Multi‑step tool use by LLM agents creates a tension between utility (completing the task) and safety (respecting domain constraints). Existing methods often block unsafe actions, which can leave the agent without a way to continue, or they ignore safety to preserve progress. Soteria is introduced to reconcile these competing objectives.

How does Soteria reconcile safety and utility?

Soteria builds a hierarchical plan before execution. The top‑level plan is verified against a formal specification of allowed behaviours. During runtime, an enforcement layer monitors each tool call, ensuring it conforms to the verified plan. If a potential violation is detected, the system can intervene with a safe alternative rather than simply halting.

What is verified hierarchical planning?

The framework first generates a symbolic plan ss that outlines the sequence of tool calls required to satisfy the user’s request. This plan is checked using formal methods to prove that, under the given specifications, no step can breach safety constraints. Only plans that pass this proof are allowed to run, providing a mathematical guarantee of compliance.

What are the practical implications for developers?

Developers can embed Soteria‑style verification into their agent pipelines to obtain provable safety without sacrificing task completion. The approach also offers a clear recovery path when an unsafe action is blocked, because the verified plan includes fallback steps. This could reduce the need for ad‑hoc safety heuristics and extensive manual testing.

How can Grafy users explore Soteria concepts?

While Grafy does not yet ship a built‑in Soteria engine, creators can experiment with the same ideas using existing tools:

  • Use the Graph app to visualise hierarchical plans as branching trees, annotating each node with safety predicates.
  • Leverage Reader to import the Soteria paper, ask the AI to summarise the verification methodology, and extract code snippets for custom enforcement logic.

These workflows let developers prototype verified planning pipelines and see how runtime enforcement could be layered on top of their own LLM agents.

Source

The findings are detailed in the pre‑print "Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents" (arXiv:2611.17880) by Mike He, Ankush Desai, Sharad Malik, and Aarti Gupta.

Frequently asked questions

Does Soteria guarantee that an LLM agent will never violate a safety policy?

Soteria guarantees that any plan that passes its formal verification will respect the specified safety constraints. However, the guarantee holds only for the parts of the system that are covered by the formal model; unforeseen bugs in the underlying tool implementations could still cause violations.

Can Soteria be used with any LLM model?

The framework is model‑agnostic; it operates on the sequence of tool calls generated by the LLM. As long as the agent produces a plan that can be expressed in the verification language, Soteria can be applied.

How does Soteria handle a situation where the safest action blocks progress?

Soteria’s hierarchical planning includes fallback branches that are also verified. If the primary action is unsafe, the runtime enforcer switches to a pre‑approved alternative, allowing the agent to continue toward the goal without breaching safety rules.


Source: Grafy Research Lab — Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents