Grafy News
Try Grafy free

PACT narrows plan‑action gaps in GUI agents, boosting mobile and desktop task success

Researchers introduce PACT, a training framework that preserves correct plans and repairs actions in GUI agents.

Illustration for the news story: PACT narrows plan‑action gaps in GUI agents, boosting mobile and desktop task successSource: Grafy Research Lab

What problem do GUI agents face?

Current GUI agents often generate a logical plan in their internal reasoning but then emit an action that does not follow that plan. According to the authors of the paper, this "execution gap" can derail multi‑step workflows because a single mismatched click or command aborts the entire task. The issue is especially acute in outcome‑supervised reinforcement learning, where a negative advantage is broadcast to all tokens in the response, unintentionally penalising the correct reasoning alongside the faulty action.

How does PACT address the execution gap?

PACT (Plan‑Action Consistency Training) introduces two complementary mechanisms while keeping the original scalar action reward unchanged. First, a rule‑based selector uses annotated reference actions to locate responses that contain a reference‑supported plan but an incorrect final action. The identified plan span is then shielded from negative policy‑gradient updates – a process the authors call reference‑aligned plan protection. Second, the framework creates a plan‑aligned action repair step: it pairs the model’s erroneous action suffix with a reference‑derived correct suffix, applying a stop‑gradient preference surrogate (inspired by Direct Preference Optimization) to favor the correct suffix. Both steps operate only during training; inference remains a single‑pass reasoning‑then‑action flow without extra reflection.

Figure 1: Execution gaps and targeted correction. Top: Both models continue the same supplied Pinterest reasoning; O launches the app, while PACT clicks the intended + co…
Figure 1. Execution gaps and targeted correction. Top: Both models continue the same supplied Pinterest reasoning; O launches the app, while PACT clicks the intended + control. Bottom: With independently generated reasoning and actions on Moon, O states a downward swipe but emits upward finger motion, while PACT executes the downward swipe. On-screen circles mark the click or gesture start; swipe arrows sho… Figure 1 of “PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents” by Zhuohan Cai, Zhangxuan Gu, Runze Li et al. (arXiv.org), CC BY 4.0.

What results does PACT achieve?

In the authors' experiments on the AndroidWorld benchmark, PACT lifts task success from 60.34% to 68.97%, an 8.63‑percentage‑point gain over the outcome‑only baseline. Rule‑based plan‑action consistency on the AndroidControl High suite improves from 83.71% to 89.74% (+6.03 pp). Extending the method to desktop environments (the OSWorld benchmark, comprising 369 tasks) raises success from 40.92% to 44.17% (+3.25 pp). Additional diagnostics show that when the plan is fixed, the repaired actions execute correctly, and that protecting plans during free generation retains more valid plans while still allowing joint repairs.

Why does this matter for developers and researchers?

The credit‑assignment conflict highlighted by the paper is a core obstacle for training reliable agents that must both think and act correctly. By disentangling plan preservation from action correction, PACT offers a principled way to improve consistency without redesigning the reward structure. For developers building AI assistants, automation scripts, or multimodal agents that interact with GUIs, the technique promises higher completion rates and fewer costly re‑tries. Researchers can also reuse the reference‑aligned masking and preference‑based repair ideas in other domains where reasoning and execution are tightly coupled, such as robotic manipulation or code generation.

How can you explore PACT with Grafy?

While Grafy does not yet provide a built‑in PACT trainer, the paper is fully accessible through Grafy Reader (link: /reader), where you can upload the PDF, ask the model to summarise sections, or query specific implementation details. If you want to experiment with the underlying concepts, you can use Grafy Chat (link: /chat) to prompt a model to generate a reasoning‑action pair and then manually apply a plan‑protection mask or a preference‑based repair step in a custom script. The modular nature of Grafy’s Code studio also lets you prototype the gradient‑masking logic using the provided code snippets from the appendix.

What are the limitations and next steps?

The authors acknowledge that PACT relies on high‑quality reference actions to identify execution gaps, which may be scarce in real‑world data. Moreover, the current rule‑based selector is conservative and could miss subtle mismatches where the plan is only partially correct. Future work could explore learned selectors, finer‑grained token‑level rewards, or integration with self‑reflection loops that allow agents to revise their own actions at inference time. Extending the approach to more complex, multi‑modal environments (e.g., web browsers or VR interfaces) is also an open research direction.


Source: https://arxiv.org/abs/2611.22591


Source: Grafy Research Lab — PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents