Decision Models Can Teach Even When They Act Poorly, New Study Shows

Decision models that answer a question with a typed output—such as a probability distribution over actions—can be used not only to pick the next move for an agent but also to evaluate past actions and to teach policies via reinforcement learning. A new paper from the Grafy Research Lab demonstrates that a model that performs poorly when acting can still be a highly effective teacher, provided it is queried in the right way.

---

What are decision models and why do they matter?

Decision models are pretrained systems that return structured answers (e.g., a list of options with associated probabilities) instead of free‑form text. According to the paper, this format makes them “quick and cheap” to run, allowing an LLM‑based agent to call the same model twice: once to choose an action (actor role) and once to evaluate an already‑taken action (judge role). The evaluations can then be turned into step‑level supervision for reinforcement‑learning agents, which otherwise rely on sparse outcome rewards that assign the same credit to every step in a trajectory.

How do decision models perform as actors?

The authors tested three dedicated decision models (JEV, Decider v1.2, GPT Decisions) and three general LLMs on the ALFWorld benchmark, which includes tasks that require ordered subgoals (e.g., heat a mug before placing it on a table). All six models struggled with ordered subgoals: JEV completed only 14.7 % of such tasks, compared with 70.8 % when the subgoals were independent. The failure is not due to lack of knowledge; when the easier subgoal is removed from the goal description, the same model’s probability of selecting the correct action jumps from 14.6 % to 77.6 %. Adding a dependency hint changes success by at most ±5 %, indicating that the core difficulty is anticipating that an easy action leaves the overall task incomplete.

Can the same models serve as judges?

When the models are asked to judge recorded trajectories, two factors change the outcome: the amount of context visible to the model (only the prefix up to the action, or the full hindsight trajectory) and the question asked (whether the task will ultimately succeed – Value – or whether the action helped achieve the goal – Progress). Figure 1 shows the three query modes (Actor, Judge, Teacher) applied to a simple “put a heated mug on a table” task. The study finds that seeing the whole trajectory (hindsight) dramatically improves a model’s ability to credit useful actions. Moreover, asking the Progress question separates helpful subgoal‑completing actions from no‑op actions, producing an average score gap of 0.36, whereas the Value question yields only a 0.05 gap because every step shares the same outcome label.

Do judges make good teachers for reinforcement‑learning agents?

Using the same decision models as teachers, the authors added the centered judge scores to the outcome advantage in a GRPO‑style RL algorithm. On ALFWorld, the JEV teacher raised a Qwen2.5‑1.5B policy’s success rate from 72.8 % (outcome rewards only) to 82.9 % when the Progress question was used with hindsight context. The same setting on the WebShop benchmark lifted success from 56.8 % to 71.4 %. Notably, the teacher‑enhanced 1.5 B policy outperformed a 7 B policy trained with outcome rewards alone, underscoring the efficiency of step‑level supervision from a frozen decision model.

The gains are largest on tasks with ordered subgoals, where the teacher‑augmented policy reaches 86.8 % success—an improvement of 6.9 % to 16.2 % over other query configurations—despite the underlying model’s low actor performance on those very tasks.

Practical takeaways for developers using Grafy

• Assess supervisors by judging ability, not just acting skill. When selecting a model to supervise RL training, run a quick evaluation of its Judge performance on recorded trajectories (e.g., via the Grafy /reader app to load the paper and the Grafy /chat app to query the model).
• Prefer hindsight context and the Progress question. Providing the full episode and asking whether each action contributed to the goal yields the strongest teaching signal.
• Leverage cheap API calls. The authors report that step‑level supervision costs roughly 22–30 USD per training run, far cheaper than labeling each step or training a learned critic.
• Integrate with Grafy’s workflow. You can store the roll‑out trajectories in the Grafy /files app, run the teacher scoring via Grafy /chat (selecting the decision model of interest), and visualize the resulting policy improvements in Grafy /graph.

For readers who want to explore the paper in depth, the Grafy /reader app can ingest the PDF directly from the arXiv link, allowing you to ask follow‑up questions about the methodology or replicate the experiments with your own models.

---

Source: https://arxiv.org/abs/2611.17167

To explore this yourself, open Reader: read, search and question PDFs and papers with AI.

Frequently asked questions

Does a decision model need to act well to be a good teacher?

No. The study shows that even models with low actor success (e.g., JEV at 14.7 % on ordered subgoals) can dramatically improve policy performance when used as teachers, provided they are asked the right hindsight question.

Which query configuration yields the strongest teaching signal?

The combination of hindsight context (the full trajectory) and the Progress question (whether the action helped achieve the goal) consistently produced the largest gains across both ALFWorld and WebShop.

How much does step‑level supervision with a decision model cost?

The authors estimate 22 to 30 USD of API queries per full training run, which is substantially cheaper than manually labeling each step or training a separate critic model.

---

Source: Grafy Research Lab — Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents