Grafy News
Try Grafy free

Self‑Improving Visual Game Agents Learn On‑The‑Fly with Code and Model Policies

A new arXiv paper shows how agents can boost performance in unseen real‑time visual games using model‑as‑policy and code‑as‑policy methods.

Illustration for the news story: Self‑Improving Visual Game Agents Learn On‑The‑Fly with Code and Model PoliciesSource: Grafy Research Lab

Pre‑trained vision‑language models can interpret a game screen and suggest moves, but they still stumble when faced with a brand‑new title. A team of researchers from several Chinese institutions proposes a way for an agent to improve itself from just a handful of its own attempts, even while the game keeps running. The work, titled Think Offline, Act in Real Time: Self‑Improving Visual Game Agents through Code, appears on arXiv on 19 November 2026.

What problem does the research aim to solve?

Current visual game agents rely on large offline training corpora and struggle with out‑of‑distribution environments. When a player launches an unfamiliar game, the agent must either guess based on prior knowledge or spend time collecting data before it can act competently. The authors ask whether an agent can self‑improve in real time—using only the visual feedback it receives and a limited number of its own trial actions—without pausing the game. This capability would make agents far more adaptable for interactive entertainment, robotics, and any setting where the environment cannot be frozen for computation.

How do the authors let an agent improve while the game runs?

The key technical hurdle is that most AI models need several milliseconds to reason, while a real‑time game proceeds frame by frame. To bridge this gap, the researchers introduce a real‑time chunking mechanism. The game loop is divided into short execution windows during which the agent’s previously decided actions are committed and continue to run. Meanwhile, the model works on the next chunk of decisions in parallel. This pipelined approach ensures that the agent never stalls the game, yet still benefits from fresh reasoning on the latest visual input.

What are the two policy approaches explored?

The paper investigates model‑as‑policy and code‑as‑policy.

Model‑as‑policy treats the neural network itself as the decision maker. The authors train a 2 billion‑parameter controller that consumes the current screen, a short history of past frames, and any internal state produced by the chunking system. The controller outputs the next action directly, allowing rapid inference within each chunk. By continually updating its hidden state, the model can incorporate lessons from earlier mistakes without re‑training from scratch.

Code‑as‑policy flips the paradigm: instead of outputting an action, the model generates executable code that modifies the agent’s behavior. The generated script can adjust heuristics, rewrite parts of the policy logic, or call external tools. Because code can be compiled or interpreted instantly, the agent can apply sophisticated updates on the fly. The paper demonstrates that even a modest code‑generation model can produce useful patches after a few failed attempts, effectively turning the agent into a self‑modifying program.

Both methods share the same chunking infrastructure, but they differ in the granularity of the output—raw actions versus higher‑level programmatic changes.

What results do the experiments show?

The authors evaluate their approaches on three benchmark visual games that were unseen during pre‑training. In each case, the baseline model (no self‑improvement) quickly plateaus at a low success rate. With the model‑as‑policy chunking pipeline, performance improves by roughly 15 percentage points after only ten self‑play episodes. The code‑as‑policy variant achieves a comparable boost, sometimes surpassing the model‑as‑policy when the game’s mechanics benefit from rule‑like adjustments.

Importantly, the improvements happen while the game continues; there is no need to pause for offline fine‑tuning. The study also notes that the 2 B controller fits comfortably on a single GPU, making the approach feasible for consumer‑grade hardware. Limitations include occasional generation of syntactically invalid code and the dependence on a relatively stable visual feed—rapid screen flicker can confuse the chunking logic.

How can Grafy creators experiment with these ideas?

Grafy users interested in self‑improving agents have several practical entry points:

  • Grafy Code – The /code studio lets you prototype the code‑as‑policy loop. You can feed a screenshot to a language model, ask it to emit a short Python function that decides the next move, and then execute that function in a sandboxed environment. This mirrors the paper’s pipeline without needing a custom game engine.

  • Grafy Game Studio – With /game you can spin up a simple visual game (e.g., a maze or platformer) that streams frames to a model. By integrating the real‑time chunking pattern—committing the last action while the model processes the next frame—you can reproduce the model‑as‑policy setup described in the study.

  • Grafy Reader – For a deeper dive, load the arXiv PDF into /reader and ask the assistant to summarize sections, extract the algorithm pseudocode, or compare the two policy approaches. This speeds up understanding before you start coding.

  • Grafy Chat – If you want to test different pre‑trained vision‑language models side by side, the /chat interface lets you switch between OpenRouter, Vertex AI, and other providers to see which one reacts best to the visual input in your prototype.

By chaining these apps, a developer can build a full “self‑improving agent sandbox”: a game provides visual frames, a model predicts actions or code, Grafy Code runs the generated script, and Grafy Reader keeps the research context at hand.

What are the broader implications for AI agents?

The paper demonstrates that real‑time self‑modification is viable even with modest model sizes. This opens the door to agents that continuously adapt in dynamic environments—think NPCs that learn player tactics on the fly, robotics that refine control policies during a single deployment, or educational software that personalises feedback without offline retraining. The distinction between model‑level and code‑level adaptation also suggests a hybrid future where agents use fast neural inference for routine decisions and invoke code generation for strategic, rule‑based updates.

Frequently asked questions

How does the real‑time chunking mechanism avoid pausing the game?

Chunking divides the game loop into short windows. Actions decided in the previous window are locked in and continue to execute while the model works on the next window’s decision. This pipelined flow keeps the frame rate stable.

Can the code‑as‑policy approach generate unsafe or harmful code?

The authors acknowledge occasional syntax errors and recommend sandboxing the generated script. In Grafy Code you can run the output in an isolated environment that restricts file system and network access, mitigating safety concerns.

Is the 2 B controller required for all games?

The 2 B controller was trained to handle the visual complexity of the benchmark games. Simpler 2D titles may work with smaller models, while more demanding 3D environments could benefit from larger architectures. Grafy’s model‑routing system can automatically select a capable yet cost‑effective model.

Source: https://arxiv.org/abs/2611.24590


Source: Grafy Research Lab — Think Offline, Act in Real Time: Self-Improving Visual Game Agents through Code