Researcher builds playable Pokémon Red world model with agent harness

A developer has successfully trained a diffusion-based world model capable of simulating playable Pokémon Red environments, paired with a dual-layer agent architecture. This project demonstrates how neural networks can move beyond static image generation to create interactive, evolving digital worlds by predicting subsequent frames based on user input.

How does the diffusion world model work?

According to a report on Reddit ML, the world model utilizes a diffusion process—the same underlying technology used in modern image generators—to simulate the game frame by frame. The model was trained on recorded gameplay footage of Pokémon Red. It operates using a 16.7-million parameter architecture, specifically a DIAMOND-style EDM U-Net, processing images at a resolution of 72×80 in grayscale.

To maintain consistency, the system functions by taking the previous four screens and a specific button press as inputs to predict and draw the next frame. To prevent the simulation from drifting too far from reality, the model re-syncs with the actual emulator every 10 steps using four real frames; otherwise, it continues to run autonomously based on its own internal "dreams." This allows a user to navigate the simulated environment using a keyboard.

What is the System 1/System 2 agent architecture?

Beyond the visual simulation, the developer implemented a game agent harness designed with a specific cognitive split. This approach follows the System 1/System 2 framework used by Krafton for their PUBG Ally system. This architecture separates fast, instinctive reactions from slower, more deliberate reasoning.

In this setup, the "System 2" component acts as a planner. According to the source, this layer can read the game's RAM to make high-level strategic decisions. Rather than just reacting to immediate visual stimuli, the System 2 layer can set long-term goals, such as traveling to a specific location, healing, or grinding to reach a certain level. It is also capable of performing information lookups, such as checking the Pokédex, to inform its decision-making process.

Why is this distinction important for AI agents?

Traditional AI agents often struggle to balance immediate interaction with long-term planning. By separating the reactive layer (System 1) from the deliberative layer (System 2), developers can create agents that are both responsive to the environment and capable of complex, multi-step reasoning. This mimics human cognition, where we handle routine tasks automatically while dedicating conscious thought to difficult problems.

In the context of this Pokémon project, the System 1 layer handles the immediate mechanics of the game, while the System 2 layer manages the overarching objectives. This dual-layer approach is a significant step toward creating more autonomous and capable agents that can operate within complex, rule-bound environments like video games or real-world simulations.

How does this relate to game development and AI?

This research highlights a shift toward "world models," where the AI does not just play a game but understands the underlying rules of the environment well enough to simulate it. This has profound implications for how games might be tested, how NPCs (non-player characters) are developed, and how synthetic training data can be generated for other AI systems.

For those interested in the intersection of AI and interactive media, exploring the brief-to-playable-game workflow can provide insight into how generative AI is already being used to move from concepts to functional software. While the Pokémon project uses a diffusion model to "dream" the world, modern tools are increasingly able to translate text descriptions into structured, playable environments.

To explore this yourself, open Game Studio: turn a brief into a playable game.

Frequently asked questions

What technology drives the Pokémon Red world model?

The model is driven by a diffusion process using a 16.7-million parameter DIAMOND-style EDM U-Net. It learns by analyzing recorded gameplay and predicts the next grayscale frame (72×80 resolution) based on the previous four frames and a user's button input, effectively "dreaming" the game state.

How does the agent's planning layer function?

The planning layer, referred to as System 2, functions by accessing the game's RAM to make strategic decisions. This allows the agent to move beyond simple reactions and instead pursue complex goals like leveling up, traveling to new areas, or utilizing the Pokédex for information.

What is the purpose of the System 1/System 2 split?

The split is designed to separate fast, reactive behaviors from slow, deliberative reasoning. By using this architecture, the agent can handle immediate environmental interactions through a reactive layer while simultaneously managing long-term objectives and complex logic through a dedicated planning layer.

---

Source: Reddit ML — I trained a playable diffusion world model of Pokémon Red, plus a game agent harness on the same System 1/2 split as Krafton's PUBG Ally. Can this be ICLR-workshop worthy? [P]