NVIDIA Cosmos 3 Adds Four Open Models for World Simulation

NVIDIA has added four open-weight checkpoints to the Cosmos 3 family for interactive world simulation, according to the Hugging Face Blog. The release matters because it gives robotics and autonomous-driving researchers shared, adaptable starting points for generating synthetic scenes, streaming observations, and studying how policies respond when actions change what a system sees next.

What are the four new Cosmos 3 checkpoints?

According to the Hugging Face Blog, the four checkpoints are all post-trained examples of Cosmos 3 Nano with 16B trainable parameters, released under OpenMDW1.1 for commercial and non-commercial use under its terms. They are Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer, and Cosmos3-Nano-Sim-Bimanual.

Each checkpoint addresses a different link between controls, actions, and generated observations. Transfer-Auto generates offline multi-sensor driving data. Sim-Auto streams synchronized camera and LiDAR observations for driving simulation. Sim-Transfer turns simulator controls and a text prompt into photorealistic RGB video. Sim-Bimanual predicts video from robot or camera actions for bimanual manipulation. Together, according to the source, they are intended as complementary ways to build environments that can respond across changing scenes and viewpoints, rather than as a single general-purpose video generator.

How does Transfer-Auto generate driving data?

Cosmos3-Nano-Transfer-Auto is described as a bidirectional model for offline synthetic driving data generation. According to the Hugging Face Blog, it can generate synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB and LiDAR generation. Per-camera World Scenario Map controls specify scene structure and traffic motion, while text prompts guide appearance. Developers can vary weather, lighting, and time of day to expand training and evaluation coverage, and optional initial RGB images can anchor scene appearance. For LiDAR, aligned HD-map range-map controls are used.

The checkpoint also serves as the foundation for adapting Cosmos to a specific sensor rig and driving domain. The recommended path is to post-train Transfer-Auto on local data to accommodate a camera layout, LiDAR configuration, and target environment. The adapted model can then be used directly for offline data generation or undergo causal training and few-step distillation for fast streaming simulation, which is the development path used to create Sim-Auto. One embedded demo video is included in this section of the source to illustrate generated driving output: demo video.

How does streaming simulation work for driving and robots?

Sim-Auto is the causally distilled streaming variant of Transfer-Auto. According to the Hugging Face Blog, it generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls to support fast streaming and long, interactive rollouts. Users provide per-camera captions and synchronized World Scenario Map controls, optionally adding initial RGB images. For LiDAR generation, HD-Map range controls aligned with the camera controls are included. Closed-loop interaction requires a simulation runtime and World Scenario Map renderer to update controls from driving inputs.

Sim-Transfer takes a different route. A simulator computes motion, contacts, and physical state, and Sim-Transfer uses its spatial controls and a text prompt to render colour observations that a policy can consume. The source lists depth, segmentation, edges, and blur as control types, with text guiding appearance while physics and ground-truth state stay with the simulator. Sim-Bimanual, by contrast, generates the next observation directly from the scene and robot actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks. One embedded demo video illustrates streaming or interactive simulation output in this context: demo video.

For Sim-Bimanual, the source notes that a mixed-action example uses 59 control values, padded to 64 during training, with a schema defining coordinate frames, rotations, gripper calibration, validity masks, and control domains. Timing must stay aligned, with T transitions needing T+1 observations. Training transforms raw captures into the model action representation, while released inference actions are already normalized. Another embedded demo video is included here to show bimanual simulation output: demo video.

What training approach makes streaming generation possible?

Cosmos 3 uses a Mixture-of-Transformers backbone that connects separate text-processing and visual-generation networks through attention, according to the Hugging Face Blog. The generator predicts each short segment from the prompt, available controls, and generated history. Denoising refines noisy, compressed video representations into a segment. Temporal-causal attention lets the model read available history and controls while blocking future target observations, and a key/value cache retains attention information for the next prediction.

The model operates on compressed video representations called latents. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk, while the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often. The four-stage training recipe begins with bidirectional adaptation from complete recorded clips, then teacher forcing from earlier recorded observations, then consistency distillation so a student generates each chunk in four denoising steps, and finally self-forcing on the student generated history. The final stage matters because a slightly mispredicted object becomes context for later frames, so errors can compound. A further embedded demo video illustrates generated segment output in this context: demo video.

How can teams adapt the models to their own data?

The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four stages, according to the Hugging Face Blog. Transfer-Auto post-training adapts to a driving sensor rig and domain for offline synthetic data. Sim-Auto applies causal training and distillation to an adapted Transfer-Auto model for fast streaming simulation. Sim-Transfer post-training covers local data preparation, stage configurations, and resume for a specific environment. Sim-Bimanual post-training covers embodiment adaptation, data validation, stage configurations, checkpoint handoffs, and export.

Teams are advised to keep full training checkpoints to resume, because exported inference weights omit optimizer and other training state. The source recommends following environment checks, validating a sample, and testing training and restart before a longer run. Evaluation should use held-out episodes, because a successful short run verifies execution rather than convergence. Requirements are Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200, which the source says is not a minimum-memory specification. The server downloads weights, while clients fetch sample inputs.

The source also stresses that the problems remain hard. The work ahead is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short. Open weights and post-training recipes are presented as a shared starting point for adapting models, comparing methods, and contributing improvements.

How can you explore this topic in Grafy?

There is no direct Grafy app for running Cosmos 3 world simulation, and Grafy does not claim to offer that capability. A practical Grafy route is to use Reader to read and question the technical report and related papers, then use Code to explore the published inference skills, notebooks, data preparation, and post-training scripts in a coding workspace. You can also use Chat to compare the four checkpoints side by side, summarise the training recipe, or turn the adaptation checklist into a project plan. That combination fits the genuine connection here: the release is primarily a code, data, and evaluation workflow, so Grafy is most useful for research, implementation planning, and documentation rather than for hosting the simulation itself.

Frequently asked questions

Are all four checkpoints post-trained from Cosmos 3 Nano?

Yes. According to the Hugging Face Blog, all four checkpoints are post-trained examples of Cosmos 3 Nano, each with 16B trainable parameters. They are released under OpenMDW1.1, which permits commercial and non-commercial use under its terms, providing a shared starting point for adaptation and comparison.

Is Sim-Auto distilled from Transfer-Auto?

Yes. According to the Hugging Face Blog, Sim-Auto is the causally distilled streaming variant of Transfer-Auto. Transfer-Auto first adapts to a sensor rig and domain, then undergoes causal training and few-step distillation to create Sim-Auto for fast streaming simulation and long interactive rollouts.

What hardware was the published vLLM-Omni setup tested on?

According to the Hugging Face Blog, the published vLLM-Omni setup was tested on one GB200. The article states this is not a minimum-memory specification. Requirements listed are Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit, with the server downloading weights and clients fetching sample inputs.

Source: Hugging Face Blog — https://huggingface.co/blog/nvidia/cosmos-interactive-world-simulation

---

Source: Hugging Face Blog — New NVIDIA Cosmos 3 Open Models for Interactive World Simulation