Masked Visual Actions
for Unified World Modeling

Every embodiment shown is unseen during training — each video generated from a single reference frame and a masked visual action.

Overview

Our video world model takes a reference frame and a masked visual action video, and outputs a video of the scene that follows that action. In the forward setting, the action is a rendering of the robot, and the model simulates the robot's effect on the scene. In the inverse setting, the action is a goal object motion, and the model synthesizes the robot motion that achieves it.

Forward Modeling

By conditioning on a rendering of the robot's actions, our video world model simulates the robot's interaction with the scene shown in the reference frame. We show that its simulated rollouts align with real-world execution (GT).

Reference

Masked visual action

Ours

Ground truth

See our full 80 real-world demonstrations alongside the model simulation across four different tasks here.

Inverse Modeling

Despite only training on masked visual actions (of robots), an emergent capability of our model is taking masked visual actions of objects and generating the robot motion that satisfies the masked visual actions.

View more results for inverse modeling →

Why Masked Visual Actions?

Generalization! As we showed earlier, Visual Masked Actions are so generalizable, they can work for objects as well as diverse unseen robots.
To further illustrate the advantage of Masked Visual Actions, we finetune the same base model on visualizations of the robot Skeleton, and End-Effector (EEF) — on the same scene. All were trained on the same data from DROID. The real-world data we captured was using the same robot as droid but with custom unseen gripper. In BEHAVIOR, we use a completely unseen embodiment to all models.

Comparison with Baselines

We compare our method conditioned on Masked Visual Actions against Ctrl-World[1] (both are trained on DROID), and also include Wan-Move[2] (trajectory conditioned video model), and Wan2.2 I2V[3] as a reference.

DROID

BEHAVIOR

BEHAVIOR uses a bimanual embodiment unseen during training for both our method and Ctrl-world. However, we find that our method generalizes more gracefully.

Extreme Generalization

For fun, we push generalization to the extreme: conditioning on the masked visual action of an orangutan — an embodiment far outside anything seen during training. The model still completes the scene to follow the provided action.

Inputs

Reference

Masked visual action

Output

Inputs

Reference

Masked visual action

Output

Failure Cases

Our model can fail in cases where precise and subtle interaction is needed and may produce unnatural interactions in those cases. Additionally, the model may produce artifacts surrounding regions completely unseen in the reference frame, as the mug example is not showing the handle in the initial image.

Reference

Masked visual action

Ours

Ground truth

References

1. Guo et al. Ctrl-World: A Controllable Generative World Model for Robot Manipulation. ICLR 2026.
2. Chu et al. Wan-Move: Motion-Controllable Video Generation via Latent Trajectory Guidance. arXiv:2512.08765, 2025.
3. Wan Team et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.

BibTeX

@misc{alzayer2026maskedvisualactionsunified,
  title         = {Masked Visual Actions for Unified World Modeling},
  author        = {Hadi Alzayer and Wenlong Huang and Haonan Chen and
                   Christopher Luey and Lvmin Zhang and Maneesh Agrawala and
                   Gordon Wetzstein and Li Fei-Fei and Yilun Du and Jiajun Wu and
                   Jia-Bin Huang},
  year          = {2026},
  eprint        = {2607.19343},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.19343}
}