How should we interact with video world models? We introduce masked visual actions, a simple representation that generalizes across unseen robots and objects. Reveal part of a video, and the model predicts how the scene unfolds.
Our video world model takes a reference frame and a masked visual action video, and outputs a video of the scene that follows that action. In the forward setting, the action is a rendering of the robot, and the model simulates the robot's effect on the scene. In the inverse setting, the action is a goal object motion, and the model synthesizes the robot motion that achieves it.
Despite only training on masked visual actions (of robots), an emergent capability of our model is taking masked visual actions of objects and generating the robot motion that satisfies the masked visual actions.
One natural source is a human demonstration. We can extract a masked visual action from a person performing the task, and our model will generate the robot motion that matches it.
Here we show additoinal results from our model in the inverse modeling setting.
Our model can be used to evaluate the policy's action without needing to execute them in the real world. Here we show how our world model's simulated output is aligned with the real world execution (GT).
We also collect demonstrations from the real world, and show that different action trajectories lead to different outcomes. On the towel example, note that the model matches the ground truth demonstration on when the towel gets picked up and when it doesn't.
Two trials starting from a similar reset state. Reset states cannot be reproduced exactly in the real world, so each trial has its own reference image.
See our full real-world demonstrations alongside the model simulation across four different tasks here.
Generalization! As we showed earlier, Visual Masked Actions are so generalizable, they can work for objects as well as diverse unseen robots. To further illustrate the advantage of Masked Visual Actions, we finetune the same base model on visualizations of the robot Skeleton, and End-Effector (EEF) — on the same scene. All were trained on the same data from DROID. The real-world data we captured was using the same robot as droid but with custom unseen gripper. In BEHAVIOR, we use a completely unseen embodiment to all models.
As expected, all variants perform similarly on the training distribution.
Here we see that unlike our method, conditioning on the skeleton or EEF causes the model to transform the robot to match the embodiment seen during training, or would simply introduce another robot into the video. The top row shows each action visualization, and below it the resulting output.
Similarly with using a bimanual robot, our method generalizes well to the unseen embodiment while the other conditioning signals completely collapse. The top row shows each action visualization, and below it the resulting output.
We compare our method conditioned on Masked Visual Actions against Ctrl-World (both are trained on DROID), and also include Wan-Move (trajectory conditioned video model), and Wan2.2 I2V as a reference.
BEHAVIOR uses a bimanual embodiment unseen during training for both our method and Ctrl-world. However, we find that our method generalizes more gracefully.
Our model can fail in cases where precise and subtle interaction is needed and may produce unnatural interactions in those cases. Additionally, the model may produce artifacts surrounding regions completely unseen in the reference frame, as the mug example is not showing the handle in the initial image.
@article{alzayer2026maskedvisualactions,
title = {Masked Visual Actions for Unified World Modeling},
author = {Alzayer, Hadi and Huang, Wenlong and Chen, Haonan and
Luey, Christopher and Zhang, Lvmin and Agrawala, Maneesh and
Wetzstein, Gordon and Fei-Fei, Li and Du, Yilun and
Wu, Jiajun and Huang, Jia-Bin},
journal = {arXiv preprint arXiv:XXXX.XXXXX},
year = {2026}
}