Geometric-Aware 4D Video Generation

Geometry-aware 4D Video Generation for Robot Manipulation

Zeyi Liu¹ Shuang Li¹ Eric Cousineau² Siyuan Feng² Benjamin Burchfiel² Shuran Song¹

¹Stanford University ²Toyota Research Institute

Understanding and predicting the dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of videos by supervising the model with cross-view pointmap alignment during training. This geometric supervision enables the model to learn a shared 3D representation of the scene, allowing it to predict future video sequences from novel viewpoints based solely on the given RGB-D observations, without requiring camera poses as inputs. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, supporting robust robot manipulation and generalization to novel camera viewpoints.

Method

4D Video Generation for Robot Manipulation. Our model takes RGB-D observations from two camera views, and predicts future pointmaps and RGB videos. To ensure cross-view consistency, we apply cross-attention in the U-Net decoders for pointmap prediction. The resulting 4D video can be used to extract the 6DoF pose of the robot end-effector using pose tracking methods, enabling downstream manipulation tasks.

Results

(a) StoreCerealBoxUnderShelf 🥣

In this task, a single robot arm picks up a cereal box from the top of a shelf and inserts it into the shelf below. Occlusions occur during insertion, especially from certain camera viewpoints, making multi-view predictions essential. Additionally, the pick-and-insert action requires spatial understanding and precision.

Ground-truth RGB-D videos

RGB

DEPTH

Generated RGB-D videos (unseen views)

RGB

DEPTH

Baselines:

OURS without cross-attention (RGB)

OURS without cross-attention (depth)

SVD

SVD w/ cross-attention

Policy Rollout (unseen views):

OURS

Diffusion Policy fails to insert the cereal box

Diffusion Policy fails to pick up the cereal box

(b) PutSpatulaOnTable 👩‍🍳

A single robot arm retrieves a spatula from a utensil crock and places it on the left side of the table. This task requires precise manipulation to successfully grasp the narrow object.

Ground-truth RGB-D videos

RGB

DEPTH

Generated RGB-D videos (unseen views)

RGB

DEPTH

Baselines:

OURS without cross-attention (RGB)

OURS without cross-attention (depth)

SVD

SVD with cross-attention

Policy Rollout (unseen views):

OURS

Diffusion Policy fails to grasp the spatula

Real World Results:

Generated RGB-D videos

RGB

DEPTH

(c) PlaceAppleFromBowlIntoBin 🍎

One robot arm picks up an apple from a bowl on the left side of the table and places it on a shelf; a second arm then picks up the apple and deposits it into a bin on the right side. This is a long-horizon, bimanual task that tests the model's ability to predict both temporally and spatially consistent trajectories.

Ground-truth RGB-D videos