Rays as Pixels: Learning A Joint Distribution of Video and Camera Trajectories

Can we bridge the gap between perceiving camera trajectories and rendering novel views within a single generative framework? Recovering camera parameters from images and rendering scenes from novel viewpoints are considered the forward and inverse problems in the field of computer vision and graphics. Previous approaches treat these problems in isolation, often failing when image coverage is sparse or camera poses are ambiguous. In this work, we propose Rays as Pixels, a specialized Video Diffusion Model (VDM) that learns a joint distribution of videos and camera trajectories. We represent cameras as dense ray pixels (raxels) and simultaneously denoise them alongside video frames using a novel Decoupled Self-Cross Attention. This joint formulation enables us to: i) generate a video from multiple input images following a defined camera trajectory, ii) perform novel view synthesis from sparse views (without necessarily requiring camera poses), and iii) predict the camera trajectory from a raw video. We evaluate our model on pose estimation, camera-controlled video generation and validate its self-consistency. Please reference supplementary material for more qualitative results.

We built a single AI model that solves two opposite problems about video and world at the same time: figuring out how the camera moved through a scene, and generating what that scene would look like from a new, unseen viewpoint. Our key idea is to represent a camera's viewpoint as an image itself, in the same form the model already uses for video frames, so the model can treat "where the camera is" and "what the scene looks like" as two halves of one problem. This matters because these two tasks have traditionally been handled by separate systems, and both tend to break when a video offers only a few viewpoints or ambiguous motion, since each task secretly depends on the other. By learning them together, our model lets them inform each other instead of failing in isolation. The result is a more robust way to recover camera motion and generate new views from limited footage, which could help 3D content creation, world models, and robotics understand the world from ordinary video.