ImmersePro: End-to-End Stereo Video Synthesis Via Implicit Disparity Learning

We introduce \textit{ImmersePro}, an innovative framework specifically designed to transform single-view videos into stereo videos. This framework utilizes a novel dual-branch architecture comprising a disparity branch and a context branch on video data by leveraging spatial-temporal attention mechanisms. \textit{ImmersePro} employs implicit disparity guidance, enabling the generation of stereo pairs from video sequences without the need for explicit disparity maps, thus reducing potential errors associated with disparity estimation models. In addition to the technical advancements, we introduce the YouTube-SBS dataset, a comprehensive collection of 423 stereo videos sourced from YouTube. This dataset is unprecedented in its scale, featuring over 7 million stereo pairs, and is designed to facilitate training and benchmarking of stereo video generation models. Our experiments demonstrate the effectiveness of \textit{ImmersePro} in producing high-quality stereo videos, offering significant improvements over existing methods. Compared to the best competitor stereo-from-mono we quantitatively improve the results by 11.76\% (L1), 6.39\% (SSIM), and 5.10\% (PSNR).

3D movies create a sense of depth by showing each eye a slightly different image, the way our two eyes naturally see the world. Because filming in 3D is expensive, most films are shot normally and converted to 3D afterward — a slow process that usually requires careful manual work by artists, and small mistakes can make the result uncomfortable to watch. We built ImmersePro, a system that automatically turns ordinary video into 3D video. Earlier automated methods first guess how far away each object is and then use that guess to build the second eye's view, so any error in the guess becomes a visible flaw. Instead, ImmersePro learns the relationship between the two eyes' views directly from examples, and uses information across neighboring video frames to keep the 3D effect smooth and stable. We also created YouTube-SBS, a large public collection of 3D videos for training and fairly comparing such systems. This makes converting flat video into immersive 3D content cheaper and more reliable, supporting wider use of virtual reality, 3D displays, and digital media.