Beyond Language Modeling: An Exploration of Multimodal Pretraining

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

The design space of multimodal models still remains largely opaque. We adopt the Transfusion framework to train a single autoregressive model using both next-token prediction for text and flow-matching for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Through controlled pretraining experiments from scratch, we yield insights into visual representations, data, world modeling, architecture, and scaling. First, we find that a high-dimensional Representation Autoencoder (RAE) provides a unified visual representation for both understanding and generation, in contrast to prior works which use separate visual representations such as VAE for generation and SigLIP for understanding. This simplifies both training and inference. Second, we find that visual and language data are complementary and even are synergistic for downstream capabilities. This contrasts with prior work in multimodal pretraining that suggests there is an inherent problem of "modality competition" --- the idea that jointly training on multiple modalities hurts performance compared to training on each modality separately. We find minimal degradation in text perplexity when adding in vision data, and minimal degradation in diffusion loss when adding text. Differences in e.g. text caption distribution affect text perplexity more than the introduction of the visual modality itself. Third, we find that world modeling emerges from multimodal pretraining rather than domain specific data. Our model handily learns navigation even if only 1% of the total data is navigation-specific. This suggests that the road towards better world models may not require large-scale collection of domain-specific data, but rather improvements in pretraining. Lastly, we find that vision and language scale differently (e.g. language is inherently more parameter hungry). Introducing sparsity via mixture-of-experts (MoE) enables efficient and effective multimodal scaling, by shifting the scaling behavior of language closer to the data-hungry regime of vision. We also find that our MoE naturally learns modality specialization in the learned FFN routing, e.g. some experts are language-heavy, some are vision-heavy, and some are multimodal, yet this routing is learned entirely from the data. Overall, our work suggests that the boundaries between multimodal and world models are blurring, and the potential of pretraining on unlabeled video remains largely untapped. Our insights provide more clarity towards the design of multimodal models that can understand and reason about the physical world, but there is still much more work to do.