MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities

Any-to-any modeling aims to flexibly relate arbitrary modalities within a single system, a requirement that arises across multimodal learning and scientific domains such as ecology and astronomy. However, existing any-to-any approaches are typically trained from scratch using encoder–decoder or diffusion architectures, limiting empirical performance and the use of pretrained models. We investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. As a consequence of this unified design, the resulting model MODUS naturally enables chained generation through intermediate modalities, cross-modal consistency verification, and analysis of visual representations by combining semantic and reconstruction features. Across a range of benchmarks, MODUS demonstrates strong out-of-the-box performance and flexible multimodal composition within a single model.

Modern AI is usually built as a collection of specialists: one model recognizes objects in a picture, another estimates depth, and another generates pictures from text. Each capability typically comes with its own architecture, training pipeline, and data format, descriptions, photographs, depth maps, edge sketches, surface normals, and segmentation masks. Given any of these as input, MODUS generates any of the others as output within the same network and training procedure. This unified design also opens up new ways of using such a model. Instead of producing an image from text in one shot, MODUS can first generate an intermediate representation — for example, an edge sketch or surface normal map — and then build the final image from it, often leading to more coherent results. The same cross-modal abilities also let MODUS check whether its own output matches the input prompt, without relying on an external judge. Our experiments show that a single decoder-based model can match or exceed specialized systems across a diverse set of tasks, suggesting that the same architecture behind modern language models can also serve as a general-purpose foundation for multimodal understanding and generation.