Twins: Learn to Predict Unified Representations with Focal Loss

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations—semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis—resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe \textit{optimization imbalance}: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to $10.57$ gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

Modern AI systems are increasingly expected to do two very different things with images: understand what is in a picture (for example, answering questions about it) and create new pictures from scratch. Today, these two abilities usually rely on two different ways of "looking at" images inside the model — one focused on meaning, the other focused on visual details — which makes it hard to combine them into a single system. We introduce Twins, a method that lets the model use both views of an image at the same time without making it slower or larger. However, we discovered that when training such a model, it tends to learn the "meaning" view quickly but neglects the "visual detail" view, leading to lower-quality generated images. We identify why this imbalance happens and design a new training objective that pays extra attention to the harder-to-learn details. As a result, Twins generates noticeably sharper images while still understanding pictures well, bringing us one step closer to AI systems that can both see and imagine through a single unified lens.