MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality

Unified visual tokenization faces a fundamental trade-off: optimizing for high-fidelity pixel reconstruction (spatial equivariance) inherently conflicts with semantic abstraction (conceptual invariance). We identify the root cause as Manifold Misalignment, where naive joint optimization leads to conflicting gradients that force a zero-sum game between these two objectives. In this paper, we propose MUSE, a framework that resolves this deadlock via Topological Orthogonality. Recognizing Structure as the orthogonal bridge, MUSE physically decouples the optimization subspaces within Transformers. We route structural gradients to refine attention topology and semantic gradients to update feature values, transforming destructive interference into Mutual Reinforcement. Extensive experiments demonstrate that MUSE breaks the trade-off, matching state-of-the-art generation (gFID 3.08) while notably outperforming its own teacher InternViT-300M in linear probing (85.2% vs. 82.5%), proving that structurally aligned reconstruction actively refines semantic perception.

When computers learn to process images, they face a fundamental problem: trying to perfectly recreate every pixel (making the image look exactly right) conflicts with recognizing high-level concepts (like knowing a picture contains a cat, regardless of lighting or pose). Our work identifies why this happens，these two goals pull the system in opposite directions, forcing it to sacrifice one for the other. We introduce MUSE, a new approach that resolves this conflict by treating the image’s structure as a neutral bridge. Instead of forcing both goals into the same learning process, MUSE separates them so they no longer interfere. In fact, they start helping each other: improving the structural details of an image actively sharpens the system’s ability to recognize what it sees. As a result, MUSE achieves top-tier image generation quality while also outperforming its own teacher model in understanding images—something previously thought impossible when optimizing for both appearance and meaning.