Hölder++: Improving Quality-Coherence Trade-off in Multimodal VAEs

Existing approaches for multimodal variational autoencoders (VAEs) face a trade-off between generative quality and coherence—i.e., they struggle to generate realistic and diverse samples that, at the same time, are semantically consistent across modalities. A recent work shows that using a simple approximation to Hölder pooling as an aggregation method improves coherence over the SOTA MMVAE+, despite assuming a single shared representation across all modalities. Yet, it slightly compromises sample diversity. Inspired by this insight, we propose Hölder++, a novel multimodal VAE that improves the generative quality-coherence trade-off through: (i) the first implementation of Hölder pooling without any approximation for multimodal VAEs; (ii) an extended architecture that models distinct shared and private (i.e., modality-specific) representations (Hölder+); and (iii) hierarchical inference that further enhances the disentanglement between the shared and private representations (Hölder++). Our experiments corroborate that Hölder++ consistently improves the generative quality-coherence trade-off, yields more structured latent spaces, and learns shared representations that are informative for downstream tasks.

AI generative systems that work with multiple modalities, such as images, text, and audio, face a fundamental challenge: generating realistic outputs for each modality while maintaining semantic consistency across all of them. For example, a model processing bird images together with descriptive captions should generate high-quality image and text pairs that refer to the same bird, but achieving both realism and semantic consistency has proven difficult in practice. In this work, we introduce Hölder++, a framework that addresses this challenge by combining information from different modalities using a mathematical formulation that explicitly models pairwise relationships between modalities, rather than treating them independently. In addition, our approach learns both private and shared representations of the data. This allows the model to explicitly separate shared semantic information (e.g., the species of a bird) from modality-specific details (e.g., photographic style or sentence structure), using a hierarchical design that naturally prevents these factors from becoming entangled. Across multiple benchmarks, Hölder++ achieves a better trade-off between generative quality and cross-modal coherence than existing approaches, while also learning structured representations that are useful for downstream tasks such as clustering and classification.