Compositional Generative Modeling from Decentralized Data

Learning the compositional nature of the physical world requires joint observation of interacting factors. However, because practical data is often decentralized, these factors are fragmented across isolated silos. Existing decentralized generative approaches focus only on modeling the union of siloed data, overlooking novel combinations implied by the collective whole. To bridge this gap, we introduce Decentralized Compositional Flow Matching (DCFM), a framework that enforces structural constraints across the global set of generative factors, without exchanging any raw data. DCFM enables novel combinations to emerge through peer interactions, even when no single data source can independently support the composition. Empirically, DCFM substantially outperforms federated learning and mixture-of-experts baselines across conditional image generation, robotic spatial planning, and medical attribute co-occurrence modeling.

To build capable AI, models need to learn from diverse, complex scenarios. However, real-world data is increasingly private and trapped in isolated silos, like individual hospitals or robots deployed in specialized environments. When data is fragmented this way, current AI systems cannot effectively piece together isolated experiences to handle completely new, unobserved situations. To tackle this, we developed Decentralized Compositional Flow Matching (DCFM). DCFM allows independent AI models to collaboratively learn how to combine different concepts without ever exchanging their raw, private data. It achieves this by teaching the models to be conditionally independent with one another, enabling them to combine their individual factors into novel outcomes. This research shows that when building a generative AI model from a decentralized and private group of data, we can learn a lot more than simply the `whole' or union of the fragmented data; we can also learn the vast combinatorial space implied by the fragments.