Timezone: »

Model Fusion with Kullback--Leibler Divergence
Sebastian Claici · Mikhail Yurochkin · Soumya Ghosh · Justin Solomon

Thu Jul 16 08:00 AM -- 08:45 AM & Thu Jul 16 08:00 PM -- 08:45 PM (PDT) @

We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.

Author Information

Sebastian Claici (MIT)
Mikhail Yurochkin (IBM Research AI)
Soumya Ghosh (IBM Research)
Justin Solomon (MIT)

More from the Same Authors