Poster
Contrasting Multiple Representations with the Multi-Marginal Matching Gap
Zoe Piran · Michal Klein · James Thornton · Marco Cuturi
Hall C 4-9 #816
[
Abstract
]
[ Project Page ]
[ Paper PDF ]
Oral
presentation:
Oral 6D Representation Learning 2
Thu 25 Jul 7:30 a.m. PDT — 8:30 a.m. PDT
Thu 25 Jul 4:30 a.m. PDT
— 6 a.m. PDT
Thu 25 Jul 7:30 a.m. PDT — 8:30 a.m. PDT
Abstract:
Learning meaningful representations of complex objects that can be seen through multiple ($k\geq 3$) views or modalities is a core task in machine learning. Existing methods use losses originally intended for paired views, and extend them to $k$ views, either by instantiating $\tfrac12k(k-1)$ loss-pairs, or by using reduced embeddings, following a *one vs. average-of-rest* strategy. We propose the multi-marginal matching gap (M3G), a loss that borrows tools from multi-marginal optimal transport (MM-OT) theory to simultaneously incorporate all $k$ views. Given a batch of $n$ points, each seen as a $k$-tuple of views subsequently transformed into $k$ embeddings, our loss contrasts the cost of matching these $n$ ground-truth $k$-tuples with the MM-OT polymatching cost, which seeks $n$ optimally arranged $k$-tuples chosen within these $n\times k$ vectors. While the exponential complexity $O(n^k$) of the MM-OT problem may seem daunting, we show in experiments that a suitable generalization of the Sinkhorn algorithm for that problem can scale to, e.g., $k=3\sim 6$ views using mini-batches of size $64~\sim128$. Our experiments demonstrate improved performance over multiview extensions of pairwise losses, for both self-supervised and multimodal tasks.
Live content is unavailable. Log in and register to view live content