Timezone: »

 
Spotlight
Geometric Multimodal Contrastive Representation Learning
Petra Poklukar · Miguel Vasco · Hang Yin · Francisco S. Melo · Ana Paiva · Danica Kragic

Wed Jul 20 01:55 PM -- 02:00 PM (PDT) @ Ballroom 1 & 2
in Deep Learning »

Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermediate representations to a latent representation space; ii) a multimodal contrastive loss function that encourages the geometric alignment of the learned representations. We experimentally demonstrate that GMC representations are semantically rich and achieve state-of-the-art performance with missing modality information on three different learning problems including prediction and reinforcement learning tasks.

Keywords: DL: Other Representation Learning · MISC: Representation Learning · MISC: Supervised Learning · MISC: Unsupervised and Semi-supervised Learning

Author Information

Petra Poklukar (KTH Royal Institute of Technology)
Miguel Vasco (INESC-ID & Instituto Superior Técnico, Universidade de Lisboa)
Miguel Vasco

Miguel Vasco is a PhD student in Computer Science at Técnico, University of Lisbon since 2018 and a researcher of the Group on Artificial Intelligence for People and Society (GAIPS) since 2016. His research focuses in robust multimodal representation learning, in particular for reinforcement learning agents acting in scenarios with changing perceptual conditions. Miguel has been selected as a member of both the AAMAS 2020 Doctoral Consortium and the RSS Pioneers 2021 cohorts. Since 2020, Miguel co-hosts the Talking Robotics seminar series, which provides visibility and network opportunities for young researchers in Robotics. In parallel to his scientific work, he enjoys composing music under the name Tomigaya and practising Shorinji Kempo.

Hang Yin (KTH)
Francisco S. Melo (IST/INESC-ID)
Ana Paiva (INESC-ID U of Lisbon)
Danica Kragic (KTH)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors