Timezone: »

 
Poster
VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix
Teng Wang · Wenhao Jiang · Zhichao Lu · Feng Zheng · Ran Cheng · chengguo yin · Ping Luo

Tue Jul 19 03:30 PM -- 05:30 PM (PDT) @ Hall E #109
in Poster Session 1 »

Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors or crawled from the internet followed by elaborate data cleaning techniques. To reduce the dependency on well-aligned image-text pairs, it is promising to directly leverage the large-scale text-only and image-only corpora. This paper proposes a data augmentation method, namely cross-modal CutMix (CMC), for implicit cross-modal alignment learning in unpaired VLP. Specifically, CMC transforms natural sentences in the textual view into a multi-modal view, where visually-grounded words in a sentence are randomly replaced by diverse image patches with similar semantics. There are several appealing proprieties of the proposed CMC. First, it enhances the data diversity while keeping the semantic meaning intact for tackling problems where the aligned data are scarce; Second, by attaching cross-modal noise on uni-modal data, it guides models to learn token-level interactions across modalities for better denoising. Furthermore, we present a new unpaired VLP method, dubbed as VLMixer, that integrates CMC with contrastive learning to pull together the uni-modal and multi-modal views for better instance-level alignments among different modalities. Extensive experiments on five downstream tasks show that VLMixer could surpass previous state-of-the-art unpaired VLP methods.

Keywords: APP: Computer Vision · DL: Self-Supervised Learning
[ Slides]  [ Poster]  [ Paper PDF] 

Author Information

Teng Wang (Southern University of Science and Technology)
Wenhao Jiang (Tencent)
Zhichao Lu (Southern University of Science and Technology)
Feng Zheng (SUSTech)
Ran Cheng (Southern University of Science and Technology)
Ran Cheng

Dr. Ran Cheng, the founder of the Evolving Machine Intelligence (EMI) Group, is currently a tenured Associate Professor with the Southern University of Science and Technology (SUSTech), China. He received the PhD degree in computer science from the University of Surrey, UK, in 2016. His research interests mainly fall into the interdisciplinary fields across evolutionary computation and other major AI branches such as statistical learning and deep learning, to provide end-to-end solutions to optimization & modeling in scientific research and engineering related applications. He is the Founding Chair of IEEE Computational Intelligence Society (CIS) Shenzhen Chapter and IEEE Symposium on Model Based Evolutionary Algorithms (IEEE MBEA). He serves as an Associated Editor/Editorial Board Member for serveral jounrlas, including: IEEE Transactions on Evolutionary Computation, IEEE Transactions on Cognitive and Developmental Systems, IEEE Transactions on Artificial Intelligence, etc. He is the recipient of the IEEE Transactions on Evolutionary Computation Outstanding Paper Awards (2018, 2021), the IEEE CIS Outstanding PhD Dissertation Award (2019), the IEEE Computational Intelligence Magazine Outstanding Paper Award (2020). He is a Senior Member of IEEE.

chengguo yin (tencent)
Ping Luo (The University of Hong Kong)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors