Timezone: »
Existing vision-language pre-training (VLP) methods primarily rely on paired image-text datasets, which are either annotated by enormous human labors or crawled from the internet followed by elaborate data cleaning techniques. To reduce the dependency on well-aligned image-text pairs, it is promising to directly leverage the large-scale text-only and image-only corpora. This paper proposes a data augmentation method, namely cross-modal CutMix (CMC), for implicit cross-modal alignment learning in unpaired VLP. Specifically, CMC transforms natural sentences in the textual view into a multi-modal view, where visually-grounded words in a sentence are randomly replaced by diverse image patches with similar semantics. There are several appealing proprieties of the proposed CMC. First, it enhances the data diversity while keeping the semantic meaning intact for tackling problems where the aligned data are scarce; Second, by attaching cross-modal noise on uni-modal data, it guides models to learn token-level interactions across modalities for better denoising. Furthermore, we present a new unpaired VLP method, dubbed as VLMixer, that integrates CMC with contrastive learning to pull together the uni-modal and multi-modal views for better instance-level alignments among different modalities. Extensive experiments on five downstream tasks show that VLMixer could surpass previous state-of-the-art unpaired VLP methods.
Author Information
Teng Wang (Southern University of Science and Technology)
Wenhao Jiang (Tencent)
Zhichao Lu (Southern University of Science and Technology)
Feng Zheng (SUSTech)
Ran Cheng (Southern University of Science and Technology)

Dr. Ran Cheng, the founder of the Evolving Machine Intelligence (EMI) Group, is currently a tenured Associate Professor with the Southern University of Science and Technology (SUSTech), China. He received the PhD degree in computer science from the University of Surrey, UK, in 2016. His research interests mainly fall into the interdisciplinary fields across evolutionary computation and other major AI branches such as statistical learning and deep learning, to provide end-to-end solutions to optimization & modeling in scientific research and engineering related applications. He is the Founding Chair of IEEE Computational Intelligence Society (CIS) Shenzhen Chapter and IEEE Symposium on Model Based Evolutionary Algorithms (IEEE MBEA). He serves as an Associated Editor/Editorial Board Member for serveral jounrlas, including: IEEE Transactions on Evolutionary Computation, IEEE Transactions on Cognitive and Developmental Systems, IEEE Transactions on Artificial Intelligence, etc. He is the recipient of the IEEE Transactions on Evolutionary Computation Outstanding Paper Awards (2018, 2021), the IEEE CIS Outstanding PhD Dissertation Award (2019), the IEEE Computational Intelligence Magazine Outstanding Paper Award (2020). He is a Senior Member of IEEE.
chengguo yin (tencent)
Ping Luo (The University of Hong Kong)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix »
Tue. Jul 19th 06:30 -- 06:35 PM Room Hall F
More from the Same Authors
-
2023 Poster: $\pi$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation »
CHENGYUE WU · Teng Wang · Yixiao Ge · Zeyu Lu · Ruisong Zhou · Ying Shan · Ping Luo -
2023 Poster: ChiPFormer: Transferable Chip Placement via Offline Decision Transformer »
Yao LAI · Jinxin Liu · Zhentao Tang · Bin Wang · Jianye Hao · Ping Luo -
2023 Poster: AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners »
Zhixuan Liang · Yao Mu · Mingyu Ding · Fei Ni · Masayoshi Tomizuka · Ping Luo -
2023 Oral: AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners »
Zhixuan Liang · Yao Mu · Mingyu Ding · Fei Ni · Masayoshi Tomizuka · Ping Luo -
2022 Poster: Flow-based Recurrent Belief State Learning for POMDPs »
Xiaoyu Chen · Yao Mu · Ping Luo · Shengbo Li · Jianyu Chen -
2022 Spotlight: Flow-based Recurrent Belief State Learning for POMDPs »
Xiaoyu Chen · Yao Mu · Ping Luo · Shengbo Li · Jianyu Chen -
2022 Poster: DynaMixer: A Vision MLP Architecture with Dynamic Mixing »
Ziyu Wang · Wenhao Jiang · Yiming Zhu · Li Yuan · Yibing Song · Wei Liu -
2022 Poster: CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer »
Yao Mu · Shoufa Chen · Mingyu Ding · Jianyu Chen · Runjian Chen · Ping Luo -
2022 Spotlight: CtrlFormer: Learning Transferable State Representation for Visual Control via Transformer »
Yao Mu · Shoufa Chen · Mingyu Ding · Jianyu Chen · Runjian Chen · Ping Luo -
2022 Spotlight: DynaMixer: A Vision MLP Architecture with Dynamic Mixing »
Ziyu Wang · Wenhao Jiang · Yiming Zhu · Li Yuan · Yibing Song · Wei Liu -
2021 Poster: Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution »
zhaoyang zhang · Wenqi Shao · Jinwei Gu · Xiaogang Wang · Ping Luo -
2021 Spotlight: Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution »
zhaoyang zhang · Wenqi Shao · Jinwei Gu · Xiaogang Wang · Ping Luo -
2021 Poster: What Makes for End-to-End Object Detection? »
Peize Sun · Yi Jiang · Enze Xie · Wenqi Shao · Zehuan Yuan · Changhu Wang · Ping Luo -
2021 Spotlight: What Makes for End-to-End Object Detection? »
Peize Sun · Yi Jiang · Enze Xie · Wenqi Shao · Zehuan Yuan · Changhu Wang · Ping Luo -
2020 Poster: Channel Equilibrium Networks for Learning Deep Representation »
Wenqi Shao · Shitao Tang · Xingang Pan · Ping Tan · Xiaogang Wang · Ping Luo -
2019 : Poster discussion »
Roman Novak · Maxime Gabella · Frederic Dreyer · Siavash Golkar · Anh Tong · Irina Higgins · Mirco Milletari · Joe Antognini · Sebastian Goldt · Adín Ramírez Rivera · Roberto Bondesan · Ryo Karakida · Remi Tachet des Combes · Michael Mahoney · Nicholas Walker · Stanislav Fort · Samuel Smith · Rohan Ghosh · Aristide Baratin · Diego Granziol · Stephen Roberts · Dmitry Vetrov · Andrew Wilson · César Laurent · Valentin Thomas · Simon Lacoste-Julien · Dar Gilboa · Daniel Soudry · Anupam Gupta · Anirudh Goyal · Yoshua Bengio · Erich Elsen · Soham De · Stanislaw Jastrzebski · Charles H Martin · Samira Shabanian · Aaron Courville · Shorato Akaho · Lenka Zdeborova · Ethan Dyer · Maurice Weiler · Pim de Haan · Taco Cohen · Max Welling · Ping Luo · zhanglin peng · Nasim Rahaman · Loic Matthey · Danilo J. Rezende · Jaesik Choi · Kyle Cranmer · Lechao Xiao · Jaehoon Lee · Yasaman Bahri · Jeffrey Pennington · Greg Yang · Jiri Hron · Jascha Sohl-Dickstein · Guy Gur-Ari -
2019 : Poster Session 1 (all papers) »
Matilde Gargiani · Yochai Zur · Chaim Baskin · Evgenii Zheltonozhskii · Liam Li · Ameet Talwalkar · Xuedong Shang · Harkirat Singh Behl · Atilim Gunes Baydin · Ivo Couckuyt · Tom Dhaene · Chieh Lin · Wei Wei · Min Sun · Orchid Majumder · Michele Donini · Yoshihiko Ozaki · Ryan P. Adams · Christian Geißler · Ping Luo · zhanglin peng · · Ruimao Zhang · John Langford · Rich Caruana · Debadeepta Dey · Charles Weill · Xavi Gonzalvo · Scott Yang · Scott Yak · Eugen Hotaj · Vladimir Macko · Mehryar Mohri · Corinna Cortes · Stefan Webb · Jonathan Chen · Martin Jankowiak · Noah Goodman · Aaron Klein · Frank Hutter · Mojan Javaheripi · Mohammad Samragh · Sungbin Lim · Taesup Kim · SUNGWOONG KIM · Michael Volpp · Iddo Drori · Yamuna Krishnamurthy · Kyunghyun Cho · Stanislaw Jastrzebski · Quentin de Laroussilhe · Mingxing Tan · Xiao Ma · Neil Houlsby · Andrea Gesmundo · Zalán Borsos · Krzysztof Maziarz · Felipe Petroski Such · Joel Lehman · Kenneth Stanley · Jeff Clune · Pieter Gijsbers · Joaquin Vanschoren · Felix Mohr · Eyke Hüllermeier · Zheng Xiong · Wenpeng Zhang · Wenwu Zhu · Weijia Shao · Aleksandra Faust · Michal Valko · Michael Y Li · Hugo Jair Escalante · Marcel Wever · Andrey Khorlin · Tara Javidi · Anthony Francis · Saurajit Mukherjee · Jungtaek Kim · Michael McCourt · Saehoon Kim · Tackgeun You · Seungjin Choi · Nicolas Knudde · Alexander Tornede · Ghassen Jerfel -
2019 Poster: Differentiable Dynamic Normalization for Learning Deep Representation »
Ping Luo · Peng Zhanglin · Shao Wenqi · Zhang ruimao · Ren jiamin · Wu lingyun -
2019 Oral: Differentiable Dynamic Normalization for Learning Deep Representation »
Ping Luo · Peng Zhanglin · Shao Wenqi · Zhang ruimao · Ren jiamin · Wu lingyun -
2017 Poster: Learning Deep Architectures via Generalized Whitened Neural Networks »
Ping Luo -
2017 Talk: Learning Deep Architectures via Generalized Whitened Neural Networks »
Ping Luo