Timezone: »
Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform `multi-grained vision language pre-training.' The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.
Author Information
Yan Zeng (ByteDance AI Lab)
Xinsong Zhang (Bytedance AI Lab)
Hang Li (Bytedance Technology)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts »
Thu. Jul 21st through Fri the 22nd Room Hall E #334
More from the Same Authors
-
2023 Poster: Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributes »
Zhaowei Zhu · Yuanshun Yao · Jiankai Sun · Hang Li · Yang Liu -
2022 Poster: VLUE: A Multi-Task Multi-Dimension Benchmark for Evaluating Vision-Language Pre-training »
Wangchunshu Zhou · Yan Zeng · shizhe diao · Xinsong Zhang -
2022 Spotlight: VLUE: A Multi-Task Multi-Dimension Benchmark for Evaluating Vision-Language Pre-training »
Wangchunshu Zhou · Yan Zeng · shizhe diao · Xinsong Zhang -
2022 Poster: Directed Acyclic Transformer for Non-Autoregressive Machine Translation »
Fei Huang · Hao Zhou · Yang Liu · Hang Li · Minlie Huang -
2022 Spotlight: Directed Acyclic Transformer for Non-Autoregressive Machine Translation »
Fei Huang · Hao Zhou · Yang Liu · Hang Li · Minlie Huang