Timezone: »

The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-language Models
Chenwei Wu · Li Li · Stefano Ermon · Patrick Haffner · Rong Ge · Zaiwei Zhang
Event URL: https://openreview.net/forum?id=WxO8cfgtCu »

Compositionality is a common property in many modalities including natural languages and images, but the compositional generalization of multi-modal models is not well-understood. In this paper, we identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We show that current attempts to improve compositional generalization rely on linguistic priors rather than on information in the image. We also propose a new metric for compositionality without such linguistic priors.

Author Information

Chenwei Wu (Duke University)
Li Li (Amazon)
Stefano Ermon (Stanford University)
Patrick Haffner (Amazon)
Rong Ge (Duke University)
Zaiwei Zhang (Amazon)

More from the Same Authors