Is Generation Required for Data-Efficient Perception?

It has been hypothesized that achieving the data efficiency of human visual perception requires a generative approach in which internal representations result from inverting a decoder. Yet today’s most successful vision models are non-generative, relying on an encoder that maps images to representations without decoder inversion. This raises the question of whether generation is necessary for data-efficient machine perception. To address this, we study to what extent generative and non-generative methods can achieve compositional generalization, a hallmark of human data efficiency. Under a compositional generative process, we formally characterize the inductive biases required for compositional generalization in decoder-based (generative) and encoder-based (non-generative) methods. We show theoretically that the inductive biases required for an encoder are substantially more complex and generally infeasible to impose explicitly through architectural constraints or regularization. By contrast, the decoder biases take a simple form that can be enforced directly. These results suggest that compositional generalization may be substantially easier to achieve through a generative paradigm of learning and inverting a decoder rather than learning an encoder directly. We examine our theoretical findings empirically by training a range of generative and non-generative methods on synthetic image data. We find that non-generative methods often fail to generalize compositionally and require large-scale pretraining to improve generalization. By comparison, generative methods yield gains in generalization without requiring additional data.

Humans can recognize familiar things in novel situations, such as a penguin in a desert, without having seen that exact combination of concepts before. Modern computer vision systems are very good at recognizing images, but they often rely on enormous training datasets to handle such new combinations. This raises the question of whether these systems can ever become as data-efficient as humans, or whether they need a different way of understanding images. We study this question by comparing two approaches to visual perception: models that directly map images to internal representations (non-generative), and models that first learn how images are generated and then infer the hidden causes behind them (generative). We show theoretically that it is generally difficult for non-generative models to generalize correctly to unseen combinations. In contrast, generative models can more naturally build in the right structure by learning how different visual factors, such as objects and backgrounds, combine to form an image. We then test these ideas experimentally on image datasets where models must recognize new combinations of familiar objects, textures, and backgrounds. Our experiments show that non-generative models often fail to generalize to novel compositions, while generative methods can achieve such generalization more naturally. These results suggest that generation may be important for building vision systems that learn more like humans.