Timezone: »

 
Spotlight
CURI: A Benchmark for Productive Concept Learning Under Uncertainty
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake

Thu Jul 22 08:45 PM -- 08:50 PM (PDT) @
in Applications 4 »

Humans can learn and reason under substantial uncertainty in a space of infinitely many compositional, productive concepts. For example, if a scene with two blue spheres qualifies as “daxy,” one can reason that the underlying concept may require scenes to have “only blue spheres” or “only spheres” or “only two objects.” In contrast, standard benchmarks for compositional reasoning do not explicitly capture a notion of reasoning under uncertainty or evaluate compositional concept acquisition. We introduce a new benchmark, Compositional Reasoning Under Uncertainty (CURI) that instantiates a series of few-shot, meta-learning tasks in a productive concept space to evaluate different aspects of systematic generalization under uncertainty, including splits that test abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, we also contribute a model-independent “compositionality gap” to evaluate the difficulty of generalizing out-of-distribution along each of these axes, allowing objective comparison of the difficulty of each compositional split. Evaluations across a range of modeling choices and splits reveal substantial room for improvement on the proposed benchmark.

Keywords: Neuroscience and Cognitive Science · Algorithms · Reinforcement Learning and Planning · Bandit Algorithms · Decision and Control

Author Information

Shanmukha Ramakrishna Vedantam (Facebook AI Research)
Arthur Szlam (Facebook)
Maximilian Nickel (Facebook AI Research)
Ari Morcos (Facebook AI Research (FAIR))
Brenden Lake (New York University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors