Timezone: »

 
Spotlight
CURI: A Benchmark for Productive Concept Learning Under Uncertainty
Shanmukha Ramakrishna Vedantam · Arthur Szlam · Maximilian Nickel · Ari Morcos · Brenden Lake

Thu Jul 22 08:45 PM -- 08:50 PM (PDT) @ None

Humans can learn and reason under substantial uncertainty in a space of infinitely many compositional, productive concepts. For example, if a scene with two blue spheres qualifies as “daxy,” one can reason that the underlying concept may require scenes to have “only blue spheres” or “only spheres” or “only two objects.” In contrast, standard benchmarks for compositional reasoning do not explicitly capture a notion of reasoning under uncertainty or evaluate compositional concept acquisition. We introduce a new benchmark, Compositional Reasoning Under Uncertainty (CURI) that instantiates a series of few-shot, meta-learning tasks in a productive concept space to evaluate different aspects of systematic generalization under uncertainty, including splits that test abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, we also contribute a model-independent “compositionality gap” to evaluate the difficulty of generalizing out-of-distribution along each of these axes, allowing objective comparison of the difficulty of each compositional split. Evaluations across a range of modeling choices and splits reveal substantial room for improvement on the proposed benchmark.

Author Information

Rama Vedantam (Facebook AI Research)
Arthur Szlam (Facebook)
Max Nickel (Facebook AI Research)
Ari Morcos (Facebook AI Research (FAIR))
Brenden Lake (New York University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors