ICML 2023

Timezone: »

Poster

MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior

Jennifer J. Sun · Markus Marks · Andrew Ulmer · Dipam Chakraborty · Brian Geuther · Edward Hayes · Heng Jia · Vivek Kumar · Sebastian Oleszko · Zachary Partridge · Milan Peelman · Alice Robie · Catherine Schretter · Keith Sheppard · Chao Sun · Param Uttarwar · Julian Wagner · Erik Werner · Joseph Parker · Pietro Perona · Yisong Yue · Kristin Branson · Ann Kennedy

Tue Jul 25 05:00 PM -- 06:30 PM (PDT) @ Exhibit Hall 1 #902

in Poster Session 2 »

Event URL: https://sites.google.com/view/computational-behavior/our-datasets/mabe2022-dataset »

We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.

[ PDF] [ Poster]

Author Information

Jennifer J. Sun (Caltech)

Markus Marks (California Institute of Technology)

Andrew Ulmer (Northwestern University)

Dipam Chakraborty (National Institute of Technology Rourkela, Dhirubhai Ambani Institute Of Information and Communication Technology)

Brian Geuther (The Jackson Laboratory)

Edward Hayes (Imperial College London, Imperial College London)

Heng Jia (Zhejiang University)

Vivek Kumar (The Jackson Laboratory)

Sebastian Oleszko (IRLAB Therapeutics)

Zachary Partridge (University of New South Wales)

Milan Peelman (Universiteit Gent)

Alice Robie (HHMI Janelia Research Campus)

Catherine Schretter (HHMI Janelia Research Campus)

Keith Sheppard (The Jackson Laboratory)

Chao Sun (Zhejiang University)

Param Uttarwar (Saarland University, Universität des Saarlandes)

Julian Wagner (California Institute of Technology)

Erik Werner (IRLAB Therapeutics)

Joseph Parker (California Institute of Technology)

Pietro Perona (caltech.edu)

Yisong Yue (Caltech & Latitude AI)

Yisong Yue is a Professor of Computing and Mathematical Sciences at Caltech and (via sabbatical) a Principal Scientist at Latitude AI. His research interests span both fundamental and applied pursuits, from novel learning-theoretic frameworks all the way to deep learning deployed in autonomous driving on public roads. His work has been recognized with multiple paper awards and nominations, including in robotics, computer vision, sports analytics, machine learning for health, and information retrieval. At Latitude AI, he is working on machine learning approaches to motion planning for autonomous driving.