Timezone: »

 
Poster
Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Sets
Lily Zhang · Veronica Tozzo · John Higgins · Rajesh Ranganath

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #524
in Poster Session 2 »

Permutation invariant neural networks are a promising tool for predictive modeling of set data. We show, however, that existing architectures struggle to perform well when they are deep. In this work, we mathematically and empirically analyze normalization layers and residual connections in the context of deep permutation invariant neural networks. We develop set norm, a normalization tailored for sets, and introduce the ``clean path principle'' for equivariant residual connections alongside a novel benefit of such connections, the reduction of information loss. Based on our analysis, we propose Deep Sets++ and Set Transformer++, deep models that reach comparable or better performance than their original counterparts on a diverse suite of tasks. We additionally introduce Flow-RBC, a new single-cell dataset and real-world application of permutation invariant prediction. We open-source our data and code here: https://github.com/rajesh-lab/deeppermutationinvariant.

Keywords: DL: Everything Else

Author Information

Lily Zhang (New York University)
Veronica Tozzo (Massachusets General Hospital Harvard Medical School)

Veronica Tozzo is a research fellow at Massachusetts General Hospital and Harvard Medical School. Her current research revolves around the understanding of blood cells systems and how single cell information can be exploited to improve clinical care. She is particularly interested in understanding red blood cell ageing kinetics with the goal of improving diagnosis and monitoring of blood related diseases with special consideration for diabetes. She obtained her PhD in Computer Science at the University of Genova with a thesis that focuses on the temporal inference of graphical models for the understanding of complex temporal systems.

John Higgins (Massachusets General Hospital - Harvard Medical School)
Rajesh Ranganath (New York University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors