Timezone: »
It is common practice in deep learning to represent a measurement of the world on a discrete grid, e.g. a 2D grid of pixels. However, the underlying signal represented by these measurements is often continuous, e.g. the scene depicted in an image. A powerful continuous alternative is then to represent these measurements using an \textit{implicit neural representation}, a neural function trained to output the appropriate measurement value for any input spatial location. In this paper, we take this idea to its next level: what would it take to perform deep learning on these functions instead, treating them as data? In this context we refer to the data as \textit{functa}, and propose a framework for deep learning on functa. This view presents a number of challenges around efficient conversion from data to functa, compact representation of functa, and effectively solving downstream tasks on functa. We outline a recipe to overcome these challenges and apply it to a wide range of data modalities including images, 3D shapes, neural radiance fields (NeRF) and data on manifolds. We demonstrate that this approach has various compelling properties across data modalities, in particular on the canonical tasks of generative modeling, data imputation, novel view synthesis and classification.
Author Information
Emilien Dupont (University of Oxford)
Hyunjik Kim (DeepMind)
S. M. Ali Eslami (DeepMind)

S. M. Ali Eslami is a staff research scientist at DeepMind working on problems related to artificial intelligence. Prior to that, he was a post-doctoral researcher at Microsoft Research in Cambridge. He did his PhD in the School of Informatics at the University of Edinburgh, during which he was also a visiting researcher in the Visual Geometry Group at the University of Oxford. His research is focused on figuring out how we can get computers to learn with less human supervision.
Danilo J. Rezende (DeepMind)

Danilo is a Senior Staff Research Scientist at Google DeepMind, where he works on probabilistic machine reasoning and learning algorithms. He has a BA in Physics and MSc in Theoretical Physics from Ecole Polytechnique (Palaiseau – France) and from the Institute of Theoretical Physics (SP – Brazil) and a Ph.D. in Computational Neuroscience at Ecole Polytechnique Federale de Lausanne, EPFL (Lausanne – Switzerland). His research focuses on scalable inference methods, generative models of complex data (such as images and video), applied probability, causal reasoning and unsupervised learning for decision-making.
Dan Rosenbaum (University of Haifa)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Spotlight: From data to functa: Your data point is a function and you can treat it like one »
Wed. Jul 20th 05:15 -- 05:20 PM Room Ballroom 1 & 2
More from the Same Authors
-
2022 : Learning to induce causal structure »
Rosemary Nan Ke · Silvia Chiappa · Jane Wang · Jorg Bornschein · Anirudh Goyal · Melanie Rey · Matthew Botvinick · Theophane Weber · Michael Mozer · Danilo J. Rezende -
2023 Poster: Learning Instance-Specific Augmentations by Capturing Local Invariances »
Ning Miao · Tom Rainforth · Emile Mathieu · Yann Dubois · Yee-Whye Teh · Adam Foster · Hyunjik Kim -
2022 Poster: Continual Repeated Annealed Flow Transport Monte Carlo »
Alexander Matthews · Michael Arbel · Danilo J. Rezende · Arnaud Doucet -
2022 Spotlight: Continual Repeated Annealed Flow Transport Monte Carlo »
Alexander Matthews · Michael Arbel · Danilo J. Rezende · Arnaud Doucet -
2021 Workshop: INNF+: Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models »
Chin-Wei Huang · David Krueger · Rianne Van den Berg · George Papamakarios · Ricky T. Q. Chen · Danilo J. Rezende -
2021 Oral: NeRF-VAE: A Geometry Aware 3D Scene Generative Model »
Adam Kosiorek · Heiko Strathmann · Daniel Zoran · Pol Moreno · Rosalia Schneider · Sona Mokra · Danilo J. Rezende -
2021 Poster: LieTransformer: Equivariant Self-Attention for Lie Groups »
Michael Hutchinson · Charline Le Lan · Sheheryar Zaidi · Emilien Dupont · Yee-Whye Teh · Hyunjik Kim -
2021 Poster: NeRF-VAE: A Geometry Aware 3D Scene Generative Model »
Adam Kosiorek · Heiko Strathmann · Daniel Zoran · Pol Moreno · Rosalia Schneider · Sona Mokra · Danilo J. Rezende -
2021 Spotlight: LieTransformer: Equivariant Self-Attention for Lie Groups »
Michael Hutchinson · Charline Le Lan · Sheheryar Zaidi · Emilien Dupont · Yee-Whye Teh · Hyunjik Kim -
2020 : Invited talk 3: Representational limitations of invertible models »
Emilien Dupont -
2020 Workshop: INNF+: Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models »
Chin-Wei Huang · David Krueger · Rianne Van den Berg · George Papamakarios · Chris Cremer · Ricky T. Q. Chen · Danilo J. Rezende -
2020 Poster: Equivariant Neural Rendering »
Emilien Dupont · Miguel Angel Bautista Martin · Alex Colburn · Aditya Sankar · Joshua M Susskind · Qi Shan -
2020 Poster: PolyGen: An Autoregressive Generative Model of 3D Meshes »
Charlie Nash · Yaroslav Ganin · S. M. Ali Eslami · Peter Battaglia -
2020 Poster: Normalizing Flows on Tori and Spheres »
Danilo J. Rezende · George Papamakarios · Sebastien Racaniere · Michael Albergo · Gurtej Kanwar · Phiala Shanahan · Kyle Cranmer -
2020 Tutorial: Representation Learning Without Labels »
S. M. Ali Eslami · Irina Higgins · Danilo J. Rezende -
2019 Workshop: Invertible Neural Networks and Normalizing Flows »
Chin-Wei Huang · David Krueger · Rianne Van den Berg · George Papamakarios · Aidan Gomez · Chris Cremer · Aaron Courville · Ricky T. Q. Chen · Danilo J. Rezende -
2019 : Poster discussion »
Roman Novak · Maxime Gabella · Frederic Dreyer · Siavash Golkar · Anh Tong · Irina Higgins · Mirco Milletari · Joe Antognini · Sebastian Goldt · Adín Ramírez Rivera · Roberto Bondesan · Ryo Karakida · Remi Tachet des Combes · Michael Mahoney · Nicholas Walker · Stanislav Fort · Samuel Smith · Rohan Ghosh · Aristide Baratin · Diego Granziol · Stephen Roberts · Dmitry Vetrov · Andrew Wilson · César Laurent · Valentin Thomas · Simon Lacoste-Julien · Dar Gilboa · Daniel Soudry · Anupam Gupta · Anirudh Goyal · Yoshua Bengio · Erich Elsen · Soham De · Stanislaw Jastrzebski · Charles H Martin · Samira Shabanian · Aaron Courville · Shorato Akaho · Lenka Zdeborova · Ethan Dyer · Maurice Weiler · Pim de Haan · Taco Cohen · Max Welling · Ping Luo · zhanglin peng · Nasim Rahaman · Loic Matthey · Danilo J. Rezende · Jaesik Choi · Kyle Cranmer · Lechao Xiao · Jaehoon Lee · Yasaman Bahri · Jeffrey Pennington · Greg Yang · Jiri Hron · Jascha Sohl-Dickstein · Guy Gur-Ari -
2018 Poster: Synthesizing Programs for Images using Reinforced Adversarial Learning »
Iaroslav Ganin · Tejas Kulkarni · Igor Babuschkin · S. M. Ali Eslami · Oriol Vinyals -
2018 Oral: Synthesizing Programs for Images using Reinforced Adversarial Learning »
Iaroslav Ganin · Tejas Kulkarni · Igor Babuschkin · S. M. Ali Eslami · Oriol Vinyals -
2018 Poster: Generative Temporal Models with Spatial Memory for Partially Observed Environments »
Marco Fraccaro · Danilo J. Rezende · Yori Zwols · Alexander Pritzel · S. M. Ali Eslami · Fabio Viola -
2018 Poster: Disentangling by Factorising »
Hyunjik Kim · Andriy Mnih -
2018 Poster: Machine Theory of Mind »
Neil Rabinowitz · Frank Perbet · Francis Song · Chiyuan Zhang · S. M. Ali Eslami · Matthew Botvinick -
2018 Poster: Conditional Neural Processes »
Marta Garnelo · Dan Rosenbaum · Chris Maddison · Tiago Ramalho · David Saxton · Murray Shanahan · Yee Teh · Danilo J. Rezende · S. M. Ali Eslami -
2018 Oral: Machine Theory of Mind »
Neil Rabinowitz · Frank Perbet · Francis Song · Chiyuan Zhang · S. M. Ali Eslami · Matthew Botvinick -
2018 Oral: Disentangling by Factorising »
Hyunjik Kim · Andriy Mnih -
2018 Oral: Generative Temporal Models with Spatial Memory for Partially Observed Environments »
Marco Fraccaro · Danilo J. Rezende · Yori Zwols · Alexander Pritzel · S. M. Ali Eslami · Fabio Viola -
2018 Oral: Conditional Neural Processes »
Marta Garnelo · Dan Rosenbaum · Chris Maddison · Tiago Ramalho · David Saxton · Murray Shanahan · Yee Teh · Danilo J. Rezende · S. M. Ali Eslami