We establish, for the first time, explicit connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.
Liwen Zhang (University of Chicago)
Gregory Naisat (The University of Chicago)
Gregory Naitzat received his B.Sc. in Physics and in Electrical Engineering in 2008, and his M.Sc. degree in Electrical Engineering in 2014, both from the Technion – Israel Institute of Technology, Haifa, Israel. He is currently a Ph.D. student in Statistics at the University of Chicago, USA. He was a machine learning intern at SAS, USA, and was an FPGA engineer in Rafael Advanced Defense Systems, Haifa, Israel, from 2008—2015.
Lek-Heng Lim (University of Chicago)
Related Events (a corresponding poster, oral, or spotlight)
2018 Oral: Tropical Geometry of Deep Neural Networks »
Thu Jul 12th 12:30 -- 12:50 PM Room K1
More from the Same Authors
2020 Poster: Recht-Re Noncommutative Arithmetic-Geometric Mean Conjecture is False »
Zehua Lai · Lek-Heng Lim