Timezone: »

The Multilinear Structure of ReLU Networks
Thomas Laurent · James von Brecht

Thu Jul 12 09:15 AM -- 12:00 PM (PDT) @ Hall B #100

We study the loss surface of neural networks equipped with a hinge loss criterion and ReLU or leaky ReLU nonlinearities. Any such network defines a piecewise multilinear form in parameter space. By appealing to harmonic analysis we show that all local minima of such network are non-differentiable, except for those minima that occur in a region of parameter space where the loss surface is perfectly flat. Non-differentiable minima are therefore not technicalities or pathologies; they are heart of the problem when investigating the loss of ReLU networks. As a consequence, we must employ techniques from nonsmooth analysis to study these loss surfaces. We show how to apply these techniques in some illustrative cases.

Author Information

Thomas Laurent (Loyola Marymount University)
James von Brecht (CSULB)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors