Timezone: »

Elementary superexpressive activations
Dmitry Yarotsky

Thu Jul 22 08:35 PM -- 08:40 PM (PDT) @ None

We call a finite family of activation functions \emph{superexpressive} if any multivariate continuous function can be approximated by a neural network that uses these activations and has a fixed architecture only depending on the number of input variables (i.e., to achieve any accuracy we only need to adjust the weights, without increasing the number of neurons). Previously, it was known that superexpressive activations exist, but their form was quite complex. We give examples of very simple superexpressive families: for example, we prove that the family {sin, arcsin} is superexpressive. We also show that most practical activations (not involving periodic functions) are not superexpressive.

Author Information

Dmitry Yarotsky (Skolkovo Institute of Science and Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors