Timezone: »

Generalization Error of Generalized Linear Models in High Dimensions
Melikasadat Emami · Mojtaba Sahraee-Ardakan · Parthe Pandit · Sundeep Rangan · Alyson Fletcher

Thu Jul 16 09:00 AM -- 09:45 AM & Thu Jul 16 08:00 PM -- 08:45 PM (PDT) @ None #None

At the heart of machine learning lies the question of generalizability of learned rules over previously unseen data.
While over-parameterized models based on neural networks are now ubiquitous in machine learning applications, our understanding of their generalization capabilities is incomplete and this task is made harder by the non-convexity of the underlying learning problems.
We provide a general framework to characterize the asymptotic generalization error for single-layer neural networks (i.e., generalized linear models) with arbitrary non-linearities, making it applicable to regression as well as classification problems. This framework enables analyzing the effect of (i) over-parameterization and non-linearity during modeling; (ii) choices of loss function, initialization, and regularizer during learning; and (iii) mismatch between training and test distributions. As examples, we analyze a few special cases, namely linear regression and logistic regression. We are also able to rigorously and analytically explain the \emph{double descent} phenomenon in generalized linear models.

Author Information

Melikasadat Emami (University of California Los Angeles)
Mojtaba Sahraee-Ardakan (UCLA)
Parthe Pandit (UCLA)
Sundeep Rangan (NYU)
Alyson Fletcher (UCLA)

More from the Same Authors