Looking at Deep Learning Phenomena Through a Telescoping Lens

Deep learning sometimes appears to work in unexpected ways. In pursuit of deeper understanding of its surprising behaviors, we investigate the utility of a tractable and accurate model of a neural network consisting of a sequence of first-order approximations telescoping out into a single empirically operational tool for practical analysis. We illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena in the literature -- including double descent, grokking, and the challenges of applying deep learning on tabular data.