The Data Manifold under the Microscope

A significant gap exists between theory and practice in deep learning. Generalization and approximation error bounds are often derived for simplified models or are too loose to be informative. Many rely on the manifold hypothesis and on geometric regularity such as intrinsic dimension, curvature, and reach. Progress requires insight into data-manifold geometry and suitable benchmarks, yet existing options are polarized: analytic manifolds with known geometry but limited applicability, or real-world datasets where geometry is only coarsely estimable. We introduce a benchmarking framework for studying data geometry. We repurpose and extend dSprites and COIL-20 with additional transformation dimensions and dense, axis-aligned sampling, and pair them with finite-difference estimators that recover curvature, reach, and volume at near-ground-truth accuracy in a regime where general-purpose estimators are unreliable or difficult to deploy. The framework is intended as a controlled testbed, useful as a calibration environment for geometric estimators and a sandbox for probing theoretical assumptions. To illustrate its use, we present two application studies, namely assessing the scaling behavior of the bounds of Genovese et al. and Fefferman et al., and tracking the layer-wise geometry of a $\beta$-VAE, highlighting the behavior of current bounds and the value of controlled benchmarks for guiding and validating future theory. A reference implementation is available at https://github.com/koulakis/manifold-microscope.

Modern AI systems can recognize images and make predictions impressively well, but our mathematical understanding of why they work still lags behind. Many explanations assume that real data has a simpler hidden shape, for example because images of the same object change smoothly as the object moves, rotates, or changes size. This idea is hard to test because clean mathematical examples are often too simple, while real image collections are too messy to measure precisely. We introduce a benchmark that creates controlled image collections where this hidden shape can be measured much more accurately. We build these collections by starting from simple object datasets and varying factors such as position, size, rotation, and viewpoint in a dense and organized way. This allows researchers to measure how large, curved, and folded the hidden shape is, and to check whether existing mathematical predictions match observed behavior. We use the benchmark to study two cases, how current error bounds scale with data size, and how a network changes data geometry inside its layers. Our work provides a practical testing ground for checking theories of deep learning and for developing better tools to measure the geometry of data.