Topologically Densified Distributions

Christoph Hofer · Florian Graf · Marc Niethammer · Roland Kwitt

Keywords: [ Deep Learning Theory ] [ Representation Learning ] [ Statistical Learning Theory ] [ Supervised Learning ] [ Deep Learning - Theory ]

[ Abstract ] [ Join Zoom
Please do not share or post zoom links


We study regularization in the context of small sample-size learning with over-parametrized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network weights, to properties of the internal representations before a linear classifier. Specifically, we impose a topological constraint on samples drawn from the probability measure induced in that space. This provably leads to mass concentration effects around the representations of training instances, i.e., a property beneficial for generalization. By leveraging previous work to impose topological constrains in a neural network setting, we provide empirical evidence (across various vision benchmarks) to support our claim for better generalization.

Chat is not available.