What Survives of Path Norms? Path-Lifting as an Intermediate Representation for ReLU Networks

More than a decade ago, Neyshabur et al. introduced path norms to define complexity measures over classes of functions implemented by ReLU networks, tightening existing bounds by factoring out intrinsic rescaling-invariances of the weight-space parameterization of such networks. While conceptually exciting, this however did not fully match expectations, as path norms bounds often remain several orders of magnitude too large to provide stand-alone quantitative bounds, e.g., on generalization error or Lipschitz constants. Path-norms are however only the most visible facet of a toolset built on top of path-lifting and path-activations, two complementary rescaling-invariant representations of ReLU networks. This short perspective paper brings these layers back to the foreground: the lasting contribution of the introduction of path norms is not a single privileged---but overly pessimistic---scalar complexity measure, but a representational toolset for formulating weight-space questions after rescaling has been factored out. As we highlight, this toolset keeps reappearing in invariant embeddings for identifiability, symmetry-aware optimization, conservation laws for gradient flow, pruning, and recent PAC-Bayes analyses, to name a few.