In dynamical systems reconstruction (DSR) we seek to infer from time series measurements a generative model of the underlying dynamical process, a prime objective in any scientific discipline. Like in other areas of deep learning, we are particularly interested in small and parsimonious models, with a low parameter load. A common strategy here is parameter pruning, removing all parameters with low weights presumably contributing only little to performance. However, here we find this strategy does not work for DSR: Even low magnitude parameters can contribute considerably to the system dynamics, and hence deleting them from the trained model may profoundly degrade performance. On the other hand, it is well known that many natural systems like the brain, ecological or social networks which generate complex dynamics have a sparse topology, e.g. "small world'', where only relatively few network links are required. Inspired by this observation, we show that geometric pruning, where in contrast to magnitude-based pruning weights with a low contribution to an attractor's geometrical structure are removed, indeed manages to reduce parameter load substantially without significantly hampering DSR quality. We further find that the networks resulting from geometric pruning have a specific type of topology, and that this topology (and not the magnitude of weights) is what is most crucial to performance. We provide an algorithm that automatically generates such topologies which can be used as priors for generative modeling of dynamical systems, and compare it to other well studied topologies like small-world or scale-free networks.