Spectrum-Informed Initialization for Multistage Neural Networks

Deep learning frameworks have become powerful tools for approaching scientific problems such as turbulent flow, which has wide-ranging applications. In practice, however, existing scientific machine learning approaches have difficulty fitting complex, multi-scale dynamical systems to very high precision, as required in scientific contexts. We propose using the novel multi-stage neural net approach with a spectrum-informed initialization to learn the residue from the previous stage, utilizing the spectral biases associated with neural nets to capture high frequency features in the residue, and successfully tackle the spectral bias of neural nets. This approach allows the neural net to fit target functions to double floating-point machine precision $O(10^{-16})$.