Interpretability and Generalization Bounds for Learning Spatial Physics

While there are many applications of machine learning (ML) to scientific problems that \emph{look} promising, the eye test can be misleading compared to the quantitative values. Using numerical analysis techniques, we rigorously quantify the accuracy, convergence rates, and generalization bounds of certain ML models applied to linear differential equations (DEs) for parameter discovery or solution finding. Beyond the quantity and discretization of data, we identify that the {function space} of the data is critical to the generalization of the model which can lead to divergence. Similar lack of generalization is empirically demonstrated for commonly used models. Surprisingly, we find that different classes of models can exhibit opposing generalization behaviors. Based on our theoretical analysis, we also introduce a new mechanistic interpretability lens on scientific models whereby Green's function representations can be extracted from the weights of black-box models. Our results inform a new cross-validation technique for measuring generalization in physical systems, and can be useful as a benchmark of future methods.

Machine learning is increasingly applied to physical simulations, predicting how heat spreads, water flows, or materials deform. These models all have to go through a process called "training" which requires large amount of data. We want to see if these models can learn information that are not in the data themselves, but whether the models are learning the physics. We concretely studies this question for a special equation called the 1D Poisson equation, which is foundational to mathematics. We showed rigorously that a model ability to learn the physics is highly dependent on the "type" of function used (the function space), and not necessarily the number of data. Thus, we're saying that the "quality" of the data highly matters to these scientific learning models. We examined this via testing on a variety of different scientific machine learning model types. We also use something called Green's function to easily extract, visualize and diagnose whether a particular model is able to generalize well. This is because Green's function is the "optimal" solution for our Poisson equation.