Data Reconstruction: Identifiability and Optimization with Sample Splitting

Training data reconstruction from KKT conditions has shown striking empirical success, yet it remains unclear when the resulting KKT equations have unique solutions and, even in identifiable regimes, how to reliably recover solutions by optimization. This work hereby focuses on these two complementary questions: identifiability and optimization. On the identifiability side, we discuss the sufficient conditions for KKT system of two-layer networks with polynomial activations to uniquely determine the training data, providing a theoretical explanation of when and why reconstruction is possible. On the optimization side, we introduce sample splitting, a curvature-aware refinement step applicable to general reconstruction objectives (not limited to KKT-based formulations): it creates additional descent directions to escape poor stationary points and refine solutions. Experiments demonstrate that augmenting several existing reconstruction methods with sample splitting consistently improves reconstruction performance. Code is available at https://github.com/mukerr/data_recon.

When a machine learning model is trained, it quietly "remembers" information about its training data — even if that data is never explicitly stored. This raises a serious privacy concern: could someone reconstruct private training examples just by analyzing a trained model? Recent work has shown this is empirically possible using mathematical conditions called KKT equations, but two fundamental questions remained open: when is reconstruction theoretically guaranteed, and how can we reliably compute it? We tackle both questions. First, we identify precise conditions under which a trained two-layer neural network uniquely determines its original training data, explaining why reconstruction succeeds in some settings but not others. Second, we introduce a technique called sample splitting, which helps reconstruction algorithms escape dead ends and find better solutions by generating new search directions when optimization gets stuck. It can be plugged into many existing reconstruction methods. Our results deepen the theoretical understanding of privacy risks in machine learning and provide a practical tool that consistently improves reconstruction performance across multiple methods.