Joint Learning in the Gaussian Single Index Model

We consider the problem of jointly learning a one-dimensional projection and a univariate function in high-dimensional Gaussian models. Specifically, we study predictors of the form $f(x)=\varphi^\star(\langle w^\star, x \rangle)$, where both the direction $w^\star \in \mathcal{S}_{d-1}$, the sphere of $\mathbb{R}^d$, and the function $\varphi^\star: \mathbb{R} \to \mathbb{R}$ are learned from Gaussian data. This setting captures a fundamental non-convex problem at the intersection of representation learning and nonlinear regression. We analyze the gradient flow dynamics of a natural alternating scheme and prove convergence, with a rate controlled by the information exponent reflecting the *Gaussian regularity* of the function $\varphi^\star$. Strikingly, our analysis shows that convergence still occurs even when the initial direction is negatively correlated with the target. On the practical side, we demonstrate that such joint learning can be effectively implemented using a Reproducing Kernel Hilbert Space (RKHS) adapted to the structure of the problem, enabling efficient and flexible estimation of the univariate function. Our results offer both theoretical insight and practical methodology for learning low-dimensional structure in high-dimensional settings.

Many modern learning problems involve discovering a low-dimensional structure hidden inside high-dimensional data. In this work, we study a fundamental setting where the prediction only depends on an unknown one-dimensional direction together with an unknown nonlinear function. Learning both objects simultaneously leads to a highly non-convex optimization problem, which is generally difficult to analyze theoretically. We show that a natural learning procedure can nevertheless reliably recover both the hidden direction and the nonlinear relationship from Gaussian data. Our analysis provides precise guarantees on the speed of convergence and reveals that learning can succeed even when the initial estimate is initially pointing in the “wrong” direction. We also show how the method can be implemented efficiently in practice using kernel-based techniques. Overall, our work provides new theoretical and practical insights into how low-dimensional structure can be learned from high-dimensional data.