Random Feature Representation Boosting

We introduce Random Feature Representation Boosting (RFRBoost), a novel method for constructing deep residual random feature neural networks (RFNNs) using boosting theory. RFRBoost uses random features at each layer to learn the functional gradient of the network representation, enhancing performance while preserving the convex optimization benefits of RFNNs. In the case of MSE loss, we obtain closed-form solutions to greedy layer-wise boosting with random features. For general loss functions, we show that fitting random feature residual blocks reduces to solving a quadratically constrained least squares problem. Through extensive numerical experiments on tabular datasets for both regression and classification, we show that RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in the small- to medium-scale regime where RFNNs are typically applied. Moreover, RFRBoost offers substantial computational benefits, and theoretical guarantees stemming from boosting theory.

A surprisingly effective family of machine learning models is the class of single hidden layer neural networks whose weights are randomly initialized and never trained, where only the final linear layer is learned. These are known as random feature neural networks (RFNNs). We asked ourselves: can we significantly improve the performance of such untrained networks by increasing their depth, without relying on end-to-end training via stochastic gradient descent?One of the key innovations behind the success of deep neural networks has been the use of residual connections: shortcuts that help information flow more easily through very deep models. However, bringing this idea into RFNNs is challenging. Naively stacking randomly initialized, untrained layers can actually degrade performance rather than improve it.We found that the answer to our question is yes. By studying neural networks from the perspective of functional gradient boosting, we developed a method that builds deep models by adding random feature residual blocks one at a time, each designed to correct the errors of the previous layers. This approach preserves the computational efficiency of RFNNs, while improving accuracy via increased depth. Through extensive experiments on tabular regression and classification tasks, we show that this approach leads to significant performance gains over existing models.