Generalized additive models via direct optimization of regularized decision stump forests

We explore ensembles of axis-aligned decision stumps, which can be viewed as a generalized additive model (GAM). In this model, stumps utilizing the same feature are grouped to form a shape function for that feature. Instead of relying on boosting or bagging, we employ alternating optimization to learn a fixed-size stump forest. We optimize the parameters of each stump exactly through enumeration, given the other stumps are fixed. For fixed stump splits, the leaf values are optimized jointly by solving a convex problem. To address the overfitting issue inherent in naive optimization of stump forests, we propose effective regularization techniques. Our regularized stump forests achieve accuracy comparable to state-of-the-art GAM methods while using fewer parameters. This work is the first to successfully learn stump forests without employing traditional ensembling techniques like bagging or boosting.

Many AI models, such as deep neural networks, are considered "black boxes" because it is difficult, or even impossible, to understand how they make decisions. This lack of transparency poses a challenge when using AI in high-stake areas like healthcare or finance, where decisions must be trusted and explained. In contrast, interpretable models are simpler and can be more easily understood by humans, making them better suited for such domains. This paper focuses on one type of interpretable model: generalized additive models (GAMs). These models make predictions by adding up individual contributions from each input feature, which makes them easier to visualize and explain. The paper introduces a new optimization-based algorithm that can train GAMs more accurately and efficiently.