Timezone: »
Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by consisting of an ensemble of effective features. In this paper, we adopt this viewpoint to construct a new gradient boosting method, which is known to be very powerful in data analysis. To do so, we formalize the boosting perspective of ResNet mathematically using the notion of functional gradients and propose a new method called ResFGB for classification tasks by leveraging ResNet perception. Two types of generalization guarantees are provided from the optimization perspective: one is the margin bound and the other is the expected risk bound by the sample-splitting technique. Experimental results show superior performance of the proposed method over state-of-the-art methods such as LightGBM.
Author Information
Atsushi Nitanda (The University of Tokyo / RIKEN)
Taiji Suzuki (The University of Tokyo / RIKEN)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Functional Gradient Boosting based on Residual Network Perception »
Thu. Jul 12th 02:40 -- 02:50 PM Room A6
More from the Same Authors
-
2023 : Benign Overfitting of Two-Layer Neural Networks under Inputs with Intrinsic Dimension »
Shunta Akiyama · Kazusato Oko · Taiji Suzuki -
2023 : Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective »
Wei Huang · Yuan Cao · Haonan Wang · Xin Cao · Taiji Suzuki -
2023 : Learning in the Presence of Low-dimensional Structure: A Spiked Random Matrix Perspective »
Jimmy Ba · Murat Erdogdu · Taiji Suzuki · Zhichao Wang · Denny Wu -
2023 : Learning Green's Function Efficiently Using Low-Rank Approximations »
Kishan Wimalawarne · Taiji Suzuki · Sophie Langer -
2023 Poster: DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning »
Tomoya Murata · Taiji Suzuki -
2023 Poster: Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems »
Atsushi Nitanda · Kazusato Oko · Denny Wu · Nobuhito Takenouchi · Taiji Suzuki -
2023 Oral: Diffusion Models are Minimax Optimal Distribution Estimators »
Kazusato Oko · Shunta Akiyama · Taiji Suzuki -
2023 Poster: Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input »
Shokichi Takakura · Taiji Suzuki -
2023 Poster: Diffusion Models are Minimax Optimal Distribution Estimators »
Kazusato Oko · Shunta Akiyama · Taiji Suzuki -
2023 Poster: Tight and fast generalization error bound of graph embedding in metric space »
Atsushi Suzuki · Atsushi Nitanda · Taiji Suzuki · Jing Wang · Feng Tian · Kenji Yamanishi -
2021 Poster: On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting »
Shunta Akiyama · Taiji Suzuki -
2021 Spotlight: On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Setting »
Shunta Akiyama · Taiji Suzuki -
2021 Poster: Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding »
Akira Nakagawa · Keizo Kato · Taiji Suzuki -
2021 Spotlight: Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding »
Akira Nakagawa · Keizo Kato · Taiji Suzuki -
2021 Poster: Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning »
Tomoya Murata · Taiji Suzuki -
2021 Spotlight: Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning »
Tomoya Murata · Taiji Suzuki -
2019 Poster: Approximation and non-parametric estimation of ResNet-type convolutional neural networks »
Kenta Oono · Taiji Suzuki -
2019 Oral: Approximation and non-parametric estimation of ResNet-type convolutional neural networks »
Kenta Oono · Taiji Suzuki