Timezone: »

 
Just-in-Time Sparsity: Learning Dynamic Sparsity Schedules
· Chiratidzo Matowe · Arnu Pretorius · Benjamin Rosman · Sara Hooker

Fri Jul 22 12:15 PM -- 01:15 PM (PDT) @
in Dynamic Neural Networks »

Sparse neural networks have various computational benefits while often being able to maintain or improve the generalization performance of their dense counterparts. Popular sparsification methods have focused on what to sparsify, i.e. which redundant components to remove from neural networks, while when to sparsify, has received less attention and is usually handled using heuristics or simple schedules. In this work, we focus on learning sparsity schedules from scratch using reinforcement learning. In simple CNNs and ResNet-18, we show that our learned schedules are diverse across layers and training steps, while achieving competitive performance when compared to naive handcrafted schedules. Our methodology is general-purpose and can be applied to learning effective sparsity schedules across any pruning implementation.

Author Information

Chiratidzo Matowe (InstaDeep)
Arnu Pretorius (InstaDeep)
Benjamin Rosman (University of the Witwatersrand, South Africa)

Benjamin Rosman received a Ph.D. degree in Informatics from the University of Edinburgh in 2014. Previously, he obtained an M.Sc. in Artificial Intelligence from the University of Edinburgh, a Bachelor of Science (Honours) in Computer Science from the University of the Witwatersrand, South Africa, and a Bachelor of Science (Honours) in Computational and Applied Mathematics, also from the University of the Witwatersrand. He is an Associate Professor in the School of Computer Science and Applied Mathematics at the University of the Witwatersrand. He is the Chair of the IEEE South African joint chapter of Control Systems, and Robotics and Automation. Prof. Rosman’s research interests focus primarily on learning and decision making in autonomous systems, in particular studying how learning can be accelerated through abstracting and generalising knowledge gained from solving previous problems. He additionally works in the area of skill and behaviour learning for robots.

Sara Hooker (Cohere)

More from the Same Authors