The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.
Adam Dziedzic (University of Chicago)
I'm a PhD student at the University of Chicago, advised by Prof. Sanjay Krishnan. I'm passionate about empowering users with data driven decisions. Currently, I work on the Band-Limited convolutional neural networks (ConvNets) and the DeepLens project. I was an intern at Microsoft Research and worked on recommendation of hybrid physical designs (B+ tree and Columnstores) for SQL Server. Currently, I work on the intersection of databases and machine/deep learning. I obtained my Bachelor's and Master's degrees from Warsaw University of Technology in Poland. I was also studying at DTU (Technical University of Denmark) and carried out research on databases in the DIAS group at EPFL, Switzerland. Previously, I had internships at CERN (Geneva, Switzerland), Barclays Investment Bank in London (UK), Microsoft Research (Redmond, USA) and Google (Madison, USA). I spend the rest of my waking hours on reading books, taking MOOC courses, especially on Machine & Deep Learning, enjoying many sports, playing a guitar and practicing Aikido.
John Paparrizos (University of Chicago)
Sanjay Krishnan (U Chicago)
Aaron Elmore (University of Chicago)
Michael Franklin (University of Chicago)
Related Events (a corresponding poster, oral, or spotlight)
2019 Oral: Band-limited Training and Inference for Convolutional Neural Networks »
Wed Jun 12th 09:20 -- 09:25 PM Room Seaside Ballroom