The convolutional layers are core building blocks of neural network architecture. In general, a convolutional filter applies to the entire frequency spectrum of an input signal. We explore artificially constraining the frequency spectra of these filters, called band-limiting, during Convolutional Neural Networks (CNN) training. The band-limiting applies to both the feedforward and backpropagation steps. Through an extensive evaluation over time-series and image datasets, we observe that CNNs are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. An extensive experimental evaluation across 1D and 2D CNN training tasks illustrates: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architecture to use unlike other compression schemes.
Author Information
Adam Dziedzic (University of Chicago)
I'm a PhD student at the University of Chicago, advised by Prof. Sanjay Krishnan. I'm passionate about empowering users with data driven decisions. Currently, I work on the Band-Limited convolutional neural networks (ConvNets) and the DeepLens project. I was an intern at Microsoft Research and worked on recommendation of hybrid physical designs (B+ tree and Columnstores) for SQL Server. Currently, I work on the intersection of databases and machine/deep learning. I obtained my Bachelor's and Master's degrees from Warsaw University of Technology in Poland. I was also studying at DTU (Technical University of Denmark) and carried out research on databases in the DIAS group at EPFL, Switzerland. Previously, I had internships at CERN (Geneva, Switzerland), Barclays Investment Bank in London (UK), Microsoft Research (Redmond, USA) and Google (Madison, USA). I spend the rest of my waking hours on reading books, taking MOOC courses, especially on Machine & Deep Learning, enjoying many sports, playing a guitar and practicing Aikido.
John Paparrizos (University of Chicago)
Sanjay Krishnan (U Chicago)
Aaron Elmore (University of Chicago)
Michael Franklin (University of Chicago)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Band-limited Training and Inference for Convolutional Neural Networks »
Wed Jun 12th 06:30 -- 09:00 PM Room Pacific Ballroom
