Continuous-time event-based GRU for activity-sparse inference and learning

The scalability of recurrent neural networks (RNNs) is hindered by the sequential dependence of each time step’s computation on the previous time step’s output. Therefore, one way to speed up and scale RNNs is to reduce the computation required at each time step independent of model size and task. In this paper, we propose a time-continuous event-based model (EGRU) that extends Gated Recurrent Units (GRU) with an event-generation mechanism. This mechanism enforces activity-sparsity in time, and allows our model’s units to compute updates only on receipt of input events from other units. The combination of activity-sparsity and event-based computation has the potential to be computationally vastly more efficient than current RNNs. Notably, activity-sparsity in our model also translates into sparse parameter updates during gradient descent, extending this compute efficiency to the training phase. This sets the stage for the next generation of recurrent networks that are more scalable and efficient.