Timezone: »

 
Spotlight
MC-LSTM: Mass-Conserving LSTM
Pieter-Jan Hoedt · Frederik Kratzert · Daniel Klotz · Christina Halmich · Markus Holzleitner · Grey Nearing · Sepp Hochreiter · Günter Klambauer

Wed Jul 21 07:25 PM -- 07:30 PM (PDT) @
in Applications 1 »

The success of Convolutional Neural Networks (CNNs) in computer vision is mainly driven by their strong inductive bias, which is strong enough to allow CNNs to solve vision-related tasks with random weights, meaning without learning. Similarly, Long Short-Term Memory (LSTM) has a strong inductive bias towards storing information over time. However, many real-world systems are governed by conservation laws, which lead to the redistribution of particular quantities — e.g.in physical and economical systems. Our novel Mass-Conserving LSTM (MC-LSTM) adheres to these conservation laws by extending the inductive bias of LSTM to model the redistribution of those stored quantities. MC-LSTMs set a new state-of-the-art for neural arithmetic units at learning arithmetic operations, such as addition tasks,which have a strong conservation law, as the sum is constant over time. Further, MC-LSTM is applied to traffic forecasting, modeling a pendulum, and a large benchmark dataset in hydrology, where it sets a new state-of-the-art for predicting peak flows. In the hydrology example, we show that MC-LSTM states correlate with real world processes and are therefore interpretable.

Keywords: Applications

Author Information

Pieter-Jan Hoedt (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)
Frederik Kratzert (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)
Daniel Klotz (Ellis Unit / University Linz)
Christina Halmich (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)
Markus Holzleitner (LIT AI Lab / University Linz)
Grey Nearing (Google Research)
Sepp Hochreiter (LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria)
Sepp Hochreiter

Sepp Hochreiter is heading the Institute for Machine Learning, the ELLIS Unit Linz, the LIT AI Lab at the JKU Linz and is director of private research institute IARAI. He is a pioneer of Deep Learning as he discovered the famous problem of vanishing or exploding gradients and invented the long short-term memory (LSTM).

Günter Klambauer (Johannes Kepler University Linz Austria)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors