Maximum-Likelihood Learning of Latent Dynamics Without Reconstruction

We address the challenge of uncovering systematic, and potentially controllable, dynamical structure underlying complex high-dimensional time-series data. Existing generative and autoregressive models have difficulty telling systematic content apart from distractors, while contrastive methods struggle to learn accurate dynamics. To address these shortcomings, we develop the Recognition-Parametrized Gaussian State-Space Model (RP-GSSM), a probabilistic framework that infers accurate latent dynamics without relying on a parametrized decoder. By eliminating explicit generative parameters, the model directs its entire representational capacity to encoding dynamically relevant state; and, being fully probabilistic, learns via maximum likelihood without auxiliary objectives or ad-hoc regularization. Combining the expressive power of a neural network encoder with exact inference under a jointly Gaussian prior allows the RP-GSSM to embed a broad class of intrinsically nonlinear dynamical systems. The RP-GSSM parameters are provably linearly identifiable in the infinite data limit. We show that the RP-GSSM recovers physically meaningful latent states from noisy video more faithfully than competing methods, more reliably identifies underlying controllable nonlinear dynamics, and remains substantially more robust to visual distractors.

Artificial and biological agents often receive data in the form of high-dimensional time series, such as video. The ability to extract relevant low-dimensional information from this data is crucial to many downstream tasks. However, relevant information may be occluded by distractors, such as changing backgrounds or movement of other agents that are irrelevant to the task. Existing approaches are generally either "generative", which reconstruct the entire video and risk focusing on irrelevant distractors, or "contrastive", which contrast different frames but do not model uncertainty and risk learning uninformative representations. We aim to combine the best of both worlds by introducing a model that accounts for uncertainty while not explicitly reconstructing frames. We show theoretically that our model is able to infer the true world state, and compare it empirically to other methods on various video tasks. The tasks include inferring dynamics of physical systems such as pendulums, tracking an agent moving around an arena, or inferring the dynamics of a moving agent from both first- and third-person point of view. Our model is also able to infer "controlled" dynamics, i.e., how an agent's actions affect its movement. In all tasks we include various distractors, such as video static, background videos, changes in camera angle, or other agents randomly moving around. In all cases, our model is able to at least match other methods' performance in the absence of distractors, and significantly outperform other methods in the presence of distractors.