Deep learning-based approaches for time series analysis notoriously suffer from interpretability and robustness issues due to their black-box nature. In this work, we propose a hybrid neural network model with embedded expert knowledge. We assume the time series are generated by a finite set of dynamics with known functional form. Our experiments show that our approach is more interpretable, and better at reconstruction than its black-box counterparts.