DRL-STAF: A Deep Reinforcement Learning Framework for State-Aware Forecasting of Complex Multivariate Hidden Markov Processes

Forecasting multivariate hidden Markov processes is challenging due to nonlinear and nonstationary observations, latent state transitions, and cross-sequence dependencies. While deep learning methods achieve strong predictive accuracy, they typically lack explicit state modeling, whereas Hidden Markov Models (HMMs) provide interpretable latent states but struggle with complex nonlinear emissions and scalability. To address these limitations, we propose DRL-STAF, a Deep Reinforcement Learning based STate-Aware Forecasting framework that jointly predicts next-step observations and estimates the corresponding hidden states for complex multivariate hidden Markov processes. Specifically, DRL-STAF models complex nonlinear emissions using deep neural networks and estimates discrete hidden states using reinforcement learning, reducing the reliance on predefined transition structures and enabling flexible adaptation to diverse temporal dynamics. In particular, DRL-STAF mitigates the state-space explosion encountered by typical multivariate HMM-based methods. Extensive experiments demonstrate that DRL-STAF outperforms HMM variants, standalone deep learning models, and existing DL-HMM hybrids in most cases, while also providing reliable hidden-state estimates.

Many systems we care about, such as financial markets, traffic flows, and industrial machines, do not follow one fixed pattern over time. They may look stable for a while, suddenly switch to another operating condition, and then change again. This makes forecasting difficult: a model must not only predict the next value, but also understand which hidden situation the system is currently in. We developed DRL-STAF to address this problem. The method combines neural networks, which are good at learning complex patterns from data, with reinforcement learning, which learns to choose hidden states through trial and error. In this way, DRL-STAF can discover useful state patterns directly from data and provide a state-aware description of how the system evolves over time. Our experiments show that DRL-STAF performs well on both forecasting and hidden-state estimation compared with several representative alternatives. This makes the model useful not only for predicting what may happen next, but also for understanding when and how a complex system changes its behavior.