Abstract

Learning and Discovery of Predictive State Representations in Dynamical Systems with Reset
Michael James - University of Michigan Satinder Singh - University of Michigan
Predictive state representations (PSRs) are a recently proposed way ofmodeling controlled dynamical systems. PSR-based models use predictions ofobservable outcomes of tests that could be done on the system as their staterepresentation, and have model parameters that define how the predictive staterepresentation changes over time as actions are taken and observations noted.Learning PSR-based models requires solving two subproblems: 1) discovery ofthe tests whose predictions constitute state, and 2) learning the modelparameters that define the dynamics. So far, there have been no resultsavailable on the discovery subproblem while for the learning subproblem anapproximate-gradient algorithm has been proposed (Singh et al., 2003) withmixed results (it works on some domains and not on others). In this paper, weprovide the first discovery algorithm and a new learning algorithm for linearPSRs for the special class of controlled dynamical systems that have a resetoperation. We provide experimental verification of our algorithms. Finally,we also distinguish our work from prior work by Jaeger (2000) on observableoperator models (OOMs).

Learning and Discovery of Predictive State Representations in Dynamical Systems with Reset

Michael James - University of Michigan
Satinder Singh - University of Michigan

Predictive state representations (PSRs) are a recently proposed way ofmodeling controlled dynamical systems. PSR-based models use predictions ofobservable outcomes of tests that could be done on the system as their staterepresentation, and have model parameters that define how the predictive staterepresentation changes over time as actions are taken and observations noted.Learning PSR-based models requires solving two subproblems: 1) discovery ofthe tests whose predictions constitute state, and 2) learning the modelparameters that define the dynamics. So far, there have been no resultsavailable on the discovery subproblem while for the learning subproblem anapproximate-gradient algorithm has been proposed (Singh et al., 2003) withmixed results (it works on some domains and not on others). In this paper, weprovide the first discovery algorithm and a new learning algorithm for linearPSRs for the special class of controlled dynamical systems that have a resetoperation. We provide experimental verification of our algorithms. Finally,we also distinguish our work from prior work by Jaeger (2000) on observableoperator models (OOMs).