Re-understanding Finite-State Representations of Recurrent Policy Networks

Mohamad H Danesh · Anurag Koul · Alan Fern · Saeed Khorram


Keywords: [ Fairness, Accountability, and Transparency ] [ Social Aspects of Machine Learning ]

[ Abstract ]
[ Slides
[ Paper ]
[ Visit Poster at Spot D3 in Virtual World ]
Tue 20 Jul 9 p.m. PDT — 11 p.m. PDT
Spotlight presentation: Reinforcement Learning 3
Tue 20 Jul 5 p.m. PDT — 6 p.m. PDT


We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine's operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.

Chat is not available.