Rationality Measurement and Theory for Reinforcement Learning Agents

This paper proposes a suite of rationality measures and associated theory for reinforcement learning agents, a property increasingly critical yet rarely explored. We define an action in deployment to be perfectly rational if it maximises the hidden true value function in the steepest direction. The expected value discrepancy of a policy's actions against their rational counterparts, culminating over the trajectory in deployment, is defined to be expected rational risk; an empirical average version in training is also defined. Their difference, termed as rational risk gap, is decomposed into (1) an extrinsic component caused by environment shifts between training and deployment, and (2) an intrinsic one due to the algorithm's generalisability in a dynamic environment. They are upper bounded by, respectively, (1) the $1$-Wasserstein distance between transition kernels and initial state distributions in training and deployment, and (2) the empirical Rademacher complexity of the value function class. Our theory suggests hypotheses on the benefits from regularisers (including layer normalisation, $\ell_2$ regularisation, and weight normalisation) and domain randomisation, as well as the harm from environment shifts. Experiments are in full agreement with these hypotheses. The code is available at \href{https://github.com/EVIEHub/Rationality}{https://github.com/EVIEHub/Rationality}.

Reinforcement learning agents are increasingly penetrating society, deployed in critical domains, such as robots, autonomous vehicles, and financial trading systems. Understanding their socioeconomic behaviour is thus crucial, for which establishing the rational model is fundamental. To this end, this paper develops the first mathematical framework in the literature for the rationality of reinforcement learning agents, to the best of our knowledge. We define a suite of measurements for rationality, develop rationality theory based on the measurements, and conduct experiments to showcase them. Our theory indicates that the irrational behaviours mainly come from two sources: the deficiency of the model and algorithm, as well as the shifts between training and deployment environments. Both theory and experiments suggest practical methods to improve rationality, including certain regularisation methods and diversifying training environments. This work sheds light on future development for detecting and mitigating the irrational behaviours of individual agents, understanding the new equilibria when AI goes into society, and informing mechanism design.