Opponent Modeling and Value of Information in Deep Reinforcement Learning for the Iterated Prisoner’s Dilemma

In the Iterated Prisoner’s Dilemma a reinforcement-learning agent must infer its opponent’s strat- egy from the running history of joint actions. The sequence model that encodes this history is therefore a first-class architectural decision with direct impact on per-turn payoff. We compare a recurrent backbone, a standard transformer architecture, a transformer variant with CLS-token readout and sinusoidal positional encoding, and a multilayer feedforward baseline that uses only auxiliary cooperation-rate features. Each architecture is trained with Double-DQN against a cor- pus of hand-coded opponents, and we summarise outcomes through seed-level confidence intervals derived from a hierarchical bootstrap that respects the dependence between seeds. At a matched training budget the recurrent backbone consistently outperforms both small transformer variants tested at this scale. A sequence-encoder ablation shows that the small transformer encoders tested here add negligible signal beyond the auxiliary-feature baseline at this budget, while the recur- rent backbone adds substantial signal beyond the same baseline. A non-learning TitForTat agent already reaches a payoff close to the recurrent mean and above both transformer variants, which places a sharp ceiling on what a learned backbone contributes on this task. The recurrent advantage is concentrated on retaliatory and adaptive opponents, while the transformer retains an advantage on pattern-detecting meta-strategies. We use Value of Information as an operational shorthand for the mean per-strategy reward gap relative to a designated reference, and we report results across blind, family-aware, and oracle-aware identity-conditioning regimes to quantify how much of that gap an explicit identity signal recovers and at what budget.