Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction

agents by decoupling sample generation from policy optimization, but it also introduces a concrete failure mode for PPO-style off-policy correction. In heterogeneous training systems, the total importance ratio should ideally decompose into two semantically distinct factors: a training--inference discrepancy term that aligns inference-side and training-side distributions at the same behavior-policy version, and a policy-staleness term that constrains the update from the historical policy to the current policy. We give an operational boundary for this failure: when delayed updates or partial rollouts make the required historical training-side logits unavailable, discrepancy repair and staleness correction become entangled. This missing-old-logit problem breaks the intended semantics of decoupled correction and makes clipping and masking thresholds interact in trace-level diagnostics such as Train-Infer Mask and PPO-CLIP activation. We study both exact and approximate repair routes. For exact repair, we examine snapshot-based version tracking, a dedicated old-logit model, and synchronization via partial rollout interruption, and report their system trade-offs. For approximate repair, we use a revised PPO-EWMA reference policy to preserve much of the benefit of decoupled correction when exact old logits are too expensive to recover. Experiments on dense and MoE backbones show that exact recovery is a useful upper bound, while PPO-EWMA improves training speed and final task performance with substantially lower system overhead.