Sparse Binary Reward Degrades Agent Uncertainty Quantification in Multi-Seed Evaluation

Uncertainty quantification (UQ) for RL-trained agents including conformal prediction, calibra- tion, and risk control implicitly requires that the agent produces consistent behaviour across train- ing conditions. We present empirical evidence that this consistency requirement is violated when agents are trained with sparse binary reward sig- nals. A Standard REINFORCE agent (binary task reward) trained on LLaVA-1.5-7B achieves Task Success Rate (TSR) = 0.360 at seed = 42 but ex- hibits severe output degeneration at seed = 123, yielding a cross-seed standard deviation of 0.129 and a 95% confidence interval of [−0.13, 0.65] , an interval that extends outside the parameter space [0, 1], indicating that the three-seed sample is insufficient to characterise agent performance. We identify reward density collapse as the likely mechanism: binary reward provides zero gradi- ent to approximately 82–88% of training steps, creating a high-variance optimisation landscape where policy trajectories diverge across seeds. A Hybrid Reward Architecture (HRA-RL) incorpo- rating continuous automated factuality grounding , cosine similarity between sentence-embedding representations of agent outputs and ground-truth references , reduces cross-seed TSR standard de- viation by 6.8× (0.019 vs. 0.129) and produces a CI of [0.281, 0.396], narrow enough to support meaningful UQ. We argue that dense, continuous reward signals are a practical design condition for agents intended to be used with downstream UQ methods, and that multi-seed CI reporting should be a prerequisite before applying conformal or calibration methods to RL-trained agents.