When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test—so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. The literature reports conflicting claims about whether model-free RL outperforms well-tuned rule-based controllers, with single-seed runs, uncalibrated baselines, and inconsistent training budgets confounding cross-study comparison. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched network architectures, training budgets, and reward functions against a properly calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization by training on one workload and deploying on five shifted distributions. Three findings challenge common assumptions: (i) a calibrated rule-based controller achieves the lowest cost on all six workloads and zero constraint violations on steady-state traffic, though it trails the best RL agents on bursty and flash patterns; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workload types, with rankings shifting by up to four positions between steady-state and bursty traffic. On bursty workloads—where RL should shine—PPO reduces constraint violations by 54% relative to the calibrated baseline, but only at 24% higher cost, suggesting that the bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.