TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.

Reinforcement learning is a way to train an AI agent by letting it try actions and learn from the results, but this trial-and-error process can be risky when some actions are unsafe. In many settings, it is hard to write down exactly what makes a behavior unsafe, but easier to give coarse feedback such as "this run was acceptable" or "something bad had happened by this point." This paper introduces TraCeS, a method that learns safety guidance from such sparse yes/no labels instead of requiring a hand-written safety rule or a known cost signal. TraCeS estimates which parts of a trajectory are likely to be responsible for safety violations, then turns this estimate into a step-by-step signal that can guide policy learning. This helps the agent avoid unsafe behavior while still improving its task performance. The method is designed for safety properties where violations are irreversible, such as entering a forbidden region, crashing, or exceeding a cumulative safety budget. Experiments on continuous-control benchmarks show that TraCeS can learn useful safety signals and train safer policies from limited trajectory-level feedback. The broader goal is to make safe learning more practical when precise safety specifications are unavailable.