Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation

Large language models (LLMs) are increasingly applied in diverse real-world applications, each governed by bespoke behavioral and safety specifications (spec) custom-tailored by users or organizations. These specifications, categorized into safety-spec and behavioral-spec, vary across scenarios and evolve with changing preferences and requirements. We formalize this challenge as specification alignment, focusing on LLMs' ability to follow dynamic, scenario-specific spec from both behavioral and safety perspectives. To address this challenge, we introduce SpecBench, a unified benchmark for measuring specification alignment, covering 5 scenarios, 103 spec, and 1,500 prompts. Experiments on 15 reasoning and 18 instruct models with several Test-Time Deliberation (TTD) methods, including Self-Refine, TPO, and MoreThink, show that SpecBench effectively reveals alignment gaps and that test-time deliberation improves specification alignment. Based on previous TTD methods, we further propose Align3, a lightweight method with hierarchical reflection and revision to reason over specification boundaries, advancing the safety-helpfulness trade-off frontier with minimal overhead. These results highlight test-time deliberation as an effective strategy for reasoning over the real-world specification boundaries. Our code and resources are available at https://github.com/zzzhr97/SpecBench.

AI assistants are expected to be both useful and safe, but the right boundaries vary by setting: a child-story app, a coding helper, a health educator, and a travel planner should not all follow the same rules. We study how well large language models can follow these changing, scenario-specific rules while still giving helpful answers. To measure this, we build SpecBench, a benchmark with five real-world scenarios, 103 rules, and 1,500 user questions that test both what the model should do and what it should avoid. We evaluate many current language models and find clear gaps: models that are very cautious often become less helpful, while more helpful models can cross safety boundaries. We also test whether models can improve by spending a little more effort thinking through the rules before answering, without retraining them. Our method, Align3, guides a model to first improve the answer, then check safety risks, and finally audit all relevant rules. This simple process improves alignment with small extra cost, including a gain of about 12 points in one representative setting. The work provides a practical way to evaluate and improve AI assistants whose rules must adapt to different users, organizations, and applications.