CUARewardBench: A Benchmark for Evaluating Reward Models for Computer-Using Agents

Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces. While script-based verifiers are widely adopted for evaluation, they suffer from limited scalability and inability to provide step-wise assessment. Reward models offer promising alternatives, but their effectiveness on CUA evaluation remains largely underexplored. To address this gap, we present CUARewardBench, comprising four key contributions: (1) First-ever Comprehensive CUA Reward Benchmark: We introduce the first benchmark for evaluating both outcome reward models (ORM) and process reward models (PRM) on CUA tasks, enabling systematic assessment across trajectory-level and step-level evaluation. (2) Diverse, Practical and Reliable Dataset: CUARewardBench encompasses trajectories from 10 software categories and 7 agent architectures with varying performance levels (25.9%-50.8% success rates). All trajectories are expertly annotated through carefully designed protocols, with rigorous quality control to ensure reliability and practical applicability. (3) Comprehensive Analysis and Insights: Through extensive experiments across 7 vision-language models and 3 prompt templates, we reveal critical limitations of current CUA RMs, including insufficient visual reasoning capabilities, knowledge deficiencies, and the superiority of general VLMs over specialized CUA models for reward evaluation. (4) Unanimous Prompt Ensemble (UPE): Based on the insights from our comprehensive analysis, we propose UPE, a novel ensemble method that significantly enhances reward model reliability through strict unanimous voting and strategic prompt-template configurations. UPE achieves 88.0% precision and 95.3% NPV for ORM, and 83.1% precision and 86.2% NPV for PRM, substantially outperforming single VLMs and traditional ensemble approaches. In a short, this work introduces both a comprehensive benchmark and a novel ensemble method that substantially enhances CUA reward model reliability.

AI assistants are beginning to use computers in the way people do: they look at a screen, click buttons, type text, and move through software to finish tasks. To improve these assistants, researchers need reliable ways to tell whether an assistant completed a task correctly and whether each step helped or hurt. Today, this checking is often done with hand-written rules, which are costly to create and hard to scale. We introduce CUARewardBench, a benchmark for testing AI-based judges that evaluate computer-using agents. It includes human-checked examples from many desktop applications and several types of agents, so researchers can measure both final task success and step-by-step correctness. We find that current AI judges still make many mistakes, especially when they must understand complex screenshots and reason about software behavior. We also propose a simple ensemble method that makes a judgment only when multiple prompts agree, improving reliability. This benchmark and method can help researchers build safer, more reliable, and more trustworthy computer-using agents.