CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that solves modern, interactive CAPTCHA challenges while retaining general GUI-agent performance. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving. Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across synthetic and real-world test sets, ReCAP substantially improves CAPTCHA-solving success over its base agents, while maintaining strong performance on general GUI-agent benchmarks.

Many websites use CAPTCHAs to check whether a user is human, but modern AI agents that can see and interact with screens are becoming increasingly capable. This raises an important question: how well can today’s general-purpose computer-use agents handle interactive CAPTCHA challenges, and what skills do they need to do so? We introduce ReCAP, an AI agent designed to study this question in a controlled research setting. ReCAP is trained using a dynamic CAPTCHA system that generates many kinds of challenges, such as reading distorted text, selecting objects in image grids, matching icons, and dragging sliders. The system automatically creates examples showing not only the correct actions, but also the reasoning behind those actions. It also uses failed attempts to teach the agent how to recognize mistakes and correct them. Our experiments show that this training substantially improves CAPTCHA-solving performance on synthetic benchmarks and transfers to several real-world CAPTCHA types, while mostly maintaining performance on general screen-interaction tasks. Rather than treating CAPTCHA solving as a standalone trick, this work shows that it can be understood as a set of visual reasoning and interaction skills. We hope this work can support future research on evaluating the robustness of human-verification systems as GUI agents continue to improve.