Agent Learning via Early Experience

A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks. However, training agents from experience data with reinforcement learning remains difficult in many environments, which either lack verifiable rewards (e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn tool use). As a result, most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly. This limitation stems from the nature of expert demonstrations: they capture only a narrow range of scenarios, and expose the agent to limited environment diversity. We address this limitation with a middle-ground paradigm we call early experience: interaction data generated by the agent's own actions, where the resulting future states serve as supervision without reward signals. Within this paradigm, we study two strategies of using such data: (1) implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making. Evaluation across eight diverse environments and multiple model families shows that our approaches consistently improve effectiveness and out-of-domain generalization, highlighting the value of early experience. Moreover, in environments with verifiable rewards, our results provide promising signals that early experience offers a strong foundation for subsequent reinforcement learning, making it a practical bridge between imitation learning and fully experience-driven agents.

Today's AI agents are being trained to do real-world tasks like browsing websites, using software, and planning trips. The standard approach is to show them recordings of humans doing each task and have them copy along, but such recordings are expensive to collect, and the agent stumbles the moment it meets a situation the recordings didn't cover. A more ambitious approach, reinforcement learning, requires the environment to tell the agent whether it succeeded, yet most websites and applications give no such feedback. We propose a middle ground we call "early experience." Starting from human examples, the agent also tries its own actions and observes what happens next, the new page, an error message, or a tool's response, and uses those observations themselves as the lesson, no score required. We study two ways to turn these observations into a training signal. In the first, the agent learns to predict what each action will lead to, building an internal sense of how its environment behaves. In the second, the agent compares its own attempts against the human's choice and writes out, in plain language, why the human's action was the better move. Across eight settings spanning household simulations, scientific experiments, travel planning, tool use, and web navigation, agents trained this way consistently outperform those trained only by imitation, handle unfamiliar situations better, and reach higher final performance when reinforcement learning is later added on top. Early experience gives agents a practical way to start learning from their own actions today, even in environments that cannot yet grade them.