Timezone: »

 
Poster
A data-driven approach for learning to control computers
Peter Humphreys · David Raposo · Tobias Pohlen · Gregory Thornton · Rachita Chhaparia · Alistair Muldal · Josh Abramson · Petko Georgiev · Adam Santoro · Timothy Lillicrap

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #836
in Poster Session 2 »

It would be useful for machines to use computers as humans do so that they can aid us in everyday tasks. This is a setting in which there is also the potential to leverage large-scale expert demonstrations and human judgements of interactive behaviour, which are two ingredients that have driven much recent success in AI. Here we investigate the setting of computer control using keyboard and mouse, with goals specified via natural language. Instead of focusing on hand-designed curricula and specialized action spaces, we focus on developing a scalable method centered on reinforcement learning combined with behavioural priors informed by actual human-computer interactions. We achieve state-of-the-art and human-level mean performance across all tasks within the MiniWob++ benchmark, a challenging suite of computer control problems, and find strong evidence of cross-task transfer. These results demonstrate the usefulness of a unified human-agent interface when training machines to use computers. Altogether our results suggest a formula for achieving competency beyond MiniWob++ and towards controlling computers, in general, as a human would.

Keywords: APP: Everything Else · RL: Deep RL · MISC: Transfer, Multitask and Meta-learning

Author Information

Peter Humphreys (Deepmind)
David Raposo (DeepMind)
Tobias Pohlen (DeepMind)
Gregory Thornton (Deepmind)
Rachita Chhaparia (Deepmind)
Alistair Muldal (DeepMind)
Josh Abramson (DeepMind)
Petko Georgiev (Deepmind)
Adam Santoro (DeepMind)
Timothy Lillicrap (Google DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors