Visual Dexterity: In-hand Dexterous Manipulation from Depth

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in unstructured environments that remain beyond the reach of current robots. Prior works built reorientation systems that assume one or many of the following specific circumstances: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasi-static manipulation, etc. We overcome these limitations and present a general object reorientation controller that is trained in simulation and evaluated in the real world. Our system uses readings from a single commodity depth camera to dynamically reorient complex objects by any amount in real time. The controller generalizes to new objects not used during training. It even demonstrates some capability of reorienting objects in the air held by a downward-facing hand that must counteract gravity during reorientation.