In many applications of autonomy in robotics, guarantees that constraints are satisfied throughout the learning process are paramount. We present a controller synthesis technique based on the computation of reachable sets, using optimal control and game theory. Then, we present methods for combining reachability with learning-based methods, to enable performance improvement while maintaining safety, and to move towards safe robot control with learned models of the dynamics and the environment. We will discuss different interaction models with other agents. Finally, we will illustrate these safe learning methods on robotic platforms at Berkeley, discussing applications in automated airspace management and air taxi operations.