A typical way to teach robots new behaviors involves offline learning from demonstrations. However, there are many other kinds of input that humans can give their robots that are often faster and more intuitive to communicate. In the first part of the talk, I will discuss one such input type that the robot can learn from online: physical corrections. In the second part of the talk, I will argue that moving beyond learning from demonstrations requires more than just thinking about other human feedback types for teaching tasks: we also need to consider new learning pipelines to facilitate seamless and aligned interaction between humans and robots.