We discuss techniques that can be used to learn how to improve AIs (dialogue models) by interacting with organic users ``in the wild''. Training models with organic data is challenging because such interactions include both high quality conversations and feedback, as well as adversarial and toxic behavior. We thus study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. We present BlenderBot 3x, an update on the conversational model BlenderBot 3, trained on 6M such interactions from participating users of the system, which we also publicly release. BlenderBot 3x is both preferred in conversation to BlenderBot 3, and is shown to produce safer responses in challenging situations. We then discuss how we believe continued use of these techniques -- and improved variants -- can lead to further gains.