Many settings in machine learning require the selection of a rotation representation. However, choosing a suitable representation from the extensive set of available possibilities remains challenging. This paper acts as a survey and guide through rotation representations. We walk through their properties that harm or benefit deep learning with gradient-based optimization. We consolidate insights from rotation-based learning, providing a comprehensive overview of learning functions with rotation representations. We provide recommendations for selecting a suitable representation depending on whether rotations are in the model's input or output.