Originally inspired by convolutional neural networks in computer vision, equivariant networks have now emerged as a successful class of models in a wide variety of domains such as protein design, drug discovery, reinforcement learning, learning physics etc. The development of such networks however, requires a careful examination of the underlying symmetries/geometric structure of the problem. In the first part of this talk, I will give an overview of some theoretical results (including ongoing work) in the area that have unified and often guided some of the development of such networks. Then I will present an efficient and very general framework to construct such networks, with an example application via spherical image recognition, and summarizing some open questions. Next, I will discuss the construction of such networks in the context of partial symmetries (groupoids) and dynamical systems. Finally, I will very briefly discuss some work on quantifying the expressivity of equivariant networks and its implications for future network design.