Physics-informed neural networks (PINNs) are computationally efficient alternatives to traditional partial differential equation (PDE) solvers.However, their reliability is dependent on the accuracy of the trained neural network. We introduce a mechanism for leveraging the symmetries of a given PDE to improve the neural solver. In particular, we propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE. Intuitively, our symmetry loss tries to ensure that infinitesimal generators of the Lie group preserve solutions of the PDE. This means that once the network learns a solution, it also learns the neighbouring solutions generated by Lie point symmetries.Our results show that symmetry is an effective inductive bias for PINNs and lead to a significant increase in sample efficiency.