Fundamental research in Neuroscience is currently undergoing a renaissance based on deep learning. The central promises of deep learning-based modeling of brain circuits are that the models shed light on evolutionary optimization problems, constraints and solutions, and generate novel predictions regarding neural phenomena. We show, through the case-study of grid cells in the entorhinal-hippocampal circuit, that one often gets neither. We begin by reviewing the principles of grid cell mechanism and function obtained from analytical and first-principles modeling efforts, then consider the claims of deep learning models of grid cells and rigorously examine their results under varied conditions. Using large-scale hyperparameter sweeps and hypothesis-driven experimentation, we demonstrate that the results of such models may reveal more about particular and non-fundamental implementation choices than fundamental truths about neural circuits or the loss function(s) they might optimize. Finally, we discuss why it is that these models of the brain cannot be expected to work without the addition of substantial amounts of inductive bias, an informal No Free Lunch theorem for Neuroscience. In conclusion, caution and consideration, together with biological knowledge, are warranted in building and interpreting deep learning models in Neuroscience.