Unreproducible Research is Reproducible

The apparent contradiction of the title is a wordplay on the different meanings attributed to the word \emph{reproducible} across different scientific fields. What we imply is that unreproducible \emph{findings} can be built upon reproducible \emph{methods}. Without denying the importance of facilitating the reproduction of \emph{methods}, we deem important to reassert that reproduction of \emph{findings} is a fundamental step of the scientific inquiry. We argue that the commendable quest towards easy deterministic reproducibility of methods and numerical results, should not have us forget the even more important necessity of ensuring the reproducibility of empirical conclusions and findings by properly accounting for essential sources of variations. We provide experiments to exemplify the limitations of current evaluation methods of models in the field of deep learning, showing that even if the results could be reproduced, a slightly different experiment would not support the findings. This work is an attempt to promote the use of more rigorous and diversified methodologies. It is not an attempt to impose a new methodology and it is not a critique on the nature of exploratory research. We hope to help clarify the distinction between exploratory and empirical research in the field of deep learning and believe more energy should be devoted to proper empirical research in our community.