ICML 2017

THE CONFERENCE

FOR AUTHORS

FOR PARTICIPANTS

Workshop

Reproducibility in Machine Learning Research

Rosemary Nan Ke · Anirudh Goyal · Alex Lamb · Joelle Pineau · Samy Bengio · Yoshua Bengio

Fri Aug 11th 08:30 AM -- 05:30 PM @ C4.10

Event URL: https://sites.google.com/view/icml-reproducibility-workshop/home »

This workshop focuses on issues of reproducibility and replication of results in the Machine Learning community. Papers from the Machine Learning community are supposed to be a valuable asset. They can help to inform and inspire future research. They can be a useful educational tool for students. They can give guidance to applied researchers in industry. Perhaps most importantly, they can help us to answer the most fundamental questions about our existence - what does it mean to learn and what does it mean to be human? Reproducibility, while not always possible in science (consider the study of a transient astrological phenomenon like a passing comet), is a powerful criteria for improving the quality of research. A result which is reproducible is more likely to be robust and meaningful and rules out many types of experimenter error (either fraud or accidental).

There are many interesting open questions about how reproducibility issues intersect with the Machine Learning community:

* How can we tell if papers in the Machine Learning community are reproducible even in theory? If a paper is about recommending news sites before a particular election, and the results come from running the system online in production - it will be impossible to reproduce the published results because the state of the world is irreversibly changed from when the experiment was ran.
* What does it mean for a paper to be reproducible in theory but not in practice? For example, if a paper requires tens of thousands of GPUs to reproduce or a large closed-off dataset, then it can only be reproduced in reality by a few large labs.
* For papers which are reproducible both in theory and in practice - how can we ensure that papers published in ICML would actually be able to replicate if such an experiment were attempted?
* What does it mean for a paper to have successful or unsuccessful replications?
* Of the papers with attempted replications completed, how many have been published?
* What can be done to ensure that as many papers which are reproducible in theory fall into the last category?
* On the reproducibility issue, what can the Machine Learning community learn from other fields?

Our aim in the following workshop is to raise the profile of these questions in the community and to search for their answers. In doing so we aim for papers focusing on the following topics:
* Analysis of the current state of reproducibility in machine learning venues
* Tools to help increase reproducibility
* Evidence that reproducibility is important for science
* Connections between the reproducibility situation in Machine Learning and other fields
* Replications, both failed and successful, of influential papers in the Machine Learning literature.

Author Information

Rosemary Nan Ke (MILA, University of Montreal)

I am a PhD student at the Universite deMontreal and Ecole Polytechnique de Montreal. I mostly work on generative models, methods for learning deep undirected graphical model, I am also interested in generative models with latent variables. I have also worked on speech recognition and deep learning at Carnegie Mellon University before coming to Montreal.

Parth Goyal (Université de Montréal)

Alex Lamb (Universite de Montreal)

Joelle Pineau (McGill University / Facebook)

Samy Bengio (Google Research Brain Team)

Yoshua Bengio (Mila / U. Montreal)

Yoshua Bengio (PhD'1991 in Computer Science, McGill University). After two post-doctoral years, one at MIT with Michael Jordan and one at AT&T Bell Laboratories with Yann LeCun, he became professor at the department of computer science and operations research at UniversitÃ© de MontrÃ©al. Author of two books (a third is in preparation) and more than 200 publications, he is among the most cited Canadian computer scientists and is or has been associate editor of the top journals in machine learning and neural networks. Since '2000 he holds a Canada Research Chair in Statistical Learning Algorithms, since '2006 an NSERC Chair, since '2005 his is a Senior Fellow of the Canadian Institute for Advanced Research and since 2014 he co-directs its program focused on deep learning. He is on the board of the NIPS foundation and has been program chair and general chair for NIPS. He has co-organized the Learning Workshop for 14 years and co-created the International Conference on Learning Representations. His interests are centered around a quest for AI through machine learning, and include fundamental questions on deep learning, representation learning, the geometry of generalization in high-dimensional spaces, manifold learning and biologically inspired learning algorithms.