The basics
We are using the Microsoft CMT system. The site for ICML 13 is: https://cmt.research.
Before you can begin your reviewing duties, you must enter your subject areas and conflict domains. These are essential pieces of information for the paper assignment process. You will not be able to change these once reviewing begins.
Conflict domains: CMT associates each user (author, reviewer, area chair) with a list of “conflict domains”. It will not assign a paper to a reviewer (or area chair) if the conflict domains of the reviewer intersect with those of any of the authors. Your conflict domains should include the institution for which you work; for instance, if you are at UC San Diego, you should include “ucsd.edu“. You might also want to include your previous place of work if you moved very recently. If you have more than one domain in your list, you should separate them with semicolons; CMT needs this in order to parse them properly.
Conflicts with specific authors: Some of you have ex-students all over the place, and have expressed an interest in identifying them as conflicts. The way to do this is to select “Manage conflicts with authors” from your main page. You will then be presented with a list of all ICML authors, and you can check off your student, relative, parole officer, etc.
Assigning papers to reviewers
Each paper has one area chair and three reviewers. We will automatically assign two reviewers per paper. You are welcome to change these assignments as you see fit, and you also need to add a third reviewer.
Here’s the procedure for managing reviewers:
- Log in, and select “Manage assigments, bids, and conflicts”. At this point, you will be able to see the papers assigned to you.
- For each paper in turn, you should click “Edit assignments”. To add a reviewer, or to change the current assignments, click the “Show all reviewers” button near the top of the page. You will then see all the reviewers, and you can use the check boxes at the right to assign/un-assign reviewers. Whenever you do this be sure to “Save changes”.
- When you are viewing all the reviewers there are two informative criteria by which you can sort them:
- The “relevance” score. This is computed by CMT, based on the subject areas declared by the reviewer.
- The “candidate suggested rank”. This is computed using a topic model provided by the Toronto Matching Service. Ignore the fact that it says you provided the ranks: this is just a side-effect of the workaround that was needed to get this ranking into the system.
- We ask that one of your three reviewers be an “expert”, and be willing to declare themselves as such (in the review confidence). We understand that such people might not be possible to find on papers that are pretty far-out: we just ask that you do your
best.
The schedule for Cycle 1
Monday Oct 15: Review assignments should be completedSome of you might not have an expert lined up by this time, but at the very least please make sure that the other two reviewers are to your liking. That is, you may add reviewers after Oct 15, but you certainly should not delete reviewer assignments.
Monday Oct 15 to Monday Nov 5: Review periodThe reviewers will be busy doing their reviews. Please keep an eye on them towards the end, to make sure things are moving along.
Monday Nov 5 to Sunday Nov 11: Area chairs clean up reviewsPlease ping delinquent reviewers to make sure there are three reviews per paper (or, in some cases, two: see “Review protocol” below) and that they are reasonable (for instance, not offensive).
Monday Nov 12 to Wednesday Nov 14: Author response period
Thursday Nov 15 to Friday Nov 30: Discussion and meta-reviews At this stage, the papers should be discussed between you and the reviewers, taking the author response into account, and reaching a final verdict. You should write a meta-review summarizing the rationale for the decision.
Review protocol
- Each paper must have at least two reviews. If they are in agreement (as regards accept/reject), and the area chair agrees with them, then a third review is not needed. If there isn’t complete agreement, then three reviews are needed.
- If the area chair agrees with two of the reviews, then the decision is simple.
- If the area chair disagrees with two of the reviews, then he/she should contact David and Sanjoy, who may, in some cases, bring in another area chair or an outside expert.
We would particularly like to draw your attention to two types of paper:
- Type 1: An incremental advance over previous work.Often the decision boils down to whether it is enough of an increment. This is your judgement call, but for our part, we would be happy if you were pretty demanding in such cases. Otherwise there is a danger of accepting vast numbers of mediocre papers with short lifespans.
- Type 2: A paper with truly novel ideas but the ideas seem insufficiently supported.For instance, there might be poor experimental corroboration; or some basic theoretical justification might be lacking. If you come across such papers, and are planning to reject them, please let us know. We might rescue them at the end.
Resubmit option
For cycle 1, the possible final outcomes for a paper are “accept”, “reject”, or “resubmit in cycle 3″. This last option is at the discretion of the area chair (the reviewers do not select it). It should be invoked sparingly, and reserved for papers where clear objective modifications are required, for instance:
- Some specific experiments needed
- Proofs seem correct at a high level but details are wrong
These papers, when resubmitted, will return to the same area chair, who can then quickly make a decision on them, if need be consulting the expert reviewer. Resubmitted papers will not automatically be re-assigned to reviewers.
List of area chairs
Aarti Singh
Alan Fern
Alekh Agarwal
Alex Ihler
Amir Globerson
Andreas Krause
Animashree Anandkumar
Arindam Banerjee
Ashutosh Saxena
Ben Taskar
Brian Kulis
Charles Elkan
Constantine Caramanis
Corinna Cortes
Csaba Szepesvari
Cynthia Rudin
Dale Schuurmans
Dan Roth
Daniel Hsu
David Sontag
Drew Bagnell
Edoardo Airoldi
Elad Hazan
Emily Fox
Eyal Amir
Fei Sha
Francis Bach
Gert Lanckriet
Gregory Shakhnarovich
Honglak Lee
Jeff Bilmes
Jerry Zhu
Joelle Pineau
John Platt
Jure Leskovec
Kamalika Chaudhuri
Karsten Borgwardt
Kilian Weinberger
Koby Crammer
Lihong Li
Lise Getoor
Luke Zettlemoyer
Marc’Aurelio Ranzato
Marina Meila
Maya Gupta
Mehryar Mohri
Miroslav Dudik
Nati Srebro
Nicolo Cesa-Bianchi
Nina Balcan
Ofer Dekel
Ohad Shamir
Pascal Poupart
Percy Liang
Peter Grunwald
Peter Auer
Phil Long
Ran El-Yaniv
Ran Gilad-Bachrach
Raquel Urtasun
Rich Caruana
Rich Sutton
Rich Zemel
Risi Kondor
Ron Parr
Ruslan Salakhutdinov
SVN Vishwanathan
Satyen Kale
Shai Ben-David
Shie Mannor
Thorsten Joachims
Tobias Scheffer
Tong Zhang
Ulrike von Luxburg
Yann Le Cun
Yisong Yue
Yoav Freund
Yoshua Bengio
Zico Kolter