ICML 2012 Survey Results
General comments
Conditioned on questionee being an area chair.
Secondary AC was not helpful Reviewer assignment should be automated (not done by AC)
secondary meta-reviews added an unreasonable amount of workload, and the instructions regarding them apparently changed during the process
It's difficult to deflate author feedback changing my mind on papers versus discussion changing my mind on papers. My mind changed, but it's hard to say which caused it. IMO, author feedback is a good forcing function for discussion, but perhaps not as useful as stats might show.
The question about the quality of the majority of reviews of our paper hides a fair bit of variance -- we had one highly-confident but wrong review, while the others were good. We were pleased to see that the ICML review process was robust to this problem.
Allowing substantial changes and uploads of new versions of submissions moves this conference too much towards a journal model. The aim should be to decide whether the snapshot of the work taken at submission time is ready for prime time or not.
As an area chair, I did not like the procedure of selecting reviewers, as for many of the papers I was responsible for the most appropriate reviewers were at their quota. One other quibble I had was that I do not think the policy of allowing the main AC, sometimes with the help of the secondary, to make the accept decisions is a good idea. Some mechanism for having broader discussions about borderline papers amongst ACs would be good.
Organizers could give a price to outstanding reviewers (as in CVPR) to encourage and reward good reviewers.
The Program Chairs did a *fantastic* job this year.