ICML 2012 Survey Results
Background
1) Did you serve as an Area chair for ICML 2012?
25/575 = 4% Yes
550/575 = 96% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
198/575 = 34% Yes
377/575 = 66% No
3) Are you an author of a paper submitted to ICML?
466/575 = 81% Yes
109/575 = 19% No
4) You are:
230/463 = 50% An author of an accepted paper
267/463 = 58% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
225/575 = 39% Yes
350/575 = 61% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
127/195 = 65% Keywords
158/195 = 81% Toronto matching service
172/195 = 88% Bids
150/195 = 77% Choices made by the Area Chairs
7) How did you like your assignments?
16/197 = 8% Very interesting
168/197 = 85% Good fit for my expertise
12/197 = 6% Mostly not my area
1/197 = 1% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
128/197 = 65% Yes
69/197 = 35% No
10) Did the supplementary material help you form a decision for any paper?
75/189 = 40% Yes
114/189 = 60% No
11) Did you have sufficient time to prepare your reviews?
160/194 = 82% Yes
34/194 = 18% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
31/196 = 16% Yes, the review period should be longer.
151/196 = 77% The review period was fine.
14/196 = 7% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
196/198 = 99% Yes
2/198 = 1% No
13_5) Was the author response informative for you?
166/196 = 85% Yes
30/196 = 15% No
14) Did you consult a new version of a paper uploaded during author response?
105/197 = 53% Yes
92/197 = 47% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
47/196 = 24% It changed my mind about a paper.
133/196 = 68% It helped clarify a few questions, but did not change my mind.
16/196 = 8% It was not useful.
0/196 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
144/196 = 73% Yes
52/196 = 27% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
188/196 = 96% Yes
8/196 = 4% No
18) Did you change any of your reviews due to the discussion?
122/195 = 63% Yes
73/195 = 37% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
7/197 = 4% ICML 2012 substantially better
43/197 = 22% Somewhat better
67/197 = 34% The same
6/197 = 3% Somewhat worse
1/197 = 1% Substantially worse
73/197 = 37% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
14/197 = 7% ICML 2012 substantially better
68/197 = 35% Somewhat better
75/197 = 38% The same
10/197 = 5% Somewhat worse
2/197 = 1% Substantially worse
28/197 = 14% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
3/196 = 2% ICML 2012 substantially less
15/196 = 8% Somewhat less
89/196 = 45% The same
48/196 = 24% Somewhat more
20/196 = 10% Substantially more
21/196 = 11% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
183/195 = 94% Yes
12/195 = 6% No
23) How many hours did you spend on the review process?
Averaged over 161 entries: 28
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
11/455 = 2% Very strong influence
92/455 = 20% Substantial influence
200/455 = 44% Marginal influence
152/455 = 33% No influence
25) Do you prefer having the author response option?
409/462 = 89% Yes
53/462 = 11% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
13/365 = 4% Very strong influence
50/365 = 14% Substantial influence
148/365 = 41% Marginal influence
154/365 = 42% No influence
27) The majority of the reviews of your paper were:
55/459 = 12% Right to the point
276/459 = 60% High quality
80/459 = 17% Low quality
48/459 = 10% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
297/457 = 65% Yes
160/457 = 35% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
160/459 = 35% Yes
145/459 = 32% Mostly
103/459 = 22% Somewhat
51/459 = 11% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
134/558 = 24% Accept fewer papers.
198/558 = 35% Add a day.
193/558 = 35% Add an additional track.
216/558 = 39% Present some papers only as posters.
241/558 = 43% Present some papers as short talk + poster.
15/558 = 3% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
151/561 = 27% Good idea, I plan on attending.
196/561 = 35% Good idea, but I won’t attend.
178/561 = 32% I don’t care.
26/561 = 5% Bad idea, ICML is too long already.
10/561 = 2% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
237/434 = 55% COLT
216/434 = 50% UAI
81/434 = 19% ECML-PKDD
118/434 = 27% SIGKDD
109/434 = 25% IJCAI
122/434 = 28% AAAI
26/434 = 6% ECAI
41/434 = 9% EMNLP-CoNLL
55/434 = 13% SIGIR
56/434 = 13% ACL
26/434 = 6% RSS
104/434 = 24% CVPR
18/434 = 4% RSS
13/434 = 3% Other (comment below)