ICML 2012 Survey Results
Conditioned on questionee being an author
Background
1) Did you serve as an Area chair for ICML 2012?
19/466 = 4% Yes
447/466 = 96% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
89/466 = 19% Yes
377/466 = 81% No
3) Are you an author of a paper submitted to ICML?
466/466 = 100% Yes
0/466 = 0% No
4) You are:
230/463 = 50% An author of an accepted paper
267/463 = 58% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
207/466 = 44% Yes
259/466 = 56% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
56/ 88 = 64% Keywords
76/ 88 = 86% Toronto matching service
78/ 88 = 89% Bids
65/ 88 = 74% Choices made by the Area Chairs
7) How did you like your assignments?
9/ 89 = 10% Very interesting
76/ 89 = 85% Good fit for my expertise
3/ 89 = 3% Mostly not my area
1/ 89 = 1% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
59/ 88 = 67% Yes
29/ 88 = 33% No
10) Did the supplementary material help you form a decision for any paper?
32/ 85 = 38% Yes
53/ 85 = 62% No
11) Did you have sufficient time to prepare your reviews?
75/ 87 = 86% Yes
12/ 87 = 14% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
10/ 88 = 11% Yes, the review period should be longer.
74/ 88 = 84% The review period was fine.
4/ 88 = 5% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
88/ 89 = 99% Yes
1/ 89 = 1% No
13_5) Was the author response informative for you?
77/ 88 = 88% Yes
11/ 88 = 13% No
14) Did you consult a new version of a paper uploaded during author response?
43/ 89 = 48% Yes
46/ 89 = 52% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
25/ 89 = 28% It changed my mind about a paper.
57/ 89 = 64% It helped clarify a few questions, but did not change my mind.
7/ 89 = 8% It was not useful.
0/ 89 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
67/ 89 = 75% Yes
22/ 89 = 25% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
87/ 89 = 98% Yes
2/ 89 = 2% No
18) Did you change any of your reviews due to the discussion?
57/ 88 = 65% Yes
31/ 88 = 35% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
4/ 89 = 4% ICML 2012 substantially better
26/ 89 = 29% Somewhat better
34/ 89 = 38% The same
4/ 89 = 4% Somewhat worse
1/ 89 = 1% Substantially worse
20/ 89 = 22% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
6/ 89 = 7% ICML 2012 substantially better
31/ 89 = 35% Somewhat better
38/ 89 = 43% The same
6/ 89 = 7% Somewhat worse
1/ 89 = 1% Substantially worse
7/ 89 = 8% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
2/ 89 = 2% ICML 2012 substantially less
2/ 89 = 2% Somewhat less
42/ 89 = 47% The same
23/ 89 = 26% Somewhat more
12/ 89 = 13% Substantially more
8/ 89 = 9% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
84/ 89 = 94% Yes
5/ 89 = 6% No
23) How many hours did you spend on the review process?
Averaged over 69 entries: 29
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
11/455 = 2% Very strong influence
92/455 = 20% Substantial influence
200/455 = 44% Marginal influence
152/455 = 33% No influence
25) Do you prefer having the author response option?
409/462 = 89% Yes
53/462 = 11% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
13/365 = 4% Very strong influence
50/365 = 14% Substantial influence
148/365 = 41% Marginal influence
154/365 = 42% No influence
27) The majority of the reviews of your paper were:
55/459 = 12% Right to the point
276/459 = 60% High quality
80/459 = 17% Low quality
48/459 = 10% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
297/457 = 65% Yes
160/457 = 35% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
160/459 = 35% Yes
145/459 = 32% Mostly
103/459 = 22% Somewhat
51/459 = 11% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
95/453 = 21% Accept fewer papers.
164/453 = 36% Add a day.
156/453 = 34% Add an additional track.
175/453 = 39% Present some papers only as posters.
189/453 = 42% Present some papers as short talk + poster.
12/453 = 3% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
135/454 = 30% Good idea, I plan on attending.
152/454 = 33% Good idea, but I won’t attend.
138/454 = 30% I don’t care.
20/454 = 4% Bad idea, ICML is too long already.
9/454 = 2% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
190/349 = 54% COLT
168/349 = 48% UAI
72/349 = 21% ECML-PKDD
84/349 = 24% SIGKDD
89/349 = 26% IJCAI
100/349 = 29% AAAI
21/349 = 6% ECAI
32/349 = 9% EMNLP-CoNLL
42/349 = 12% SIGIR
40/349 = 11% ACL
23/349 = 7% RSS
89/349 = 26% CVPR
17/349 = 5% RSS
12/349 = 3% Other (comment below)