ICML 2012 Survey Results
Conditioned on questionee being an area chair
Background
1) Did you serve as an Area chair for ICML 2012?
25/ 25 = 100% Yes
0/ 25 = 0% No
2) Did you serve as a Reviewer / PC member for ICML 2012?
25/ 25 = 100% Yes
0/ 25 = 0% No
3) Are you an author of a paper submitted to ICML?
19/ 25 = 76% Yes
6/ 25 = 24% No
4) You are:
17/ 19 = 89% An author of an accepted paper
10/ 19 = 53% An author of a rejected paper
5) Are you going to attend ICML 2012 in Edinburgh?
18/ 25 = 72% Yes
7/ 25 = 28% No
Paper Assignment
6) Which information do you support using for assignment of papers to PC members? (choose all that apply)
15/ 25 = 60% Keywords
24/ 25 = 96% Toronto matching service
22/ 25 = 88% Bids
21/ 25 = 84% Choices made by the Area Chairs
7) How did you like your assignments?
2/ 25 = 8% Very interesting
21/ 25 = 84% Good fit for my expertise
1/ 25 = 4% Mostly not my area
1/ 25 = 4% Terrible
Paper Reviewing
9) Did you consult supplementary material for any of the papers you reviewed?
12/ 24 = 50% Yes
12/ 24 = 50% No
10) Did the supplementary material help you form a decision for any paper?
8/ 24 = 33% Yes
16/ 24 = 67% No
11) Did you have sufficient time to prepare your reviews?
22/ 24 = 92% Yes
2/ 24 = 8% No
12) Would you prefer if the paper submission deadline was earlier, to allow more time for the initial review period?
2/ 24 = 8% Yes, the review period should be longer.
18/ 24 = 75% The review period was fine.
4/ 24 = 17% No, in fact the review period could be shorter.
Author Response
13) Did you read the author responses for any of the papers you reviewed?
25/ 25 = 100% Yes
0/ 25 = 0% No
13_5) Was the author response informative for you?
23/ 25 = 92% Yes
2/ 25 = 8% No
14) Did you consult a new version of a paper uploaded during author response?
9/ 25 = 36% Yes
16/ 25 = 64% No
15) Did you change any of your (meta-)reviews due to the author response or new version?
11/ 25 = 44% It changed my mind about a paper.
13/ 25 = 52% It helped clarify a few questions, but did not change my mind.
1/ 25 = 4% It was not useful.
0/ 25 = 0% I did not see it.
16) Do you think the author feedback has an influence on the decision of acceptance/rejection?
21/ 24 = 88% Yes
3/ 24 = 13% No
Discussion
17) Did you participate in the discussion for any of the papers you reviewed?
24/ 25 = 96% Yes
1/ 25 = 4% No
18) Did you change any of your reviews due to the discussion?
21/ 24 = 88% Yes
3/ 24 = 13% No
Assessment of Reviewing Process
19) Do you think the ICML 2012 reviews were different in quality from the reviews at previous ICMLs?
1/ 25 = 4% ICML 2012 substantially better
9/ 25 = 36% Somewhat better
12/ 25 = 48% The same
0/ 25 = 0% Somewhat worse
0/ 25 = 0% Substantially worse
3/ 25 = 12% Don't know
20) Do you think the ICML 2012 reviews were different in quality from the reviews at other similar conferences?
1/ 25 = 4% ICML 2012 substantially better
9/ 25 = 36% Somewhat better
13/ 25 = 52% The same
0/ 25 = 0% Somewhat worse
0/ 25 = 0% Substantially worse
2/ 25 = 8% Don't know
21) Compared to other conferences, how much effort was it to participate in the program committee for ICML 2012?
1/ 25 = 4% ICML 2012 substantially less
1/ 25 = 4% Somewhat less
9/ 25 = 36% The same
9/ 25 = 36% Somewhat more
5/ 25 = 20% Substantially more
0/ 25 = 0% Don't know
22) Throughout the reviewing process, was it always clear to you what needed to be done?
21/ 25 = 84% Yes
4/ 25 = 16% No
23) How many hours did you spend on the review process?
Averaged over 13 entries: 43
24) If you submitted a rebuttal during the author response period, do you believe it had some influence on the final decision?
0/ 17 = 0% Very strong influence
5/ 17 = 29% Substantial influence
9/ 17 = 53% Marginal influence
3/ 17 = 18% No influence
25) Do you prefer having the author response option?
17/ 19 = 89% Yes
2/ 19 = 11% No
26) If you uploaded a new file during the author response period, do you believe it had some influence on the final decision?
0/ 15 = 0% Very strong influence
1/ 15 = 7% Substantial influence
6/ 15 = 40% Marginal influence
8/ 15 = 53% No influence
27) The majority of the reviews of your paper were:
1/ 18 = 6% Right to the point
15/ 18 = 83% High quality
1/ 18 = 6% Low quality
1/ 18 = 6% The reviewers did not understand my paper.
28) Do you prefer the option to revise the paper during author response?
9/ 19 = 47% Yes
10/ 19 = 53% No
29) Do you think that the meta-reviews adequately summarized the reviewers' opinions and made the right decision?
8/ 18 = 44% Yes
6/ 18 = 33% Mostly
4/ 18 = 22% Somewhat
0/ 18 = 0% No, there was no justification of the decision.
Conference Format
30) With 242 accepted papers, ICML can no longer offer a full talk to all accepted papers (assuming 5 parallel tracks over 3 days). Which of the following do you support (choose all that apply):
8/ 24 = 33% Accept fewer papers.
3/ 24 = 13% Add a day.
8/ 24 = 33% Add an additional track.
9/ 24 = 38% Present some papers only as posters.
9/ 24 = 38% Present some papers as short talk + poster.
1/ 24 = 4% Other (briefly comment below)
30-c) Other
31) The workshop program has been extended to 2-days. What is your opinion of this new format?
10/ 25 = 40% Good idea, I plan on attending.
6/ 25 = 24% Good idea, but I won’t attend.
7/ 25 = 28% I don’t care.
1/ 25 = 4% Bad idea, ICML is too long already.
1/ 25 = 4% Bad idea, I don’t like workshops.
32) Which conferences would like to see in co-location with ICML in the future? (choose all that apply):
13/ 20 = 65% COLT
15/ 20 = 75% UAI
3/ 20 = 15% ECML-PKDD
6/ 20 = 30% SIGKDD
7/ 20 = 35% IJCAI
9/ 20 = 45% AAAI
1/ 20 = 5% ECAI
6/ 20 = 30% EMNLP-CoNLL
5/ 20 = 25% SIGIR
6/ 20 = 30% ACL
1/ 20 = 5% RSS
8/ 20 = 40% CVPR
1/ 20 = 5% RSS
1/ 20 = 5% Other (comment below)