Result Summary for Survey:ICML 2010 Feedback

ICML 2010 Feedback

1. Did you serve as a reviewer at ICML 2010?
	answered question		195
	skipped question		0
		Response Percent	Response Count
Yes		100,0%	195
No		0,0%	0

2. Are you going to attend ICML 2010 in Haifa?
	answered question		195
	skipped question		0
		Response Percent	Response Count
Yes		25,6%	50
No		74,4%	145
viewMy main reason for not attending is:			121

3. What model do you prefer for assigning area chairs to papers? (multiple selections possible)
	answered question		167
	skipped question		28
		Response Percent	Response Count
Authors supply area keywords, and area chairs bid on papers primarily in the areas they agreed to cover. The assignment of area chairs to areas is public to the authors. (this year's model)		83,2%	139
Authors supply area keywords, and the system matches area chairs to papers by keyword overlap.		15,6%	26
Authors bid on area chairs. (last year's model)		17,4%	29
viewI have the following alternative proposal:		3,6%	6

4. What model do you prefer for assigning reviewers to papers? (multiple selections possible)
	answered question		174
	skipped question		21
		Response Percent	Response Count
Two-phase model mixing bidding and manual assignment. In Phase I, reviewer bidding is used to assign two reviewers to each paper. In Phase II, area chairs manually assign additional reviewers based on the Phase I reviews and the author rebuttal. (this year's model)		62,1%	108
Two-phase model like above, but area chairs manually assign reviewers in both phases.		16,1%	28
Single-phase model, where reviewers are assigned to papers based on their bids.		19,5%	34
Single-phase model, where reviewers are manually assigned by the area chairs.		10,9%	19
viewI have the following alternative proposal:		5,2%	9

5. What reviewing model do you prefer?
	answered question		175
	skipped question		20
		Response Percent	Response Count
double-blind (reviewers do not know the authors, and vice versa) (as this year)		76,0%	133
single-blind (reviewers know the authors, but authors don't know the reviewers)		17,7%	31
open (reviewers and authors know each other's identities)		6,3%	11

6. Do you think the Phase I reviews were different in quality from the Phase II reviews?
	answered question		150
	skipped question		45
		Response Percent	Response Count
Phase I substantially better		6,7%	10
Phase I slightly better		16,0%	24
Both were the same		52,0%	78
Phase II slightly better		19,3%	29
Phase II substantially better		6,0%	9

7. Do you think the ICML2010 reviews were different in quality from the reviews at previous ICMLs?
	answered question		146
	skipped question		49
		Response Percent	Response Count
ICML2010 substantially better		1,4%	2
ICML2010 slightly better		14,4%	21
About the same		74,0%	108
Previous ICMLs slightly better		8,2%	12
Previous ICMLs substantially better		2,1%	3
viewAdditional comments:			13

8. Do you think the ICML2010 reviews were different in quality from those at related conferences?
	answered question		156
	skipped question		39
		Response Percent	Response Count
ICML2010 substantially better		7,1%	11
ICML2010 slightly better		35,3%	55
About the same		49,4%	77
Other conferences slightly better		6,4%	10
Other conferences substantially better		1,9%	3
viewWhat conference are you comparing to and what is the subject area?			37

9. What do you think of early rejects by area chairs (i.e., rejecting a paper based on two reviews and author rebuttal)?
	answered question		173
	skipped question		22
		Response Percent	Response Count
I object to early rejects. Rejected papers should get at least three reviews.		9,8%	17
I support early rejects, but only in very clear cases (approximately 13% of submissions this year).		76,3%	132
Area chairs should make more liberal use of early rejects.		13,9%	24

10. What do you think of early accepts (i.e., accepting a paper after only two reviews and author rebuttal)?
	answered question		173
	skipped question		22
		Response Percent	Response Count
We should not allow this (as it was this year).		57,8%	100
We should allow this for papers with exceptionally good reviews.		42,2%	73

11. Do you think the author feedback has an influence on the decision of acceptance/rejection?
	answered question		176
	skipped question		19
		Response Percent	Response Count
Very strong influence.		1,1%	2
Substantial influence.		21,0%	37
Marginal influence.		67,6%	119
No influence.		10,2%	18

12. When do you think should authors be asked to provide their feedback?
	answered question		173
	skipped question		22
		Response Percent	Response Count
After Phase I of the reviewing process, so it can inform the selection of Phase II reviewers and is immediately available during the discussion. (this year's model)		49,7%	86
After Phase II of the reviewing process, so authors can respond to all reviews, not just the Phase I reviews.		17,9%	31
Ask for author feedback both after Phase I and Phase II, even though the paper submission deadline would need to be moved to January 15.		14,5%	25
Do not ask for author feedback at all.		8,1%	14
viewI have the following alternative proposal:		9,8%	17

13. How did you like your assignments in the first reviewing phase?
	answered question		164
	skipped question		31
		Response Percent	Response Count
very interesting		9,8%	16
good fit with my expertise		84,1%	138
mostly not my area		6,1%	10
terrible		0,0%	0

14. How did you like your assignments in the second reviewing phase?
	answered question		141
	skipped question		54
		Response Percent	Response Count
very interesting		12,8%	18
good fit with my expertise		80,9%	114
mostly not my area		6,4%	9
terrible		0,0%	0

15. How would you like to receive your assignment?
	answered question		170
	skipped question		25
		Response Percent	Response Count
An area chair who knows me should assign them to me		12,9%	22
I want to bid for my papers		32,9%	56
We should try a mix of both models		51,2%	87
Don't care		2,9%	5

16. Did you look at the supplementary materials of the papers?
	answered question		171
	skipped question		24
		Response Percent	Response Count
always		22,2%	38
sometimes		33,9%	58
never		8,8%	15
my papers didn't have any		28,7%	49
I don't know whether my papers had any		6,4%	11

17. As a reviewer, how did you rate the author feedback of your papers?
	answered question		168
	skipped question		27
		Response Percent	Response Count
it changed my mind about a paper		7,7%	13
it helped to clarify a few questions, but did not change my mind		75,0%	126
it was not useful		14,9%	25
I did not see it		2,4%	4

18. Throughout the reviewing process, was it always clear to you what needed to be done?
	answered question		170
	skipped question		25
		Response Percent	Response Count
Yes.		66,5%	113
Most of the time.		32,9%	56
Mostly not clear.		0,6%	1
viewComments:			7

19. Which conference tool do you like best?
	answered question		106
	skipped question		89
		Response Percent	Response Count
CMT (this year)		72,6%	77
CyberChair		20,8%	22
START		6,6%	7
viewOther (please specify)			15

20. Did you find CMT
	answered question		162
	skipped question		33
		Response Percent	Response Count
great		7,4%	12
comfortable		40,7%	66
o.k.		42,6%	69
clumsy		6,8%	11
awful		2,5%	4

21. Do you have any suggestions for improving CMT that we should pass on to its developers?
	answered question	26
	skipped question	169
		Response Count
view		26

22. With ca. 160 accepted and invited papers, the program currently fills 5 parallel tracks on three days to the limit. Should we allow the conference to grow further?
	answered question		154
	skipped question		41
		Response Percent	Response Count
No, we already accept too many papers.		18,8%	29
No, we should not accept more papers even if we get substantially more submissions.		28,6%	44
Yes, but include accepted papers that will only be presented as posters.		27,3%	42
Yes, add another parallel track.		13,0%	20
Yes, add another day of sessions to the program.		12,3%	19
viewOther suggestions?			8

23. This year, we invited 7 application papers, which got a free entry to the program and a slot in the proceedings. Do you think this is a good idea?
	answered question		161
	skipped question		34
		Response Percent	Response Count
Yes, we should keep that.		9,9%	16
Sounds good in principle, but I have not yet seen the papers.		52,8%	85
No, application papers are fine, but we do not need additional incentives for attracting them		21,1%	34
No, we should find another way for attracting application papers.		16,1%	26
viewOther comments?			12

24. At this year´s conference, proceedings on paper, USB or CD will not be available, but papers can be downloaded from the Web page. Does this work for you?
	answered question		168
	skipped question		27
		Response Percent	Response Count
Yes.		88,7%	149
Yes, but I think it's a bad idea.		6,5%	11
No.		4,8%	8

25. As in previous years, printed proceedings will be available from Omnipress (this year only after the conference). This service is not free (about 6$ per registration). Do you think we should still produce printed proceedings for archival purposes?
	answered question		165
	skipped question		30
		Response Percent	Response Count
Yes.		33,3%	55
Yes, but try to find a cheaper way.		9,7%	16
No.		57,0%	94

26. What other conferences do you regularly attend?
	answered question		120
	skipped question		75
		Response Percent	Response Count
NIPS		58,3%	70
AIStats		20,0%	24
UAI		20,0%	24
COLT		12,5%	15
ECML/PKDD		40,8%	49
SIGKDD		15,8%	19
ICDM		9,2%	11
IJCAI		21,7%	26
AAAI		24,2%	29
ECAI		10,0%	12
EMNLP		6,7%	8
CoNLL		0,8%	1
SIGIR		3,3%	4
viewOther (please specify)			21