Invited Talk: Thwarting Dr. Deceit's Malicious Activities in Conference Peer Review
Abstract
Peer review is an essential part of scientific research, and has a considerable influence on careers of researchers. Hence enter Dr. Deceit, who by various dishonest means, tries to game the peer review system (yes, this does happen in reality). Our goal is to thwart Dr. Deceit's malicious activities.
Dr. Deceit: As a reviewer, I will manipulate the scores or rankings of the papers that I review in order to increase the chances of my own paper getting accepted. Ha ha ha!
Us: We will use an impartial mechanism, e.g., via a partition-based method, which guarantees a reviewer cannot influence their own paper's outcome. We show via an analysis on ICLR data that such a mechanism is feasible in conference peer review, despite the complexity and constraints of the conference peer-review process.
Dr. Deceit: But using such a mechanism reduces the efficiency of the process. So if there is no deceitful reviewer like me in the conference, the mechanism will hurt the efficiency of the peer review. Would you really want to use it then?
Us: We can help make that decision -- we design statistical tests to detect the existence of such strategic behavior in peer assessment.
Dr. Deceit: Ok so you will stop me from manipulating my reviews to help my own paper. But I will strike a quid pro quo deal with another potential reviewer for my paper: the reviewer will try to get to review my paper and give a positive review, and in exchange I'll do the same for them in another conference. Your impartial mechanisms can't do anything about this.
Us: We also design randomized reviewer-assignment algorithms which optimally mitigate such arbitrary reviewer-author collusions. Evaluations on data from four conferences show their promise for use in practice.
Dr. Deceit: Fine. I will recruit not one, but multiple such reviewers.
Us: Hmm..then we get into computational-hardness-land. But there is probably some structure on your colluders (e.g., colluding reviewers are at the same institution). Then we have optimal mitigating strategies computable in polynomial time. Keep trying in vain, Dr. Deceit!
Throughout the talk, Dr. Deceit will also throw some more important challenges at us whose solutions are yet unknown.