Please refer to the detailed schedule on the following link:

For some questions, such as whether extraterrestrial life exists, it is uncertain if and when the answer will be known. Asking experts for their opinion yields two practical problems. First, how can truth-telling be incentivized if the correct answer is unknowable? Second, if experts disagree, who should be trusted? This paper solves both problems simultaneously. Experts decide whether to endorse a statement and trade an asset whose value depends on the endorsement rate. The respective payoffs of buyers and sellers indicate whom to trust. We demonstrate theoretically and illustrate empirically that ``following the money" outperforms selecting the majority opinion.

Learning and estimation techniques draw insights about unknown underlying relations based on statistical properties of the observed data. Factors that can change the statistical properties of the observed data thus affect the conclusions drawn by these techniques. One such factor is the strategic nature of data holders. The strategic behavior of data holders can alter the observed data even when the underlying relations are unchanged, hence leading to inaccurate conclusions. The cause of the strategic behavior varies from cost of providing data to vested interests in the learning outcomes. In this talk, I will discuss a few attempts to account for the strategic behavior of data holders in statistical estimation and learning. It calls for a more integrated approach for thinking of data and learning.

The talk is available on the following link:

People give massive amounts of their personal data to companies every day and these data are used to generate tremendous business values. Some economists, politicians, and activists argue that people should be paid for their contributions—but the million-dollar question is: by how much? In this talk, I will present some recent work on data valuation. I will start by introducing a principled notion for data value and then present a suite of algorithms that we developed to efficiently compute the data value. I will also discuss the applications of our data valuation techniques to the tasks beyond data pricing, such as detecting bad training data.

The talk is available on the following link:

In the setting where participants are asked multiple similar possibly subjective multi-choice questions (e.g. Do you like Panda Express? Y/N; do you like Chick-fil-A? Y/N), a series of peer prediction mechanisms are designed to incentivize honest reports and some of them achieve dominantly truthfulness: truth-telling is a dominant strategy and strictly dominate other ``non-permutation strategy'' with some mild conditions. However, a major issue hinders the practical usage of those mechanisms: they require the participants to perform an infinite number of tasks. When the participants perform a finite number of tasks, these mechanisms only achieve approximated dominant truthfulness. The existence of a dominantly truthful multi-task peer prediction mechanism that only requires a finite number of tasks remains to be an open question that may have a negative result, even with full prior knowledge.

This work answers this open question by proposing a new mechanism, Determinant based Mutual Information Mechanism (DMI-Mechanism), that is dominantly truthful when the number of tasks is at least 2C. C is the number of choices for each question (C=2 for binary-choice questions). DMI-Mechanism also pays truth-telling higher than any strategy profile and strictly higher than uninformative strategy profiles (informed truthfulness). In addition to the truthfulness properties, DMI-Mechanism is also easy to implement since it does not require any prior knowledge (detail-free) and only requires at least two participants. The core of DMI-Mechanism is a novel information measure, Determinant based Mutual Information (DMI). DMI generalizes Shannon's mutual information and the square of DMI has a simple unbiased estimator. In addition to incentivizing honest reports, DMI-Mechanism can also be transferred into an information evaluation rule that identifies high-quality information without verification when there are at least three participants.

Peer review is an essential part of scientific research, and has a considerable influence on careers of researchers. Hence enter Dr. Deceit, who by various dishonest means, tries to game the peer review system (yes, this does happen in reality). Our goal is to thwart Dr. Deceit's malicious activities.

Dr. Deceit: As a reviewer, I will manipulate the scores or rankings of the papers that I review in order to increase the chances of my own paper getting accepted. Ha ha ha!

Us: We will use an impartial mechanism, e.g., via a partition-based method, which guarantees a reviewer cannot influence their own paper's outcome. We show via an analysis on ICLR data that such a mechanism is feasible in conference peer review, despite the complexity and constraints of the conference peer-review process.

Dr. Deceit: But using such a mechanism reduces the efficiency of the process. So if there is no deceitful reviewer like me in the conference, the mechanism will hurt the efficiency of the peer review. Would you really want to use it then?

Us: We can help make that decision -- we design statistical tests to detect the existence of such strategic behavior in peer assessment.

Dr. Deceit: Ok so you will stop me from manipulating my reviews to help my own paper. But I will strike a quid pro quo deal with another potential reviewer for my paper: the reviewer will try to get to review my paper and give a positive review, and in exchange I'll do the same for them in another conference. Your impartial mechanisms can't do anything about this.

Us: We also design randomized reviewer-assignment algorithms which optimally mitigate such arbitrary reviewer-author collusions. Evaluations on data from four conferences show their promise for use in practice.

Dr. Deceit: Fine. I will recruit not one, but multiple such reviewers.

Us: Hmm..then we get into computational-hardness-land. But there is probably some structure on your colluders (e.g., colluding reviewers are at the same institution). Then we have optimal mitigating strategies computable in polynomial time. Keep trying in vain, Dr. Deceit!

Throughout the talk, Dr. Deceit will also throw some more important challenges at us whose solutions are yet unknown.

We consider the design of forecasting competitions in which multiple forecasters make predictions about one or more independent events and compete for a single prize. We have two objectives: (1) to award the prize to the most accurate forecaster, and (2) to incentivize forecasters to report truthfully, so that forecasts are informative and forecasters need not spend any cognitive effort strategizing about reports. Proper scoring rules incentivize truthful reporting if all forecasters are paid according to their scores. However, incentives become distorted if only the best-scoring forecaster wins a prize, since forecasters can often increase their probability of having the highest score by reporting extreme beliefs. In this paper, we introduce a truthful forecaster selection mechanism. We lower-bound the probability that our mechanism selects the most accurate forecaster, and give rates for how quickly this bound approaches 1 as the number of events grows. Our techniques can be generalized to the related problems of outputting a ranking over forecasters and hiring a forecaster with high accuracy on future events.

The talk is available on the following link:

The talk is available on the following link: