Abstract

Alice submits a number of papers to a machine learning conference and has knowledge of the quality of her papers. Given noisy grades provided by independent reviewers, can Bob obtain accurate estimates of the ground-truth quality of the papers by asking Alice a question about the ground truth? In this talk, we address this when the payoff of Alice is additive convex utility over all her papers. First, if Alice would truthfully answer the question because by doing so her payoff is maximized, we show that the questions must be formulated as pairwise comparisons between her papers. Moreover, if Alice is required to provide a ranking of her papers, which is the most fine-grained question via pairwise comparisons, we prove that she would be truth-telling. By incorporating the ground-truth ranking, we show that Bob can obtain an estimator with the optimal squared error in certain regimes based on any possible ways of truthful information elicitation. Moreover, the estimated grades are substantially more accurate than the raw grades when the number of papers is large and the raw grades are very noisy. Finally, we conclude the talk with several extensions and some refinements for practical considerations. This is based on the paper “A Truthful Owner-Assisted Scoring Mechanism” (tinyurl.com/4f7pnfk6).

Biography

Weijie Su is an assistant professor, and will be an associate professor starting this July, in the Department of Statistics and Data Science of the Wharton School and, by courtesy, in the Department of Computer and Information Science, at the University of Pennsylvania. He is a co-director of Penn Research in Machine Learning. Prior to joining Penn, he received his Ph.D. in statistics from Stanford University in 2016 and his bachelor’s degree in mathematics from Peking University in 2011. His research interests span privacy-preserving data analysis, optimization, high-dimensional statistics, and deep learning theory. He is a recipient of the Stanford Theodore Anderson Dissertation Award in 2016, an NSF CAREER Award in 2019, an Alfred Sloan Research Fellowship in 2020, and the Society for Industrial and Applied Mathematics (SIAM) Early Career Prize in Data Science in 2022.

• Admission

Zoom Meeting ID: 873 0707 1043

Password: 773480