Skip to content

Opening book details…

Can I read Measurement Integrity in Peer Prediction: A Peer Assessment Case Study on EtoBox?

Measurement Integrity in Peer Prediction: A Peer Assessment Case Study by Burrell, Noah; Schoenebeck, Grant is a scholarly article available to read on EtoBox.

What is Measurement Integrity in Peer Prediction: A Peer Assessment Case Study about?

We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many natural applications. Like robustness against strategic reporting, the property that has been the primary focus of the peer prediction literature, measurement integrity is an important consideration for understanding the practical performance of peer prediction mechanisms. We perform computational experiments, both with an agent-based model and with real data, to empirically evaluate peer prediction mechanisms according to both of these important properties. Our evaluations simulate the application of peer prediction mechanisms to peer assessment -- a setting in which ex post fairness concerns are particularly salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate significant measurement integrity in our experiments. We also find that theoretical properties concerning robustness against strategic reporting are somewhat noisy predictors of empirical performance. Further, there is an apparent trade-off between our two dimensions of analysis. The best-performing mechanisms in terms of measur

Author
Burrell, Noah; Schoenebeck, Grant
Published
2021
Language
EN

More by Burrell, Noah; Schoenebeck, Grant

Browse all works by Burrell, Noah; Schoenebeck, Grant