Inducing Honest Behaviour on Amazon's Mechanical Turk

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a large group of workers at a reduced cost. It has been widely reported, however, that crowd workers are not necessarily honest when performing a task in the absence of a well-chosen incentive structure. A crucial question that then arises is how to incentivize self-interested workers to behave honestly in crowdsourcing settings.

In this talk, I will describe how economic incentives might induce honest behaviour on Amazon’s Mechanical Turk, currently one of the most popular crowdsourcing platforms. In particular, I will show how in theory a simple payment function that works by rewarding agreements between workers’ reports is enough to induce honest behaviour on Amazon’s Mechanical Turk.

From an empirical perspective, I will describe the results of a content-analysis experiment on Amazon’s Mechanical Turk that indicate that the presence of economic incentives based on pairwise comparisons results in reports that are more accurate than when such incentives are not present.

Biography:
Arthur Carvalho is a PhD candidate in the David R. Cheriton School of Computer Science at the University of Waterloo. His research interests are in the design of economic mechanisms to incentivize self-interested agents to behave in desirable ways, and in the design of methods to elicit and aggregate information from agents in order to build collective intelligence.