Culling the Herding: Using Real World Randomised Experiments to Measure Social Bias with Known Costly Goods

Peer-rating systems have become an increasingly popular way for consumers to learn about the quality of products. Prior literature has argued that the accuracy of social signals is limited by herding behaviour -- social signals may drive the popularity of products and services based on initial conditions independently of their inherent quality. However, most of these results have been obtained without asking users to risk their money. Thus it is not clear whether they apply to more real world settings with costly known goods. In this paper we design and implement a randomized field experiment to determine how ``likes'' influence the sales of movies in a real-world Video-on-Demand (VoD) marketplace. Movies in the VoD system of our cable provider partner are costly and known to the public at large. This VoD system suggests movies to subscribers when they log in. These movies are shown from left to right in decreasing order of the number of likes given by other users. During our experiment, movies were primarily placed in their true slots and shown alongside with their true number of likes. However, sometimes some movies had their positions swapped. These movies, randomly chosen, were displayed out of order and with a fake number of likes.  We found that promoting a movie by one slot increased weekly sales by 4% on average and that better-known movies were less sensitive to these manipulations than other movies were. We also found that on average a movie promoted (demoted) to a fake slot sold 15.9% less (27.7% more) than a true movie placed at that slot. Likewise, we found that a movie promoted (demoted) to a fake slot received 33.1% fewer (30.1% more) likes than a true movie at that slot. Therefore, manipulated movies tend to move back to their true slots over time. In our case, this process is likely to converge quickly, which might lead the cable provider to promote different movies over time. Hence, we find that the self-fulfilling prophecies widely discussed in the literature on the effect of ratings on sales are hard to sustain in a real-world market in which goods are costly and sufficiently well known.