"How not to sort by average rating" (2009): https://news.ycombinator.com/item?id=3792627 For thumbs-up/thumbs-down systems, suggests using the lower bound of a Wilson confidence interval for a Bernoulli distribution, which is what Reddit does now. Convincingly refuted by How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective, http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... by Dell Zhang et al., which argues for simple smoothing with a Dirichlet prior (i.e. (upvotes + x) รท (upvotes + x + downvotes + y)), which was also suggested by several people in the comments.
Perhaps related: HotOrNot started out displaying the mean of the rankings as the rating of each photo (after you clicked on it). But they found that there was a gradual drift down in ratings: they started with around 1-5 (out of a theoretical max of 10), then ended up around 1-3, etc., with the predictable damaging effects on egos, people's willingness to post their photos, and the information content of the ratings. The solution they adopted was to display not the mean of ratings but the percentile: a photo rated higher than 76% of other photos would have its "average" displayed as "7.6", even if the mean was 4.5. This trained the users to flatten the histogram!
http://www.nashcoding.com/2011/10/28/hackernews-needs-honeyp... suggested that fake "products" to attract ratings could distinguish intelligent ratings from unintelligent ones. Although written about thumbs-up/down systems, it applies to multi-star systems as well.
"How not to sort by average rating" (2009): https://news.ycombinator.com/item?id=3792627 For thumbs-up/thumbs-down systems, suggests using the lower bound of a Wilson confidence interval for a Bernoulli distribution, which is what Reddit does now. Convincingly refuted by How to Count Thumb-Ups and Thumb-Downs: User-Rating based Ranking of Items from an Axiomatic Perspective, http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2... by Dell Zhang et al., which argues for simple smoothing with a Dirichlet prior (i.e. (upvotes + x) รท (upvotes + x + downvotes + y)), which was also suggested by several people in the comments.
In 2010, William Morgan wrote http://masanjin.net/blog/how-to-rank-products-based-on-user-... partly in response, applying Bayesian statistics to the problem of ranking things rated using 5-star rating systems.
Perhaps related: HotOrNot started out displaying the mean of the rankings as the rating of each photo (after you clicked on it). But they found that there was a gradual drift down in ratings: they started with around 1-5 (out of a theoretical max of 10), then ended up around 1-3, etc., with the predictable damaging effects on egos, people's willingness to post their photos, and the information content of the ratings. The solution they adopted was to display not the mean of ratings but the percentile: a photo rated higher than 76% of other photos would have its "average" displayed as "7.6", even if the mean was 4.5. This trained the users to flatten the histogram!
http://www.nashcoding.com/2011/10/28/hackernews-needs-honeyp... suggested that fake "products" to attract ratings could distinguish intelligent ratings from unintelligent ones. Although written about thumbs-up/down systems, it applies to multi-star systems as well.