The 2016 Wines of Portugal Challenge: general implications of more than 8400 wine-score observations

TLDR: The Wines of Portugal Challenge as discussed by the authors is an annual competition among wines produced by over 1000 vintners in over 30 of the country's wine growing regions, and judges assigned scores to over 1300 wines resulting in over 8400 wine-score observations.

Abstract: The Wines of Portugal Challenge is an annual competition among wines produced by over 1000 vintners in over 30 of the country’s wine growing regions. In 2016, judges assigned scores to over 1300 wines resulting in over 8400 wine-score observations. Analysis of that large sample yields implications about wine judges’ ratings that are difficult to detect with statistical significance in the small samples that are typical of most wine tastings. The Challenge’s frequency distribution of scores showed left skewness and local peaks just below the score thresholds for bronze, silver and gold awards. Student’s t-tests showed that there were no significant differences in scores assigned by gender-of-judge, nationality-of-judge and to wines from different regions. However, judges did assign higher scores to sweet wines than to other types of wine. While the dispersion in scores was material, p-values showed that the aggregate order of rating was very unlikely to be random and the distributions of mean scores showed that the strengths of judges’ preferences against the least-preferred wines were stronger than those in favor of the most-preferred wines. Ties between wines’ mean scores were common and could be broken by several methods including the preference probabilities implied by a Plackett-Luce model.