Judging the judges: evaluating the accuracy and national bias of international gymnastics judges

We design, describe and implement a statistical engine to analyze the performance of gymnastics judges with three objectives: (1) provide constructive feedback to judges, executive committees and national federations; (2) assign the best judges to the most important competitions; (3) detect bias and persistent misjudging. Judging a gymnastics routine is a random process, and we model this process using heteroscedastic random variables. The developed marking score scales the difference between the mark of a judge and the true performance level of a gymnast as a function of the intrinsic judging error variability estimated from historical data for each apparatus. This dependence between judging variability and performance quality has never been properly studied. We leverage the intrinsic judging error variability and the marking score to detect outlier marks and study the national bias of judges favoring athletes of the same nationality. We also study ranking scores assessing to what extent judges rate gymnasts in the correct order. Our main observation is that there are significant differences between the best and worst judges, both in terms of accuracy and national bias. The insights from this work have led to recommendations and rule changes at the Fédération Internationale de Gymnastique.

Estimating player value in American football using plus–minus models

Calculating the value of football player’s on-field performance has been limited to scouting methods while data-driven methods are mostly limited to quarterbacks. A popular method to calculate player value in other sports are Adjusted Plus–Minus (APM) and Regularized Adjusted Plus–Minus (RAPM) models. These models have been used in other sports, most notably basketball (Rosenbaum, D. T. 2004. Measuring How NBA Players Help Their Teams Win. http://www.82games.com/comm30.htm#_ftn1; Kubatko, J., D. Oliver, K. Pelton, and D. T. Rosenbaum. 2007. “A Starting Point for Analyzing Basketball Statistics.” Journal of Quantitative Analysis in Sports 3 (3); Winston, W. 2009. Player and Lineup Analysis in the NBA. Cambridge, Massachusetts; Sill, J. 2010. “Improved NBA Adjusted +/− Using Regularization and Out-Of-Sample Testing.” In Proceedings of the 2010 MIT Sloan Sports Analytics Conference) to estimate each player’s value by accounting for those in the game at the same time. Football is less amenable to APM models due to its few scoring events, few lineup changes, restrictive positioning, and small quantity of games relative to the number of teams. More recent methods have found ways to incorporate plus–minus models in other sports such as Hockey (Macdonald, B. 2011. “A Regression-Based Adjusted Plus-Minus Statistic for NHL players.” Journal of Quantitative Analysis in Sports 7 (3)) and Soccer (Schultze, S. R., and C.-M. Wellbrock. 2018. “A Weighted Plus/Minus Metric for Individual Soccer Player Performance.” Journal of Sports Analytics 4 (2): 121–31 and Matano, F., L. F. Richardson, T. Pospisil, C. Eubanks, and J. Qin (2018). Augmenting Adjusted Plus-Minus in Soccer with Fifa Ratings. arXiv preprint arXiv:1810.08032). These models are useful in coming up with results-oriented estimation of each player’s value. In American football, many positions such as offensive lineman have no recorded statistics which hinders the ability to estimate a player’s value. I provide a fully hierarchical Bayesian plus–minus (HBPM) model framework that extends RAPM to include position-specific penalization that solves many of the shortcomings of APM and RAPM models in American football. Cross-validated results show the HBPM to be more predictive out of sample than RAPM or APM models. Results for the HBPM models are provided for both Collegiate and NFL football players as well as deeper insights into positional value and position-specific age curves.

Towards a more objective time standard in competitive rowing

Rowing needs a standardized Gold Medal Standard (GMS) to clearly compare performance across boat classes in competition. Here, we report a method to factor out environmental effects, developing a fairer GMS for individual rowing events. We used results from World Rowing Championships and Olympics Games (2005–2016) to calculate the difference between the fastest winning time of the day and other event winning times on the same day. From this, we calculated a prognostic GMS time for each event via repeated k-fold cross-validation linear regression. Then, we compared these values with the 10-year average winning time and the World Best Time (WBT). We repeated this process to develop prognostic podium standard (PS) times. The prognostic GMS times (RMSE = 9.47; R 2 = 0.875) were universally slower than the WBT (current GMS) by 6.2 s on average but faster than the 10-year average by 12.3 s. The prognostic PS times (RMSE = 10.5; R 2 = 897) were also slower than the WBT but faster than the 10-year average, by 12.2 and 6.3 s respectively. Our time-difference prediction model based on historical data generates non-outlier prognostic times. With the utilization of relative time difference, this approach promises a selection standard independent of environmental conditions, easily applicable across different sports.

Evaluating probabilistic forecasts of football matches: the case against the ranked probability score

A scoring rule is a function of a probabilistic forecast and a corresponding outcome used to evaluate forecast performance. There is some debate as to which scoring rules are most appropriate for evaluating forecasts of sporting events. This paper focuses on forecasts of the outcomes of football matches. The ranked probability score (RPS) is often recommended since it is ‘sensitive to distance’, that is it takes into account the ordering in the outcomes (a home win is ‘closer’ to a draw than it is to an away win). In this paper, this reasoning is disputed on the basis that it adds nothing in terms of the usual aims of using scoring rules. A local scoring rule is one that only takes the probability placed on the outcome into consideration. Two simulation experiments are carried out to compare the performance of the RPS, which is non-local and sensitive to distance, the Brier score, which is non-local and insensitive to distance, and the Ignorance score, which is local and insensitive to distance. The Ignorance score outperforms both the RPS and the Brier score, casting doubt on the value of non-locality and sensitivity to distance as properties of scoring rules in this context.

Opening up the court: analyzing player performance across tennis Grand Slams

In tennis, the Australian Open, French Open, Wimbledon, and US Open are the four most prestigious events (Grand Slams). These four Grand Slams differ in the composition of the court surfaces, when they are played in the year, and which city hosts the players. Individual Grand Slams come with different expectations, and it is often thought that some players achieve better results at some Grand Slams than others. It is also thought that differences in results may be attributed, at least partially, to surface type of the courts. For example, Rafael Nadal, Roger Federer, and Serena Williams have achieved their best results on clay, grass, and hard courts, respectively. This paper explores differences among Grand Slams, while adjusting for confounders such as tour, competitor strength, and player attributes. More specifically, we examine the effect of the Grand Slam on player performance for matches from 2013 to 2019. We take two approaches to modeling these data: (1) a mixed-effects model accounting for both player and tournament features and (2) models that emphasize individual performance. We identify differences across the Grand Slams at both the tournament and individual player level.

Distributed lag models to identify the cumulative effects of training and recovery in athletes using multivariate ordinal wellness data

Subjective wellness data can provide important information on the well-being of athletes and be used to maximize player performance and detect and prevent against injury. Wellness data, which are often ordinal and multivariate, include metrics relating to the physical, mental, and emotional status of the athlete. Training and recovery can have significant short- and long-term effects on athlete wellness, and these effects can vary across individual. We develop a joint multivariate latent factor model for ordinal response data to investigate the effects of training and recovery on athlete wellness. We use a latent factor distributed lag model to capture the cumulative effects of training and recovery through time. Current efforts using subjective wellness data have averaged over these metrics to create a univariate summary of wellness, however this approach can mask important information in the data. Our multivariate model leverages each ordinal variable and can be used to identify the relative importance of each in monitoring athlete wellness. The model is applied to professional referee daily wellness, training, and recovery data collected across two Major League Soccer seasons.

The middle-seed anomaly: why does it occur in some sports tournaments but not others?

Previously published statistical analyses of NCAA Division I Men’s Tournament (“March Madness”) game outcomes have revealed that the relationship between tournament seed and the time-aggregated number of third-round (“Sweet 16”) appearances for the middle half of the seeds exhibits a statistically and practically significant departure from monotonicity. In particular, the 8- and 9-seeds combined appear less often than any one of seeds 10–12. In this article, we show that a similar “middle-seed anomaly” also occurs in the NCAA Division I Women’s Tournament but does not occur in two other major sports tournaments that are similar in structure to March Madness. We offer explanations for the presence of a middle-seed anomaly in the NCAA basketball tournaments, and its absence in the others, that are based on the combined effects of the functional form of the relationship between team strength and seed specific to each tournament, the degree of parity among teams, and certain elements of tournament structure. Although these explanations account for the existence of middle-seed anomalies in the NCAA basketball tournaments, their larger-than-expected magnitudes, which arise mainly from the overperformance of seeds 10–12 in the second round, remain enigmatic.