A construction principle for proper scoring rules

Proper scoring rules enable decision-theoretically principled comparisons of probabilistic forecasts. New scoring rules can be constructed by identifying the predictive distribution with an element of a parametric family and then applying a known scoring rule. We introduce a condition which ensures propriety in this construction and thereby obtain novel proper scoring rules.

Evaluating probabilistic forecasts of football matches: the case against the ranked probability score

A scoring rule is a function of a probabilistic forecast and a corresponding outcome used to evaluate forecast performance. There is some debate as to which scoring rules are most appropriate for evaluating forecasts of sporting events. This paper focuses on forecasts of the outcomes of football matches. The ranked probability score (RPS) is often recommended since it is ‘sensitive to distance’, that is it takes into account the ordering in the outcomes (a home win is ‘closer’ to a draw than it is to an away win). In this paper, this reasoning is disputed on the basis that it adds nothing in terms of the usual aims of using scoring rules. A local scoring rule is one that only takes the probability placed on the outcome into consideration. Two simulation experiments are carried out to compare the performance of the RPS, which is non-local and sensitive to distance, the Brier score, which is non-local and insensitive to distance, and the Ignorance score, which is local and insensitive to distance. The Ignorance score outperforms both the RPS and the Brier score, casting doubt on the value of non-locality and sensitivity to distance as properties of scoring rules in this context.

Scoring Probability Forecasts by a User’s Bets Against a Market Consensus

The purpose of our paper is to describe a probability scoring rule that reflects the economic performance of a hypothetical investor who acts upon the probability forecasts emanating from a given model or human expert by trading against a market-clearing consensus of competing models and forecasts. The probability forecasts being compared are aggregated by an equilibrium condition into a market consensus reflecting the wisdom of the crowd. A good forecaster (model or human expert) is measured as one who allows the user to bet profitably against the market consensus. By asking forecasts to beat the market, forecasters are discouraged from herding and motivated to obtain better information than rival forecasters. We illustrate and prove that each trader’s personal incentive to hedge or fudge disappears when the number of forecasts in the market is sufficiently large. Our score exhibits the forecaster’s ability to assist economically profitable action and reveals how the user’s profits depend strongly on the accuracy of the forecasts and the decision rule (boldness) with which they are acted upon.

Understanding Persuasion Cascades in Online Product Rating Systems: Modeling, Analysis, and Inference

Online product rating systems have become an indispensable component for numerous web services such as Amazon, eBay, Google Play Store, and TripAdvisor. One functionality of such systems is to uncover the product quality via product ratings (or reviews) contributed by consumers. However, a well-known psychological phenomenon called “ message-based persuasion ” lead to “ biased ” product ratings in a cascading manner (we call this the persuasion cascade ). This article investigates: (1) How does the persuasion cascade influence the product quality estimation accuracy? (2) Given a real-world product rating dataset, how to infer the persuasion cascade and analyze it to draw practical insights? We first develop a mathematical model to capture key factors of a persuasion cascade. We formulate a high-order Markov chain to characterize the opinion dynamics of a persuasion cascade and prove the convergence of opinions. We further bound the product quality estimation error for a class of rating aggregation rules including the averaging scoring rule, via the matrix perturbation theory and the Chernoff bound. We also design a maximum likelihood algorithm to infer parameters of the persuasion cascade. We conduct experiments on both synthetic data and real-world data from Amazon and TripAdvisor. Experiment results show that our inference algorithm has a high accuracy. Furthermore, persuasion cascades notably exist, but the average scoring rule has a small product quality estimation error under practical scenarios.

Characterising scoring rules by their solution in iteratively undominated strategies

We characterize voting procedures according to the social choice correspondence they implement when voters cast ballots strategically, applying iteratively undominated strategies. In elections with three candidates, the Borda Rule is the unique positional scoring rule that satisfies unanimity (U) (i.e., elects a candidate whenever it is unanimously preferred) and is majoritarian after eliminating a worst candidate (MEW)(i.e., if there is a unanimously disliked candidate, the majority-preferred among the other two is elected). In a larger class of rules, Approval Voting is characterized by a single axiom that implies both U and MEW but is weaker than Condorcet-consistency (CON)—it is the only direct mechanism scoring rule that is majoritarian after eliminating a Pareto-dominated candidate (MEPD)(i.e., if there is a Pareto-dominated candidate, the majority-preferred among the other two is elected); among all finite scoring rules that satisfy MEPD, Approval Voting is the most decisive. However, it fails a desirable monotonicity property: a candidate that is elected for some preference profile, may lose the election once she gains further in popularity. In contrast, the Borda Rule is the unique direct mechanism scoring rule that satisfies U, MEW and monotonicity (MON). There exists no direct mechanism scoring rule that satisfies both MEPD and MON and no finite scoring rule satisfying CON.

Predicting Recurrent Venous Thromboembolism in Patients With Deep-Vein Thrombosis: Development and Internal Validation of a Potential New Prediction Model (Continu-8)

Background: Previous prediction models for recurrent thromboembolism (VTE) are often complicated to apply and have not been implemented widely.Aim: To develop and internally validate a potential new prediction model for recurrent VTE that can be used without stopping anticoagulant treatment for D-dimer measurements in patients with provoked and unprovoked DVT.Methods: Cohort data of 479 patients treated in a clinical care pathway at Maastricht University Medical Center were used. Predictors for the Cox proportional hazards model (unprovoked DVT, male gender, factor VIII levels) were derived from literature and using forward selection procedure. The scoring rule was internally validated using bootstrapping techniques and the predictive ability was compared to existing prediction models.Results: Patients were followed for a median of 3.12 years after stopping anticoagulation treatment (IQR 0.78, 3.90). Sixty-four of 479 patients developed recurrent VTE (13%). The scoring rule consisted of unprovoked DVT (yes: 2 points), male sex (yes: 1 point), and factor VIII > 213 % (yes: 2 points) and was categorized into three groups [i.e., low risk (score 0), medium risk (scores 1, 2, or 3) and high risk (scores 4 and 5)]. The concordance statistic was 0.68 (95% CI: 0.61, 0.75).Conclusion: The discriminative ability of the new Continu-8 score was adequate. Future studies shall verify this score in an independent setting without stopping anticoagulation treatment.

Incumbent and Entrant Bidding in Scoring Rule Auctions: A Study on Italian Canteen Services

We empirically investigate incumbents’ and entrants’ bids on an original dataset of 192 scoring rule auctions for canteen services in Italy. Our findings show that winning rebates are lower (i.e., prices paid by the public buyer are higher) when the contract is awarded to the incumbent supplier. This result is not explained by the observable characteristics of the auction or the service awarded. We develop a simple theoretical model showing that the result is consistent with a setting in which the buyer exploits specific information on the incumbent supplier’s production cost.