Comparing probabilistic accounts of probability judgments

Bayesian theories of cognitive science hold that cognition is fundamentally probabilistic, but people’s explicit probability judgments often violate the laws of probability. Two recent proposals, the “Probability Theory plus Noise” (Costello & Watts, 2014) and “Bayesian Sampler” (Zhu et al., 2020) theories of probability judgments, both seek to account for these biases while maintaining that mental credences are fundamentally probabilistic. These theories fit quite differently into the larger project of Bayesian cognitive science, but their many similarities complicate comparisons of their predictive accuracy. In particular, comparing the models demands a careful accounting of model complexity. Here, I cast these theories into a Bayesian data analysis framework that supports principled model comparison using information criteria. Comparing the fits of both models on data collected by Zhu and colleagues (2020) I find the data are best explained by a modified version of the Bayesian Sampler model under which people may hold informative priors about probabilities.

Bayesianism: Objections and Rebuttals

While the laws of probability are rarely disputed, the question of how we should interpret probability judgments is less straightforward. Broadly, there are two ways to conceive of probability—either as an objective feature of the world, or as a subjective measure of our uncertainty. Both notions have their place in science, but it is the latter subjective notion (the Bayesian approach) that is crucial in legal reasoning. This chapter explains the advantages of using Bayesian networks in adjudicative factfinding. It addresses a number of common objections to the Bayesian approach, such as “There is no such thing as a probability of a single specified event”; “The Bayesian approach only works with statistical evidence”; “The Bayesian approach is too difficult for legal factfinders to comprehend”; and “A Bayesian network can never capture the full complexity of a legal case.” Fenton and Lagnado offer rebuttals to each of these objections.

When alternative hypotheses shape your beliefs: Context effects in probability judgments

When people are asked to estimate the probability of an event occurring, they sometimes make different subjective probability (SP) judgments for different descriptions of the same event. This implies the evidence or support recruited to make SPs is based on the descriptions of the events (hypotheses) instead of the events themselves, as captured by Tversky and Koehler's (1994) support theory. However, is the support assigned to a hypothesis invariant, as support theory assumes? Here, across two studies where participants were asked to estimate the probability that an event would occur, we show that the support people recruit about the target hypothesis also depends on the other hypotheses under consideration. The first study shows that the presence of a distractor---a hypothesis objectively dominated by the target hypothesis---boosts the SP assigned to the target hypothesis. The second study shows that the presence of a resembler---a hypothesis that is objectively similar to the target hypothesis---detracts more from the SP assigned to the target hypothesis than the competing hypothesis. These context effects invalidate the regularity and the strong independence assumptions of support theory and more generally suggest a similar process that drives the construction of preference also underlies belief.

The Ape Lottery: Chimpanzees Fail To Consider Spatial Information When Drawing Statistical Inferences

Humans and nonhuman great apes share a sense for intuitive statistics, making intuitive probability judgments based on proportional information. This ability is of tremendous importance, in particular for predicting the outcome of events using prior information and for inferring general regularities from limited numbers of observations. Already in infancy, humans functionally integrate intuitive statistics with other cognitive domains, rendering this type of reasoning a powerful tool to make rational decisions in a variety of contexts. Recent research suggests that chimpanzees are capable of one type of such cross-domain integration: The integration of statistical and social information. Here, we investigated whether apes can also integrate physical information into their statistical inferences. We tested 14 sanctuary-living chimpanzees in a new task setup consisting of two “gumball machine”-apparatuses that were filled with different combinations of preferred and non-preferred food items. In four test conditions, subjects decided which of two apparatuses they wanted to operate to receive a random sample, while we varied both the proportional composition of the food items as well as their spatial configuration above and below a barrier. To receive the more favorable sample, apes needed to integrate proportional and spatial information. Chimpanzees succeeded in conditions in which we provided them either with proportional information or spatial information, but they failed to correctly integrate both types of information when they were in conflict. Whether these limitations in chimpanzees' performance reflect true limits of cognitive competence or merely performance limitations due to accessory task demands is still an open question.

The Response Vector for Mastery Method of Standard Setting

Proposed is a new method of standard setting referred to as response vector for mastery (RVM) method. Under the RVM method, the task of panelists that participate in the standard setting process does not involve conceptualization of a borderline examinee and probability judgments as it is the case with the Angoff and bookmark methods. Also, the RVM-based computation of a cut-score is not based on a single item (e.g., marked in an ordered item booklet) but, instead, on a response vector (1/0 scores) on items and their parameters calibrated in item response theory or under the recently developed D-scoring method. Illustrations with hypothetical and real-data scenarios of standard setting are provided and methodological aspects of the RVM method are discussed.

What Can the Conjunction Fallacy Tell Us about Human Reasoning?

This chapter briefly summarizes some the main results obtained from more than three decades of studies on the conjunction fallacy. It shows that this striking and widely discussed reasoning error is a robust phenomenon that can systematically affect the probabilistic inferences of both laypeople and experts, and it introduces an explanation based on the notion of evidential impact in terms of contemporary Bayesian confirmation theory. Finally, the chapter tackles the open issue of the greater accuracy and reliability of impact assessments over posterior probability judgments and outlines how further research on the role of evidential reasoning in the acceptability of explanations might contribute to the development of effective human-like computing.

Effects of experience-based frequency learning on posterior probability judgments

Aging is associated with an increase in the frequency of medical screening tests. Bayesian inference is used to estimate posterior probabilities of medical tests such as positive or negative predictive values (PPVs or NPVs). Both laypeople and experts are typically poor at estimating PPVs and NPVs when relevant probabilities are communicated descriptively. Decision making research has revealed dissociations between described and experience-based judgments. This study examined the accuracy of posterior probability estimates of 80 younger and 81 older adults when statistical information was presented through description or experience. Results show that both younger and older adults can make more accurate posterior probability estimates if they experience probabilities compared to when probabilities are described as either natural frequencies or conditional probabilities. Results also indicate that most people prefer to rely on physicians to make their medical decisions regardless of how confident they are in their judgments of probabilities.

Effects of experience-based frequency learning on posterior probability judgments

Aging is associated with an increase in the frequency of medical screening tests. Bayesian inference is used to estimate posterior probabilities of medical tests such as positive or negative predictive values (PPVs or NPVs). Both laypeople and experts are typically poor at estimating PPVs and NPVs when relevant probabilities are communicated descriptively. Decision making research has revealed dissociations between described and experience-based judgments. This study examined the accuracy of posterior probability estimates of 80 younger and 81 older adults when statistical information was presented through description or experience. Results show that both younger and older adults can make more accurate posterior probability estimates if they experience probabilities compared to when probabilities are described as either natural frequencies or conditional probabilities. Results also indicate that most people prefer to rely on physicians to make their medical decisions regardless of how confident they are in their judgments of probabilities.

The Wisdom of the Coherent: Improving Correspondence with Coherence-Weighted Aggregation

Previous research shows that variation in coherence (i.e., degrees of respect for axioms of probability calculus), when used as a basis for performance-weighted aggregation, can improve the accuracy of probability judgments. However, many aspects of coherence-weighted aggregation remain a mystery, including both prescriptive issues (e.g. how best to use coherence measures) and theoretical issues (e.g. why coherence-weighted aggregation is effective). Using data from previous experiments employing either general-knowledge or statistical information-integration tasks, we addressed many of these issues. Of prescriptive relevance, we examined the effectiveness of coherence-weighted aggregation as a function of judgment elicitation method, group size, weighting function, and the aggressivity of the function’s tuning parameter. Of descriptive relevance, we propose that coherence-weighted aggregation can improve accuracy via two distinct, task-dependent routes: a deterministic route in which the bases for scoring accuracy depend on conformity to coherence principles (e.g., Bayesian information integration) and a diagnostic route in which coherence serves as a cue to correct knowledge. The findings provide support for the efficacy of both routes, but they also highlight why coherence weighting, especially the most aggressive forms, sometimes impose costs to accuracy. We conclude by sketching a decision-theoretic approach to how the wisdom of the coherent within the wisdom of the crowd can be sensibly leveraged.