Coherence of probability judgments from uncertain evidence: Does ACH help?

Although the Analysis of Competing Hypotheses method (ACH) is a structured analytic technique promoted in several intelligence communities for improving the quality of probabilistic hypothesis testing, it has received little empirical testing. Whereas previous evaluations have used numerical evidence assumed to be perfectly credible, in the present experiment we tested the effectiveness of ACH using a judgment task that presented participants with uncertain evidence varying in source reliability and information credibility. Participants (N = 227) assigned probabilities to two alternative hypotheses across six cases that systematically varied case features. Across multiple tests of coherence, the ACH group showed no advantage over a no-technique control group. Both groups showed evidence of subadditivity, unreliability (which was significantly worse in the ACH group), and overly conservative non-Bayesian judgments. The ACH group also showed pseudo-diagnostic weighting of evidence. The findings do not support the claim that ACH is effective at improving probabilistic judgment.

The Oxford Handbook of Probability and Philosophy

Probability theory is a key tool of the physical, mathematical, and social sciences. It has also been playing an increasingly significant role in philosophy: in epistemology, philosophy of science, ethics, social philosophy, philosophy of religion, and elsewhere. This Handbook encapsulates and furthers the influence of philosophy on probability, and of probability on philosophy. Nearly forty articles summarize the state of play and present new insights in various areas of research at the intersection of these two fields. The volume begins with a primer on those parts of probability theory that we believe are most important for philosophers to know, and the rest is divided into seven main sections: history; formalism; alternatives to standard probability theory; interpretations and interpretive issues; probabilistic judgment and its applications; applications of probability: science; and applications of probability: philosophy.

Introduction

This is an exciting time for the philosophy of probability, and probability theory’s value to philosophy has never been as appreciated as it is nowadays. The introduction to this Handbook sets out the context of the current debate in this area and provides a primer on those parts of probability theory that are most important for philosophers to know. It then goes on to introduce the seven main sections of the handbook: History; Formalism; Alternatives to Standard Probability Theory; Interpretations and Interpretive Issues; Probabilistic Judgment and Its Applications; Applications of Probability: Science; and Applications of Probability: Philosophy.

Probability

Rather than entailing that a particular outcome will occur, many scientific theories only entail that an outcome will occur with a certain probability. Because scientific evidence inevitably falls short of conclusive proof, when choosing between different theories it is standard to make reference to how probable the various options are in light of the evidence. A full understanding of probability in science needs to address both the role of probabilities in theories, or chances, as well as the role of probabilistic judgment in theory choice. In this chapter, the author introduces and distinguishes the two sorts of probability from one another and attempt to offer a satisfactory characterization of how the different uses for probability in science are to be understood. A closing section turns to the question of how views about the chance of some outcome should guide our confidence in that outcome.

Probability

Rather than entailing that a particular outcome will occur, many scientific theories only entail that an outcome will occur with a certain probability. Because scientific evidence inevitably falls short of conclusive proof, when choosing between different theories it is standard to make reference to how probable the various options are in light of the evidence. A full understanding of probability in science needs to address both the role of probabilities in theories, or chances, as well as the role of probabilistic judgment in theory choice. In this chapter, the author introduces and distinguishes the two sorts of probability from one another and attempt to offer a satisfactory characterization of how the different uses for probability in science are to be understood. A closing section turns to the question of how views about the chance of some outcome should guide our confidence in that outcome.

The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges

We have been oversold on the base rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint. At the empirical level, a thorough examination of the base rate literature (including the famous lawyer–engineer problem) does not support the conventional wisdom that people routinely ignore base rates. Quite the contrary, the literature shows that base rates are almost always used and that their degree of use depends on task structure and representation. Specifically, base rates play a relatively larger role in tasks where base rates are implicitly learned or can be represented in frequentist terms. Base rates are also used more when they are reliable and relatively more diagnostic than available individuating information. At the normative level, the base rate fallacy should be rejected because few tasks map unambiguously into the narrow framework that is held up as the standard of good decision making. Mechanical applications of Bayes's theorem to identify performance errors are inappropriate when (1) key assumptions of the model are either unchecked or grossly violated, and (2) no attempt is made to identify the decision maker's goals, values, and task assumptions. Methodologically, the current approach is criticized for its failure to consider how the ambiguous, unreliable, and unstable base rates of the real world are and should be used. Where decision makers' assumptions and goals vary, and where performance criteria are complex, the traditional Bayesian standard is insufficient. Even where predictive accuracy is the goal in commonly defined problems, there may be situations (e.g., informationally redundant environments) in which base rates can be ignored with impunity. A more ecologically valid research program is called for. This program should emphasize the development of prescriptive theory in rich, realistic decision environments.