The Efficacy of ACH in Mitigating Serial Position Effects and Confirmation Bias in an Intelligence Analysis Scenario1

Chapter three provides details of an experimental study conducted in 2016 to provide an evaluation of the efficacy of ACH in mitigating the cognitive biases of serial position effects and confirmation bias using the scoring systems of credibility of information and diagnostic value of information. The study is based on a disguised version of the intelligence case for both the biological and nuclear weapons capabilities of Saddam Hussein’s regime that was used to support the US decision to invade Iraq in 2003. The study shows that the version of ACH taught by the PHIA to the UK’s intelligence community between 2016-2017 has no statistically significant mitigating effect on the occurrence of serial position effects or confirmation bias.

Introduction

and Keywords to be supplied.

Intelligence and Belief Acquisition

Chapter one examines whether belief acquisition in intelligence a unique epistemological act. It examines the nature of intelligence analysis as an act of belief acquisition, providing a theoretical context grounded in epistemology, the philosophical study of the nature of belief and knowledge, and examining the sources of information on which intelligence analysis is based, and the mechanisms by which knowledge is gained. The chapter argues that: 1) the essential nature of intelligence is epistemological: that it is necessarily defined by the attempt to acquire justified beliefs and knowledge; 2) intelligence is necessarily a covert activity; 3) intelligence does not require unique methods of gaining knowledge and does not derive from unique sources, and; 4) the covert characteristic of intelligence means that belief acquisition in intelligence is likely to differ in the degree of epistemic complexity it faces, and that this may produce a difference in the degree to which intelligence analysis is vulnerable to cognitive bias.

Predicting Serial Position Effects1

Chapter four provides details of a meta-analysis conducted into serial position effects. The meta-analysis also identified whether there are any significant differences in the proportion of bias under different analytical conditions between belief adjustment conducted in an intelligence scenario and belief adjustment conducted in non-intelligence scenarios. The chapter argues that there is no compelling evidential basis to suggest that serial position effects or confirmation bias affect intelligence analysis differently from non-intelligence analysis. It will show that the analytical conditions of volume of information, reliance on recall, accountability and type of information likely have an impact on serial position effects. The results undermine key assumptions in predominant predictions models. This includes the belief-adjustment model for serial position effects (Hogarth and Einhorn 1992).

The Efficacy of ACH in Establishing Epistemic Justification and Mitigating Cognitive Bias

The chapter examines the ability of current versions of the structured analytical technique, assessment of competing hypotheses (ACH) to provide theoretically valid methodologies for establishing justified beliefs in intelligence analysis, to cope with the higher risk of epistemic complexity in intelligence analysis, and to mitigate or reduce cognitive bias. It argues that no current version of ACH provides a theoretically valid mechanism to establish justification for beliefs, or to cope with epistemic complexity, but that the method can be adapted to do so. It also argues that no current version of ACH provides a theoretically valid mechanism to mitigate the risk of cognitive bias, but that the method could be adapted to make the occurrence of some cognitive biases visible to peer review.

Predicting Confirmation Bias

Chapter five provides details of the meta-analyses into confirmation bias. It will show that the analytical conditions of diagnostic weighting of initial information, consistency of information, hypothesis testing instructions and type of information likely have an impact on confirmation bias. It will also show that the results undermine key assumptions in predominant predictions models the inability to identify diagnostic value of information theory concerning confirmation bias (Koslowski and Maqueda 1993, and Kuhn et al. 1988). The chapter proposes alternative models for predicting serial position effects and confirmation bias. These models argue that whilst the risk of occurrence of serial position effects and confirmation bias are impacted by different analytical conditions, they share an underlying cognitive process: a force towards forming a focal hypothesis early on in belief acquisition.

Reducing the Risk of Cognitive Bias in Intelligence Analysis

Recommendations for future research and best practice for intelligence analysis.