Analysis and Decision

This chapter explains how decision makers can incorporate assessments of uncertainty into high-stakes foreign policy choices. It begins by describing a simple analytic tool called break-even analysis, with which leaders can use explicit probability assessments as a point of leverage for determining whether or not a risky decision is worthwhile. The chapter then explains how transparent probabilistic reasoning is especially important for assessing strategic progress. In some cases, it can actually be impossible to make rigorous judgments about the extent to which foreign policies are making acceptable progress without assessing subjective probabilities in detail. This argument departs from a large body of existing scholarship on learning in international politics that assumes leaders can use a straightforward logic of trial and error to determine how they should update their strategic perceptions over time. The chapter provides examples of these dynamics drawn from the U.S. occupation of Iraq.

The Value of Precision in Probability Assessment

This chapter analyzes a database containing nearly one million geopolitical forecasts. These data show that foreign policy analysts are surprisingly effective at estimating subjective probabilities. Fine-grained distinctions in probability estimates convey meaningful information about world politics, not arbitrary detail. By extension, the chapter shows that common qualitative expressions of uncertainty (including expressions currently recommended for use in intelligence analysis and military planning) systematically degrade the value of foreign policy discourse. The ability of foreign policy analysts to achieve “returns to precision” in probability assessment does not appear to depend on easy questions, short time horizons, or special cognitive attributes. Instead, the value of precision in probability assessment appears to be a generalizable skill that foreign policy analysts can cultivate through training, effort, and experience.

Introduction

This chapter describes widespread skepticism regarding the value of assessing uncertainty in international politics. “Agnostics” argue that assessments of uncertainty in international politics are too unreliable to be useful for shaping major foreign policy decisions. “Rejectionists” argue that attempting to assess uncertainty in international politics can be counterproductive, surrounding foreign policy analyses with illusions of rigor or exposing foreign policy analysts to excessive criticism. “Cynics” claim that foreign policy analysts and decision makers have self-interested motives to avoid assessing uncertainty. The chapter explains how these ideas lead many scholars, practitioners, and pundits to avoid holding careful debates about the risks surrounding major foreign policy choices. The chapter describes how this aversion to probabilistic reasoning appears in several high-profile cases, such as President Kennedy’s decision to authorize the Bay of Pigs invasion and President Obama’s decision to raid Osama bin Laden’s compound in Abbottabad, Pakistan.

Subjective Probability and International Politics

This chapter explores the theoretical foundations of assessing uncertainty in international politics. It begins by explaining that virtually all important assessments of uncertainty in international politics are inherently subjective. The chapter’s second section explains how it is possible to believe that these subjective judgments are meaningless, but that this argument carries logical implications that no foreign policy analyst could accept. The chapter’s third section demonstrates that, conditional on believing that assessments of subjective probability contain any meaningful insight, it is always possible to express that insight in clear and structured ways, including through the use of numeric percentages. The chapter shows how its theoretical framework can help to resolve difficult analytic problems, focusing in particular on debates among U.S. intelligence analysts about the chances that Osama bin Laden was living in Abbottabad, Pakistan, in the spring of 2011.

Placing Probability Front and Center in Foreign Policy Discourse

The book’s concluding chapter explores practical opportunities for improving assessments of uncertainty in foreign policy debates. It focuses on the value of creating norms that place probabilistic reasoning front and center in foreign policy debates. There are three main ways to pursue this goal: foreign policy organizations can improve formal analytic standards for assessing uncertainty; decision makers can actively encourage foreign policy analysts to assess uncertainty in clear and structured ways; and critics can press their opponents to describe the uncertainty that surrounds controversial policy proposals. This last dynamic is especially important, because it shows that the goal of improving assessments of uncertainty in international politics is not just an issue for government officials; it is also a matter of how scholars, journalists, and pundits can raise the standards of public discourse.

Pathologies of Probability Assessment

This chapter describes how foreign policy analysts often avoid assessing uncertainty in a manner that supports sound decision making. This aversion to probabilistic reasoning can take three forms: imprecise judgments that do not establish clear meaning; judgments of relative probability, which frame assessments of uncertainty against unspecified baselines; and conditioning, a practice that identifies the assumptions that must hold for a statement to be true but says nothing about the probability that a conclusion is actually correct. The chapter shows how official doctrine for intelligence analysis and military planning encourages foreign policy analysts to assess uncertainty in these problematic ways. It then shows how these problematic methods of assessing uncertainty shaped the highest levels of U.S. decision making during the Vietnam War, focusing on how senior leaders crafted military strategies without carefully assessing the chances that they would succeed.

The Politics of Uncertainty and Blame

This chapter explores the politics of assessing uncertainty in international affairs, particularly the notion that clear probability assessments expose foreign policy analysts to excessive criticism. Although this idea is widespread among scholars and practitioners of international relations, there is just as much reason to believe that the opposite is true. If foreign policy discourse is truly as polarized as the conventional wisdom suggests, then leaving key judgments vague could actually increase blame exposure by giving critics the opportunity to make ambiguous statements seem more mistaken than they really are. The chapter supports this claim by combing experimental evidence in a historical review of perceived intelligence failures such as the Bay of Pigs invasion, the Yom Kippur War, and assessments of Iraq’s weapons of mass destruction programs.

Dispelling Illusions of Rigor

This chapter explores the psychology of assessing uncertainty in international politics. It presents a series of survey experiments that test important concerns about the extent to which transparent probabilistic reasoning could potentially warp the quality of foreign policy analysis and decision making. The most important of these concerns is the idea that clear assessments of uncertainty create illusions of rigor that make leaders insensitive to risk. Contrary to this assertion, the chapter’s experimental evidence suggests that decision makers’ choices are sensitive to subtle variations in probabilistic reasoning, and that making this reasoning more transparent encourages decision makers to be more cautious when placing lives and resources at risk. These experiments involve hundreds of real national security professionals and thousands of non-elite respondents.