How to Improve Your Own Scientific Thinking

Chapter 14 suggests concrete ways to improve your scientific thinking about your own hypotheses and how to identify them in the scientific publications of others. The chapter continues exploration of the effects of unconscious mental processes on scientific thinking by emphasizing methods for minimizing such effects. Practical exercises include finding and diagramming hypotheses, illustrated by a detailed example from the literature. Building on the notions that scientists’ own intellectual productions entail significant ego investment and are subject to the concerns from behavioral economics that were raised in Chapter 11, this chapter suggests ways for scientists to step back and view their own scientific thinking skills objectively. The goal is to help promote sound thinking by calling readers’ attention to subtle intrinsic forces that can undermine it. Strategies for improvement include avoiding the “curse of knowledge,” taking the “outside” view, and ignoring the “sunk cost fallacy” when it comes to their own ideas.

Advantages of the Hypothesis

This chapter makes the case for the scientific hypothesis from two quite different points of view: statistical and cognitive. The consideration of statistical advantages picks up from the discussion of the Reproducibility Crisis in the previous chapter. First, it explores reasoning that shows that hypothesis-based research will, as a general rule, be much more reliable than, for example, open-ended gene searches. It also revives a procedure, Fisher’s Method for Combining Results that, though rarely used nowadays, underscores the strengths of multiple testing of hypotheses. Second, the chapter goes into many cognitive advantages of hypothesis-based research that exist because the human mind is inherently and continually at work trying to understand the world. The hypothesis is a natural way of channeling this drive into science. It is also a powerful organizational tool that serves as a blueprint for investigations and helps organize scientific thinking and communications.

The Scientific Hypothesis Today

Chapter 2 begins by reviewing the concept of the Scientific Method, as well as many outdated definitions of “hypothesis.” The discussion leads to the modern definition of the hypothesis as a conjectural explanation for a phenomenon; it is testable and falsifiable. The hypothesis serves as a blueprint and a summary of an investigation. Certain criticisms of the hypothesis and hypothesis-driven research are based on the older definitions of the term, and the book returns to them later. This chapter identifies and defines, with simple, nontechnical examples, concepts associated with the hypothesis, such as prediction and direct and indirect measurements. The philosophical programs of Karl Popper and John Platt, Critical Rationalism and Strong Inference, respectively, form a major focus of the chapter. The chapter explores the complexities of the concepts of falsification and corroboration and the importance of having multiple hypotheses. The chapter introduces the idea of the implicit hypothesis and ends with the presentation and discussion of key features of a good hypothesis.

What Scientists Think About Scientific Thinking

This chapter reports original data from two surveys conducted to find out how scientists view the hypothesis and related concepts. One was an online survey of hundreds of members of biological research societies. The scientists reported on their knowledge of and training in the use of the hypothesis, their views about hypothesis testing as compared with other modes of conducting science, such as Discovery Science, open-ended questioning, and Big Data methods. Respondents estimated how the various scientific modes influenced their work and how much they relied on each one. Most respondents,70% of them, reported having received little or no training in scientific thinking; however, 90% felt confident about their thinking skills. Nevertheless, more than 90% felt that formal training in this area would be helpful. The great majority relied on hypotheses in their research work. The second survey, analyzed all (more than 150) neuroscience research papers that appeared sequentially in top journals during 2015 to determine how the papers were structured, particularly with respect to the hypothesis. Only 33% of the papers had an explicitly stated hypothesis, whereas, in 45%, the hypothesis was “implicit.” A minor, though notable, fraction of the papers misused the term “hypothesis.” The results are germane to several topics covered in the remainder of the book.

Philosophical Background Matters

This chapter discusses the rift between science and philosophy, but argues that scientists can benefit from philosophical insights without becoming philosophers. It presents an elementary introduction to philosophical concepts that recur throughout the book, including deduction, induction, inference, and others. It covers the problem of induction and the Uniformity of Nature assumption, and reviews Hume’s critique of induction. Other technical issues that confuse the public debate about science, concern explanation, uncertainty, and levels of organization of science, are in here as well. A central issue is the question of how we can resolve two opposing notions: the widely agreed on principle that scientific findings are never completely certain, and our conviction that some findings are certain: e.g., the earth goes around the sun in an elliptical orbit. The chapter sorts out this and other common misunderstandings.

The Reproducibility Crisis

This chapter reviews and evaluates reports that scientists often cannot repeat, or “reproduce” published work. It begins by defining what “reproducibility” means and how reproducibility applies to various kinds of science. The focus then shifts to the Reproducibility Project: Psychology, which was a systematic effort to repeat published findings in psychology, and which gave rise to many of the present concerns about reproducibility. The chapter critically examines the Reproducibility Project and points out how the nature of science and the complexity of nature can stymie the best attempts at reproducibility. The chapter also reviews the statistical criticisms of science that John Ioannidis and Katherine Button and their colleagues have raised. The hypothesis is a central issue because it is inconsistently defined across various branches of science. The statisticians’ strongest attacks are directed against work that differs from most laboratory experimental science. A weak point in the reasoning behind the Reproducibility Project and the statistical arguments is the assumption that a multi-pronged scientific investigation can be legitimately criticized by close examination of one of its components. Experimental science relies on multiple tests and multiple hypotheses to arrive at its conclusions. Reproducibility is a valid concern for science; it is not a “crisis.”

Kinds of Science

This chapter reviews distinctions between kinds of science, which is especially relevant to the book’s topic because it is an area that Karl Popper did not consider in detail. This creates a problem since critics of the hypothesis often do not distinguish between true hypothesis-based science and other kinds that don’t depend on hypotheses, and the traditional divisions of science miss the main points. The chapter distinguishes among several modern kinds of science including Big Science/Small Science and how they relate to Big Data and Little Data, and why Discovery Science is different from hypothesis-testing science. It separates “exploratory” from “confirmatory” studies and explains why this terminology can create confusion in trying to understand science. The differences between applied and basic science are genuine and meaningful because these two kinds of science have different goals and apply different, though overlapping, standards to achieve their goals.

Bayesian Basics and the Scientific Hypothesis

This chapter covers the basics of Bayesian statistics, emphasizing the conceptual framework for Bayes’ Theorem. It works through several iterations of the theorem to demonstrate how the same equation is applied in different circumstances, from constructing and updating models to parameter evaluation, to try to establish an intuitive feel for it. The chapter also covers the philosophical underpinnings of Bayesianism and compares them with the frequentist perspective described in Chapter 5. It addresses the question of whether Bayesians are inductivists. Finally, the chapter shows how the Bayesian procedures of model selection and comparison can be pressed into service to allow Bayesian methods to be used in hypothesis testing in essentially the same way that various p-tests are used in the frequentist hypothesis testing framework.

Statistics and Statistical Hypotheses

This chapter covers statistics from a conceptual rather than a quantitative perspective. The objective is to distinguish statistical from scientific hypotheses and provide background for later discussions. The fact that scientific and statistical hypotheses are dissimilar is often overlooked, and the chapter shows how they differ and why the differences matter. The chapter uses simple, nontechnical examples to present the main ideas in frequentist statistics; the next chapter covers the basics of Bayesian statistics and explores how the divergent philosophical viewpoints of frequentists and Bayesians affect scientific reasoning. The present chapter covers statistical error and its role in null hypothesis significance testing, which is still the dominant mode of scientific significance testing, as well as the origins, weaknesses of, and alternatives to, null-hypothesis testing. A philosophical objective of the chapter is to show how the concept of probability fits with Karl Popper’s program of falsification. The background material in this chapter will be especially important for evaluating the Reproducibility Crisis in Chapter 7.

Automatic Thinking

This chapter introduces the role of “automatic thinking,” in hypothesis formation. Automatic thinking refers to the default operations of our mind that affect how we view and understand the world. These operations are behind cognitive biases and heuristic reasoning, inductive reasoning, and, most importantly, our tendency to engage in continual, conscious and unconscious, hypothesis generation. Unconscious hypotheses allow us to predict what will happen next and see why things “make sense.” They also account for our susceptibility to the “cognitive illusions,” that lead us astray. The chapter also questions whether “inductive reasoning” should count as a form of “reasoning” at all, since the ability to recognize and respond to regularities in the environment appears to be an adaptive trait shared by all animals and, perhaps, plants as well. This chapter argues that much of the criticism of the hypothesis has failed to take these automatic mental processes into account. The chapter suggests that a better sense of our automatic mental activity can lead to improvements in scientific thinking.