Protecting against researcher bias in secondary data analysis: challenges and potential solutions

Analysis of secondary data sources (such as cohort studies, survey data, and administrative records) has the potential to provide answers to science and society’s most pressing questions. However, researcher biases can lead to questionable research practices in secondary data analysis, which can distort the evidence base. While pre-registration can help to protect against researcher biases, it presents challenges for secondary data analysis. In this article, we describe these challenges and propose novel solutions and alternative approaches. Proposed solutions include approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) help ensure that pre-registered analyses will be appropriate for the data, and (4) address difficulties arising from reduced analytic flexibility in pre-registration. For each solution, we provide guidance on implementation for researchers and data guardians. The adoption of these practices can help to protect against researcher bias in secondary data analysis, to improve the robustness of research based on existing data.

Announcing a New Registered Report Article Type at the Journal of Speech, Language, and Hearing Research

Purpose: This editorial introduces the new registered reports article type for the Journal of Speech, Language, and Hearing Research . The goal of registered reports is to create a structural solution to address issues of publication bias toward results that are unexpected and sensational, questionable research practices that are used to produce novel results, and a peer-review process that occurs at the end of the research process when changes in fundamental design are difficult or impossible to implement. Conclusion: Registered reports can be a positive addition to scientific publications by addressing issues of publication bias, questionable research practices, and the late influence of peer review. This article type does so by requiring reviewers and authors to agree in advance that the experimental design is solid, the questions are interesting, and the results will be publishable regardless of the outcome. This procedure ensures that replication studies and null results make it into the published literature and that authors are not incentivized to alter their analyses based on the results that they obtain. Registered reports represent an ongoing commitment to research integrity and finding structural solutions to structural problems inherent in a research and publishing landscape in which publications are such a high-stakes aspect of individual and institutional success.

Managing the terror of publication bias: A comprehensive p-curve analysis of the Terror Management Theory literature

A key prediction of Terror Management Theory is that people affirm their cultural worldview after they are reminded of death. This mortality salience (MS) hypothesis has been widely explored, yet the presence of questionable research practices may impact the replicability of this literature. We assess the evidential value of the MS hypothesis by conducting a pre-registered p-curve analysis of 860 published studies. Our results suggest that there are nonzero effects in this literature and that power is larger for studies conducted with multiple delays between the independent and dependent variables, for studies that test for main effects in comparison to those that test for interactions, and for studies conducted more recently. However, since the estimated average power of MS studies is 26%, direct replications are unlikely to succeed. We recommend researchers consider our evidence when planning their samples, and that they anticipate smaller effects by increasing their sample sizes.

What Do Participants Think of Our Research Practices? An Examination of Behavioral Psychology Participants’ Preferences

What research practices should be considered acceptable? Historically, scientists have set the standards for what constitutes acceptable research practices. However, there is value in considering non-scientists’ perspectives, including research participants’. 1,873 participants from MTurk and university subject pools were surveyed after their participation in one of eight minimal-risk studies. We asked participants how they would feel if common research practices were applied to their data: p-hacking/cherry-picking results, selective reporting of studies, Hypothesizing After Results are Known (HARKing), committing fraud, conducting direct replications, sharing data, sharing methods, and open access publishing. An overwhelming majority of psychology research participants think questionable research practices (e.g., p-hacking, HARKing) are unacceptable (68.3--81.3%), and were supportive of practices to increase transparency and replicability (71.4--80.1%). A surprising number of participants expressed positive or neutral views toward scientific fraud, raising concerns about the quality of our data. We grapple with this concern and interpret our results in light of the limitations of our study. Despite ambiguity in our results, we argue that there is evidence (from our study and others’) that researchers may be violating participants’ expectations and should be transparent with participants about how their data will be used.

Grant writing and grant peer review as questionable research practices

A large part of governmental research funding is currently distributed through the peer review of project proposals. In this paper, we argue that such funding systems incentivize and even force researchers to violate five moral values, each of which is central to commonly used scientific codes of conduct. Our argument complements existing epistemic arguments against peer-review project funding systems and, accordingly, strengthens the mounting calls for reform of these systems.

Questionable Research Practices and Misconduct Among Norwegian Researchers

This article presents results from the national survey conducted in 2018 for the project Research Integrity in Norway (RINO). A total of 31,206 questionnaires were sent out to Norwegian researchers by e-mail, and 7291 responses were obtained. In this paper, we analyse the survey data to determine attitudes towards and the prevalence of fabrication, falsification and plagiarism (FFP) and contrast this with attitudes towards and the prevalence of the more questionable research practices (QRPs) surveyed. Our results show a relatively low percentage of self-reported FFPs (0.2–0.3%), while the number of researchers who report having committed one of the QRPs during the last three years reached a troublesome 40%. The article also presents a ranking of the perceived severity of FFP and QRPs among Norwegian researchers. Overall, there is a widespread normative consensus, where FFP is considered more troublesome than QRPs.

Is Preregistration Worthwhile? A Survey on Personal Experiences

The preregistration of research protocols and analysis plans is a main reform innovation to counteract confirmation bias in the social and behavioral sciences. While theoretical reasons to preregister are frequently discussed in the literature, the individually experienced advantages and disadvantages of this method remain largely unexplored. The goal of this exploratory study was to identify the benefits and challenges of preregistration from the researcher's perspective. To this aim, we surveyed 355 researchers, 299 of whom had used preregistration in their own work. The researchers indicated the experienced or expected effects of preregistration on their workflow. The results show that experiences and expectations are mostly positive. Researchers in our sample believe that implementing preregistration improves or is likely to improve the quality of their projects, and that preregistration makes it easier to avoid questionable research practices. Criticism of preregistration is primarily related to the increase in work-related stress and the overall duration of the project. The majority of researchers with experience in preregistration reported that the benefits outweigh the challenges. However, the majority of researchers without preregistration would not consider preregistration for future projects or recommend the practice to colleagues. Our interpretation of the results is that preregistration can have positive side-effects as it adds an extra preparatory step in researchers' workflow, thus requiring researchers to think through the theoretical and practical aspects of their project.

Research on illicit cultural artefacts

This paper examines whether we can have confidence in the scientific integrity of a research effort that could potentially be part of the illicit trade in cultural artefacts. As an example, I use the research on the ancient clay tablets from the Schøyen Collection. A closer study of the research product reveals questionable research practices, and the latter issue is then put into a wider context. After highlighting the importance of the research community as a social institution in shaping the norms and values of its members, and its influence on what is desirable research, I explore how these expectations and guidelines impact research conducted on illicit cultural artefacts.

Open Science, Replicability, and Transparency in Modelling

Recent years have seen large changes to research practices within psychology and a variety of other empirical fields in response to the discovery (or rediscovery) of the pervasiveness and potential impact of questionable research practices, coupled with well-publicised failures to replicate published findings. In response to this, and as part of a broader open science movement, a variety of changes to research practice have started to be implemented, such as publicly sharing data, analysis code, and study materials, as well as the preregistration of research questions, study designs, and analysis plans. This chapter outlines the relevance and applicability of these issues to computational modelling, highlighting the importance of good research practices for modelling endeavours, as well as the potential of provenance modelling standards, such as PROV, to help discover and minimise the extent to which modelling is impacted by unreliable research findings from other disciplines.

Psychology and Law, Meet Open Science

As the field of psychology and law begins to embrace more transparent and accessible science, many questions arise about what open science actually is and how to do it. In this chapter, we contextualize this reform by examining fundamental concerns about psychological research—irreproducibility and replication failures, false-positive errors, and questionable research practices—that threaten its validity and credibility. Next, we turn to psychology’s response by reviewing the concept of open science and explaining how to implement specific practices—preregistration, registered reports, open materials/data/code, and open access publishing—designed to make research more transparent and accessible. We conclude by weighing the implications of open science for the field of psychology and law, specifically with respect to how we conduct and evaluate research, as well as how we train the next generation of psychological scientists and share scientific findings in applied settings.