The file drawer effect – a long-lasting issue in science

Most of the scientific outputs produced by researchers are inaccessible since they are not published in scientific journals: they remain in the researchers' drawers, forming what we call the Dark Science. This is a long-standing issue in research, creating a misleading view of the scientific facts. Contrary to the current literature overfed with positive findings, the Dark Science is nurtured with null findings, replications, flawed experimental designs and other research outputs. Publishers, researchers, institutions and funders all play an important role in the accumulation of those unpublished works, but it is only once we understand the reasons and the benefits of publishing all the scientific findings that we can collectively act to solve the Dark Science problem. In this article, we discuss the causes and consequences of the Dark Science expansion, arguing that science and scientists would benefit from getting all their findings to the light of publication.

Questionable Research Practices in Single-Case Experimental Designs: Examples and Possible Solutions

Questionable research practices (QRPs) are a variety of research choices that introduce bias into the body of scientific literature. Researchers have documented widespread presence of QRPs across disciplines and promoted practices aimed at preventing them. More recently, Single-Case Experimental Design (SCED) researchers have explored how QRPs could manifest in SCED research. In the chapter, we describe QRPs in participant selection, independent variable selection, procedural fidelity documentation, graphical depictions of behavior, and effect size measures and statistics. We also discuss QRPs in relation to the file drawer effect, publication bias, and meta-analyses of SCED research. We provide recommendations for researchers and the research community to promote practices for preventing QRPs in SCED.

Norm vigilance in reaction to control threat: A file drawer report

In times of threat people often turn to social groups to fulfill various needs. In situations where this threat is related to people’s personal sense of control, the model of group-based control provides a social-identity-based account of why thinking and acting in terms of group membership should become more likely to occur. We set out to extend this perspective to the perception and processing of information about norms. Our initial hypothesis, the norm vigilance hypothesis, was that threat to people’s personal sense of control should induce a state where people become more vigilant for information about relevant social norms. This should become evident in more accurate recall of specifically this type of information under conditions of threat. In a series of four studies, we investigated this hypothesis with different paradigms, but were unable to find convincing evidence for the notion of norm vigilance in terms of enhanced accuracy. In a fifth study, we investigated an alternative hypothesis of motivated intergroup distortion of information about social norms after threat, but did not find convincing evidence for this mechanism either.

Using the Comparative Method in Democratic Theory: A Solution to the “File Drawer Problem”?

There are currently dozens of conceptions in democratic theory of what constitutes democratic progress and even more concepts of the crisis of democracy. This plethora of ideas is both good news and bad news. The boom in theorizing means that specializations have allowed political scientists to fine-tune their in-depth analysis in order to capture smaller and more specific movements of democratic progress and regress. But the multiplicity of models of the crisis and transformation of democracy also spells some bad news, which is far less often acknowledged and far less understood. This article seeks to shed light on the dangers of not comparing and integrating various models of democracy and to extrapolate the benefits of using the comparative method to do so.

A Short Null Report on the Relationship Between Online Learning and Various Outcomes

Using COVID Pulse Data collected by the U.S. Census Bureau I establish that there are weak to nocorrelational relationships between a household reporting a child attending virtual or in-person school andvarious outcomes including expectations of loss of employment, child hunger, anxiety. Due to the coarsenessof the data, it is unclear if this is an artifact of the data or a reflection of the lack of underlying causalrelationships between mode of schooling and the outcomes. Therefore, these results should not be used tomake policy decisions or draw substantive conclusions about the decision to reopen schools and are reportedonly to avoid the file-drawer effect.

Crowdsourced meta-analysis in psychological science: An open science approach to compiling and synthesizing scientific knowledge

This article introduces the concept of an open science meta-analysis that functions through crowdsourced imputation of data and is thereby perpetually updating. This is proposed to supplement the current journal article-based system of knowledge storage and synthesis and will, a) increase the consumptive capabilities of researchers (i.e., the amount of research one is exposed to), b) minimize cognitive biases that influence scientific knowledge, c) reduce the file-drawer problem, and d) create new knowledge through mass synthesis of existing research. The proposed infrastructure, much like the recent norm of publicly available data, may be viewed as an industry standard in the near future.

Coordinated Data Analysis: Knowledge Accumulation in Lifespan Developmental Psychology

Coordinated analysis is a powerful form of integrative analysis, and is well suited in its capacity to promote cumulative scientific knowledge, particularly in subfields of psychology that focus on the processes of lifespan development and aging. Coordinated analysis uses raw data from individual studies to create similar hypothesis tests for a given research question across multiple datasets, thereby making it less vulnerable to common criticisms of meta-analysis such as file drawer effects or publication bias. Coordinated analysis can sometimes use random effects meta-analysis to summarize results, which does not assume a single true effect size for a given statistical test. By fitting parallel models in separate datasets, coordinated analysis preserves the heterogeneity among studies, and provides a window into the generalizability and external validity of a set of results. The current paper achieves three goals: First, it describes the phases of a coordinated analysis so that interested researchers can more easily adopt these methods in their labs. Second, it discusses the importance of coordinated analysis within the context of the credibility revolution in psychology. Third, it encourages the use of existing data networks and repositories for conducting coordinated analysis, in order to enhance accessibility and inclusivity. Subfields of research that require time- or resource- intensive data collection, such as longitudinal aging research, would benefit by adopting these methods.

Seek and ye shall be fine: Attitudes toward political perspective-seekers

Six pre-registered studies (N = 2421) examine how people respond to co-partisan political perspective-seekers: political allies who attempt to hear from shared opponents and better understand their views. We find North American adults and students generally like co-partisan seekers (meta-analytic Cohen’s d = .83 across 4231 participants, including an emptied file drawer). People like co-partisan seekers because they seem tolerant, cooperative, and rational, but this liking is diminished because seekers seem to validate—and may even adopt—opponents’ illegitimate views. Participants liked co-partisan seekers across a range of different motivations guiding these seekers’ actions but, consistent with our theorizing, their liking decreased (though rarely disappeared entirely) when seekers lacked partisan commitments, or when they sought especially illegitimate beliefs. Despite evidence of rising political intolerance in recent decades, these findings suggest people nonetheless celebrate political allies who tolerate and seriously consider their opponents’ views.

Sorting the File Drawer: A Typology for Describing Unpublished Studies

A typology of unpublished studies is presented to describe various types of unpublished studies and the reasons for their nonpublication. Reasons for nonpublication are classified by whether they stem from an awareness of the study results (result-dependent reasons) or not (result-independent reasons) and whether the reasons affect the publication decisions of individual researchers or reviewers/editors. I argue that result-independent reasons for nonpublication are less likely to introduce motivated reasoning into the publication decision process than are result-dependent reasons. I also argue that some reasons for nonpublication would produce beneficial as opposed to problematic publication bias. The typology of unpublished studies provides a descriptive scheme that can facilitate understanding of the population of study results across the field of psychology, within subdisciplines of psychology, or within specific psychology research domains. The typology also offers insight into different publication biases and research-dissemination practices and can guide individual researchers in organizing their own file drawers of unpublished studies.