‘Trust Us’: Open Data and Preregistration in Political Science and International Relations

The scientific method is predicated on transparency -- yet the pace at which transparent research practices are being adopted by the scientific community is slow. The replication crisis in psychology showed that published findings employing statistical inference are threatened by undetected errors, data manipulation, and data falsification. To mitigate these problems and bolster research credibility, open data and preregistration have increasingly been adopted in the natural and social sciences. While many political science and international relations journals have committed to implementing these reforms, the extent of open science practices is unknown. We bring large-scale text analysis and machine learning classifiers to bear on the question. Using population-level data -- 93,931 articles across the top 160 political science and IR journals between 2010 and 2021 -- we find that approximately 21% of all statistical inference papers have open data, and 5% of all experiments are preregistered. Despite this shortfall, the example of leading journals in the field shows that change is feasible and can be effected quickly.

Legacy of Jon Tennant, “Open science is just good science”

This work is a transcribed and edited collection of the message delivered by Jon Tennant in his talk “Open science is just good science”, May 21st, 2018. In readable form with many links, this paper provides a primer on open science and the open science movement. It details the problems with closed access science as it is still practiced today, and how big publishing as an industry is largely responsible. It talks about the ethics behind open science practices. It provides many statistics and links to information about paywalls, movements such as Project DEAL, workflows, and personal and community issues such as fear and cultural inertia that may prevent us from adopting better science practices.

What do incoming university students believe about open science practices in psychology?

Background: Understanding students’ naive conceptions about how science works and the norms that guide scientific best practice is important so that teachers can adapt their teaching to students’ existing understandings. Objective: To describe what incoming undergraduate students of psychology believe about reproducibility and open science practices in psychology. Method: International online survey with participants who were about to start their first course in psychology at a university (N = 239). Results: When asked about how research should be done, most students endorsed most (but not all) of ten open science practices. When asked to estimate the proportion of published psychological studies that apply each of a set of 10 open science practices, participants’ estimates tended to average near 50%. Only 18% of participants had heard of the term “replication crisis.” Conclusion: Despite relatively significant media attention on the replication crisis, few incoming psychology students are familiar with the term. Incoming students nevertheless appear to be sympathetic toward most open science practices, although they may overestimate the prevalence of these practices in psychology. Teaching Implications: Teaching materials aimed at incoming psychology students should not assume pre-existing knowledge about open science or replicability.

Open science practices and risks arising from misuse of biological research

The proliferation of open science may inadvertently increase the chance of deliberate or accidental misuse of research. Here, we examine the interaction between open science practices and biosecurity and biosafety to identify risks and opportunities for risk mitigation. We argue that open data, code, and materials may increase risks from research with misuse potential, despite their general importance. For instance, increased access to protocols, datasets, and computational methods for viral engineering may increase the risk of release of enhanced pathogens. For this dangerous subset of research, both open science and biosecurity goals may be achieved by using access-controlled repositories or application programming interfaces. The increased use of preprints could challenge any strategy for risk mitigation that relies on assessment at the publication stage, emphasising the need for earlier oversight in the research lifecycle. Preregistration of research, a practice promoted by the open science community, provides an opportunity for achieving biosecurity risk assessment at the conception of research. Open science and biosecurity experts have an important role to play in enabling responsible research with maximal societal benefit.

Assessing Open Science practices in physical activity behaviour change intervention evaluations

Objectives: Concerns on the lack of reproducibility and transparency in science have led to a range of research practice reforms, broadly referred to as Open Science. The extent that physical activity interventions are embedding Open Science practices is currently unknown. In this study, we randomly sampled 100 reports of recent physical activity behaviour change interventions to estimate the prevalence of Open Science practices. Methods: One hundred reports of randomised controlled trial physical activity behaviour change interventions published between 2018-2021 were identified. Open Science practices were coded in identified reports, including: study pre-registration, protocol sharing, data-, materials- and analysis scripts-sharing, replication of a previous study, open access publication, funding sources and conflict of interest statements. Coding was performed by two independent researchers, with inter-rater reliability calculated using Krippendorffs alpha. Results: 78% of the 100 reports provided details of study pre-registration and 41% provided evidence of a published protocol. 4% provided accessible open data, 8% provided open materials and 1% provided open analysis scripts. 73% of reports were published as open access and no studies were described as replication attempts. 93% of reports declared their sources of funding and 88% provided conflicts of interest statements. A Krippendorffs alpha of 0.73 was obtained across all coding. Conclusion: Open data, materials, analysis and replication attempts are currently rare in physical activity behaviour change intervention reports, whereas funding source and conflict of interest declarations are common. Future physical activity research should increase the reproducibility of their methods and results by incorporating more Open Science practices.

Transparency, Documentation, and Open Science

Some GSA journals are especially interested in promoting transparency and open science practices, reflecting how some subdisciplines in aging are moving toward open science practices faster than others. In this talk, I will consider the transparency and open science practices that seem most relevant to aging researchers, such as preregistration, open data, open materials and code, sample size justification and analytic tools for considering null effects. I will also discuss potential challenges to implementing these practices as well as reasons why it is important to do so despite these challenges. The focus will be on pragmatic suggestions for researchers planning and conducting studies now that they hope to publish later.

Survey on Open Science Practices in Functional Neuroimaging

Replicability and reproducibility of scientific findings is paramount for sustainable progress in neuroscience. Preregistration of the hypotheses and methods of an empirical study before analysis, the sharing of primary research data, and compliance with data standards such as the Brain Imaging Data Structure (BIDS), are considered effective practices to secure progress and to substantiate quality of research. We investigated the current level of adoption of open science practices in neuroimaging and the difficulties that prevent researchers from using them. Email invitations to participate in the survey were sent to addresses received through a PubMed search of human functional magnetic resonance imaging studies between 2010 and 2020. 283 persons completed the questionnaire. Although half of the participants were experienced with preregistration, the willingness to preregister studies in the future was modest. The majority of participants had experience with the sharing of primary neuroimaging data. Most of the participants were interested in implementing a standardized data structure such as BIDS in their labs. Based on demographic variables, we compared participants on seven subscales, which had been generated through factor analysis. It was found that experienced researchers at lower career level had higher fear of being transparent, researchers with residence in the EU had a higher need for data governance, and researchers at medical faculties as compared to other university faculties reported a higher need for data governance and a more unsupportive environment. The results suggest growing adoption of open science practices but also highlight a number of important impediments.