scholarly journals Researcher Degrees of Freedom and a Lack of Transparency Contribute to Unreliable Results of Nonparametric Statistical Analyses Across SPSS, SAS, Stata, and R

2020 ◽  
Author(s):  
Cooper Hodges ◽  
Hannah Michelle Lindsey ◽  
Paula Johnson ◽  
Bryant M Stone ◽  
James carter
Keyword(s):  
Degrees Of Freedom ◽  
Statistical Procedure ◽  
Statistical Analyses ◽  
Computational Error ◽  
Statistical Software ◽  
Replication Studies ◽  
The Social ◽  
Replication Crisis ◽  
Hand Calculation ◽  
Nonparametric Statistical

The replication crisis within the social and behavioral sciences has called into question the consistency of research methodology. A lack of attention to minor details in replication studies may limit researchers’ abilities to reproduce the results. One such overlooked detail is the statistical programs used to analyze the data. In the current investigation, we compared the results of several nonparametric analyses and measures of normality conducted on a large sample of data in SPSS, SAS, Stata, and R with results obtained through hand-calculation using the raw computational formulas. Multiple inconsistencies were found in the results produced between statistical packages due to algorithmic variation, computational error, and lack of clarity and/or specificity in the statistical output generated. We also highlight similar inconsistencies in supplementary analyses conducted on subsets of the data, which reflect realistic sample sizes. These inconsistencies were largely due to algorithmic variations used within packages when the analyses are performed on data from small- or medium-sized samples. We discuss how such inconsistencies may influence the conclusions drawn from the results of statistical analyses depending on the statistical software used, and we urge researchers to analyze their data across multiple packages, report details regarding the statistical procedure used for data analysis and consider these details when conducting direct replications studies.

scholarly journals Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme

2020 ◽  
Vol 7 (7) ◽  
pp. 200566
Author(s):  
Michael Gordon ◽  
Domenico Viganola ◽  
Michael Bishop ◽  
Yiling Chen ◽  
Anna Dreber ◽  
...  
Keyword(s):  
Large Scale ◽  
Academic Discipline ◽  
Empirical Studies ◽  
Survey Response ◽  
Replication Rate ◽  
Replication Studies ◽  
Research Fields ◽  
The Social ◽  
Replication Crisis ◽  
Independent Replication

The Defense Advanced Research Projects Agency (DARPA) programme ‘Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.

Statistical Significance and the Replication Crisis in the Social Sciences

2019 ◽  
Author(s):  
Anna Dreber ◽  
Magnus Johannesson
Keyword(s):  
Social Sciences ◽  
Statistical Power ◽  
Degrees Of Freedom ◽  
Statistical Significance ◽  
Academic Community ◽  
The Other ◽  
Main Indicator ◽  
The Social ◽  
Replication Crisis ◽  
High Statistical Power

The recent “replication crisis” in the social sciences has led to increased attention on what statistically significant results entail. There are many reasons for why false positive results may be published in the scientific literature, such as low statistical power and “researcher degrees of freedom” in the analysis (where researchers when testing a hypothesis more or less actively seek to get results with p < .05). The results from three large replication projects in psychology, experimental economics, and the social sciences are discussed, with most of the focus on the last project where the statistical power in the replications was substantially higher than in the other projects. The results suggest that there is a substantial share of published results in top journals that do not replicate. While several replication indicators have been proposed, the main indicator for whether a results replicates or not is whether the replication study using the same statistical test finds a statistically significant effect (p < .05 in a two-sided test). For the project with very high statistical power the various replication indicators agree to a larger extent than for the other replication projects, and this is most likely due to the higher statistical power. While the replications discussed mainly are experiments, there are no reasons to believe that the replicability would be higher in other parts of economics and finance, if anything the opposite due to more researcher degrees of freedom. There is also a discussion of solutions to the often-observed low replicability, including lowering the p value threshold to .005 for statistical significance and increasing the use of preanalysis plans and registered reports for new studies as well as replications, followed by a discussion of measures of peer beliefs. Recent attempts to understand to what extent the academic community is aware of the limited reproducibility and can predict replication outcomes using prediction markets and surveys suggest that peer beliefs may be viewed as an additional reproducibility indicator.

Ethnography, Data Transparency, and the Information Age

2021 ◽  
Vol 47 (1) ◽  
Author(s):  
Alexandra K. Murphy ◽  
Colin Jerolmack ◽  
DeAnna Smith
Keyword(s):  
Social Life ◽  
New Technologies ◽  
Information Age ◽  
Annual Review ◽  
Publication Date ◽  
Ethical Implications ◽  
The Social ◽  
Share Data ◽  
Replication Crisis ◽  
Data Transparency

The conventions ethnographers follow to gather, write about, and store their data are increasingly out of sync with contemporary research expectations and social life. Despite technological advancements that allow ethnographers to observe their subjects digitally and record interactions, few follow subjects online and many still reconstruct quotes from memory. Amid calls for data transparency, ethnographers continue to conceal subjects’ identities and keep fieldnotes private. But things are changing. We review debates, dilemmas, and innovations in ethnography that have arisen over the past two decades in response to new technologies and calls for transparency. We focus on emerging conversations around how ethnographers record, collect, anonymize, verify, and share data. Considering the replication crisis in the social sciences, we ask how ethnographers can enable others to reanalyze their findings. We address ethical implications and offer suggestions for how ethnographers can develop standards for transparency that are consistent with their commitment to their subjects and interpretive scholarship. Expected final online publication date for the Annual Review of Sociology, Volume 47 is July 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

scholarly journals Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

2019 ◽  
Author(s):  
Jessica Kay Flake ◽  
Eiko I Fried
Keyword(s):  
Construct Validity ◽  
Degrees Of Freedom ◽  
Scientific Research ◽  
Replication Studies ◽  
Psychological Science ◽  
Rigorous Research ◽  
Statistical Conclusion ◽  
Measurement Practices

In this paper, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

scholarly journals How to Talk to Your Friends and Colleagues About Replication

2020 ◽  
Author(s):  
Jonathan Plucker ◽  
Matthew C. Makel
Keyword(s):  
Social Sciences ◽  
Educational Psychology ◽  
Research Integrity ◽  
Research Quality ◽  
Psychology Research ◽  
Replication Studies ◽  
The Social

Replicability and the importance of enhanced research rigor are foundational issues across the social sciences, and educational psychology is no exception. Yet strategies for increasing research quality are not widespread in the field, including the use of replication studies. In this manuscript, we examine the nature and scope of replication problems in educational psychology research, and how these issues threaten research integrity and transparency. We also examine strategies to mitigate these problems in educational psychology. Finally, we discuss several on-going challenges that contribute to replication problems, and which need additional attention from researchers.

scholarly journals Researcher Degrees of Freedom in the Psychology of Religion

2018 ◽  
Author(s):  
Sarah Jane Charles ◽  
James Edward Bartlett ◽  
Kyle J. Messick ◽  
Thomas Joseph Coleman ◽  
Alex Uzdavines
Keyword(s):  
Systematic Review ◽  
Degrees Of Freedom ◽  
Psychology Of Religion ◽  
Open Science ◽  
Science Practices ◽  
Replication Crisis ◽  
The Way

There is a push in psychology toward more transparent practices, stemming partially as a response to the replication crisis. We argue that the psychology of religion should help lead the way toward these new, more transparent practices to ensure a robust and dynamic subfield. One of the major issues that proponents of Open Science practices hope to address is researcher degrees of freedom (RDF). We pre-registered and conducted a systematic review of the 2017 issues from three psychology of religion journals. We aimed to identify the extent to which the psychology of religion has embraced Open Science practices and the role of RDF within the subfield. We found that many of the methodologies that help to increase transparency, such as pre-registration, have yet to be adopted by those in the subfield. In light of these findings, we present recommendations for addressing the issue of transparency in the psychology of religion and outline how to move toward these new Open Science practices.

scholarly journals A STUDY ON THE DYNAMICS TO WIN THE NATIONAL CHAMPION TITLE IN ATHLETICS IN THE LAST CENTURY, IN A DISTINCT REGION IN ROMANIA

GYMNASIUM ◽  
2017 ◽  
Vol XVII (2) ◽  
Author(s):  
Dan-Iulian Alexe
Keyword(s):  
Social Context ◽  
Statistical Analyses ◽  
Distinct Region ◽  
Sports Performance ◽  
Research Subjects ◽  
Demographic Situation ◽  
The Social ◽  
Gender Categories

Reaching sports performance in a sports branch, discipline or event is dependent and conditioned also by the social context, the demographic situation of the area, region, the athlete’s country, age, gender or race, experience and the “migration of the sports population” etc. Previous research has shown that statistics can provide evidence in sport on the dynamics over a certain period of time, but statistics can also be a benchmark for future predictions. This paper aims to highlight various statistical analyses, for a period of 101 years, based on various criteria that have contributed more or less to the “record” of national titles in a distinct region of Romania, namely Bacau County. The research subjects are represented by 162 performance athletes registered according to the Romanian Athletics Federation regulations into four age categories (Junior II, Junior I, Youth and Senior) and into two gender categories (male, female). By comparing the results highlighted by the analysis of 712 titles, we can declare that the dynamics of sports performance over a century of athletics in Bacau can be clearly traced only by analyzing the data as a whole. The results, reported in detailed analyses of certain criteria, illustrate trends which do not allow us to generalize.

scholarly journals Determinants of the Attitude of Vocational High School Students in Using Computers in Jeneponto Regency

2021 ◽  
Vol 58 (1) ◽  
pp. 419-427
Author(s):  
Grace T. Pontoh Et al.
Keyword(s):  
High School Students ◽  
Social Cognitive ◽  
Cognitive Theory ◽  
Survey Method ◽  
Vocational High School ◽  
School Students ◽  
Statistical Software ◽  
Positive Attitudes ◽  
The Social ◽  
Technological Developments

The purpose of this study was to analyze the attitudes of students towards the use of computers as one of the subjects in school. This research used a survey method with data collection techniques by questionnaires with a sample of 210 students and data analyzed using AMOS statistical software. The results indicate: (1) CSE had an influence on attitudes through PEOU; (2) CA has an influence on attitudes through PEOU; (3) PE has an influence on attitudes through PEOU; and (4) PEOU has an influence on attitudes in using computers. This research contributes to the progress of science and technology with TAM and the Social Cognitive Theory which theoretically can provide evidence and support for the advancement of research in the scope of behavioral accounting. Also teachers can form positive attitudes of students in the learning process so as to produce outstanding graduates who are able to accept technological developments.

scholarly journals Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

2020 ◽  
Vol 3 (4) ◽  
pp. 456-465
Author(s):  
Jessica Kay Flake ◽  
Eiko I. Fried
Keyword(s):  
Construct Validity ◽  
Degrees Of Freedom ◽  
Scientific Research ◽  
Replication Studies ◽  
Psychological Science ◽  
Rigorous Research ◽  
Statistical Conclusion ◽  
Measurement Practices

In this article, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons, including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical-conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, and pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

CAVE: Making Collective Virtual Narrative: Best Paper Award

Leonardo ◽  
2019 ◽  
Vol 52 (4) ◽  
pp. 349-356 ◽  
Author(s):  
Kris Layng ◽  
Ken Perlin ◽  
Sebastian Herscher ◽  
Corinne Brenner ◽  
Thomas Meduri
Keyword(s):  
Virtual Reality ◽  
Degrees Of Freedom ◽  
Paper Award ◽  
Six Degrees Of Freedom ◽  
The Social ◽  
Future Path ◽  
Shared Narrative ◽  
Specific Pairing

CAVE is a shared narrative six degrees of freedom (6DoF) virtual reality experience. In 3.5 days, 1,927 people attended its premiere at SIGGRAPH 2018. Thirty participants at a time each saw and heard the same narrative from their own individual location in the room, as they would when attending live theater. CAVE set out to disruptively change how audiences collectively experience immersive art and entertainment. Inspired by the social gatherings of theater and cinema, CAVE resonated with viewers in powerful and meaningful ways. Its specific pairing of colocated audiences and physically shared immersive narrative suggests a possible future path for shared cinematic experiences.

Sign in / Sign up

Export Citation Format

Share Document