Why Hypothesis Testers Should Spend Less Time Testing Hypotheses

For almost half a century, Paul Meehl educated psychologists about how the mindless use of null-hypothesis significance tests made research on theories in the social sciences basically uninterpretable (Meehl, 1990). In response to the replication crisis, reforms in psychology have focused on formalising procedures for testing hypotheses. These reforms were necessary and impactful. However, as an unexpected consequence, psychologists have begun to realise that they may not be ready to test hypotheses. Forcing researchers to prematurely test hypotheses before they have established a sound ‘derivation chain’ between test and theory is counterproductive. Instead, various non-confirmatory research activities should be used to obtain the inputs necessary to make hypothesis tests informative. Before testing hypotheses, researchers should spend more time forming concepts, developing valid measures, establishing the causal relationships between concepts and their functional form, and identifying boundary conditions and auxiliary assumptions. Providing these inputs should be recognised and incentivised as a crucial goal in and of itself. In this article, we discuss how shifting the focus to non-confirmatory research can tie together many loose ends of psychology’s reform movement and help us lay the foundation to develop strong, testable theories, as Paul Meehl urged us to.

Download Full-text

Editor’s note: replication crisis in the social sciences

International Gambling Studies ◽

10.1080/14459795.2019.1673786 ◽

2019 ◽

Vol 19 (3) ◽

pp. 359-361 ◽

Cited By ~ 2

Author(s):

Alex Blaszczynski ◽

Sally M. Gainsbury

Keyword(s):

Social Sciences ◽

The Social ◽

Replication Crisis

Download Full-text

Have psychologists increased reporting of outliers in response to the reproducibility crisis?

10.31219/osf.io/jq8eu ◽

2018 ◽

Author(s):

Kathrene D Valentine ◽

Erin Michelle Buchanan ◽

Arielle Cunningham ◽

Tabetha Gaile Hopke ◽

Addie Wikowsky ◽

...

Keyword(s):

Social Sciences ◽

Statistical Analysis ◽

Inferential Statistics ◽

Questionable Research Practices ◽

Reporting Practices ◽

Time Points ◽

The Social ◽

Problems And Solutions ◽

Replication Crisis ◽

Data Points

Psychology is currently experiencing a "renaissance" where the replication and reproducibility of published reports are at the forefront of conversations in the field. While researchers have worked to discuss possible problems and solutions, work has yet to uncover how this new culture may have altered reporting practices in the social sciences. As outliers and other errant data points can bias both descriptive and inferential statistics, the search for these data points is essential to any analysis using these parameters. We quantified the rates of reporting of outliers and other data within psychology at two time points: 2012 when the replication crisis was born, and 2017, after the publication of reports concerning replication, questionable research practices, and transparency. A total of 2235 experiments were identified and analyzed, finding an increase in reporting from only 15.7% of experiments in 2012 to 25.0% in 2017. We investigated differences across years given the psychological field or statistical analysis that experiment employed. Further, we inspected whether data exclusions mentioned were whole participant observations or data points, and what reasons authors gave for stating the observation was deviant. We conclude that while report rates are improving overall, there is still room for improvement in the reporting practices of psychological scientists which can only aid in strengthening our science.

Download Full-text

Breaking the F-barrier: How the use of a visual representation of Fisher’s F-ratio can aid student comprehension of orthodox statistics

مجلة جامعة فلسطين التقنية للأبحاث ◽

10.53671/pturj.v6i2.61 ◽

2018 ◽

Vol 6 (2) ◽

pp. 5-10

Author(s):

Rory Allen

Keyword(s):

Social Sciences ◽

Null Hypothesis ◽

Statistical Tests ◽

Real Life ◽

Deep Understanding ◽

Significance Test ◽

Algebraic Manipulation ◽

Life Challenges ◽

The Social ◽

Degree Of Confidence

Universal laws are notoriously hard to discover in the social sciences, but there is one which can be stated with a fair degree of confidence: “all students hate statistics”. Students in the social sciences often need to learn basic statistics as part of a research methods module, and anyone who has ever been responsible for teaching statistics to these students will soon discover that they find it to be the hardest and least popular part of any social science syllabus. A typical problem for students is the use of Fisher’s F-test as a significance test, which even in the simple case of a one-factor analysis of variance (ANOVA) presents difficulties. These are two in number. Firstly, the test is presented as a test of the null hypothesis, that is, that there is no effect of one variable (the independent variable, IV) on the other, dependent variable (DV). This highlights the opposite of what one generally wants to prove, the experimental hypothesis, which is usually that there is an effect of the IV on the DV. Students, if they think about the question at all, may be tempted to ask “why not try to prove the experimental hypothesis directly rather than using this back-to-front approach?” Secondly, the F-ratio itself is presented in the form of an algebraic manipulation, involving the ratio of two mean sums of squares, and these means are themselves moderately complicated to understand. Even students specializing in mathematics often find algebra difficult, and to non- athematicians this formula is simply baffling. Instructors do not usually make a serious attempt to remedy this confusion by attempting to explain what the F-ratio is attempting to measure, and when they do, the explanation is not usually very enlightening. Students may struggle with the statement that the F-ratio is the ratio of “two different estimates of the variance of the population being sampled from, under the null hypothesis”. So what? The result is that students frequently end up applying statistical analysis programs such as SPSS and R, without having the faintest understanding of how the mathematics works. They use the results in a mechanical way, according to a procedure learned by rote memory, and may overlook different tests which might be more appropriate for their data. This might be called the cookbook approach to data analysis, and it is the opposite of the ultimate aim of high quality teaching, which is to provide a deep understanding of principles, which will allow the student to use these principles flexibly in real life challenges, without violating the assumptions of the statistical tests being employed.

Download Full-text

An Evaluation of Research Needs for Agricultural Economists

Journal of Agricultural and Applied Economics ◽

10.1017/s0081305200024419 ◽

1969 ◽

Vol 1 (1) ◽

pp. 151-158

Author(s):

R. J. Hildreth ◽

Roland R. Robinson

Keyword(s):

Social Sciences ◽

Relative Importance ◽

Qualitative Information ◽

Research Activity ◽

Department Of Agriculture ◽

Social Scientists ◽

Research Activities ◽

Land Grant Institutions ◽

The Social ◽

Professional Research

An attempt will be made to indicate: (1) the future needs for social scientists, in general, and agricultural economists, in particular, for research in the land-grant institutions and the U.S. Department of Agriculture, (2) the location of these needs for agricultural economists, and (3) the types of research activities that are gaining in relative importance in the social sciences. The quantitative and qualitative information presented should provide some guidance in locating professional research workers where they are most needed.Let us examine the present and prospective allocation of social scientists among the various areas of research activity, as well as the relative importance of these areas.

Download Full-text

DIGITAL SKILLS FOR EDUCATORS AND PARENTS IN THE CONTEXT OF HIGH - QUALITY EDUCATION

SOCIETY INTEGRATION EDUCATION Proceedings of the International Scientific Conference ◽

10.17770/sie2021vol3.6400 ◽

2021 ◽

Vol 3 ◽

pp. 185-194

Author(s):

Ilga Prudnikova

Keyword(s):

Social Sciences ◽

Quantitative Methods ◽

Digital Technologies ◽

Quality Education ◽

Future Research ◽

Empirical Methods ◽

High Quality Education ◽

Research Activities ◽

The Social ◽

Digital Skills

The purpose of the research is to identify commonalities and possible differences in the assessment of educators’ and parents’ attitudes towards digital technologies, reasons for their usage, and identify motivation to improve their digital skills. The study is built on research activities and there are used both theoretical and empirical methods. Quantitative methods in the form of questionnaires are used during the study. The researcher is more important to identify precedents and learn about the character of educators’ and parents’ attitudes. Dinamic environment for teaching should be supported by positive attitude to tehnologies. The statistical programme used for the analyses and presentation of data in this research is the Statistical Package for the Social Sciences (SPSS). In conclusion: the results from this study will be used to support interesting directions for future research in the context of higt –quility education.

Download Full-text

Replicability in Empirical Legal Research

Annual Review of Law and Social Science ◽

10.1146/annurev-lawsocsci-121620-085055 ◽

2021 ◽

Vol 17 (1) ◽

Author(s):

Jason M. Chin ◽

Kathryn Zeiler

Keyword(s):

Social Sciences ◽

Social Science ◽

Science Volume ◽

Reform Movement ◽

Annual Review ◽

Publication Date ◽

Legal Research ◽

Current State ◽

The Social ◽

Statistical Issues

As part of a broader methodological reform movement, scientists are increasingly interested in improving the replicability of their research. Replicability allows others to perform replications to explore potential errors and statistical issues that might call the original results into question. Little attention, however, has been paid to the state of replicability in the field of empirical legal research (ELR). Quality is especially important in this field because empirical legal researchers produce work that is regularly relied upon by courts and other legal bodies. In this review, we summarize the current state of ELR relative to the broader movement toward replicability in the social sciences. As part of that aim, we summarize recent collective replication efforts in ELR and transparency and replicability guidelines adopted by journals that publish ELR. Based on this review, ELR seems to be lagging other fields in implementing reforms. We conclude with suggestions for reforms that might encourage improved replicability. Expected final online publication date for the Annual Review of Law and Social Science, Volume 17 is October 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Download Full-text

Breaking the F-barrier: How the use of a visual representation of Fisher’s F-ratio can aid student comprehension of orthodox statistics

مجلة جامعة فلسطين التقنية للأبحاث ◽

10.53671/ptukrj.v6i2.61 ◽

2018 ◽

Vol 6 (2) ◽

pp. 5-10

Author(s):

Rory Allen

Keyword(s):

Social Sciences ◽

Null Hypothesis ◽

Statistical Tests ◽

Real Life ◽

Deep Understanding ◽

Significance Test ◽

Algebraic Manipulation ◽

Life Challenges ◽

The Social ◽

Degree Of Confidence

Universal laws are notoriously hard to discover in the social sciences, but there is one which can be stated with a fair degree of confidence: “all students hate statistics”. Students in the social sciences often need to learn basic statistics as part of a research methods module, and anyone who has ever been responsible for teaching statistics to these students will soon discover that they find it to be the hardest and least popular part of any social science syllabus. A typical problem for students is the use of Fisher’s F-test as a significance test, which even in the simple case of a one-factor analysis of variance (ANOVA) presents difficulties. These are two in number. Firstly, the test is presented as a test of the null hypothesis, that is, that there is no effect of one variable (the independent variable, IV) on the other, dependent variable (DV). This highlights the opposite of what one generally wants to prove, the experimental hypothesis, which is usually that there is an effect of the IV on the DV. Students, if they think about the question at all, may be tempted to ask “why not try to prove the experimental hypothesis directly rather than using this back-to-front approach?” Secondly, the F-ratio itself is presented in the form of an algebraic manipulation, involving the ratio of two mean sums of squares, and these means are themselves moderately complicated to understand. Even students specializing in mathematics often find algebra difficult, and to non- athematicians this formula is simply baffling. Instructors do not usually make a serious attempt to remedy this confusion by attempting to explain what the F-ratio is attempting to measure, and when they do, the explanation is not usually very enlightening. Students may struggle with the statement that the F-ratio is the ratio of “two different estimates of the variance of the population being sampled from, under the null hypothesis”. So what? The result is that students frequently end up applying statistical analysis programs such as SPSS and R, without having the faintest understanding of how the mathematics works. They use the results in a mechanical way, according to a procedure learned by rote memory, and may overlook different tests which might be more appropriate for their data. This might be called the cookbook approach to data analysis, and it is the opposite of the ultimate aim of high quality teaching, which is to provide a deep understanding of principles, which will allow the student to use these principles flexibly in real life challenges, without violating the assumptions of the statistical tests being employed.

Download Full-text

Cross-Validation: A Method Every Psychologist Should Know

Advances in Methods and Practices in Psychological Science ◽

10.1177/2515245919898466 ◽

2020 ◽

Vol 3 (2) ◽

pp. 248-263 ◽

Cited By ~ 4

Author(s):

Mark de Rooij ◽

Wouter Weeda

Keyword(s):

Social Sciences ◽

Model Selection ◽

Null Hypothesis ◽

Cross Validation ◽

R Package ◽

Statistical Procedure ◽

General Sense ◽

Null Hypothesis Testing ◽

The Social ◽

Research Problems

Cross-validation is a statistical procedure that every psychologist should know. Most are possibly familiar with the procedure in a global way but have not used it for the analysis of their own data. We introduce cross-validation for the purpose of model selection in a general sense, as well as an R package we have developed for this kind of analysis, and we present examples illustrating the use of this package for types of research problems that are often encountered in the social sciences. Cross-validation can be an easy-to-use alternative to null-hypothesis testing, and it has the benefit that it does not make as many assumptions.

Download Full-text

Replication and Pre-Registration in Comparative Psychology

International Journal of Comparative Psychology ◽

10.46867/ijcp.2018.31.01.09 ◽

2018 ◽

Vol 31 ◽

Author(s):

Michael J. Beran

Keyword(s):

Social Sciences ◽

Experimental Research ◽

Comparative Psychology ◽

Good Practices ◽

The Social ◽

Replication Crisis ◽

History Of ◽

Crisis In Psychology

There is growing interest and pressure in the social sciences to find ways to address the so-called “replication crisis” in psychology. This includes increasing transparency and good practices in all areas of experimental research, and in particular to promote attempts at replication. Comparative psychology has a long history of efforts to replicate and extend previous research, but it is often difficult to do this when highly specialized methods or uncommon species are being studied. I propose that comparative researchers make greater use of pre-registration as a way to ensure good practices, and I outline some of the ways in which this can be accomplished.

Download Full-text