Research Commentary: A Rejoinder: A Reflection on the Evolution of a Replication Study

2018 ◽  
Vol 49 (1) ◽  
pp. 111-115 ◽  
Author(s):  
Faiza M. Jamil
Keyword(s):  
Reward System ◽  
Past Research ◽  
Original Study ◽  
Replication Study ◽  
Expectancy Effects ◽  
Opportunity To Respond ◽  
Replication Studies ◽  
Teacher Expectancy ◽  
Do So

I appreciate the opportunity to respond to the thoughtful comments made by Alan Schoenfeld (2018) and Jon Star (2018) in their commentaries on replication studies in this issue of JRME, including their comments on our study of teacher expectancy effects (Jamil, Larsen, & Hamre, 2018). I have decided to write this rejoinder in the form of a personal reflection. As academics, we carry the tremendous burden of expertise, and perhaps that is partly why, as pointed out by Schoenfeld (2018), the academic reward system focuses so heavily on novelty and innovation. With our expertise, we are supposed to have all the answers, solve all the problems, and do so in brilliant, new ways. Replication studies are undervalued because they not only, by definition, recreate past research but, perhaps, also bring into question another scholar‧s expertise. Star (2018) even states that one of the three criteria of an outstanding replication study is that it “convincingly shows that there is reason to believe that the results of the original study may be flawed” (p. 99). Although this rigorous examination is precisely the way to build trust in the quality of our findings and move the field forward, it is also what makes it challenging to have candid conversations about what we do not know.

scholarly journals Bayesian evaluation of replication studies

2020 ◽  
Author(s):  
Hidde Jelmer Leplaa ◽  
Charlotte Rietbergen ◽  
Herbert Hoijtink
Keyword(s):  
Bayes Factor ◽  
Open Science ◽  
Original Study ◽  
Replication Study ◽  
Research Results ◽  
Replication Studies ◽  
Science Collaboration ◽  
Using Data

In this paper a method is proposed to determine whether the result from an original study is corroborated in a replication study. The paper is illustrated using data from the reproducibility project psychology by the Open Science Collaboration. This method emphasizes the need to determine what one wants to replicate: the hypotheses as formulated in the introduction of the original paper, or hypotheses derived from the research results presented in the original paper. The Bayes factor will be used to determine whether the hypotheses evaluated in/resulting from the original study are corroborated by the replication study. Our method to assess the successfulness of replication will better fit the needs and desires of researchers in fields that use replication studies.

scholarly journals General claims require generalized effects: A reply to Ruiz et al.’s (2020) ‘A systematic and critical response to Pendrous et al. (2020) replication study’

2020 ◽  
Author(s):  
Ian Hussey
Keyword(s):  
Scientific Community ◽  
Original Study ◽  
Replication Study ◽  
High Quality ◽  
Null Results ◽  
Critical Response ◽  
Replication Studies ◽  
Predictable Pattern ◽  
Made In

Recent evidence suggests the results of psychology studies replicate less than half the time. Replies are often then written by the authors of the original studies. This discourse often follows a predictable pattern: highly general claims are made in an original study, the replication produces null results, and the response by the original authors primarily focuses on reasons to disqualify the replication’s results from requiring consideration, rather than acknowledging that the original study’s finding may not in fact be replicable, or that the generality of the original claims may require revision or constraint. I illustrate these points using the example of a recently published trio of original study, failed replication, and response by original authors. I argue that our scientific goals would be better served by efforts to avoid falling into these writing tropes, and to instead move the discourse forward and reinforce the behaviours we want to see in our scientific community, such as the conduction of high-quality replication studies.

More Than One Replication Study Is Needed for Unambiguous Tests of Replication

2019 ◽  
Vol 44 (5) ◽  
pp. 543-570 ◽  
Author(s):  
Larry V. Hedges ◽  
Jacob M. Schauer
Keyword(s):  
High Power ◽  
Statistical Test ◽  
Original Study ◽  
Replication Study ◽  
Experimental Results ◽  
Replication Studies

The problem of assessing whether experimental results can be replicated is becoming increasingly important in many areas of science. It is often assumed that assessing replication is straightforward: All one needs to do is repeat the study and see whether the results of the original and replication studies agree. This article shows that the statistical test for whether two studies obtain the same effect is smaller than the power of either study to detect an effect in the first place. Thus, unless the original study and the replication study have unusually high power (e.g., power of 98%), a single replication study will not have adequate sensitivity to provide an unambiguous evaluation of replication.

scholarly journals A re-examination of online journal quality and investigation of the possible impact of poor electronic surrogate quality on researchers

2018 ◽  
Vol 13 (3) ◽  
pp. 53-68 ◽  
Author(s):  
Ken Ladd
Keyword(s):  
Best Practice ◽  
Web Of Science ◽  
Poor Quality ◽  
Original Study ◽  
Electronic Journal ◽  
Research Papers ◽  
Online Journal ◽  
Content Information ◽  
Do So

Objective – This study re-examines the findings of a paper (Ladd, 2010) that investigated whether evidence indicated print equivalent journal collections needed to be preserved, based on the quality of their electronic surrogates. The current study investigates whether: 1) electronic surrogate articles that failed (i.e., the print equivalent article needed to be consulted to view all the content/information) in the first study had improved in quality; and 2) there was evidence that poor-quality electronic surrogates could impact on research if the print equivalent articles did not exist. Methods – Each of the 198 PDF documents identified in the 2010 study as failing were re-examined to assess whether any change in quality had occurred. To assess the possible impact for researchers if they needed to rely solely on poor-quality electronic journal surrogates, citation data were collected for each of the failed scholarly PDFs using Web of Science and Scopus, and usage count data were collected from Web of Science. Results – Across the electronic journal backfiles/archives examined, there were 13.6% fewer failures of electronic surrogates for all PDF documents than in the original study, while for scholarly PDF documents (e.g., research papers) there were 13.8% fewer failures. One electronic journal archive accounted for 91.7% of the improvement for scholarly PDF documents. A second archive accounted for all the observed improvement for non-scholarly PDF documents. The study found that for the failed scholarly PDF documents from the original study, 58.7% had been cited or had Web of Science usage counts from 2010 onward. Conclusion – The study demonstrates a continued need for retaining print equivalent journal titles for the foreseeable future, while poor-quality electronic surrogates are being replaced and digitally preserved. There are still poor-quality images, poor-quality scans of text-only articles, missing pages, and even content of PDF documents that could not be explained (e.g., incorrect text for images when compared to the print). While it is known that not all researchers will consult each of the papers that they cite, although it is best practice to do so, the extent of citations of the failed scholarly PDF documents indicate that having to rely solely on electronic surrogates could pose a problem for researchers.

General claims require generalized effects: A reply to Ruiz et al.’s (2020) ‘A systematic and critical response to Pendrous et al. (2020) replication study’

2020 ◽  
Author(s):  
Ian Hussey
Keyword(s):  
Scientific Community ◽  
Original Study ◽  
Replication Study ◽  
High Quality ◽  
Null Results ◽  
Critical Response ◽  
Replication Studies ◽  
Predictable Pattern ◽  
Made In

Recent evidence suggests the results of psychology studies replicate less than half the time. Replies are often then written by the authors of the original studies. This discourse often follows a predictable pattern: highly general claims are made in an original study, the replication produces null results, and the response by the original authors primarily focuses on reasons to disqualify the replication’s results from requiring consideration, rather than acknowledging that the original study’s finding may not in fact be replicable, or that the generality of the original claims may require revision or constraint. I illustrate these points using the example of a recently published trio of original study, failed replication, and response by original authors. I argue that our scientific goals would be better served by efforts to avoid falling into these writing tropes, and to instead move the discourse forward and reinforce the behaviours we want to see in our scientific community, such as the conduction of high-quality replication studies.

scholarly journals A call for replications of addiction research: Which studies should we replicate & what constitutes a “successful” replication?

2020 ◽  
Author(s):  
Robert Heirene
Keyword(s):  
Problem Gambling ◽  
Experimental Research ◽  
Original Study ◽  
Replication Study ◽  
High Quality ◽  
Extant Literature ◽  
Entire Field ◽  
Replication Studies ◽  
Successful Replication

Several prominent researchers in the problem gambling field have recently called for high-quality replications of existing gambling studies. This call should be extended to the entire field of addiction research: there is a need to focus on ensuring that the understanding of addiction and related phenomena gained through the extant literature is robust and replicable. This article discusses two important questions addictions researchers should consider before proceeding with replication studies: [1] which studies should we attempt to replicate? And: [2] how should we interpret the findings of a replication study in relation to the original study? In answering these questions, a focus is placed on experimental research, though the discussion may still serve as a useful introduction to the topic of replications for addictions researchers using any methodology.

scholarly journals Beïnvloedt een meer of minder sympathieke protagonist de transportatie van de lezer? : Twee nieuwe replicatiestudies

2019 ◽  
Vol 41 (3) ◽  
pp. 515-546
Author(s):  
Carel Jansen ◽  
Anneke de Graaf ◽  
Lettica Hustinx ◽  
Joëlle Ooms ◽  
Merel Schreinemakers ◽  
...  
Keyword(s):  
Statistical Tests ◽  
Meta Analysis ◽  
Original Study ◽  
Replication Study ◽  
Limited Extent ◽  
Replication Studies ◽  
Disposition Theory ◽  
Limited Power ◽  
Affective Disposition Theory ◽  
Positive Effect

Abstract Does a more or less sympathetic protagonist influence transportation of the reader? Two new replication studiesThree previous studies into presenting a protagonist in a story as more or less sympathetic have not provided a clear picture of the effects that the portrayal of the protagonist may have on transportation, and via transportation on story-consistent beliefs. Results from a first study (N = 83) by De Graaf and Hustinx (2015) suggest that the way the protagonist is portrayed ‐ as sympathetic, unsympathetic or neutral ‐ influences the extent to which readers are transported into a story. No significant effects on beliefs of the readers were found, however. In a direct replication study (N = 79) and in a conceptual replication study (N = 81), Jansen, Nederhoff, and Ooms (2017) found results that supported the hypotheses from the original study to a limited extent. In view of the relatively small numbers of participants in these three studies and the resulting limited power of the statistical tests two new, larger-scaled replication studies were conducted. A direct replication study was performed (N = 238) with the same versions of the story as used in the original study, and also a conceptual replication study (N = 248) with three versions of a new story. Again, the hypotheses from the original study were supported to a limited extent. A meta-analysis of all five studies revealed a large indirect positive effect of story version on transportation via empathy, when comparing the versions with a sympathetic protagonist with the versions with an unsympathetic protagonist. When comparing the neutral story versions with the versions with an unsympathetic protagonist, the meta-analytic indirect effect was medium sized. Other than what the Affective Disposition Theory (Raney, 2004; Zillmann, 1994; 2006) claims, the story versions with a neutral protagonist did not lead to an absence of emotional responses. Furthermore, the outcomes add to the Transportation-Imagery Model (Green & Brock, 2002; Van Laer, De Ruyter, Visconti, & Wetzels, 2014). While this model does not include concrete suggestions of story characteristics that lead to transportation, our studies show that a protagonist who is portrayed as sympathetic may contribute to the level of transportation that readers experience, be it indirectly through empathy.

Holding replication studies to mainstream standards of evidence

2018 ◽  
Vol 41 ◽  
Author(s):  
Duane T. Wegener ◽  
Leandre R. Fabrigar
Keyword(s):  
Replication Studies ◽  
Replication Research ◽  
Standards Of Evidence ◽  
Do So

AbstractReplications can make theoretical contributions, but are unlikely to do so if their findings are open to multiple interpretations (especially violations of psychometric invariance). Thus, just as studies demonstrating novel effects are often expected to empirically evaluate competing explanations, replications should be held to similar standards. Unfortunately, this is rarely done, thereby undermining the value of replication research.

Collaborative Remembering in Dementia: A Focus on Joint Activities

2018 ◽  
Author(s):  
Lars-Christer Hydén ◽  
Mattias Forsblad
Keyword(s):  
Everyday Life ◽  
Past Research ◽  
Research Field ◽  
Future Research ◽  
Research Needs ◽  
Joint Activities ◽  
Do So ◽  
Dementia Diseases

In this chapter we consider collaborative remembering and joint activates in everyday life in the case of people living with dementia. First, we review past research of practices that scaffolds the participation of persons with dementia in everyday chores under different stages of dementia diseases. We do so by suggesting three analytical types of scaffolding: when the scaffolding practices (i) frame the activity, (ii) guide actions, or (iii) are part of repair activities. Second, we review two aspects of collaborative remembering that are especially important in the case of dementia: training of scaffolding practices, and the sustaining and presentation of identities through collaborative storytelling. Finally, theoretical and methodological tendencies of the research field are summarized and future research needs are formulated.

Sign in / Sign up

Export Citation Format

Share Document