A new method to explore inferential risks associated with each study in a meta-analysis: An approach based on Design Analysis

In the last ten years, scientific research has experienced an unprecedented “credibility’s crisis” of results. This means that researchers couldn't find the same results as in the original ones when conducting replication studies. In fact, the results showed that effects size were often not as strong as in the original studies and sometimes no effect was found. However, an important side-effect of the replicability crisis is that it increased the awareness of the problematic issues in the published literature and it promoted the development of new practices which would guarantee rigour, transparency and reproducibility. In the current work, the aim is to propose a new method to explore the inferential risks associated with each study in a meta-analysis. Specifically, this method is based on Design Analysis, a power analysis approach developed by @gelmanPowerCalculationsAssessing2014, which allows to analyse two other type of errors that are not commonly considered: the Type M (Magnitude) error and the Type S (Sign) error, concerning the magnitude and direction of the effects. We chose the Design Analysis approach because it allows to put more emphasis on the estimate of the effect size and it can be a valid tool available to researchers to make more conscious and informed decisions.

A Survey of Using YouTube as Supplementary Material with University English Language Learners in Vietnam: A Replication Study

There is a growing body of international literature that has shown positive student perceptions of using YouTube as a source of supplementary teaching and learning material. The literature further shows that YouTube use in Vietnamese educational settings is growing. However, empirical investigations in this region are lacking. As such, the question arises whether international studies’ results may be generalizable to this context. However, no replication studies have been undertaken to explore this. This mixed-methods external approximate replication of the seminal work in this field (Kelsen, 2009) was conducted to address this gap. The triangulated results (descriptive statistics, inferential statistics, qualitative interpretation) demonstrated that the current study showed markedly more positive perceptions than the initial study. Following this, it is suggested that, while the initial study is historically important, generalizability may not be transferable to the current Vietnamese context. It is further suggested the current study’s results are employed as a reference for this context and a starting point for future investigations and discussions. The results’ relevancy to the broader Asian context and suggestions for further study are also discussed.

Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology

As part of the Reproducibility Project: Cancer Biology, we published Registered Reports that described how we intended to replicate selected experiments from 29 high-impact preclinical cancer biology papers published between 2010 and 2012. Replication experiments were completed and Replication Studies reporting the results were submitted for 18 papers, of which 17 were accepted and published by eLife with the rejected paper posted as a preprint. Here, we report the status and outcomes obtained for the remaining 11 papers. Four papers initiated experimental work but were stopped without any experimental outcomes. Two papers resulted in incomplete outcomes due to unanticipated challenges when conducting the experiments. For the remaining five papers only some of the experiments were completed with the other experiments incomplete due to mundane technical or unanticipated methodological challenges. The experiments from these papers, along with the other experiments attempted as part of the Reproducibility Project: Cancer Biology, provides evidence about the challenges of repeating preclinical cancer biology experiments and the replicability of the completed experiments.

ARE REPLICATION STUDIES INFREQUENT BECAUSE OF NEGATIVE ATTITUDES?

Replication is a research methodology designed to verify, consolidate, and generalize knowledge and understanding within empirical fields of study. In second language studies, however, reviews share widespread concern about the infrequency of replication. A common but speculative explanation for this situation is that replication studies are not valued because they lack originality and/or innovation. To better understand and respond to the infrequency of replication in our field, 354 researchers were surveyed about their attitudes toward replication and their practices conducting replication studies. Responses included worldwide participation from researchers with and without replication experience. Overall, replications were evaluated as relevant and valuable to the field. Claims that replication studies lack originality/innovation were not supported. However, dissemination issues were identified: half of published replication studies lacked explicit labeling and one quarter of completed replications were unpublished. Explicit labeling of replication studies and training in research methodology and dissemination can address this situation.

The Validity of the Residuals Approach to Measuring Resilience to Adverse Childhood Experiences

Background: Resilience is broadly defined as the ability to maintain or regain functioning in the face of adversity. Recent work to harmonise the quantification and definition of resilience quantifies resilience as the residual variance in psychosocial functioning that remains after accounting for adversity exposure. However, there have been no published studies that have formally investigated the validity of this approach. Considering this, we examine the construct and predictive validity of the residuals approach using participants from ALSPAC. Methods: We regressed exposures of adolescent adversity on adolescent psychopathology scores using the Strength and Difficulties Questionnaire and obtained the residual variance. We investigated construct validity by analysing whether previously identified demographic and resilience factors significantly predicted resilience. Predictive validity of resilience was investigated by comparing the predictive power of resilience with other determinants of psychosocial functioning on two developmental outcomes: depressive symptoms at 18 years, measured by the Short Moods and Feelings Questionnaire, and NEET (Not in Employment, Education or Training) status at 17 and 23 years. The associations between depressive symptoms at 18, resilience, ACEs and covariates were tested using multiple linear regression. NEET status at 17 and 23 were run as separate binary multiple logistic regression models to test associations with resilience and known demographics previously associated with NEET status.Results: Seven previously identified protective factors, including self-esteem, positive sibling relationship, temperament, and positive perception of school, significantly predicted resilience to adolescent psychopathology, thus providing strong construct validity. Resilience significantly predicted a reduction in depressive symptoms at 18 years, and significantly decreased the likelihood of having NEET status at both 17 years and 23 years, even after taking into account early childhood adversity and other risk factors. None of the socioeconomic factors were significantly associated with resilience.Conclusions: Our study demonstrates that the residuals method of operationalising resilience has good construct and predictive validity yet recommend replication studies. It has the potential to advance research into the mechanisms and modifiability of resilience.Trial Registration: Not applicable

Prediction of Suicidality According to Serum Folate Levels in Depressive Patients Receiving Stepwise Pharmacotherapy

Background: The effects of serum folate levels on suicidal behavior, strongly associated with depression, have not been investigated. Therefore, this study investigated the associations between serum folate levels and suicidal behavior in patients with depressive disorders.Methods: Serum folate levels were measured at baseline in 1,094 patients with depressive disorder, 884 of whom were followed during a 12-month period of stepwise pharmacotherapy. Suicidal behaviors evaluated at baseline were (i) previous suicide attempt and (ii) baseline suicidal severity; behaviors evaluated at follow-up were (iii) increased suicidal severity and iv) fatal/non-fatal suicide attempt. Associations of serum folate levels with four types of suicidal behaviors were analyzed using logistic regression models after adjustment for relevant covariates; they were also examined using area under receiver operating characteristic (AUROC) curve analyses.Results: Reduced serum folate levels (<6.0 ng/mL) were independently associated with all four types of suicidal behaviors. AUROC curve analyses indicated that discriminant or prognostic values of reduced serum folate levels were fair for fatal/non-fatal suicide attempt during follow-up, whereas they were modest for previous suicide attempt, baseline suicidal severity, and increased suicidal severity.Conclusions: Serum folate levels could serve as a biomarker of suicidal behavior in depressive patients. However, it should be used as an adjunct rather than a substitute for prediction of suicidal behavior considering its low prognostic values. Further replication studies are needed for its clinical utilization.

What has Royal Society Open Science achieved in its first few years?

It has been a pleasure and a privilege to serve as the first Editor-in-Chief of Royal Society Open Science for the past 6 years. I step down at the end of December 2021, having completed two 3-year terms, and am taking the opportunity here to reflect on some of the successes and challenges that the journal has experienced and the innovations that we have introduced. When I was first approached back in 2015, the breadth of the journal, covering the whole of science, resonated with my own interests: my research career has ranged across the entire landscape of chemistry, while my leadership roles have embraced all of science, technology and medicine. The open access ethos, the objective refereeing policy that rejects the idea of only publishing what is in fashion, and the opportunities offered by a new venture that could transcend traditional disciplinary boundaries also all appealed to me. Among our successful innovations are Registered Reports, Replication Studies and the new ‘Science, Society and Policy' section. The challenges have included the transition to paid article processing charges (APCs), whether to resist pressure to retract a controversial paper, and bullying of young female authors by established senior males in the same field. I explore all of these below, provide some statistics on the journal's performance, also cover some of the notable papers we have published, and provide some concluding thoughts.

Higher Limbic and Basal Ganglia volumes in surviving COVID-negative patients and the relations to fatigue.

Severe Acute Respiratory Syndrome Coronavirus 2 (SARSCoV2) has caused a global pandemic. Among several systemic abnormalities, little is known about the critical attack on the central nervous system (CNS). Several patient reports with multiple pathologies, ischemic strokes, mild infarcts, encephalitis, cerebrovascular abnormalities, cerebral inflammation, and loss of consciousness, indicate CNS involvement. However, due to limited neuroimaging studies, conclusive group level effects are scarce in the literature and replication studies are necessary to verify if these effects persist in surviving acute-COVID patients. Furthermore, recent reports indicate fatigue is highly prevalent among slowly recovering patients. How early structural changes relate to fatigue need to be investigated. Our goal was to address this by scanning COVID subjects two weeks after hospital discharge. We hypothesized these surviving patients will demonstrate altered gray matter volume (GMV) when compared to healthy controls and further demonstrate correlation of GMV with fatigue. Voxel-based morphometry was applied to T1-weighted MRI images between 46 patients with COVID and 32 healthy controls. Significantly higher GMV in the Limbic System and Basal Ganglia regions were observed in surviving COVID-19 patients when compared to healthy controls. Moreover, within the patient group, there was a significant positive correlation between GMV and self-reported fatigue scores during work, within the ventral Basal Ganglia and Ventromedial Prefrontal Cortex regions. Therefore, our results align with both single case acute patient reports and current group level neuroimaging findings. Finally, we newly report a positive correlation of GMV with fatigue in COVID survivors.