How to report E-values for meta-analyses: Recommended improvements and additions to the new GRADE approach

In a recent concept paper (Verbeek et al., 2021), the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group provides a preliminary proposal to improve its existing guidelines for assessing sensitivity to uncontrolled confounding in meta-analyses of nonrandomized studies. The new proposal centers on reporting the E-value for the meta-analytic mean and on comparing this E-value to a measured “reference confounder” to determine whether residual uncontrolled confounding in the meta-analyzed studies could or could not plausibly explain away the meta-analytic mean. Although we agree that E-value analogs for meta-analyses could be an informative addition to future GRADE guidelines, we suggest improvements to the Verbeek et al. (2021)’s specific proposal regarding: (1) their interpretation of comparisons between the E-value and the strengths of associations of a reference confounder; (2) their characterization of evidence strength in meta-analyses in terms of only the meta-analytic mean; and (3) the possibility of confounding bias that is heterogeneous across studies.

Probable Pain on the Pain Assessment in Impaired Cognition (PAIC15) Instrument: Assessing Sensitivity and Specificity of Cut-Offs against Three Standards

Observational pain scales can help to identify pain in persons with dementia who may have difficulty expressing pain verbally. The Pain Assessment in Impaired Cognition-15 (PAIC15) covers 15 items that indicate pain, but it is unclear how probable pain is, for each summed score (range 0–45). We aimed to determine sensitivity and specificity of cut-offs for probable pain on the PAIC15 against three standards: (1) self-report when able, (2) the established Pain Assessment in Advanced Dementia (PAINAD) cut-off of 2, and (3) observer’s overall estimate based on a series of systematic observations. We used data of 238 nursing home residents with dementia who were observed by their physician in training or nursing staff in the context of an evidence-based medicine (EBM) training study, with re-assessment after 2 months in 137 residents. The area under the ROC curve was excellent against the PAINAD cut-off (≥0.8) but acceptable or less than acceptable for the other two standards. Across standards and criteria for optimal sensitivity and specificity, PAIC15 scores of 3 and higher represent possible pain for screening in practice, with sensitivity and specificity against self-report in the 0.5 to 0.7 range. While sensitivity for screening in practice may be too low, a cut-off of 4 is reasonable to indicate probable pain in research.

Reliability and Validity of a 3-minute Psychomotor Vigilance Task (PVT) in Assessing Sensitivity to Sleep Loss and Alcohol: Fitness for Duty in Aviation and Transportation

Study Objectives The psychomotor vigilance task (PVT) is a widely used objective method to measure sustained attention, but the standard 10-min version is often impractical in operational settings. We investigated the reliability and validity of a 3-min PVT administered on a portable handheld device assessing sensitivity to sleep loss and alcohol in relation to a 10-min PVT and to applied tasks. Methods Forty-seven healthy volunteers underwent a 12 consecutive days sleep lab protocol. A cross-over design was adopted including total sleep deprivation (TSD, 38 hours awake), sleep restriction (SR, 4 h sleep opportunity), acute alcohol consumption, and SR after alcohol intake (SR/Alc 4 h sleep opportunity). Participants performed a 10-min and 3-min PVT and operationally-relevant tasks related to demands in aviation and transportation. Results Sleep loss resulted in significant performance impairments compared to baseline measurements detected by both PVT versions – particularly for mean speed (both p < .001) - and the operationally-relevant tasks. Similar effects were observed due to alcohol intake (speed: both p < .001). The 3-min and 10-min PVT results were highly correlated (speed: between r = .72 and r = .89). Three of four aviation related tasks showed robust correlations with the 3-min PVT. Correlations with the parameters of the task related to transportation were lower, but mainly significant. Conclusion The 3-min PVT showed a high reliability and validity in assessing sleep loss and alcohol induced impairments in cognitive performance. Thus, our results underline its usefulness as potential fitness for duty self-monitoring tool in applied settings.

Assessing sensitivity to Condition A in the case of Chinese reflexives

There are two reflexives in Mandarin Chinese, ziji (‘self’) and ta-ziji (‘s/he-self’). It is often assumed that ziji can be bound by a non-local antecedent while ta-ziji cannot. This is because ziji can be used as an exempt anaphor licensed by discourse-pragmatic conditions. However, prior research shows that, in contexts without perspectival cues, ziji tends to be interpreted as a ‘regular’ syntactically bound reflexive, exhibiting a similar locality bias as ta-ziji. However, prior studies comparing the locality biases of ziji and ta-ziji present divergent results. In this study, we report two forced choice judgment experiments to assess which reflexive, ziji or ta-ziji, exhibits a stronger locality bias. Overall, our results fit better with claims that in local contexts, ta-ziji is preferred over ziji; we find no clear evidence of ziji being preferred over ta-ziji in local contexts. Our results are compatible with the idea that ta-ziji, rather than ziji, is more constrained by Condition A.

Targeted Undersmoothing: Sensitivity Analysis for Sparse Estimators

This paper proposes a procedure for assessing sensitivity of inferential conclusions for functionals of sparse high-dimensional models following model selection. The proposed procedure is called targeted undersmoothing. Functionals considered include dense functionals that may depend on many or all elements of the highdimensional parameter vector. The sensitivity analysis is based on systematic enlargements of an initially selected model. By varying the enlargements, one can conduct sensitivity analysis about the strength of empirical conclusions to model selection mistakes. We illustrate the procedure's performance through simulation experiments and two empirical examples.

Assessing Sensitivity of MERRA-2 to AMSU-A in the Upper Stratosphere

Microwave temperature sounders provide key observations in data assimilation, both in the current and historical global observing systems, as they provide the largest amount of horizontal and vertical temperature information due to their insensitivity to clouds. In the Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2), microwave sounder radiances from the Advanced Microwave Sounding Unit-A (AMSU-A) are assimilated beginning with NOAA-15 and continuing through the current period. The time series of observation minus background statistics for AMSU-A channels sensitive to upper stratosphere and lower mesosphere show variabilities due to changes in AMSU-A constellation in the early AMSU-A period. Noted discrepancies are seen at the onset and exit of AMSU-A observations on the NOAA-15, NOAA-16, NOAA-17, and NASA EOS Aqua satellites. This effort characterizes the sensitivity, both in terms of the observations and the MERRA-2 analysis. Furthermore, it explores the use of reprocessed and inter-calibrated datasets to evaluate whether these homogenized observations can reduce the disparity due to change in instrumental biases against the model background. The results indicate that the AMSU-A radiances used in MERRA-2 are the fundamental cause of this inter-platform sensitivity which can be mitigated by using reprocessed data. The results explore the importance of the reprocessing of the AMSU-A radiances as well as their inter-calibration.