Dissociation Between Speech and Emotion Effects in Short-Term Memory: A Data Reanalysis.

Performance in visual serial recall tasks is often impaired by irrelevant auditory distracters. The duplex-mechanism account of auditory distraction states that if the distracters provide order cues, these interfere with the processing of the order cues in the serial recall task (interference by process). In contrast, the unitary account states that distracters capture only attention on a general level (attentional distraction) without interfering specifically withorder processing. Marsh et al. (2018, Journal of Experimental Psychology-Learning Memory and Cognition, 44, 882-897) reported finding a dissociation between the effects of serial recall tasks and those of a missing-item task on the disruptive effects of speech and of emotional words, as predicted by the duplex-mechanism account. Critically, the reported analyses did not test specifically for the claimed dissociation. Therefore, I reanalyzed the Marsh et al. data and conducted the appropriate analyses. I also tested the dissociation more directly and added a Bayesian hypothesis test to measure the strength of the evidence for a dissociation. Results provided strong evidence for a dissociation (i.e., crossover interaction) between effects of speech and of emotion. Because the duplex-mechanism account predicts this dissociation between speech effects (interference by process) and emotion effects (attentionaldiversion) whereas the unitary account does not, Marsh et al.’s data support the duplex-mechanism account. However, to show that this dissociation is robust, researchers are advised to replicate this dissociation in an adversarial registered report.

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

On the Treatment of Missing Item Responses in Educational Large-scale Assessment Data: The Case of PISA 2018 Mathematics

Missing item responses are prevalent in educational large-scale assessment studies like the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians advocated a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. This article uses the PISA 2018 mathematics dataset to investigate the consequences of different missing data treatments on country means. Obtained country means can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

Construct Validity and Clinical Utility of World Health Organization Disability Assessment Schedule 2.0 in Older Patients Discharged From Emergency Departments

Introduction: The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is designed to measure functioning and disability in six domains. It is included in the International Classification of Diseases 11th revision (ICD-11). The objective of the study was to examine the construct validity of WHODAS 2.0 and describe its clinical utility for the assessment of functioning and disability among older patients discharged from emergency departments (EDs).Material and Methods: This cross-sectional study is based on data from 129 older patients. Patients completed the 36-item version of WHODAS 2.0 together with the Barthel-20, the Assessment of Motor and Process Skills (AMPS), Timed Up and Go (TUG), and the 30-Second Chair Stand Test (30 s-CST). Construct validity was examined through hypothesis testing by correlating the WHODAS with the other instruments and specifically the mobility domain in WHODAS 2.0 with the TUG and 30 s-CST tests. The clinical utility of WHODAS 2.0 was explored through floor/ceiling effect and missing item responses.Results: WHODAS 2.0 correlated fair with Barthel-20 (r = −0.49), AMPS process skills (r = −0.26) and TUG (r=0.30) and correlated moderate with AMPS motor skills (r = −0.58) and 30s-CST (r = −0.52). The WHODAS 2.0 mobility domain correlated fair with TUG (r = 0.33) and moderate with 30s-CST (r = −0.60). Four domains demonstrated floor effect: D1 “Cognition,” D3 “Self-care,” D4 “Getting along,” and D5 “Household.” Ceiling effect was not identified. The highest proportion of missing item responses were present for Item 3.4 (Staying by yourself for a few days), Item 4.4 (Making new friends), and Item 4.5 (Sexual activities).Conclusion: WHODAS 2.0 had fair-to-moderate correlations with Barthel-20, AMPS, TUG, and 30s-CST and provides additional aspects of disability compared with commonly used instruments. However, the clinical utility of WHODAS 2.0 applied to older patients discharged from EDs poses some challenges due to floor effect and missing item responses. Accordingly, patient and health professional perspectives need further investigation.

Estimating Probabilities of Passing for Examinees With Incomplete Data in Mastery Tests

Administrative problems such as computer malfunction and power outage occasionally lead to missing item scores and hence to incomplete data on mastery tests such as the AP and U.S. Medical Licensing examinations. Investigators are often interested in estimating the probabilities of passing of the examinees with incomplete data on mastery tests. However, there is a lack of research on this estimation problem. The goal of this article is to suggest two new approaches—one each based on classical test theory and item response theory—for estimating the probabilities of passing of the examinees with incomplete data on mastery tests. The two approaches are demonstrated to have high accuracy and negligible misclassification rates.

Dissociation Between Speech and Emotion Effects in Short-Term Memory: A Data Reanalysis

Performance in visual serial recall tasks is often impaired by irrelevant auditory distracters. The duplex-mechanism account of auditory distraction states that if the distracters provide order cues, these interfere with the processing of the order cues in the serial recall task (interference by process). In contrast, the unitary account states that distracters capture only attention on a general level (attentional distraction) without interfering specifically with order processing. Marsh et al. (2018, Journal of Experimental Psychology-Learning Memory and Cognition, 44, 882-897) reported finding a dissociation between the effects of serial recall tasks and those of a missing-item task on the disruptive effects of speech and of emotional words, as predicted by the duplex-mechanism account. Critically, the reported analyses did not test specifically for the claimed dissociation. Therefore, I reanalyzed the Marsh et al. data and conducted the appropriate analyses. I also tested the dissociation more directly and added a Bayesian hypothesis test to measure the strength of the evidence for a dissociation. Results provided strong evidence for a dissociation (i.e., crossover interaction) between effects of speech and of emotion. Because the duplex-mechanism account predicts this dissociation between speech effects (interference by process) and emotion effects (attentional diversion) whereas the unitary account does not, Marsh et al.’s data support the duplex-mechanism account. However, to show that this dissociation is robust, researchers are advised to replicate this dissociation in an adversarial registered report.

About Still Nonignorable Consequences of (Partially) Ignoring Missing Item Responses in Large-scale Assessment

In recent literature, alternative models for handling missing item responses in large-scale assessments are proposed. In principle, based on simulations and arguments based test theory (Rose, 2013). In those approaches, it is argued that missing item responses should never be scored as incorrect, but rather treated as ignorable (e.g., Pohl et al., 2014). The present contribution shows that these arguments have limited validity and illustrates the consequences in a country comparison in the PIRLS 2011 study. A different treatment of missing item responses than recoding them as incorrect leads to significant changes in country rankings, which induces nonignorable consequences regarding the results' validity. Additionally, two alternative item response models based on different assumptions for missing item responses are proposed.

Missing item responses in latent growth analysis: Item response theory versus classical test theory

In medical research, repeated questionnaire data is often used to measure and model latent variables across time. Through a novel imputation method, a direct comparison is made between latent growth analysis under classical test theory and item response theory, while also including effects of missing item responses. For classical test theory and item response theory, by means of a simulation study the effects of item missingness on latent growth parameter estimates are examined given longitudinal item response data. Several missing data mechanisms and conditions are evaluated in the simulation study. The additional effects of missingness on differences in classical test theory- and item response theory-based latent growth analysis are directly assessed by rescaling the multiple imputations. The multiple imputation method is used to generate latent variable and item scores from the posterior predictive distributions to account for missing item responses in observed multilevel binary response data. It is shown that a multivariate probit model, as a novel imputation model, improves the latent growth analysis, when dealing with missing at random (MAR) in classical test theory. The study also shows that the parameter estimates for the latent growth model using item response theory show less bias and have smaller MSE’s compared to the estimates using classical test theory.