When (Not) to Rely on the Reliable Change Index

Mapping Intimacies ◽

10.31219/osf.io/3kthg ◽

2021 ◽

Author(s):

Andrew Athan McAleavey

Keyword(s):

Measurement Error ◽

Clinical Psychology ◽

False Positives ◽

Reliable Change Index ◽

Applied Psychology ◽

False Negatives ◽

Statistical Tool ◽

Reliable Change ◽

Difference Scores

The reliable change index (RCI) is a widely used statistical tool designed to account for measurement error when evaluating difference scores. Because of its conceptual simplicity and computational ease, it persists in research and applied psychology. However, researchers have repeatedly demonstrated ways that the RCI is insufficient or invalid for various applications. This is a problem in research and clinical psychology since this common tool is potentially problematic. The aims of this manuscript are to non-technically describe the formulation and assumptions of the RCI, to offer guidance as to when the RCI is (and is not) appropriate, and to identify what is needed for proper calculation of the RCI when it is used. Several criteria are identified to help determine whether the RCI is appropriate for a specific use. It is apparent that the RCI is the best available method only in a small number of situations, is frequently miscalculated, and produces incorrect inferences more often than simple alternatives, largely because it is highly insensitive to real changes. Specific alternatives are offered which may better operationalize common inferential tasks, including when more than two observations are available and when false negatives are equally costly to false positives.

Download Full-text

Comparison of false positive and false negative rates of two indices of individual reliable change: Jacobson-Truax and Hageman-Arrindell methods

10.31234/osf.io/rzbhv ◽

2021 ◽

Author(s):

Rodrigo Ferrer

Keyword(s):

False Positive ◽

False Negative ◽

False Positives ◽

Effect Sizes ◽

Reliable Change Index ◽

Individual Change ◽

False Negatives ◽

Reliable Change ◽

Change Effect ◽

Two Indices

Background: The quantification of the change is crucial to correctly estimate the effect of a treatment and, for to distinguish random or non-systematic from substantive changes. The objective of the present study was to learn about the performance of two distribution-based methods (the Jacobson-Truax Reliable Change Index [RCI] and the Hageman-Arrindell [HA] approach) designed to evaluate individual change (reliable change).Methods: A pre-post design was simulated with the purpose to evaluate the false positive and false negative rates of RCI and HA methods. In this design, a first measurement is obtained before treatment and a second measurement is obtained after treatment, in the same group of subjects.Results: The rate of false positives, only the HA statistic provided acceptable results. Regarding the rate of false negatives, both statistics offered similar results and both could claim to offer acceptable rates when Ferguson’s stringent criteria were used to define effect sizes as opposed to when the conventional criteria advanced by Cohen were employed. Conclusions: Since the HA statistic appeared to be a better option than the RCI statistic, we have developed and presented an Excel macro so that the greater complexity of calculating HA would not represent an obstacle for the non-expert user.Key words: Individual reliable change, false positives, false negatives, assessment of change, effect size.

Download Full-text

Standard error in the Jacobson and Truax Reliable Change Index: The “classical approach” leads to poor estimates

Journal of the International Neuropsychological Society ◽

10.1017/s1355617704106115 ◽

2004 ◽

Vol 10 (6) ◽

pp. 899-901 ◽

Cited By ~ 14

Author(s):

NANCY R. TEMKIN

Keyword(s):

Measurement Error ◽

Classical Theory ◽

Error Rate ◽

Standard Error ◽

Practice Effect ◽

Practice Effects ◽

Reliable Change Index ◽

Classical Approach ◽

Reliable Change

Different authors have used different estimates of variability in the denominator of the Reliable Change Index (RCI). Maassen attempts to clarify some of the differences and the assumptions underlying them. In particular he compares the ‘classical’ approach using an estimate SEd supposedly based on measurement error alone with an estimate SDiff based on the variability of observed differences in a population that should have no true change. Maassen concludes that not only is SEd based on classical theory, but it properly estimates variability due to measurement error and practice effect while SDiff overestimates variability by accounting twice for the variability due to practice. Simulations show Maassen to be wrong on both accounts. With an error rate nominally set to 10%, RCI estimates using SDiff wrongly declare change in 10.4% and 9.4% of simulated cases without true change while estimates using SEd wrongly declare change in 17.5% and 12.3% of the simulated cases (p < .000000001 and p < .008, respectively). In the simulation that separates measurement error and practice effects, SEd estimates the variability of change due to measurement error to be .34, when the true variability due to measurement error was .014. Neuropsychologists should not use SEd in the denominator of the RCI. (JINS, 2004, 10, 899–901.)

Download Full-text

The standard error in the Jacobson and Truax Reliable Change Index: The classical approach to the assessment of reliable change

Journal of the International Neuropsychological Society ◽

10.1017/s1355617704106097 ◽

2004 ◽

Vol 10 (6) ◽

pp. 888-893 ◽

Cited By ~ 73

Author(s):

GERARD H. MAASSEN

Keyword(s):

Standard Error ◽

Real Data ◽

Practice Effects ◽

Reliable Change Index ◽

Classical Approach ◽

Reliable Change ◽

Difference Scores

Researchers and clinicians using Jacobson and Truax's index to assess the reliability of change in patients, or its counterpart by Chelune et al., which takes practice effects into account, are confused by the different ways of calculating the standard error encountered in the literature (see the discussion started in this journal by Hinton-Bayre). This article compares the characteristics of (1) the standard error used by Jacobson and Truax, (2) the standard error of difference scores used by Temkin et al. and (3) an adaptation of Jacobson and Truax's approach that accounts for difference between initial and final variance. It is theoretically demonstrated that the last variant is preferable, which is corroborated by real data. (JINS, 2004, 10, 888–893.)

Download Full-text

Die Wirksamkeit eines Frühinterventionsprogramms für Jugendliche mit Computerspiel- und Internetabhängigkeit: Mittelfristige Effekte der PROTECT+ Studie

Zeitschrift für Kinder- und Jugendpsychiatrie und Psychotherapie ◽

10.1024/1422-4917/a000673 ◽

2020 ◽

Vol 48 (1) ◽

pp. 3-14 ◽

Cited By ~ 3

Author(s):

Carolin Szász-Janocha ◽

Eva Vonderlin ◽

Katajun Lindenberg

Keyword(s):

Internet Use ◽

Health Problems ◽

Reliable Change Index ◽

Gaming Disorder ◽

Statistical Classification ◽

Reliable Change ◽

Classification Of Diseases ◽

International Statistical Classification

Zusammenfassung. Fragestellung: Das junge Störungsbild der Computerspiel- und Internetabhängigkeit hat in den vergangenen Jahren in der Forschung zunehmend an Aufmerksamkeit gewonnen. Durch die Aufnahme der „Gaming Disorder“ in die ICD-11 (International Statistical Classification of Diseases and Related Health Problems) wurde die Notwendigkeit von evidenzbasierten und wirksamen Interventionen avanciert. PROTECT+ ist ein kognitiv-verhaltenstherapeutisches Gruppentherapieprogramm für Jugendliche mit Symptomen der Computerspiel- und Internetabhängigkeit. Die vorliegende Studie zielt auf die Evaluation der mittelfristigen Effekte nach 4 Monaten ab. Methodik: N = 54 Patientinnen und Patienten im Alter von 9 bis 19 Jahren (M = 13.48; SD = 1.72) nahmen an der Frühinterventionsstudie zwischen April 2016 und Dezember 2017 in Heidelberg teil. Die Symptomschwere wurde zu Beginn, zum Abschluss der Gruppentherapie sowie nach 4 Monaten anhand von standardisierten Diagnostikinstrumenten erfasst. Ergebnisse: Mehrebenenanalysen zeigten eine signifikante Reduktion der Symptomschwere anhand der Computerspielabhängigkeitsskala (CSAS) nach 4 Monaten. Im Selbstbeurteilungsbogen zeigte sich ein kleiner Effekt (d = 0.35), im Elternurteil ein mittlerer Effekt (d = 0.77). Der Reliable Change Index, der anhand der Compulsive Internet Use Scale (CIUS) berechnet wurde, deutete auf eine starke Heterogenität im individuellen Symptomverlauf hin. Die Patientinnen und Patienten bewerteten das Programm zu beiden Follow-Up-Messzeitpunkten mit einer hohen Zufriedenheit. Schlussfolgerungen: Die vorliegende Arbeit stellt international eine der wenigen Studien dar, die eine Reduktion der Symptome von Computerspiel- und Internetabhängigkeit im Jugendalter über 4 Monate belegen konnte.

Download Full-text

Sky Segmentation for Enhanced Depth Reconstruction and Bokeh Rendering with Efficient Architectures

Electronic Imaging ◽

10.2352/issn.2470-1173.2020.14.coimg-378 ◽

2020 ◽

Vol 2020 (14) ◽

pp. 378-1-378-7

Author(s):

Tyler Nuanes ◽

Matt Elsey ◽

Radek Grzeszczuk ◽

John Paul Shen

Keyword(s):

Real Time ◽

Mobile Device ◽

Computational Cost ◽

False Positives ◽

Compact Model ◽

High Quality ◽

False Negatives ◽

Trade Off ◽

Depth Reconstruction ◽

Binary Classifiers

We present a high-quality sky segmentation model for depth refinement and investigate residual architecture performance to inform optimally shrinking the network. We describe a model that runs in near real-time on mobile device, present a new, highquality dataset, and detail a unique weighing to trade off false positives and false negatives in binary classifiers. We show how the optimizations improve bokeh rendering by correcting stereo depth misprediction in sky regions. We detail techniques used to preserve edges, reject false positives, and ensure generalization to the diversity of sky scenes. Finally, we present a compact model and compare performance of four popular residual architectures (ShuffleNet, MobileNetV2, Resnet-101, and Resnet-34-like) at constant computational cost.

Download Full-text

Automatic Extraction of Acronyms from Text

10.26686/wgtn.12922298 ◽

2020 ◽

Author(s):

Stuart Yeates

Keyword(s):

Digital Library ◽

False Positives ◽

Automatic Extraction ◽

False Negatives ◽

Library Research ◽

Communications Theory ◽

Textual Content

A brief introduction to acronyms is given and motivation for extracting them in a digital library environment is discussed. A technique for extracting acronyms is given with an analysis of the results. The technique is found to have a low number of false negatives and a high number of false positives. Introduction Digital library research seeks to build tools to enable access of content, while making as few as possible assumptions about the content, since assumptions limit the range of applicability of the tools. Generally, the broader the assumptions the more widely applicable the tools. For example, keyword based indexing [5] is based on communications theory and applies to all natural human textual languages (allowances for differences in character sets and similar localisation issues not withstanding) . The algorithm described in this paper makes much stronger assumptions about the content. It assumes textual content that contains acronyms, an assumption which is known to hold for...

Download Full-text

Avoiding Interest-Based Revenues While Constructing Shariah-Compliant Portfolios: False Negatives and False Positives

SSRN Electronic Journal ◽

10.2139/ssrn.2975790 ◽

2017 ◽

Author(s):

zggr Arslan-Ayaydin ◽

Kris Boudt ◽

Muhammad Wajid Raza

Keyword(s):

False Positives ◽

False Negatives

Download Full-text

Evaluation of Positive T- and B-Cell Gene Rearrangement Studies Among Patients Without a Definitive Diagnosis by Other Assays

American Journal of Clinical Pathology ◽

10.1093/ajcp/aqz112.067 ◽

2019 ◽

Vol 152 (Supplement_1) ◽

pp. S35-S36

Author(s):

Hadrian Mendoza ◽

Christopher Tormey ◽

Alexa Siddon

Keyword(s):

T Cell ◽

False Positive ◽

Gene Rearrangement ◽

Hematologic Malignancy ◽

False Negative ◽

False Positives ◽

False Negatives ◽

True Negative ◽

Flow Cytometric ◽

Pathology Reports

Abstract In the evaluation of bone marrow (BM) and peripheral blood (PB) for hematologic malignancy, positive immunoglobulin heavy chain (IG) or T-cell receptor (TCR) gene rearrangement results may be detected despite unrevealing results from morphologic, flow cytometric, immunohistochemical (IHC), and/or cytogenetic studies. The significance of positive rearrangement studies in the context of otherwise normal ancillary findings is unknown, and as such, we hypothesized that gene rearrangement studies may be predictive of an emerging B- or T-cell clone in the absence of other abnormal laboratory tests. Data from all patients who underwent IG or TCR gene rearrangement testing at the authors’ affiliated VA hospital between January 1, 2013, and July 6, 2018, were extracted from the electronic medical record. Date of testing; specimen source; and morphologic, flow cytometric, IHC, and cytogenetic characterization of the tissue source were recorded from pathology reports. Gene rearrangement results were categorized as true positive, false positive, false negative, or true negative. Lastly, patient records were reviewed for subsequent diagnosis of hematologic malignancy in patients with positive gene rearrangement results with negative ancillary testing. A total of 136 patients, who had 203 gene rearrangement studies (50 PB and 153 BM), were analyzed. In TCR studies, there were 2 false positives and 1 false negative in 47 PB assays, as well as 7 false positives and 1 false negative in 54 BM assays. Regarding IG studies, 3 false positives and 12 false negatives in 99 BM studies were identified. Sensitivity and specificity, respectively, were calculated for PB TCR studies (94% and 93%), BM IG studies (71% and 95%), and BM TCR studies (92% and 83%). Analysis of PB IG gene rearrangement studies was not performed due to the small number of tests (3; all true negative). None of the 12 patients with false-positive IG/TCR gene rearrangement studies later developed a lymphoproliferative disorder, although 2 patients were later diagnosed with acute myeloid leukemia. Of the 14 false negatives, 10 (71%) were related to a diagnosis of plasma cell neoplasms. Results from the present study suggest that positive IG/TCR gene rearrangement studies are not predictive of lymphoproliferative disorders in the context of otherwise negative BM or PB findings. As such, when faced with equivocal pathology reports, clinicians can be practically advised that isolated positive IG/TCR gene rearrangement results may not indicate the need for closer surveillance.

Download Full-text