Receiver operating characteristic plot and area under the curve with binary classifiers: pragmatic analysis of cognitive screening instruments

Screening Test ◽

Odds Ratio ◽

Addenbrooke’S Cognitive Examination

Area Under The Curve ◽

Diagnostic Odds Ratio ◽

Test Accuracy ◽

Cognitive Screening ◽

Binary Classifiers ◽

Aim: To examine whether receiver operating characteristic plots and area under the curve (AUC) values may be potentially misleading when assessing cognitive screening instruments as binary predictors rather than as categorical or continuous scales. Materials & methods: AUC was calculated using different methods (rank-sum, diagnostic odds ratio) using data from test accuracy studies of two binary classifiers of cognitive status (applause sign, attended with sign), a screener producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke’s Cognitive Examination). Results: For all screeners, AUC calculated using diagnostic odds ratio method was greater than using rank-sum method. When Codex and Mini-Addenbrooke’s Cognitive Examination were analyzed as binary (single fixed threshold) tests, AUC using rank-sum method was lower than when screeners were analyzed as categorical or continuous scales, respectively. Conclusion: If cognitive screeners producing categorical or continuous measures are dichotomized, calculated AUC may be an underestimate, thus affecting screening test accuracy.

Defining ‘optimal’ test cut-off using global test metrics: evidence from a cognitive screening instrument

Neurodegenerative Disease Management ◽

10.2217/nmt-2020-0003 ◽

2020 ◽

Vol 10 (4) ◽

pp. 223-230

Author(s):

Andrew J Larner

Keyword(s):

Receiver Operating Characteristic Curve ◽

Characteristic Curve ◽

Test Accuracy ◽

Cognitive Screening ◽

Global Test ◽

Operating Characteristic Curve ◽

Addenbrooke’S Cognitive Examination ◽

Aim: To examine the variation of several global metrics of test accuracy with test cut-off for the diagnosis of dementia. These metrics included some based on the receiver operating characteristic curve, such as Youden index, and some independent of receiver operating characteristic curve, such as correct classification accuracy. Materials & methods: Data from a test accuracy study of Mini-Addenbrooke’s Cognitive Examination were used to calculate and plot each global measure against test cut-off. Results: Different ‘optimal’ cut-points were identified for the different global measures, with a spread of ten points in observed optimal cut-off in the 30-point Mini-Addenbrooke’s Cognitive Examination scale. Using these optima gave a large variation in test sensitivity from very high (diagnostic odds ratio) to very low (likelihood to be diagnosed or misdiagnosed), but all had high negative predictive value. Conclusion: The method used to determine the cut-off of cognitive screening instruments may have significant implications for test performance.

Systematic Review with Meta-Analysis: Fecal Calprotectin as a Surrogate Marker for Predicting Relapse in Adults with Ulcerative Colitis

Mediators of Inflammation ◽

10.1155/2019/2136501 ◽

2019 ◽

Vol 2019 ◽

pp. 1-10 ◽

Cited By ~ 2

Author(s):

Jiajia Li ◽

Xiaojing Zhao ◽

Xueting Li ◽

Meijiao Lu ◽

Hongjie Zhang

Keyword(s):

Ulcerative Colitis ◽

Likelihood Ratio ◽

Odds Ratio ◽

Diagnostic Odds Ratio ◽

Fecal Calprotectin ◽

Clinical Relapse ◽

Sensitivity Specificity ◽

The clinical course of ulcerative colitis (UC) is featured by remission and relapse, which remains unpredictable. Recent studies revealed that fecal calprotectin (FC) could predict clinical relapse for UC patients in remission, which has not yet been well accepted. To detect the predictive value of FC for clinical relapse in adult UC patients based on updated literature, we carried out a comprehensive electronic search of PubMed, Web of Science, Embase, and the Cochrane Library to identify all eligible studies. Diagnostic accuracy including pooled sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), and pooled area under the receiver operating characteristic (AUROC) was calculated using a random effects model. Heterogeneity across studies was assessed by the I2 metric. Sources of heterogeneity were detected using subgroup analysis. Metaregression was used to test potential factors correlated to DOR. Publication bias was assessed using Deek’s funnel plots. In our study, 14 articles enrolling a total of 1110 participants were finally included, and all articles underwent a quality assessment. Pooled sensitivity, specificity, PLR, and NLR with 95% confidence intervals (CIs) were 0.75 (95% CI: 0.70–0.79), 0.77 (95% CI: 0.74–0.80), 3.45 (95% CI: 2.31–5.14), and 0.37 (95% CI: 0.28–0.49) respectively. The area under the summary receiver operating characteristic (sROC) curve was 0.82, and the diagnostic odds ratio was 10.54 (95% CI: 6.16–18.02). Our study suggested that FC is useful in predicting clinical relapse for adult UC patients in remission as a simple and noninvasive marker.

Screening for dementia: Q* index as a global measure of test accuracy revisited

10.1101/2020.04.01.20050567 ◽

2020 ◽

Author(s):

Andrew J Larner

Keyword(s):

Odds Ratio ◽

Meta Analysis ◽

Roc Curves ◽

Diagnostic Odds Ratio ◽

Test Accuracy ◽

Cognitive Screening ◽

Optimal Test ◽

Global Parameter ◽

Accuracy Study ◽

Q Index

AbstractReceiver operating characteristic (ROC) curves intersect the downward diagonal through ROC space at a point, the Q* index, where by definition sensitivity and specificity are equal. Aside from its use in meta-analysis, Q* index has also been suggested as a possible global parameter summarising test accuracy of cognitive screening instruments and as a definition for optimal test cut-off. Area under the ROC curve (AUC ROC) is a recognised measure of test accuracy. This study compared different methods for determining Q* index (both graphical and calculation from diagnostic odds ratio) and AUC ROC (integration and calculation from diagnostic odds ratio) using the dataset of a prospective screening test accuracy study of the Mini-Addenbrooke’s Cognitive Examination. The different methods did not agree. DOR-based calculations gave a very sensitive cut-off but with poorer global metrics than the graphical method. DOR-based calculations are not recommended for defining optimal test cut-off or test accuracy.

ROC plot and AUC with binary classifiers: pragmatic analysis of cognitive screening instruments

10.1101/2021.03.09.21253194 ◽

2021 ◽

Author(s):

Gashirai K Mbizvo ◽

Andrew J Larner

Keyword(s):

Test Performance ◽

False Positive Rate ◽

Area Under The Curve ◽

True Positive Rate ◽

Cognitive Status ◽

Test Accuracy ◽

Cognitive Screening ◽

Continuous Scale ◽

Positive Rate ◽

Binary Classifiers

Receiver operating characteristic (ROC) plots are a performance graphing method showing the relative trade-off between test benefits (true positive rate) and costs (false positive rate) with the area under the curve (AUC) giving a scalar value of test performance. It has been suggested that ROC and AUC may be potentially misleading when examining binary predictors rather than continuous scales. The purpose of this study was to examine ROC plots and AUC values for two binary classifiers of cognitive status (applause sign, attended with sign), a cognitive screening instrument producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke's Cognitive Examination), the latter two also analysed with single fixed threshold tests. For each of these plots, AUC was calculated using different methods. The findings indicate that if categorical or continuous measures are dichotomised then the calculated AUC may be an underestimate, thus affecting screening or diagnostic test accuracy which in the context of clinical practice may prove to be misleading.

Addenbrooke’S Cognitive Examination

The Mini-Addenbrooke’s Cognitive Examination (M-ACE) as a brief cognitive screening instrument in Mild Cognitive Impairment and mild Alzheimer’s disease

Dementia & Neuropsychologia ◽

10.1590/1980-57642018dn12-040005 ◽

2018 ◽

Vol 12 (4) ◽

pp. 368-373

Author(s):

Diane da Costa Miranda ◽

Sonia Maria Dozzi Brucki ◽

Mônica Sanches Yassuda

Keyword(s):

Alzheimer’S Disease ◽

Alzheimer's Disease ◽

Cognitive Impairment ◽

Mild Cognitive Impairment ◽

Screening Test ◽

Area Under The Curve ◽

Cognitive Screening ◽

Healthy Elderly ◽

Administration Time ◽

ABSTRACT The Mini-Addenbrooke’s Cognitive Examination (M-ACE) is a brief cognitive screening test that evaluates four main cognitive domains (orientation, memory, language and visuospatial function) with a maximum score of 30 points and administration time of five minutes. Objective: To assess the performance of healthy elderly, MCI patients and mild AD patients using the Brazilian version of the M-ACE. Methods: The test was applied to a group of 36 Mild Cognitive Impairment (MCI), 23 mild Alzheimer’s Disease (AD) and 25 cognitive healthy elderly. All participants were aged ≥60 years. Results: The M-ACE displayed high internal consistency (Cronbach alpha >0.8; 95% CI 0.7-0.8) and proved effective for differentiating the AD group from MCI and control groups, providing superior accuracy than the MMSE (the cut-off point of 20 points had the highest sensitivity and specificity – 95.6% and 90.16% respectively, with a high area under the curve – AUC=0.8; 95% CI 0.7-0.9). Performance on the M-ACE was strongly correlated with that of the MMSE and Functional Activities Questionnaire (FAQ). The M-ACE was not accurate in discriminating MCI from control subjects. Conclusion: The M-ACE is a brief screening test which provided high accuracy for diagnosing AD in this sample. The suggested cut-off point in this study was 20 points for AD.

Evaluation of alpha-l-fucosidase for the diagnosis of hepatocellular carcinoma based on meta-analysis

LaboratoriumsMedizin ◽

10.1515/labmed-2019-0152 ◽

2020 ◽

Vol 0 (0) ◽

Cited By ~ 1

Author(s):

Lei Xi ◽

Chunqing Yang

Keyword(s):

Hepatocellular Carcinoma ◽

Meta Analysis ◽

Area Under The Curve ◽

Diagnostic Odds Ratio ◽

Diagnostic Value ◽

Receiver Operating Characteristic Curves ◽

Sensitivity Specificity ◽

Review Manager

AbstractObjectivesThe main aim of the present study was to assess the diagnostic value of alpha-l-fucosidase (AFU) for hepatocellular carcinoma (HCC).MethodsStudies that explored the diagnostic value of AFU in HCC were searched in EMBASE, SCI, and PUBMED. The sensitivity, specificity, and DOR about the accuracy of serum AFU in the diagnosis of HCC were pooled. The methodological quality of each article was evaluated with QUADAS-2 (quality assessment for studies of diagnostic accuracy 2). Receiver operating characteristic curves (ROC) analysis was performed. Statistical analysis was conducted by using Review Manager 5 and Open Meta-analyst.ResultsEighteen studies were selected in this study. The pooled estimates for AFU vs. α-fetoprotein (AFP) in the diagnosis of HCC in 18 studies were as follows: sensitivity of 0.7352 (0.6827, 0.7818) vs. 0.7501 (0.6725, 0.8144), and specificity of 0.7681 (0.6946, 0.8283) vs. 0.8208 (0.7586, 0.8697), diagnostic odds ratio (DOR) of 7.974(5.302, 11.993) vs. 13.401 (8.359, 21.483), area under the curve (AUC) of 0.7968 vs. 0.8451, respectively.ConclusionsAFU is comparable to AFP for the diagnosis of HCC.

Identification of an optimal prolactin threshold to determine prolactinoma size using receiver operating characteristic analysis

Scientific Reports ◽

10.1038/s41598-021-89256-7 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Bianca M. Leca ◽

Maria Mytilinaiou ◽

Marina Tsoli ◽

Andreea Epure ◽

Simon J. B. Aylwin ◽

...

Keyword(s):

Tumour Size ◽

Area Under The Curve ◽

Serum Prolactin ◽

Pituitary Neoplasms ◽

Characteristic Analysis ◽

Roc Curve Analysis ◽

Baseline Serum ◽

AbstractProlactinomas represent the most common type of secretory pituitary neoplasms, with a therapeutic management that varies considerably based on tumour size and degree of hyperprolactinemia. The aim of the current study was to evaluate the relationship between serum prolactin (PRL) concentrations and prolactinoma size, and to determine a cut-off PRL value that could differentiate micro- from macro-prolactinomas. A retrospective cohort study of 114 patients diagnosed with prolactinomas between 2007 and 2017 was conducted. All patients underwent gadolinium enhanced pituitary MRI and receiver operating characteristic (ROC) analyses were performed. 51.8% of patients in this study were men, with a mean age at the time of diagnosis of 42.32 ± 15.04 years. 48.2% of the total cohort were found to have microadenomas. Baseline serum PRL concentrations were strongly correlated to tumour dimension (r = 0.750, p = 0.001). When performing the ROC curve analysis, the area under the curve was 0.976, indicating an excellent accuracy of the diagnostic method. For a value of 204 μg/L (4338 mU/L), sensitivity and specificity were calculated at 0.932 and 0.891, respectively. When a cut off value of 204 μg/L (4338 mU/L) was used, specificity was 93.2%, and sensitivity 89.1%, acceptable to reliably differentiate between micro- and macro- adenomas.

Impact of the preoperative prognostic nutritional index as a predictor for postoperative complications after resection of locally recurrent rectal cancer

BMC Cancer ◽

10.1186/s12885-021-08160-5 ◽

2021 ◽

Vol 21 (1) ◽

Author(s):

Masakatsu Paku ◽

Mamoru Uemura ◽

Masatoshi Kitakaze ◽

Shiki Fujino ◽

Takayuki Ogino ◽

...

Keyword(s):

Risk Factors ◽

Rectal Cancer ◽

Postoperative Complications ◽

Blood Loss ◽

Odds Ratio ◽

Inflammatory Markers ◽

Recurrent Rectal Cancer ◽

Prognostic Nutrition Index

Abstract Background Local recurrence is common after curative resections for rectal cancer. Surgical intervention is among the best treatment choices. However, achieving a negative resection margin often requires extensive pelvic organ resections; thus, the postoperative complication rate is quite high. Recent studies have reported that the inflammatory index could predict postoperative complications. This study aimed to validate the correlation between clinical factors, including inflammatory markers, and severe complications after surgery for local recurrent rectal cancer. Methods This retrospective study included 99 patients that underwent radical resections for local recurrences of rectal cancer. Postoperative complications were graded according to the Clavien-Dindo classification. Grades ≥3 were defined as severe complications. Risk factors for severe complications were identified with univariate and multivariate logistic regression models and assessed with receiver-operating characteristic curves. Results Severe postoperative complications occurred in 38 patients (38.4%). Analyses of correlations between inflammatory markers and severe postoperative complications revealed that the strongest correlation was found between the prognostic nutrition index and severe postoperative complications. The receiver-operating characteristic analysis showed that the optimal prognostic nutrition index cut-off value was 42.2 (sensitivity: 0.790, specificity: 0.508). In univariate and multivariate analyses, a prognostic nutrition index ≤44.2 (Odds ratio: 3.007, 95%CI:1.171–8.255, p = 0.02) and a blood loss ≥2850 mL (Odds ratio: 2.545, 95%CI: 1.044–6.367, p = 0.04) were associated with a significantly higher incidence of severe postoperative complications. Conclusions We found that a low preoperative prognostic nutrition index and excessive intraoperative blood loss were risk factors for severe complications after surgery for local recurrent rectal cancer.

Risikoabschätzung einer akuten Exazerbation bei COPD-Patienten im Rahmen einer pneumologischen Anschluss-Rehabilitation anhand der Prävalenz und Schwergradausprägung von Komorbiditäten

Pneumologie ◽

10.1055/a-1346-5504 ◽

2021 ◽

Author(s):

P. Luu ◽

S. Tulka ◽

S. Knippschild ◽

W. Windisch ◽

M. Spielmanns

Keyword(s):

Charlson Comorbidity Index ◽

Rating Scale ◽

Area Under The Curve ◽

Sensitivität Und Spezifität ◽

Comorbidity Index ◽

Zusammenfassung Einleitung Akute COPD-Exazerbationen (AECOPD) im Rahmen einer pneumologischen Rehabilitation (PR) sind häufige und gefährliche Komplikationen. Neben Einschränkungen der Lebensqualität führen sie zu einem Unterbrechung der PR und gefährden den PR-Erfolg. Eine Abhängigkeit zwischen dem Krankheitsstatus und einem erhöhten Risiko für eine AECOPD ist beschrieben. Dabei stellt sich die Frage, ob der Charlson Comorbidity Index (CCI) oder die Cumulative Illness Rating Scale (CIRS) dafür geeignet sind, besonders exazerbationsgefährdete COPD-Patienten in der PR im Vorfeld zu detektieren. Patienten und Methoden In einer retrospektiven Untersuchung wurden die Daten von COPD-Patienten, welche im Jahr 2018 eine PR erhielten, analysiert. Primärer Endpunkt der Untersuchung war die Punktzahl im CCI. Alle Daten wurden dem Klinikinformationssystem Phönix entnommen und COPD-Exazerbationen erfasst. Die laut Fallzahlplanung benötigten 44 Patienten wurden zufällig (mittels Zufallsliste für jede Gruppe) aus diesem Datenpool rekrutiert: 22 Patienten mit und 22 ohne Exazerbation während der PR. CCI und CIRS wurden für die eingeschlossenen Fälle für beide Gruppen bestimmt. Die Auswertung des primären Endpunktes (CCI) erfolgte durch den Gruppenvergleich der arithmetischen Mittel und der Signifikanzprüfung (Welch-Tests). Weitere statistische Lage- und Streuungsmaße wurden ergänzt (Median, Quartile, Standardabweichung).Zusätzlich wurde mittels Receiver Operating Characteristic (ROC)-Analyse sowohl für den CCI als auch für den CIRS ein optimaler Cutpoint zur Diskriminierung in AECOPD- und Nicht-AECOPD-Patienten gesucht. Ergebnisse 244 COPD-Patienten erhielten eine stationäre PR von durchschnittlich 21 Tagen, wovon 59 (24 %) während der PR eine behandlungspflichtige AECOPD erlitten. Die ausgewählten 22 Patienten mit einer AECOPD hatten einen mittleren CCI von 6,77 (SD: 1,97) und die 22 Patienten ohne AECOPD von 4,32 (SD: 1,17). Die Differenz von –2,45 war zu einem Signifikanzniveau von 5 % statistisch signifikant (p < 0,001; 95 %-KI: [–3,45 ; –1,46]). Die ROC-Analyse zeigte einen optimalen Cutpoint für den CCI bei 6 mit einer Sensitivität zur Feststellung einer AECOPD von 81,8 % und einer Spezifität von 86.,4 % mit einem Wert der AUC (area under the curve) von 0,87. Der optimale Cutpoint für den CIRS war 19 mit einer Sensitivität von 50 %, einer Spezifität von 77,2 % und einer AUC von 0,65. Schlussfolgerung COPD-Patienten mit einer akuten Exazerbation während der pneumologischen Rehabilitation haben einen höheren CCI. Mithilfe des CCI lässt sich mit einer hohen Sensitivität und Spezifität das Risiko einer AECOPD von COPD-Patienten im Rahmen eines stationären PR-Programms einschätzen.

The Optimal Cutoff Value of D-dimer Levels to Predict in Hospital Mortality in Severe Cases of Coronavirus Disease 2019

Open Access Macedonian Journal of Medical Sciences ◽

10.3889/oamjms.2021.7600 ◽

2021 ◽

Vol 9 (B) ◽

pp. 1561-1564

Author(s):

Ngakan Ketut Wira Suastika ◽

Ketut Suega

Keyword(s):

Hospital Mortality ◽

Receiver Operating Characteristic Curve ◽

Characteristic Curve ◽

Area Under The Curve ◽

Optimal Cutoff ◽

Cutoff Value ◽

Operating Characteristic Curve ◽

D Dimer

Introduction: Coronavirus disease 2019 (Covid-19) can cause coagulation parameters abnormalities such as an increase of D-dimer levels especially in severe cases. The purpose of this study is to determine the differences of D-dimer levels in severe cases of Covid-19 who survived and non-survived and determine the optimal cut-off value of D-dimer levels to predict in-hospital mortality. Method: Data were obtained from confirmed Covid-19 patients who were treated from June to September 2020. The Mann-Whitney U test was used to determine differences of D-dimer levels in surviving and non-surviving patients. The optimal cut-off value and area under the curve (AUC) of the D-dimer level in predicting mortality were obtained by the receiver operating characteristic curve (ROC) method. Results: A total of 80 patients were recruited in this study. Levels of D-dimer were significantly higher in non-surviving patients (median 3.346 mg/ml; minimum – maximum: 0.939 – 50.000 mg/ml) compared to surviving patients (median 1.201 mg/ml; minimum – maximum: 0.302 – 29.425 mg/ml), p = 0.012. D-dimer levels higher than 1.500 mg/ml are the optimal cut-off value for predicting mortality in severe cases of Covid-19 with a sensitivity of 80.0%; specificity of 64.3%; and area under the curve of 0.754 (95% CI 0.586 - 0.921; p = 0.010). Conclusions: D-dimer levels can be used as a predictor of mortality in severe cases of Covid-19.