Sensitivity, specificity and ROC-curves in multiple reader diagnostic trials—A unified, nonparametric approach

Abstract Purpose To assess Strain Ratio (SRE) and Shear Wave Elastography (SWE) accuracy alone and with TIRADS classification, for the risk stratification of indeterminate thyroid nodules. Materials and methods 128 Patients with 128 indeterminate nodules candidates for thyroidectomy underwent preoperative staging neck ultrasound and were classified according to K-TIRADS score. After TIRADS evaluation, semi-quantitative (SRE) and quantitative (SWE expressed in kPa) elastosonography were performed and relative diagnostic performances, alone and in combination, were compared through ROC curves analysis. In order to maximize the SRE and SWE sensitivity and specificity, their cut-off values were calculated using the Liu test. Bonferroni test was used to evaluate statistically significant differences with a p value < 0.05. Results Sensitivity, specificity, PPV and NPV were, respectively, 71.4%, 82.4%, 62.5%, 87.5% for K-TIRADS baseline US, 85.7%, 94.1%, 85.7%, 94.1% for SRE and 57.1%, 79.4%, 53.3%, 81.8% for SWE (kPa expressed). SRE evaluation showed the best diagnostic accuracy compared to the SWE (kPa expressed) (p < 0.05) and to the K-TIRADS (p > 0.05). The association of SRE with conventional ultrasound with K-TIRADS score increased sensitivity (92.9% vs 71.4%) but decreased the specificity than conventional US alone (76.5% vs 82.4%). Conclusion Strain Elastosonography can be associated with K-TIRADS US examination in the thyroid nodule characterization with indeterminate cytology; in fact, adding the SRE to K-TIRADS assessment significantly increases its sensitivity and negative predictive value. However, further multicenter studies on larger population are warranted.

Download Full-text

The Child Behavior Checklist as a Screening Instrument for PTSD in Refugee Children

Children ◽

10.3390/children8060521 ◽

2021 ◽

Vol 8 (6) ◽

pp. 521

Author(s):

Ina Nehring ◽

Heribert Sattel ◽

Maesa Al-Hallak ◽

Martin Sack ◽

Peter Henningsen ◽

...

Keyword(s):

Child Behavior ◽

Child Behavior Checklist ◽

High Sensitivity ◽

Roc Curves ◽

Structured Interview ◽

Screening Instrument ◽

Screening Tools ◽

Refugee Children ◽

Post Traumatic Stress ◽

Sensitivity Specificity

Thousands of refugees who have entered Europe experienced threatening conditions, potentially leading to post traumatic stress disorder (PTSD), which has to be detected and treated early to avoid chronic manifestation, especially in children. We aimed to evaluate and test suitable screening tools to detect PTSD in children. Syrian refugee children aged 4–14 years were examined using the PTSD-semi-structured interview, the Kinder-DIPS, and the Child Behavior Checklist (CBCL). The latter was evaluated as a potential screening tool for PTSD using (i) the CBCL-PTSD subscale and (ii) an alternative subscale consisting of a psychometrically guided selection of items with an appropriate correlation to PTSD and a sufficient prevalence (presence in more than 20% of the cases with PTSD). For both tools we calculated sensitivity, specificity, and a receiver operating characteristic (ROC) curve. Depending on the sum score of the items, the 20-item CBCL-PTSD subscale as used in previous studies yielded a maximal sensitivity of 85% and specificity of 76%. The psychometrically guided item selection resulted in a sensitivity of 85% and a specificity of 83%. The areas under the ROC curves were the same for both tools (0.9). Both subscales may be suitable as screening instrument for PTSD in refugee children, as they reveal a high sensitivity and specificity.

Download Full-text

Beijing Friendship Hospital Osteoporosis Self-Assessment Tool for Elderly Male (BFH-OSTM) vs Fracture Risk Assessment Tool (FRAX) for identifying painful new osteoporotic vertebral fractures in older Chinese men: a cross-sectional study

BMC Musculoskeletal Disorders ◽

10.1186/s12891-021-04476-2 ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Ning An ◽

Ji Sheng Lin ◽

Qi Fei

Keyword(s):

Assessment Tool ◽

Roc Curves ◽

Cross Sectional Study ◽

Self Assessment ◽

Cross Sectional ◽

Fracture Group ◽

Friendship Hospital ◽

Fracture Risk Assessment Tool ◽

Older Chinese ◽

Sensitivity Specificity

Abstract Background To compare the validation of four tools for identifying painful new osteoporotic vertebral compression fractures (PNOVCFs) in older Chinese men: bone mineral density (BMD), Asian osteoporosis self-assessment tool (OSTA), World Health Organization fracture risk assessment tool (FRAX) (without BMD) and Beijing Friendship Hospital Osteoporosis Self-Assessment Tool (BFH-OSTM). Methods A cross sectional study was conducted from 2013 to 2019. A total of 846 men aged ≥50 were included and were divided into two groups: Fracture Group (patients with PNOVCFs underwent percutaneous vertebroplasty surgery) and Non-Fracture Group (community dwelled subjects for healthy examination). All subjects accepted a dual-energy X-ray BMD test and a structured questionnaire. The results of BMD, OSTA, FRAX and BFH-OSTM scores were assessed and receiver-operating characteristic (ROC) curves were generated to compare the validity of four tools for identifying PNOVCFs. Optimal cutoff points, sensitivity, specificity, and areas under the ROC curves (AUCs) were determined. Results There were significant differences including BMD T score (femoral neck, total hip and L1-L4), OSTA, FRAX and BFH-OSTM scores between Fracture group and Non-fracture group. Compared to BMD and OSTA, BFH-OSTM and FRAX had better predictive value, the sensitivity, specificity and AUC value are 0.841, 81.29%, 70.67% and 0.796, 74.85%, 78.52%, respectively. Compared with FRAX, the BFH-OSTM has a better AUC value. Conclusions Both BFH-OSTM and FRAX can be used to identify POVCFs, However, BFH-OSTM model may be a more simple and effective tool to identify the risk of POVCFs in Chinese elderly men.

Download Full-text

Diagnostic performances of manual and automated reticulocyte parameters in anaemic cats

Journal of Feline Medicine and Surgery ◽

10.1177/1098612x17699067 ◽

2017 ◽

Vol 20 (2) ◽

pp. 122-127 ◽

Cited By ~ 2

Author(s):

Saverio Paltrinieri ◽

Marco Fossati ◽

Valentina Menaballi

Keyword(s):

Receiver Operating Characteristic ◽

Sensitivity And Specificity ◽

Operating Characteristic ◽

Positive Likelihood Ratio ◽

Roc Curves ◽

Blood Samples ◽

Production Index ◽

Instrumental Measurement ◽

Sensitivity Specificity ◽

Reticulocyte Production

Objectives The objective of this study was to evaluate the diagnostic performances of manual and instrumental measurement of reticulocyte percentage (Ret%), reticulocyte number (Ret#) and reticulocyte production index (RPI) to differentiate regenerative anaemia (RA) from non-regenerative anaemia (NRA) in cats. Methods Data from 106 blood samples from anaemic cats with manual counts (n = 74; 68 NRA, six RA) or instrumental counts of reticulocytes (n = 32; 25 NRA, seven RA) collected between 1995 and 2013 were retrospectively analysed. Sensitivity, specificity and positive likelihood ratio (LR+) were calculated using either cut-offs reported in the literature or cut-offs determined from receiver operating characteristic (ROC) curves. Results All the reticulocyte parameters were significantly higher in cats with RA than in cats with NRA. All the ROC curves were significantly different ( P <0.001) from the line of no discrimination, without significant differences between the three parameters. Using the cut-offs published in literature, the Ret% (cut-off: 0.5%) was sensitive (100%) but not specific (<75%), the RPI (cut-off: 1.0) was specific (>92%) but not sensitive (<15%), and the Ret# (cut-off: 50 × 10³/µl) had a sensitivity and specificity >80% and the highest LR+ (manual count: 14; instrumental count: 6). For all the parameters, sensitivity and specificity approached 100% using the cut-offs determined by the ROC curves. These cut-offs were higher than those reported in the literature for Ret% (manual: 1.70%; instrumental: 3.06%), lower for RPI (manual: 0.39; instrumental: 0.59) and variably different, depending on the method (manual: 41 × 10³/µl; instrumental: 57 × 10³/µl), for Ret#. Using these cut-offs, the RPI had the highest LR+ (manual: 22.7; instrumental: 12.5). Conclusions and relevance This study indicated that all the reticulocyte parameters may confirm regeneration when the pretest probability is high, while when this probability is moderate, RA should be identified using the RPI providing that cut-offs <1.0 are used.

Download Full-text

MR-Proadrenomedullin as biomarker of renal damage in urinary tract infection in children

BMC Pediatrics ◽

10.1186/s12887-021-02765-2 ◽

2021 ◽

Vol 21 (1) ◽

Author(s):

Rafael Peñalver Penedo ◽

Marta Rupérez Lucas ◽

Luis Antonio Álvarez-Sala Walther ◽

Alicia Torregrosa Benavent ◽

María Luisa Casas Losada ◽

...

Keyword(s):

Urinary Tract Infection ◽

Urinary Tract ◽

Tract Infection ◽

Pediatric Patients ◽

Pediatric Population ◽

Adult Population ◽

Roc Curves ◽

Renal Scintigraphy ◽

Reactive Protein ◽

Sensitivity Specificity

Abstract Background Midregional-proadrenomedullin (MR-proADM) is a useful prognostic peptide in severe infectious pathologies in the adult population. However, there are no studies that analyze its utility in febrile urinary tract infection (fUTI) in children. An accurate biomarker would provide an early detection of patients with kidney damage, avoiding other invasive tests like renal scintigraphy scans. Our objective is to study the usefulness of MR-proADM as a biomarker of acute and chronic renal parenchymal damage in fUTI within the pediatric population. Methods A prospective cohort study was conducted in pediatric patients with fUTI between January 2015 and December 2018. Plasma and urine MR-proADM levels were measured at admission in addition to other laboratory parameters. After confirmation of fUTI, renal scintigraphy scans were performed during the acute and follow-up stages. A descriptive study has been carried out and sensitivity, specificity and ROC curves for MR-proADM, C-reactive protein, and procalcitonin were calculated. Results 62 pediatric patients (34 female) were enrolled. Scintigraphy showed acute pyelonephritis in 35 patients (56.5%). Of those patients, the median of plasmatic MR-proADM (P-MR-proADM) showed no differences compared to patients without pyelonephritis. 7 patients (11.3%) developed renal scars (RS). Their median P-MR-proADM levels were 1.07 nmol/L (IQR 0.66–1.59), while in patients without RS were 0.48 nmol/L (0.43–0.63) (p < 0.01). The AUC in this case was 0.92 (95% CI 0.77–0.99). We established an optimal cut-off point at 0.66 nmol/L with sensitivity 83.3% and specificity 81.8%. Conclusion MR-ProADM has demonstrated a poor ability to diagnose pyelonephritis in pediatric patients with fUTI. However, P-MR-proADM proved to be a very reliable biomarker for RS prediction.

Download Full-text

Identification of the most specific markers to differentiate primary pulmonary carcinoma from metastatic gastrointestinal carcinoma to the lung

Diagnostic Pathology ◽

10.1186/s13000-021-01184-2 ◽

2022 ◽

Vol 17 (1) ◽

Author(s):

Bachar Alabdullah ◽

Amir Hadji-Ashrafy

Keyword(s):

Gastrointestinal Tract ◽

Predictive Value ◽

Regression Models ◽

Area Under The Curve ◽

Roc Curves ◽

Multivariate Models ◽

Gastrointestinal Carcinoma ◽

Lung Tumours ◽

Napsin A ◽

Sensitivity Specificity

Abstract Background A number of biomarkers have the potential of differentiating between primary lung tumours and secondary lung tumours from the gastrointestinal tract, however, a standardised panel for that purpose does not exist yet. We aimed to identify the smallest panel that is most sensitive and specific at differentiating between primary lung tumours and secondary lung tumours from the gastrointestinal tract. Methods A total of 170 samples were collected, including 140 primary and 30 non-primary lung tumours and staining for CK7, Napsin-A, TTF1, CK20, CDX2, and SATB2 was performed via tissue microarray. The data was then analysed using univariate regression models and a combination of multivariate regression models and Receiver Operating Characteristic (ROC) curves. Results Univariate regression models confirmed the 6 biomarkers’ ability to independently predict the primary outcome (p < 0.001). Multivariate models of 2-biomarker combinations identified 11 combinations with statistically significant odds ratios (ORs) (p < 0.05), of which TTF1/CDX2 had the highest area under the curve (AUC) (0.983, 0.960–1.000 95% CI). The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 75.7, 100, 100, and 37.5% respectively. Multivariate models of 3-biomarker combinations identified 4 combinations with statistically significant ORs (p < 0.05), of which CK7/CK20/SATB2 had the highest AUC (0.965, 0.930–1.000 95% CI). The sensitivity, specificity, PPV, and NPV were 85.1, 100, 100, and 41.7% respectively. Multivariate models of 4-biomarker combinations did not identify any combinations with statistically significant ORs (p < 0.05). Conclusions The analysis identified the combination of CK7/CK20/SATB2 to be the smallest panel with the highest sensitivity (85.1%) and specificity (100%) for predicting tumour origin with an ROC AUC of 0.965 (p < 0.001; SE: 0.018, 0.930–1.000 95% CI).

Download Full-text

P227 Defining faecal calprotectin thresholds to predict endoscopic and histological healing in ulcerative colitis (UC) by using advanced optical enhancement techniques

Journal of Crohn s and Colitis ◽

10.1093/ecco-jcc/jjz203.356 ◽

2020 ◽

Vol 14 (Supplement_1) ◽

pp. S255-S256

Author(s):

R Cannatelli ◽

D Zardo ◽

O Nardone ◽

A Bazarova ◽

U Shivaji ◽

...

Keyword(s):

Ulcerative Colitis ◽

Surrogate Marker ◽

Scoring Systems ◽

Roc Curves ◽

Mucosal Healing ◽

Faecal Calprotectin ◽

Area Under Roc Curve ◽

Index Of Severity ◽

Sensitivity Specificity ◽

A Prospective Study

Abstract Background Faecal calprotectin (FC) is the most common surrogate marker of mucosal healing (MH) in UC. A number of endoscopic and histologic scoring systems in UC have been developed for defining MH. We report the optimum FC thresholds for defining MH using all the assessment methods. Methods In a prospective study we collected all clinical, endoscopic and histologic data and FC from 76 UC patients (mean age 44.2y, 50.0% male) who attended endoscopy unit for colitis assessment or surveillance. Endoscopic scores were determined by the same endoscopist (MI) and included Mayo Endoscopic Score (MES), Ulcerative Colitis Endoscopic Index of Severity (UCEIS) and PICaSSO (Paddington International virtual ChromoendoScopy ScOre). Histological activity was scored by the Robarts Histology Index (RHI) and Nancy Index by the same pathologist (DZ). Faecal calprotectin was assayed using Buhlmann faecal turbo test, particle enhanced turbidimetric immunoassay. ROC curves were performed to evaluate sensitivity, specificity and accuracy of the optimum cut-off of FC to predict endoscopic and histological healing. Results The best cut-off for FC to predict endoscopic healing calculated as Picasso≤3 was 161 μg/g with Area Under ROC curve (AUROC) of 85.3% (95% CI 76.2, 94.4). Sensitivity, specificity and accuracy were 87.9% (95% CI 57.6, 100), 76.7% (95% CI 53.5, 90.7) and 81.6% (95% CI 68.4, 89.5), respectively. While, the best threshold of FC to predict UCEIS≤1 was 148 μg/g with AUROC of 89.2 (95% CI 81.9, 96.5). Sensitivity was 93.5% (95% CI 50.5, 100), specificity 82.2% (95% CI 53.3, 91.1) and accuracy 86.8% (95% CI 69.7, 92.1). The best threshold for FC to predict MES equal to 0, was 112 μg/g, with AUROC of 89.6 μg/g, (95% CI 82.5, 96.7). Sensitivity, specificity and accuracy were 89.7%ww (95% CI 39.2, 100), 85.1% (95% CI 55.3, 93.6) and 86.9% (95% CI 68.4, 92.1), respectively. The best value of FC to predict histological healing with RHI≤3 was 112μg/g with AUROC of 88.0% (95% CI 80.6, 95.4). Sensitivity, specificity and accuracy were 88.5% (95% CI 53.8, 100), 80.0% (95% CI 62.0, 90.0) and 82.9% (95% CI 72.5, 89.5), respectively. When used Nancy≤1 FC cut-off to predict healing was 172 μg/g with AUROC of 87.1% (95% CI 78.6, 95.6). Sensitivity was 96.4% (95% CI 60.7, 100), specificity 72.9% (54.2, 85.4) and accuracy 81.6% (69.7, 89.5). Conclusion Advanced enhancement technologies can accurately define the level of FC to predict endoscopic and histological healing in UC. The optimum FC threshold for MH by PICaSSO and by Nancy was similar (161 and 172 μg/g respectively), while the FC threshold for mucosal healing by MES and by RHI was 112 μg/g. The FC threshold for determining MH in clinical practice should be lower than at least 200 μg/g.

Download Full-text

Assessment of Endoscopic Healing by Using Advanced Technologies Reflects Histological Healing in Ulcerative Colitis

Journal of Crohn s and Colitis ◽

10.1093/ecco-jcc/jjaa056 ◽

2020 ◽

Vol 14 (9) ◽

pp. 1282-1289 ◽

Cited By ~ 2

Author(s):

Marietta Iacucci ◽

Rosanna Cannatelli ◽

Xianyong Gui ◽

Davide Zardo ◽

Alina Bazarova ◽

...

Keyword(s):

Ulcerative Colitis ◽

Roc Curves ◽

Mucosal Healing ◽

Confocal Laser Endomicroscopy ◽

Confocal Laser ◽

High Definition ◽

Endoscopic Score ◽

Virtual Chromoendoscopy ◽

Sensitivity Specificity ◽

A Prospective Study

Abstract Background Several studies have reported that ulcerative colitis [UC] patients with endoscopic mucosal healing may still have histological inflammation. We investigated the relationship between mucosal healing defined by modified PICaSSO [Paddington International Virtual ChromoendoScopy ScOre], Mayo Endoscopic Score [MES] and probe-based confocal laser endomicroscopy [pCLE] with histological indices in UC. Methods A prospective study enrolling 82 UC patients [male 66%] was conducted. High-definition colonoscopy was performed to evaluate the activity of the disease with MES assessed with High-Definition MES [HD-MES] and modified PICaSSO and targeted biopsies were taken; pCLE was then performed. Receiver operating characteristic [ROC] curves were plotted to determine the best thresholds for modified PICaSSO and pCLE scores that predicted histological healing according to the Robarts Histopathology Index [RHI] and ECAP ‘Extension, Chronicity, Activity, Plus’ histology score. Results A modified PICaSSO of ≤ 4 predicted histological healing at RHI ≤ 3, with sensitivity, specificity, accuracy and area under the ROC curve [AUROC] of 89.8%, 95.7%, 91.5% and 95.9% respectively. The sensitivity, specificity, accuracy and AUROC of HD-MES to predict histological healing by RHI were 81.4%, 95.7%, 85.4% and 92.1%, respectively. A pCLE ≤ 10 predicted histological healing with sensitivity of 94.9%, specificity of 91.3%, accuracy of 93.9% and AUROC of 96.5%. An ECAP of ≤ 10 was predicted by modified PICaSSO ≤ 4 with accuracy of 91.5% and AUROC of 95.9%. Conclusion Histological healing by RHI and ECAP is accurately predicted by HD-MES and modified virtual electronic chromoendoscopy PICaSSO, endoscopic score; and the use of pCLE did not improve the accuracy any further.

Download Full-text

Deep-learning for predicting C-shaped canals in mandibular second molars on panoramic radiographs

Dentomaxillofacial Radiology ◽

10.1259/dmfr.20200513 ◽

2021 ◽

pp. 20200513

Author(s):

Su-Jin Jeon ◽

Jong-Pil Yun ◽

Han-Gyeol Yeom ◽

Woo-Sang Shin ◽

Jong-Hyun Lee ◽

...

Keyword(s):

Deep Learning ◽

Cone Beam Ct ◽

Area Under The Curve ◽

Roc Curves ◽

Learning System ◽

Operating Characteristics ◽

Panoramic Radiographs ◽

Test Sets ◽

Sensitivity Specificity ◽

Deep Learning Model

Objective: The aim of this study was to evaluate the use of a convolutional neural network (CNN) system for predicting C-shaped canals in mandibular second molars on panoramic radiographs. Methods: Panoramic and cone beam CT (CBCT) images obtained from June 2018 to May 2020 were screened and 1020 patients were selected. Our dataset of 2040 sound mandibular second molars comprised 887 C-shaped canals and 1153 non-C-shaped canals. To confirm the presence of a C-shaped canal, CBCT images were analyzed by a radiologist and set as the gold standard. A CNN-based deep-learning model for predicting C-shaped canals was built using Xception. The training and test sets were set to 80 to 20%, respectively. Diagnostic performance was evaluated using accuracy, sensitivity, specificity, and precision. Receiver-operating characteristics (ROC) curves were drawn, and the area under the curve (AUC) values were calculated. Further, gradient-weighted class activation maps (Grad-CAM) were generated to localize the anatomy that contributed to the predictions. Results: The accuracy, sensitivity, specificity, and precision of the CNN model were 95.1, 92.7, 97.0, and 95.9%, respectively. Grad-CAM analysis showed that the CNN model mainly identified root canal shapes converging into the apex to predict the C-shaped canals, while the root furcation was predominantly used for predicting the non-C-shaped canals. Conclusions: The deep-learning system had significant accuracy in predicting C-shaped canals of mandibular second molars on panoramic radiographs.

Download Full-text