Evaluation of automated microvascular flow analysis software AVA 4: a validation study

Abstract Background Real-time automated analysis of videos of the microvasculature is an essential step in the development of research protocols and clinical algorithms that incorporate point-of-care microvascular analysis. In response to the call for validation studies of available automated analysis software by the European Society of Intensive Care Medicine, and building on a previous validation study in sheep, we report the first human validation study of AVA 4. Methods Two retrospective perioperative datasets of human microcirculation videos (P1 and P2) and one prospective healthy volunteer dataset (V1) were used in this validation study. Video quality was assessed using the modified Microcirculation Image Quality Selection (MIQS) score. Videos were initially analyzed with (1) AVA software 3.2 by two experienced investigators using the gold standard semi-automated method, followed by an analysis with (2) AVA automated software 4.1. Microvascular variables measured were perfused vessel density (PVD), total vessel density (TVD), and proportion of perfused vessels (PPV). Bland–Altman analysis and intraclass correlation coefficients (ICC) were used to measure agreement between the two methods. Each method’s ability to discriminate between microcirculatory states before and after induction of general anesthesia was assessed using paired t-tests. Results Fifty-two videos from P1, 128 videos from P2 and 26 videos from V1 met inclusion criteria for analysis. Correlational analysis and Bland–Altman analysis revealed poor agreement and no correlation between AVA 4.1 and AVA 3.2. Following the induction of general anesthesia, TVD and PVD measured using AVA 3.2 increased significantly for P1 (p < 0.05) and P2 (p < 0.05). However, these changes could not be replicated with the data generated by AVA 4.1. Conclusions AVA 4.1 is not a suitable tool for research or clinical purposes at this time. Future validation studies of automated microvascular flow analysis software should aim to measure the new software’s agreement with the gold standard, its ability to discriminate between clinical states and the quality thresholds at which its performance becomes unacceptable.

Download Full-text

Evaluation of Automated Microvascular Flow Analysis Software AVA 4: A Validation Study

10.21203/rs.3.rs-71474/v2 ◽

2021 ◽

Author(s):

Christian S. Guay ◽

Mariam Khebir ◽

Shiva Shahiri ◽

Ariana Szilagyi ◽

Erin Elizabeth Cole ◽

...

Keyword(s):

General Anesthesia ◽

Validation Study ◽

Validation Studies ◽

Gold Standard ◽

Flow Analysis ◽

Automated Analysis ◽

Altman Analysis ◽

Analysis Software ◽

Bland Altman Analysis ◽

Microvascular Flow

Abstract BACKGROUND: Real-time automated analysis of videos of the microvasculature is an essential step in the development of research protocols and clinical algorithms that incorporate point-of-care microvascular analysis. In response to the call for validation studies of available automated analysis software by the European Society of Intensive Care Medicine, and building on a previous validation study in sheep, we report the first human validation study of AVA 4.METHODS:Two retrospective perioperative datasets of human microcirculation videos (P1 and P2) and one prospective healthy volunteer dataset (V1) were used in this validation study. Video quality was assessed using the modified Microcirculation Image Quality Selection (MIQS) score. Videos were initially analyzed with (1) AVA software 3.2 by two experienced investigators using the gold standard semi-automated method, followed by an analysis with (2) AVA automated software 4.1. Microvascular variables measured were perfused vessel density (PVD), total vessel density (TVD), and proportion of perfused vessels (PPV). Bland-Altman analysis and intraclass correlation coefficients (ICC) were used to measure agreement between the two methods. Each method’s ability to discriminate between microcirculatory states before and after induction of general anesthesia was assessed using paired t-tests. RESULTS: Fifty-two videos from P1, 128 videos from P2 and 26 videos from V1 met inclusion criteria for analysis. Correlational analysis and Bland Altman analysis revealed poor agreement and no correlation between AVA 4.1 and AVA 3.2. Following the induction of general anesthesia, TVD and PVD measured using AVA 3.2 increased significantly for P1 (p < 0.05) and P2 (p < 0.05). However, these changes could not be replicated with the data generated by AVA 4.1. CONCLUSIONS: AVA 4.1 is not a suitable tool for research or clinical purposes at this time. Future validation studies of automated microvascular flow analysis software should aim to measure the new software’s agreement with the gold standard, its ability to discriminate between clinical states and the quality thresholds at which its performance becomes unacceptable.

Download Full-text

Evaluation of Automated Microvascular Flow Analysis Software AVA 4.0: A Validation Study

10.21203/rs.3.rs-71474/v1 ◽

2020 ◽

Author(s):

Christian S. Guay ◽

Mariam Khebir ◽

Shiva Shahiri ◽

Ariana Szilagyi ◽

Erin Elizabeth Cole ◽

...

Keyword(s):

Validation Study ◽

Validation Studies ◽

Intraclass Correlation ◽

Flow Analysis ◽

Automated Analysis ◽

Vessel Density ◽

Altman Analysis ◽

Analysis Software ◽

Bland Altman Analysis ◽

Microvascular Flow

Abstract BACKGROUND: Real-time automated analysis of videos of the microvasculature is an essential step in the development of research protocols and clinical algorithms that incorporate point-of-care microvascular analysis. Validation studies on the software packages developed to perform these analyses have reported low agreement with the current referent standard semi-automated analysis method. In response to the call for validation studies of available automated analysis software by the European Society of Intensive Care Medicine, we report the first human validation study of AVA 4.0. METHODS: Two retrospective perioperative datasets of human microcirculation videos (P1 and P2) and one prospective healthy volunteer dataset (V1) were used. Video quality was assessed using the Microcirculation Image Quality Selection (MIQS) score. Videos were initially analysed with (1) AVA software 3.2 by two experienced users through a semi-manual method, followed by an analysis with (2) AVA automated software 4.0 for perfused vessel density (PVD), total vessel density (TVD), and proportion of perfused vessels (PPV). Bland-Altman analysis and intraclass correlation coefficients (ICC) were used to measure agreement between the two methods. Each method’s ability to discriminate between microcirculatory states before and after induction of general anesthesia was assessed using paired t-tests. RESULTS: Fifty-two videos from P1, 128 videos from P2 and 26 videos from V1 met inclusion criteria for analysis. Correlational analysis and Bland Altman analysis revealed poor agreement and no correlation between AVA 4.0 and AVA 3.2. Increasing video length did not improve agreement. Automated analysis consistently underestimated measures of vessel density. Following the induction of anesthesia, TVD and PVD measured using AVA 3.2 increased significantly for P1 and P2 (p < 0.05). However, these changes could not be replicated with the data generated by AVA 4.0. CONCLUSIONS: AVA 4.0 is not a suitable tool for research or clinical purposes at this time. Future validation studies of automated microvascular flow analysis software should aim to measure the new software’s agreement with the referent standard, its ability to discriminate between clinical states and the quality thresholds at which its performance becomes unacceptable.

Download Full-text

Assessing the relative validity of the Scottish Collaborative Group FFQ for measuring dietary intake in adults

Public Health Nutrition ◽

10.1017/s1368980016002421 ◽

2016 ◽

Vol 20 (3) ◽

pp. 449-455 ◽

Cited By ~ 9

Author(s):

Jenna L Hollis ◽

Leone CA Craig ◽

Stephen Whybrow ◽

Heather Clark ◽

Janet AM Kyle ◽

...

Keyword(s):

Validation Study ◽

Large Scale ◽

Weighted Kappa ◽

Collaborative Group ◽

Altman Analysis ◽

Food Diary ◽

Cross Sectional ◽

Relative Validity ◽

Bland Altman Analysis ◽

Spearman’S Correlation

AbstractObjectiveTo assess the relative validity of the latest version of the Scottish Collaborative Group (SCG) FFQ (version 6.6) in adults living in Scotland.DesignA cross-sectional validation study. Participants completed the self-administered, 169-item SCG FFQ followed by a 7 d, non-weighed food diary. Energy and energy-adjusted macronutrients and micronutrients were examined for relative validity through Spearman’s correlation, the percentage of classification into thirds of intake, Cohen’s weighted kappa (κw) and Bland–Altman analysis.SettingGeneral population living in Scotland.SubjectsNinety-six adults aged 18–65 years.ResultsSpearman’s correlation coefficients ranged from 0·21 (retinol) to 0·71 (Mg). A median of 52 % of adults were correctly classified into thirds of intake (range: 42 % (PUFA, MUFA and Fe) to 64 % (percentage energy from carbohydrates)) and 8 % were grossly misclassified into opposite thirds of intake (range: 3 % (carbohydrates, percentage energy from carbohydrates) to 19 % (thiamin)). Values of κw ranged between 0·20 (PUFA, β-carotene) to 0·55 (percentage energy from carbohydrates). In the Bland–Altman analysis, the smallest limits of agreement, when expressed as a percentage of the mean intake from the FFQ and food diary, were seen for the main macronutrients carbohydrates, fat and protein.ConclusionsAs in the previous validation study more than 10 years ago, the FFQ gave higher estimates of energy and most nutrients than the food diary, but after adjustment for energy intake the FFQ could be used in place of non-weighed food diaries for most macronutrients and many micronutrients in large-scale epidemiological studies.

Download Full-text

Comparing School-Aged Refraction Measurements Using the 2WIN-S Portable Refractor in Relation to Cycloplegic Retinoscopy: A Cross-Sectional Study

Journal of Ophthalmology ◽

10.1155/2021/6612476 ◽

2021 ◽

Vol 2021 ◽

pp. 1-8

Author(s):

Ziming Liu ◽

Emmanuel Eric Pazo ◽

Hong Ye ◽

Cui Yu ◽

Ling Xu ◽

...

Keyword(s):

Gold Standard ◽

Intraclass Correlation ◽

Cross Sectional Study ◽

Mean Value ◽

Spherical Equivalent ◽

Sectional Study ◽

Altman Analysis ◽

Cross Sectional ◽

Bland Altman Analysis ◽

The Mean

Purpose. To assess the repeatability and agreement of refractive measurements using 2WIN-S photoscreening with the gold-standard cycloplegic retinoscope refraction. Design. Single centre, cross-sectional study. Methods. Spherical, cylindrical, axis, and spherical equivalent of 194 bilateral eyes of 97 children were assessed using a retinoscope and 2WIN-S. One week later, another operator repeated the 2WIN-S measurements. The primary outcome measures were to assess the repeatability and agreement between spherical equivalent, J0, and J45 readings of 2WIN-S. The repeatability of measurements was assessed by the within-subject standard deviation (2.77 Sw) and intraclass correlation coefficient (ICC). The agreement between devices was assessed using 95% limits of agreement. The extent of the agreement between cycloplegic retinoscopy and noncycloplegic 2WIN-S measurements was assessed using Bland–Altman analysis. Results. The mean age ± SD was 10.3 ± 2.46 year (range, 4–14 years). The sphere, cylinder, and spherical equivalent measurements were found to be consistent with both apparatus (r value >0.86). ICC for SE, J0, and J45 was 0.900, 0.666, and 0.639, respectively; Sw for SE, J0, and J45 was 0.61D, 0.30D, and 0.31D, respectively; Bland–Altman analysis of retinoscopy with cycloplegia and 2WIN-S for SE was 184/194 (95%) in 95% confidence interval, and the mean value was 0.46. J0 was 184/194 (95%), and the mean value is −0.04. J45 was 181/194 (93%), and the mean value is −0.15. Conclusion. The objective refractive measurement of 2WIN-S had good reliability and high agreement with the gold-standard retinoscopy refraction in children and adolescents. While consistency was observed, it is essential to take into consideration that it is a screening tool.

Download Full-text

Mobile, Remote, and Individual Focused: Comparing Breath Carbon Monoxide Readings and Abstinence Between Smartphone-Enabled and Stand-Alone Monitors

Nicotine & Tobacco Research ◽

10.1093/ntr/ntaa203 ◽

2020 ◽

Author(s):

Breanna M Tuck ◽

Joshua L Karelitz ◽

Rachel L Tomko ◽

Jennifer Dahne ◽

Patrick Cato ◽

...

Keyword(s):

Carbon Monoxide ◽

Gold Standard ◽

Smoking Status ◽

Smoking Abstinence ◽

Altman Analysis ◽

Operating Characteristics ◽

Bland Altman Analysis ◽

Urine Cotinine ◽

Study Inclusion

Abstract Introduction Newly available, smartphone-enabled carbon monoxide (CO) monitors are lower in cost than traditional stand-alone monitors and represent a marked advancement for smoking research. New products are promising, but data are needed to compare breath CO readings between smartphone-enabled and stand-alone monitors. The purpose of this study was to (1) determine the agreement between the mobile iCO (Bedfont Scientific Ltd) with two other monitors from the same manufacturer (Micro+ pro and Micro+ basic) and (2) determine optimal, monitor-specific, cotinine-confirmed abstinence cutoff values. Methods Adult (≥18) smokers (n = 26) and nonsmokers (n = 21) provided three breath CO samples (using three different monitors) in each of 10 sessions, and urine cotinine was measured for gold standard determination of abstinence. CO comparisons (N = 437) were analyzed using regression-based Bland–Altman Analysis of Agreement; receiver operating characteristics curves were used to determine optimal abstinence cutoffs. Results Bland–Altman analyses indicated that the iCO monitor provided higher CO results than both Micro+ monitors. Sensitivity and specificity analyses showed that the optimal CO cutoff for determining abstinence was <3 ppm for the Micro+ pro (88% sensitivity, 93% specificity) and Micro+ basic (83% sensitivity, 98% specificity), but was higher for the iCO (<6 ppm; 73% sensitivity, 100% specificity). Conclusions Relative to both Micro+ monitors, the smartphone-enabled iCO provided systematically higher CO values and required a higher cutoff to reliably determine smoking abstinence. This does not indicate that CO values obtained using the iCO are not valid; instead, these results suggest that monitor-specific abstinence cutoffs are needed to ensure accurate bioverification of smoking status. Implications Results from this study indicate that CO values from the smartphone-enabled iCO should not be used interchangeably with the stand-alone Micro+ pro and Micro+ basic, particularly when lower CO values (<10 ppm) are critical (ie, determination of abstinence vs confirming smoking status for study inclusion). Optimal CO cutoffs recommended for determining abstinence on Micro+ and iCO monitors are at <3 and <6 ppm, respectively.

Download Full-text

The ensemble learning model is not better than the Asian modified CKD-EPI equation for glomerular filtration rate estimation in Chinese CKD patients in the external validation study

BMC Nephrology ◽

10.1186/s12882-021-02595-5 ◽

2021 ◽

Vol 22 (1) ◽

Author(s):

Li Zhao ◽

Jing-jing Zhang ◽

Xin Tian ◽

Jian-min Huang ◽

Peng Xie ◽

...

Keyword(s):

Chronic Kidney Disease ◽

Ensemble Learning ◽

Validation Study ◽

Diagnostic Performance ◽

Filtration Rate ◽

External Validation ◽

Learning Model ◽

Altman Analysis ◽

Bland Altman Analysis ◽

External Validation Study

Abstract Objective To assess the clinical practicability of the ensemble learning model established by Liu et al. in estimating glomerular filtration rate (GFR) and validate whether it is a better model than the Asian modified Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation in a cohort of Chinese chronic kidney disease (CKD) patients in an external validation study. Methods According to the ensemble learning model and the Asian modified CKD-EPI equation, we calculated estimated GFRensemble and GFRCKD-EPI, separately. Diagnostic performance of the two models was assessed and compared by correlation coefficient, regression equation, Bland–Altman analysis, bias, precision and P30 under the premise of 99mTc-diethylenetriaminepentaacetic acid (99mTc-DTPA) dual plasma sample clearance method as reference method for GFR measurement (mGFR). Results A total of 158 Chinese CKD patients were included in our external validation study. The GFRensemble was highly related with mGFR, with the correlation coefficient of 0.94. However, regression equation of GFRensemble = 0.66*mGFR + 23.05, the regression coefficient was far away from one, and the intercept was wide. Compared with the Asian modified CKD-EPI equation, the diagnostic performance of the ensemble learning model also demonstrated a wider 95% limit of agreement in Bland-Altman analysis (52.6 vs 42.4 ml/min/1.73 m2), a poorer bias (8.0 vs 1.0 ml/min/1.73 m2, P = 0.02), an inferior precision (18.4 vs 12.7 ml/min/1.73 m2, P < 0.001) and a lower P30 (58.9% vs 74.1%, P < 0.001). Conclusions Our study showed that the ensemble learning model cannot replace the Asian modified CKD-EPI equation for the first choice for GFR estimation in overall Chinese CKD patients.

Download Full-text

Reply: Bland-Altman analysis for pupillometer comparison

Journal of Cataract & Refractive Surgery ◽

10.1016/j.jcrs.2010.08.002 ◽

2010 ◽

Vol 36 (10) ◽

pp. 1803-1804

Author(s):

Magdalena Scheffel ◽

Christoph Kuehne ◽

Thomas Kohnen

Keyword(s):

Altman Analysis ◽

Bland Altman Analysis

Download Full-text

Dental and Skeletal Imaging in Forensic Age Estimation: Disparities in Current Approaches and the Continuing Search for Optimization

Seminars in Musculoskeletal Radiology ◽

10.1055/s-0040-1701495 ◽

2020 ◽

Vol 24 (05) ◽

pp. 510-522

Author(s):

Jannick De Tobel ◽

Christian Ottow ◽

Thomas Widek ◽

Isabella Klasinc ◽

Håkan Mörnstad ◽

...

Keyword(s):

Young Adults ◽

Medical Imaging ◽

Gold Standard ◽

Age Estimation ◽

Statistical Processing ◽

Automated Analysis ◽

Forensic Age Estimation ◽

European Research ◽

Research Groups ◽

Case Examples

AbstractMedical imaging for forensic age estimation in living adolescents and young adults continues to be controversial and a subject of discussion. Because age estimation based on medical imaging is well studied, it is the current gold standard. However, large disparities exist between the centers conducting age estimation, both between and within countries. This review provides an overview of the most common approaches applied in Europe, with case examples illustrating the differences in imaging modalities, in staging of development, and in statistical processing of the age data. Additionally, the review looks toward the future because several European research groups have intensified studies on age estimation, exploring four strategies for optimization: (1) increasing sample sizes of the reference populations, (2) combining single-site information into multifactorial information, (3) avoiding ionizing radiation, and (4) conducting a fully automated analysis.

Download Full-text

Population-based input function for TSPO quantification and kinetic modeling with [11C]-DPA-713

EJNMMI Physics ◽

10.1186/s40658-021-00381-8 ◽

2021 ◽

Vol 8 (1) ◽

Author(s):

Mercy I. Akerele ◽

Sara A. Zein ◽

Sneha Pandya ◽

Anastasia Nikolopoulou ◽

Susan A. Gauthier ◽

...

Keyword(s):

Healthy Volunteers ◽

Kinetic Analysis ◽

Kinetic Modeling ◽

Input Function ◽

Brain Regions ◽

Population Based ◽

Patient Specific ◽

Systematic Bias ◽

Altman Analysis ◽

Bland Altman Analysis

Abstract Introduction Quantitative positron emission tomography (PET) studies of neurodegenerative diseases typically require the measurement of arterial input functions (AIF), an invasive and risky procedure. This study aims to assess the reproducibility of [11C]DPA-713 PET kinetic analysis using population-based input function (PBIF). The final goal is to possibly eliminate the need for AIF. Materials and methods Eighteen subjects including six healthy volunteers (HV) and twelve Parkinson disease (PD) subjects from two [11C]-DPA-713 PET studies were included. Each subject underwent 90 min of dynamic PET imaging. Five healthy volunteers underwent a test-retest scan within the same day to assess the repeatability of the kinetic parameters. Kinetic modeling was carried out using the Logan total volume of distribution (VT) model. For each data set, kinetic analysis was performed using a patient-specific AIF (PSAIF, ground-truth standard) and then repeated using the PBIF. PBIF was generated using the leave-one-out method for each subject from the remaining 17 subjects and after normalizing the PSAIFs by 3 techniques: (a) Weightsubject×DoseInjected, (b) area under AIF curve (AUC), and (c) Weightsubject×AUC. The variability in the VT measured with PSAIF, in the test-retest study, was determined for selected brain regions (white matter, cerebellum, thalamus, caudate, putamen, pallidum, brainstem, hippocampus, and amygdala) using the Bland-Altman analysis and for each of the 3 normalization techniques. Similarly, for all subjects, the variabilities due to the use of PBIF were assessed. Results Bland-Altman analysis showed systematic bias between test and retest studies. The corresponding mean bias and 95% limits of agreement (LOA) for the studied brain regions were 30% and ± 70%. Comparing PBIF- and PSAIF-based VT estimate for all subjects and all brain regions, a significant difference between the results generated by the three normalization techniques existed for all brain structures except for the brainstem (P-value = 0.095). The mean % difference and 95% LOA is −10% and ±45% for Weightsubject×DoseInjected; +8% and ±50% for AUC; and +2% and ± 38% for Weightsubject×AUC. In all cases, normalizing by Weightsubject×AUC yielded the smallest % bias and variability (% bias = ±2%; LOA = ±38% for all brain regions). Estimating the reproducibility of PBIF-kinetics to PSAIF based on disease groups (HV/PD) and genotype (MAB/HAB), the average VT values for all regions obtained from PBIF is insignificantly higher than PSAIF (%difference = 4.53%, P-value = 0.73 for HAB; and %difference = 0.73%, P-value = 0.96 for MAB). PBIF also tends to overestimate the difference between PD and HV for HAB (% difference = 32.33% versus 13.28%) and underestimate it in MAB (%difference = 6.84% versus 20.92%). Conclusions PSAIF kinetic results are reproducible with PBIF, with variability in VT within that obtained for the test-retest studies. Therefore, VT assessed using PBIF-based kinetic modeling is clinically feasible and can be an alternative to PSAIF.

Download Full-text

Comparison of IOP Measurement by Goldmann Applanation Tonometer, ICare Rebound Tonometer, and Tono-Pen in Keratoconus Patients after MyoRing Implantation

Journal of Ophthalmology ◽

10.1155/2019/1964107 ◽

2019 ◽

Vol 2019 ◽

pp. 1-7

Author(s):

Mahmoud Rateb ◽

Mahmoud Abdel-Radi ◽

Zeiad Eldaly ◽

Mohamed Nagy Elmohamady ◽

Asaad Noor El Din

Keyword(s):

Central Corneal Thickness ◽

Pearson Correlation ◽

Corneal Thickness ◽

Altman Analysis ◽

Bland Altman Analysis ◽

Rebound Tonometer ◽

Goldmann Applanation Tonometer ◽

Significant Difference ◽

Applanation Tonometer ◽

Icare Rebound Tonometer

Purpose. To evaluate the different IOP readings by Goldmann applanation tonometer (GAT), ICare rebound tonometer, and Tono-Pen in keratoconus patients after MyoRing implantation. To assess the influence of central corneal thickness (CCT) and thinnest corneal location (TCL) on IOP measurements by different tonometers. Setting. Prospective observational study was conducted in two private centers in Egypt from February 2015 to November 2016. Methods. Seventeen eyes of 10 patients suffering from keratoconus and who underwent MyoRing implantation were recruited. All subjects underwent GAT, ICare, and Tono-Pen IOP measurements in random order. Central corneal thickness and thinnest corneal location were assessed by Pentacam. Difference in mean in IOP readings was assessed by T-test. Correlation between each pair of devices was evaluated by Pearson correlation coefficient. The Bland–Altman analysis was used to assess intertonometer agreement. Results. Seventeen eyes (10 patients) were evaluated. The mean IOP reading was 13.9 ± 3.68, 12.41 ± 2.87, and 14.29 ± 1.31 mmHg in GAT, ICare, and Tono-Pen group, respectively. There was a significant difference between IOP readings by GAT/ICare and Tono-Pen/ICare (p value: 0.032 and 0.002, respectively) with no significant difference between GAT/Tono-Pen (p value: 0.554). Mean difference in IOP measurements between GAT/ICare was 1.49 ± 2.61 mmHg, Tono-Pen/ICare was 1.89 ± 2.15 mmHg, and GAT/Tono-Pen was −0.39 ± 2.59 mmHg. There was no significant correlation between the difference in IOP readings among any pair of devices and CCC or TCL. The Bland–Altman analysis showed a reasonable agreement between any pair of tonometers.

Download Full-text