scholarly journals Spectrum bias in algorithms derived by artificial intelligence: a case study in detecting aortic stenosis using electrocardiograms

2021 ◽  
Author(s):  
Andrew S Tseng ◽  
Michal Shelly-Cohen ◽  
Itzhak Z Attia ◽  
Peter A Noseworthy ◽  
Paul A Friedman ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Aortic Stenosis ◽  
Test Performance ◽  
Severe Aortic Stenosis ◽  
Model Performance ◽  
Target Population ◽  
Artificial Intelligence Algorithm ◽  
Sensitivity Specificity ◽  
Spectrum Model

Abstract Aims Spectrum bias can arise when a diagnostic test is derived from study populations with different disease spectra than the target population, resulting in poor generalizability. We used a real-world artificial intelligence-derived algorithm to detect severe aortic stenosis to experimentally assess the effect of spectrum bias on test performance. Methods and Results All adult patients at the Mayo Clinic between January 1st, 1989 to September 30th, 2019 with transthoracic echocardiograms within 180 days after electrocardiogram were identified. Two models were developed from two distinct patient cohorts: a whole-spectrum cohort comparing severe AS to any non-severe AS and an extreme-spectrum cohort comparing severe AS to no AS at all. Model performance was assessed. Overall, 258,607 patients had valid ECG and echocardiograms pairs. The area under the receiver operator curve was 0.87 and 0.91 for the whole-spectrum and extreme-spectrum models respectively. Sensitivity and specificity for the whole-spectrum model was 80% and 81% respectively, while for the extreme-spectrum model it was 84% and 84% respectively. When applying the AI-ECG derived from the extreme-spectrum cohort to patients in the whole-spectrum cohort, the sensitivity, specificity and AUC dropped to 83%, 73% and 0.86 respectively. Conclusion While the algorithm performed robustly in identifying severe AS, this study shows that limiting datasets to clearly positive or negative labels leads to overestimation of test performance when testing an artificial intelligence algorithm in the setting of classifying severe AS using ECG data. While the effect of the bias may be modest in this example, clinicians should be aware of the existence of such a bias in AI-derived algorithms.

scholarly journals UNDERSTANDING SPECTRUM BIAS IN ALGORITHMS DERIVED BY ARTIFICIAL INTELLIGENCE A CASE STUDY IN DETECTING AORTIC STENOSIS USING ELECTROCARDIOGRAMS

2021 ◽  
Vol 77 (18) ◽  
pp. 3241
Author(s):  
Andrew S. Tseng ◽  
Michal Shelly-Cohen ◽  
Zachi Itzhak Attia ◽  
Peter Noseworthy ◽  
Paul Friedman ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Aortic Stenosis

scholarly journals Feasibility and Validity of Computed Tomography-Derived Fractional Flow Reserve in Patients With Severe Aortic Stenosis

2020 ◽  
Author(s):  
Michael Michail ◽  
Abdul-Rahman Ihdayhid ◽  
Andrea Comella ◽  
Udit Thakur ◽  
James D. Cameron ◽  
...  
Keyword(s):  
Aortic Stenosis ◽  
Diagnostic Accuracy ◽  
Operating Characteristic ◽  
Severe Aortic Stenosis ◽  
Characteristic Curve ◽  
Fractional Flow ◽  
Predictive Values ◽  
Flow Reserve ◽  
Artery Disease ◽  
Sensitivity Specificity

Background: Coronary artery disease is common in patients with severe aortic stenosis. Computed tomography-derived fractional flow reserve (CT-FFR) is a clinically used modality for assessing coronary artery disease, however, its use has not been validated in patients with severe aortic stenosis. This study assesses the safety, feasibility, and validity of CT-FFR in patients with severe aortic stenosis. Methods: Prospectively recruited patients underwent standard-protocol invasive FFR and coronary CT angiography (CTA). CTA images were analyzed by central core laboratory (HeartFlow, Inc) for independent evaluation of CT-FFR. CT-FFR data were compared with FFR (ischemia defined as FFR ≤0.80). Results: Forty-two patients (68 vessels) underwent FFR and CTA; 39 patients (92.3%) and 60 vessels (88.2%) had interpretable CTA enabling CT-FFR computation. Mean age was 76.2±6.7 years (71.8% male). No patients incurred complications relating to premedication, CTA, or FFR protocol. Mean FFR and CT-FFR were 0.83±0.10 and 0.77±0.14, respectively. CT calcium score was 1373.3±1392.9 Agatston units. On per vessel analysis, there was positive correlation between FFR and CT-FFR (Pearson correlation coefficient, R =0.64, P <0.0001). Sensitivity, specificity, positive predictive value, and negative predictive values were 73.9%, 78.4%, 68.0%, and 82.9%, respectively, with 76.7% diagnostic accuracy. The area under the receiver-operating characteristic curve for CT-FFR was 0.83 (0.72–0.93, P <0.0001), which was higher than that of CTA and quantitative coronary angiography ( P =0.01 and P <0.001, respectively). Bland-Altman plot showed mean bias between FFR and CT-FFR as 0.059±0.110. On per patient analysis, the sensitivity, specificity, positive predictive, and negative predictive values were 76.5%, 77.3%, 72.2%, and 81.0% with 76.9% diagnostic accuracy. The per patient area under the receiver-operating characteristic curve analysis was 0.81 (0.67–0.95, P <0.0001). Conclusions: CT-FFR is safe and feasible in patients with severe aortic stenosis. Our data suggests that the diagnostic accuracy of CT-FFR in this cohort potentially enables its use in clinical practice and provides the foundation for future research into the use of CT-FFR for coronary evaluation pre-aortic valve replacement.

scholarly journals Real-World Verification of Artificial Intelligence Algorithm-Assisted Auscultation of Breath Sounds in Children

2021 ◽  
Vol 9 ◽  
Author(s):  
Jing Zhang ◽  
Han-Song Wang ◽  
Hong-Yuan Zhou ◽  
Bin Dong ◽  
Lei Zhang ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Medical Center ◽  
Age Groups ◽  
Pulmonary Diseases ◽  
Clinical Environment ◽  
Electronic Stethoscope ◽  
Breath Sounds ◽  
Artificial Intelligence Algorithm ◽  
Sensitivity Specificity ◽  
Better Than

Objective: Lung auscultation plays an important role in the diagnosis of pulmonary diseases in children. The objective of this study was to evaluate the use of an artificial intelligence (AI) algorithm for the detection of breath sounds in a real clinical environment among children with pulmonary diseases.Method: The auscultations of breath sounds were collected in the respiratory department of Shanghai Children's Medical Center (SCMC) by using an electronic stethoscope. The discrimination results for all chest locations with respect to a gold standard (GS) established by 2 experienced pediatric pulmonologists from SCMC and 6 general pediatricians were recorded. The accuracy, sensitivity, specificity, precision, and F1-score of the AI algorithm and general pediatricians with respect to the GS were evaluated. Meanwhile, the performance of the AI algorithm for different patient ages and recording locations was evaluated.Result: A total of 112 hospitalized children with pulmonary diseases were recruited for the study from May to December 2019. A total of 672 breath sounds were collected, and 627 (93.3%) breath sounds, including 159 crackles (23.1%), 264 wheeze (38.4%), and 264 normal breath sounds (38.4%), were fully analyzed by the AI algorithm. The accuracy of the detection of adventitious breath sounds by the AI algorithm and general pediatricians with respect to the GS were 77.7% and 59.9% (p &lt; 0.001), respectively. The sensitivity, specificity, and F1-score in the detection of crackles and wheeze from the AI algorithm were higher than those from the general pediatricians (crackles 81.1 vs. 47.8%, 94.1 vs. 77.1%, and 80.9 vs. 42.74%, respectively; wheeze 86.4 vs. 82.2%, 83.0 vs. 72.1%, and 80.9 vs. 72.5%, respectively; p &lt; 0.001). Performance varied according to the age of the patient, with patients younger than 12 months yielding the highest accuracy (81.3%, p &lt; 0.001) among the age groups.Conclusion: In a real clinical environment, children's breath sounds were collected and transmitted remotely by an electronic stethoscope; these breath sounds could be recognized by both pediatricians and an AI algorithm. The ability of the AI algorithm to analyze adventitious breath sounds was better than that of the general pediatricians.

scholarly journals TCT-58 Using Artificial Intelligence to Predict the Progression from Moderate to Severe Aortic Stenosis

2021 ◽  
Vol 78 (19) ◽  
pp. B24
Author(s):  
Soundos Moualla ◽  
Patrick McCarthy ◽  
James Thomas ◽  
Michael Dobbles ◽  
Madalina Petrescu ◽  
...  
Keyword(s):  
Artificial Intelligence ◽  
Aortic Stenosis ◽  
Severe Aortic Stenosis

scholarly journals Improving Artificial Intelligence Forecasting Models Performance with Data Preprocessing: European Union Allowance Prices Case Study

Energies ◽  
2021 ◽  
Vol 14 (23) ◽  
pp. 7845
Author(s):  
Miguel A. Jaramillo-Morán ◽  
Daniel Fernández-Martínez ◽  
Agustín García-García ◽  
Diego Carmona-Fernández
Keyword(s):  
Artificial Intelligence ◽  
European Union ◽  
Time Series ◽  
Mean Value ◽  
Percentage Error ◽  
Intrinsic Mode Functions ◽  
Mode Decomposition ◽  
New Business ◽  
Artificial Intelligence Algorithm

European Union Allowances (EUAs) are rights to emit CO2 that may be sold or bought by enterprises. They were originally created to try to reduce greenhouse gas emissions, although they have become assets that may be used by financial intermediaries to seek for new business opportunities. Therefore, forecasting the time evolution of their price is very important for agents involved in their selling or buying. Neural Networks, an artificial intelligence paradigm, have been proved to be accurate and reliable tools for time series forecasting, and have been widely used to predict economic and energetic variables; two of them are used in this work, the Multilayer Preceptron (MLP) and the Long Short-Term Memories (LSTM), along with another artificial intelligence algorithm (XGBoost). They are combined with two preprocessing tools, decomposition of the time series into its trend and fluctuation and decomposition into Intrinsic Mode Functions (IMF) by the Empirical Mode Decomposition (EMD). The price prediction is obtained by adding those from each subseries. These two tools are combined with the three forecasting tools to provide 20 future predictions of EUA prices. The best results are provided by MLP-EMD, which is able to achieve a Mean Absolute Percentage Error (MAPE) of 2.91% for the first predicted datum and 5.65% for the twentieth, with a mean value of 4.44%.

scholarly journals Hearing Artificial Intelligence: Sonification Guidelines & Results From a Case-study in Melanoma Diagnosis

2019 ◽  
Author(s):  
R. Michael Winters ◽  
Ankur Kalra ◽  
Bruce N. Walker
Keyword(s):  
Artificial Intelligence ◽  
Data Driven ◽  
Learning Effects ◽  
General Technique ◽  
Melanoma Diagnosis ◽  
The Mean ◽  
Sensitivity Specificity ◽  
Improved Accuracy ◽  
Full Classification

The applications of artificial intelligence are becoming more and more prevalent in everyday life. Although many AI systems can operate autonomously, their goal is often assisting humans. Knowledge from the AI system must somehow be perceptualized. Towards this goal, we present a case-study in the application of data-driven non-speech audio for melanoma diagnosis. A physician photographs a suspicious skin lesion, triggering a sonification of the system’s penultimate classification layer. We iterated on sonification strategies and coalesced around designs representing three general approaches. We tested each in a group of novice listeners (n=7) for mean sensitivity, specificity, and learning effects. The mean accuracy was greatest for a simple model, but a trained dermatologist preferred a perceptually compressed model of the full classification layer. We discovered that training the AI on sonifications from this model improved accuracy further. We argue for perceptual compression as a general technique and for a comprehensible number of simultaneous streams.

scholarly journals Simplifying Research Oriented to Practitioners and Policymakers: A Case Study of Predictive Modelling with Naive Bayes

2021 ◽  
Author(s):  
Troy Smith
Keyword(s):  
Model Comparison ◽  
Performance Metrics ◽  
Naive Bayes ◽  
Model Performance ◽  
Predictive Modelling ◽  
Target Population ◽  
Naïve Bayes ◽  
Offender Profiling ◽  
High Crime

The study examines the applicability of Naïve Bayes in predictive classification modelling using a case study of cybercrime victimization data. The goal of which was a targeted presentation of the benefits of Bayesian analysis in crime research geared to policymakers. The method is assessed using a Model-Comparison Approach and model performance metrics. The study shows that Naïve Bayes can be useful in predictive classification where the target population is small or difficult to acquire such as offender profiling and analysis of high crime areas. This is important as it provides a plausible option to traditional Frequentist methods, that overcome statistical limitations and provides results in a form easily conveyable to policymakers. Further, the conditional probability data produced makes future prediction transparent and can foster confidence in predicted outcomes. In particular, Directed Acyclic Graph can be easily used to represent the Naïve Bayes output allowing visualization of the relationships between variables.

scholarly journals Enhanced Diagnosis of Severe Aortic Stenosis Using Artificial Intelligence: A Proof-of-Concept Study of 530,871 Echocardiograms

2020 ◽  
Vol 13 (4) ◽  
pp. 1087-1090 ◽  
Author(s):  
David Playford ◽  
Edward Bordin ◽  
Razali Mohamad ◽  
Simon Stewart ◽  
Geoff Strange
Keyword(s):  
Artificial Intelligence ◽  
Aortic Stenosis ◽  
Severe Aortic Stenosis ◽  
Proof Of Concept ◽  
Concept Study

scholarly journals Development of an Optimised Dataset for Training a Deep Neural Network

2021 ◽  
Author(s):  
Callum Newman ◽  
Jon Petzing ◽  
Yee Mey Goh ◽  
Laura Justham
Keyword(s):  
Neural Network ◽  
Artificial Intelligence ◽  
Neural Networks ◽  
Computer Vision ◽  
Light Intensity ◽  
Test Performance ◽  
Deep Neural Network ◽  
Training Dataset ◽  
Multiple Factors

Artificial intelligence in computer vision has focused on improving test performance using techniques and architectures related to deep neural networks. However, improvements can also be achieved by carefully selecting the training dataset images. Environmental factors, such as light intensity, affect the image’s appearance and by choosing optimal factor levels the neural network’s performance can improve. However, little research into processes which help identify optimal levels is available. This research presents a case study which uses a process for developing an optimised dataset for training an object detection neural network. Images are gathered under controlled conditions using multiple factors to construct various training datasets. Each dataset is used to train the same neural network and the test performance compared to identify the optimal factors. The opportunity to use synthetic images is introduced, which has many advantages including creating images when real-world images are unavailable, and more easily controlled factors.

Sign in / Sign up

Export Citation Format

Share Document