Positive Predictive Value Surfaces as a Complementary Tool to Assess the Performance of Virtual Screening Methods

2020 ◽  
Vol 20 (14) ◽  
pp. 1447-1460
Author(s):  
Juan F. Morales ◽  
Sara Chuguransky ◽  
Lucas N. Alberca ◽  
Juan I. Alice ◽  
Sofía Goicoechea ◽  
...  
Keyword(s):  
Virtual Screening ◽  
Positive Predictive Value ◽  
Ensemble Learning ◽  
Predictive Value ◽  
Experimental Testing ◽  
Screening Tools ◽  
Screening Methods ◽  
Learning Approaches ◽  
Classification Problems ◽  
Screening Experiments

Background: Since their introduction in the virtual screening field, Receiver Operating Characteristic (ROC) curve-derived metrics have been widely used for benchmarking of computational methods and algorithms intended for virtual screening applications. Whereas in classification problems, the ratio between sensitivity and specificity for a given score value is very informative, a practical concern in virtual screening campaigns is to predict the actual probability that a predicted hit will prove truly active when submitted to experimental testing (in other words, the Positive Predictive Value - PPV). Estimation of such probability is however, obstructed due to its dependency on the yield of actives of the screened library, which cannot be known a priori. Objective: To explore the use of PPV surfaces derived from simulated ranking experiments (retrospective virtual screening) as a complementary tool to ROC curves, for both benchmarking and optimization of score cutoff values. Methods: The utility of the proposed approach is assessed in retrospective virtual screening experiments with four datasets used to infer QSAR classifiers: inhibitors of Trypanosoma cruzi trypanothione synthetase; inhibitors of Trypanosoma brucei N-myristoyltransferase; inhibitors of GABA transaminase and anticonvulsant activity in the 6 Hz seizure model. Results: Besides illustrating the utility of PPV surfaces to compare the performance of machine learning models for virtual screening applications and to select an adequate score threshold, our results also suggest that ensemble learning provides models with better predictivity and more robust behavior. Conclusion: PPV surfaces are valuable tools to assess virtual screening tools and choose score thresholds to be applied in prospective in silico screens. Ensemble learning approaches seem to consistently lead to improved predictivity and robustness.

scholarly journals A machine learning approach for predicting suicidal thoughts and behaviours among college students

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Melissa Macalli ◽  
Marie Navarro ◽  
Massimiliano Orri ◽  
Marie Tournier ◽  
Rodolphe Thiébaut ◽  
...  
Keyword(s):  
Mental Health ◽  
College Students ◽  
Positive Predictive Value ◽  
Predictive Value ◽  
Screening Tools ◽  
Receiver Operating Curve ◽  
Depression Symptoms ◽  
Suicidal Thoughts ◽  
One Year

AbstractSuicidal thoughts and behaviours are prevalent among college students. Yet little is known about screening tools to identify students at higher risk. We aimed to develop a risk algorithm to identify the main predictors of suicidal thoughts and behaviours among college students within one-year of baseline assessment. We used data collected in 2013–2019 from the French i-Share cohort, a longitudinal population-based study including 5066 volunteer students. To predict suicidal thoughts and behaviours at follow-up, we used random forests models with 70 potential predictors measured at baseline, including sociodemographic and familial characteristics, mental health and substance use. Model performance was measured using the area under the receiver operating curve (AUC), sensitivity, and positive predictive value. At follow-up, 17.4% of girls and 16.8% of boys reported suicidal thoughts and behaviours. The models achieved good predictive performance: AUC, 0.8; sensitivity, 79% for girls, 81% for boys; and positive predictive value, 40% for girls and 36% for boys. Among the 70 potential predictors, four showed the highest predictive power: 12-month suicidal thoughts, trait anxiety, depression symptoms, and self-esteem. We identified a parsimonious set of mental health indicators that accurately predicted one-year suicidal thoughts and behaviours in a community sample of college students.

scholarly journals EVALUATING THE PREDICTIVE VALIDITY OF THE NIPISSING DISTRICT DEVELOPMENTAL SCREEN IN PRIMARY CARE SETTINGS AT THE 18-MONTH HEALTH SUPERVISION VISIT

2018 ◽  
Vol 23 (suppl_1) ◽  
pp. e37-e37
Author(s):  
Vinusha Gunaseelan ◽  
Patricia Parkin ◽  
Imaan Bayoumi ◽  
Patricia Jiang ◽  
Alexandra Medline ◽  
...  
Keyword(s):  
Primary Care ◽  
Positive Predictive Value ◽  
Diagnostic Accuracy ◽  
Developmental Delay ◽  
Predictive Validity ◽  
Predictive Value ◽  
Screening Tools ◽  
Developmental Screening ◽  
Research Personnel ◽  
Health Supervision

Abstract BACKGROUND The Canadian Paediatric Society (CPS) recommends that every Canadian physician caring for young children provide an enhanced 18-month well-baby visit including the use of a developmental screening tool, such as the Nipissing District Developmental Screen (NDDS). The Province of Ontario implemented an enhanced 18-month well-baby visit specifically emphasizing the NDDS, which is now widely used in Ontario primary care. However, the diagnostic accuracy of the NDDS in identifying early developmental delays in real-world clinical settings is unknown. OBJECTIVES To assess the predictive validity of the NDDS in primary care for identifying developmental delay and prompting a specialist referral at the 18-month health supervision visit. DESIGN/METHODS This was a prospective longitudinal cohort study enrolling healthy children from primary care practices. Parents completed the 18-month NDDS during their child’s scheduled health supervision visit between January 2012 and February 2015. Using a standardized data collection form, research personnel abstracted data from the child’s health records regarding the child’s developmental outcomes following the 18-month assessment. Data collected included confirmed diagnoses of a development delay, specialist referrals, family history, and interventions. Research personnel were blind to the results of the NDDS. We assessed the diagnostic test properties of the NDDS with a confirmed diagnosis of developmental delay as the criterion measure. The specificity, sensitivity, positive predictive value, and negative predictive value were calculated, with 95% confidence intervals. RESULTS We included 255 children with a mean age of 18.5 months (range, 17.5–20.6) and 139 (55%) were male. 102 (40%) screened positive (1+ flag result on their NDDS). A total of 48 (19%) children were referred, and 23 (9%) had a confirmed diagnosis of a developmental delay (speech and language: 14; gross motor: 4; autism spectrum disorder: 3; global developmental delay: 1; developmental delay: 1). The sensitivity was 74% (95% CI: 52–90%), specificity was 63% (95% CI: 57–70%), positive predictive value was 17% (95% CI:10–25%), and the negative predictive value was 96% (95% CI: 92–99%). CONCLUSION For developmental screening tools, sensitivity between 70%-80% and specificity of 80% have been suggested. The NDDS has moderate sensitivity and specificity in identifying developmental delay at the 18-month health supervision visit. The 1+NDDS flag cut-point may lead to overdiagnosis with more children with typical development being referred, leading to longer wait times for specialist referrals among children in need. Future work includes investigating the diagnostic accuracy of combining the NDDS with other screening tools.

Comparison of eight screening tools to detect interactions between herbal supplements and oncology agents

2020 ◽  
Vol 26 (8) ◽  
pp. 1843-1849
Author(s):  
Faisal Shakeel ◽  
Fang Fang ◽  
Kelley M Kidwell ◽  
Lauren A Marcath ◽  
Daniel L Hertz
Keyword(s):  
Drug Interaction ◽  
Positive Predictive Value ◽  
Drug Interactions ◽  
Predictive Value ◽  
Screening Tools ◽  
Herbal Supplements ◽  
Interaction Detection ◽  
Natural Medicine ◽  
Interaction Screening ◽  
Drug Drug Interaction

Introduction Patients with cancer are increasingly using herbal supplements, unaware that supplements can interact with oncology treatment. Herb–drug interaction management is critical to ensure optimal treatment outcomes. Several screening tools exist to detect drug–drug interactions, but their performance to detect herb–drug interactions is not known. This study compared the performance of eight drug–drug interaction screening tools to detect herb–drug interaction with anti-cancer agents. Methods The herb–drug interaction detection performance of four subscription (Micromedex, Lexicomp, PEPID, Facts & Comparisons) and free (Drugs.com, Medscape, WebMD, RxList) drug–drug interaction tools was assessed. Clinical relevance of each herb–drug interaction was determined using Natural Medicine and each drug–drug interaction tool. Descriptive statistics were used to calculate sensitivity, specificity, positive predictive value, and negative predictive value. Linear regression was used to compare performance between subscription and free tools. Results All tools had poor sensitivity (<0.20) for detecting herb–drug interaction. Lexicomp had the highest positive predictive value (0.98) and best overall performance score (0.54), while Medscape was the best performing free tool (0.52). The worst subscription tools were as good as or better than the best free tools, and as a group subscription tools outperformed free tools on all metrics. Using an average subscription tool would detect one additional herb–drug interaction for every 10 herb–drug interactions screened by a free tool. Conclusion Lexicomp is the best available tool for screening herb–drug interaction, and Medscape is the best free alternative; however, the sensitivity and performance for detecting herb–drug interaction was far lower than for drug–drug interactions, and overall quite poor. Further research is needed to improve herb–drug interaction screening performance.

Automated screening tool for Subcutaneous Implantable Defibrillator in Brugada syndrome has a high eligibility rate which is predicted by simple electrocardiographic parameters

2020 ◽  
Vol 41 (Supplement_2) ◽  
Author(s):  
S Marelli ◽  
D Kukavica ◽  
A Mazzanti ◽  
T Chargeishvili ◽  
A Trancuccio ◽  
...  
Keyword(s):  
Positive Predictive Value ◽  
Brugada Syndrome ◽  
Predictive Value ◽  
Screening Tool ◽  
Screening Tools ◽  
Molecular Medicine ◽  
Funding Source ◽  
S Wave ◽  
High Positive Predictive Value ◽  
Automated Screening

Abstract Background Manual electrocardiographic (ECG) screening tools for the use of subcutaneous cardiac defibrillator (S-ICD) have been associated with high ineligibility rates in Brugada syndrome patients (BrS). Although recent works identified ECG parameters for S-ICD eligibility in general population, automated screening tool (AST) for S-ICD eligibility have not even been assessed in large series of patients with BrS. Purpose This study evaluates the AST-derived eligibility rates for an S-ICD in patients with BrS, and ECG parameters associated with S-ICD eligibility. Methods Screening for S-ICD eligibility was performed using AST in 194 consecutive patients with BrS. Eligibility was defined when at least one of the three vectors was acceptable both in supine and standing position. Twelve-lead ECGs were registered during the screening. ECG parameters associated with AST eligibility were identified using multivariable logistical regression. Results Our study population consisted of 194 patients, with male preponderance (n=165/194; 85%); and were 43±12 years old at the time of screening. Majority of patients presented a spontaneous type 1 pattern during screening (n=128/194; 66%), with an average pattern height of 3±3 mm. Remarkably, 93% of patients passed the screening with AST. No differences in eligibility rates in terms of gender (93% males vs. 93% females eligible; p=1) and age (48±9 years non-eligible vs. 42±12 eligible; p=0.07) existed. Notably, our eligibility rate was 2.5 times higher than rates reported in literature when using manual screening tools (p=0.023). Independent 12-lead ECG parameters (Table) associated with AST eligibility were duration of S wave &lt;80 ms in aVF and R/T ratio ≥3 in lead II (Figure), which have a high positive predictive value (97% and 99%, respectively) for screening eligibility. Conclusions Most BrS patients (93%) are eligible for S-ICD when AST is used. S wave &lt;80 ms in aVF, and R/T ratio ≥3 in lead II have a high positive predictive value for S-ICD eligibility. Funding Acknowledgement Type of funding source: Public grant(s) – National budget only. Main funding source(s): The Italian Ministry of Research and University Dipartimenti di Eccellenza 2018–2022 grant to the Molecular Medicine Department (University of Pavia)

scholarly journals Virtual screening web servers: designing chemical probes and drug candidates in the cyberspace

2020 ◽  
Author(s):  
Natesh Singh ◽  
Ludovic Chaput ◽  
Bruno O Villoutreix
Keyword(s):  
Virtual Screening ◽  
Drug Repositioning ◽  
Pharmaceutical Research ◽  
Bioactive Molecules ◽  
Screening Methods ◽  
Web Servers ◽  
Web Based ◽  
Drug Candidates ◽  
Screening Experiments ◽  
Molecule Docking

Abstract The interplay between life sciences and advancing technology drives a continuous cycle of chemical data growth; these data are most often stored in open or partially open databases. In parallel, many different types of algorithms are being developed to manipulate these chemical objects and associated bioactivity data. Virtual screening methods are among the most popular computational approaches in pharmaceutical research. Today, user-friendly web-based tools are available to help scientists perform virtual screening experiments. This article provides an overview of internet resources enabling and supporting chemical biology and early drug discovery with a main emphasis on web servers dedicated to virtual ligand screening and small-molecule docking. This survey first introduces some key concepts and then presents recent and easily accessible virtual screening and related target-fishing tools as well as briefly discusses case studies enabled by some of these web services. Notwithstanding further improvements, already available web-based tools not only contribute to the design of bioactive molecules and assist drug repositioning but also help to generate new ideas and explore different hypotheses in a timely fashion while contributing to teaching in the field of drug development.

Attended With and Head-Turning Sign can be clinical markers of cognitive impairment in older adults

2017 ◽  
Vol 29 (11) ◽  
pp. 1763-1769 ◽  
Author(s):  
Pinar Soysal ◽  
Cansu Usarel ◽  
Gul Ispirli ◽  
Ahmet Turan Isik
Keyword(s):  
Older Adults ◽  
Cognitive Impairment ◽  
Positive Predictive Value ◽  
Negative Predictive Value ◽  
Predictive Value ◽  
Cognitive Assessment ◽  
Screening Methods ◽  
Operating Characteristics ◽  
Head Turning ◽  
Sensitivity Specificity

ABSTRACTBackground:Comprehensive neurocognitive assessment may not be performed in clinical practice, as it takes too much time and requires special training. Development of easily applicable, time-saving, and cost effective screening methods has allowed identifying the individuals that require further evaluation. The aim of present study was to assess the diagnostic value of the Attended With (AW) and Head-Turning Sign (HTS) for screening cognitive impairment (CI).Methods:Comprehensive geriatric assessment was performed in 529 elderly outpatients, and the presence or absence of AW and HTS was investigated in them all.Results:Of the 529 patients, of whom the mean age was 75.67 ± 8.29 years, 126 patients were considered as CI (102 dementia, 24 mild CI). The patients with positive AW had significantly lower scores on Mini-Mental State Examination, Cognitive State Test, and Montreal Cognitive Assessment, and activities of daily living compared to AW (−) patients (p < 0.001). Similar significant findings were obtained in the patients with positive and negative HTS (p < 0.001). The sensitivity, specificity, positive predictive value, and negative predictive value of AW in detecting CI were 92%, 37%, 31.4%, and 93.7%, respectively. The sensitivity, specificity, positive predictive value, and negative predictive value of HTS were 80%, 64%, 41.8%, and 91.5%, respectively. The area under the receiver-operating characteristics curve was 0.90 for AW and 0.82 for HTS.Conclusion:AW and HTS are fast, simple, effective, and sensitive methods for detecting CI. Therefore, they can be used for older adults attending the primary care settings with memory loss. Those with positive AW or HTS can be referred to the relevant centers for detailed cognitive assessment.

Metabolomics for discovery of biomarkers for cystic fibrosis: Towards MS-based primary screening methods with improved positive predictive value

2014 ◽  
Vol 47 (15) ◽  
pp. 143 ◽  
Author(s):  
Alicia DiBattista ◽  
Adriana N. Macedo ◽  
Osama Y. Al-Dirbashi ◽  
Pranesh Chakraborty ◽  
Philip Britz-McKibbin
Keyword(s):  
Cystic Fibrosis ◽  
Positive Predictive Value ◽  
Predictive Value ◽  
Screening Methods ◽  
Primary Screening

scholarly journals Virtual Screening of Conjugated Polymers for Organic Photovoltaic Devices Using Support Vector Machines and Ensemble Learning

2019 ◽  
Vol 2019 ◽  
pp. 1-7
Author(s):  
Fang-Chung Chen
Keyword(s):  
Virtual Screening ◽  
Conjugated Polymers ◽  
Ensemble Learning ◽  
High Performance ◽  
Learning Algorithm ◽  
Predictive Performance ◽  
Machine Learning Algorithms ◽  
Organic Photovoltaic ◽  
Support Vector ◽  
Learning Approaches

Herein, we report virtual screening of potential semiconductor polymers for high-performance organic photovoltaic (OPV) devices using various machine learning algorithms. We particularly focus on support vector machine (SVM) and ensemble learning approaches. We found that the power conversion efficiencies of the device prepared with the polymer candidates can be predicted with their structure fingerprints as the only inputs. In other words, no preliminary knowledge about material properties was required. Additionally, the predictive performance could be further improved by “blending” the results of the SVM and random forest models. The resulting ensemble learning algorithm might open up a new opportunity for more precise, high-throughput virtual screening of conjugated polymers for OPV devices.

scholarly journals Predicting kinase inhibitors using bioactivity matrix derived informer sets

2019 ◽  
Author(s):  
Huikun Zhang ◽  
Spencer S. Ericksen ◽  
Ching-pei Lee ◽  
Gene E. Ananiev ◽  
Nathan Wlodarchak ◽  
...  
Keyword(s):  
Experimental Data ◽  
Drug Discovery ◽  
Virtual Screening ◽  
Kinase Inhibitor ◽  
Experimental Testing ◽  
Chemical Diversity ◽  
Training Data ◽  
Initial Screening ◽  
Screening Experiments ◽  
Selection Of

AbstractPrediction of compounds that are active against a desired biological target is a common step in drug discovery efforts. Virtual screening methods seek some active-enriched fraction of a library for experimental testing. Where data are too scarce to train supervised learning models for compound prioritization, initial screening must provide the necessary data. Commonly, such an initial library is selected on the basis of chemical diversity by some pseudo-random process (for example, the first few plates of a larger library) or by selecting an entire smaller library. These approaches may not produce a sufficient number or diversity of actives. An alternative approach is to select an informer set of screening compounds on the basis of chemogenomic information from previous testing of compounds against a large number of targets.We compare different ways of using chemogenomic data to choose a small informer set of compounds based on previously measured bioactivity data. We develop this Informer-Based-Ranking (IBR) approach using the Published Kinase Inhibitor Sets (PKIS) as the chemogenomic data to select the informer sets. We test the informer compounds on a target that is not part of the chemogenomic data, then predict the activity of the remaining compounds based on the experimental informer data and the chemogenomic data. Through new chemical screening experiments, we demonstrate the utility of IBR strategies in a prospective test on two kinase targets not included in the PKIS. Using limited training data in both retrospective and prospective tests, bioactivity fingerprints based on chemogenomic data outperform chemical fingerprints in predicting active compounds in both standard virtual screening metrics and accurate identification of hits from novel chemical classes.Author SummaryIn the early stages of drug discovery efforts, computational models are used to predict activity and prioritize compounds for experimental testing. New targets commonly lack the data necessary to build effective models, and the screening needed to generate that experimental data can be costly. We seek to improve the efficiency of the initial screening phase, and of the process of prioritizing compounds for subsequent screening.We choose a small informer set of compounds based on publicly available prior screening data on distinct (though related) targets. We then use experimental data on these informer compounds to predict the activity of other compounds in the set against the target of interest. Computational and statistical tools are needed to identify informer compounds and to prioritize other compounds for subsequent phases of screening. Using limited training data, we find that selection of informer compounds on the basis of bioactivity data from previous screening efforts is superior to the traditional approach of selection of a chemically diverse subset of compounds. We demonstrate the success of this approach in retrospective tests on the Published Kinase Inhibitor Sets (PKIS) chemogenomic data and in prospective experimental screens against two additional non-human kinase targets.

scholarly journals Developing a Primary Care EMR-based Frailty Definition using Machine Learning

2018 ◽  
Vol 3 (4) ◽  
Author(s):  
Sylvia Aponte-Hao ◽  
Bria Mele ◽  
Dave Jackson ◽  
Alan Katz ◽  
Charles Leduc ◽  
...  
Keyword(s):  
Machine Learning ◽  
Primary Care ◽  
Positive Predictive Value ◽  
Predictive Value ◽  
Laboratory Data ◽  
Screening Tools ◽  
Geriatric Syndrome ◽  
Surveillance Network ◽  
Clinical Frailty Scale ◽  
Definition Of

IntroductionFrailty is a geriatric syndrome that is predictive of heightened vulnerability for disability, hospitalization, and mortality. Annually an estimated 250,000 frail Canadians die, and this estimate is expected to double in the next 40 years, as Canadians grow older. Currently there is no single accepted clinical definition of frailty. Objectives and ApproachThe objective of this study was to develop an operational definition of frailty using machine learning that can be applied to a primary care electronic medical record (EMR) database. The Canadian Primary Care Sentinel Surveillance Network (CPCSSN) is a pan-Canadian network of primary care practices that collect de-identified patient information (such as encounter diagnoses, health conditions, and laboratory data) from EMRs. 780 patients from CPCSSN have were randomly selected and assessed by physicians using the Rockwood Clinical Frailty Scale (as frail or not frail), and their clinical characteristics from CPCSSN used to develop the definition using machine-learning. ResultsA total of 8,044 clinical features were extracted from these tables: billing, problem list, encounter diagnosis, labs, medications and referrals. A chi-squared automatic interaction detector (CHAID) approach was selected as the best approach. The bootstrapping process used a cost matrix that prioritized high sensitivity and positive predictive value. 10-fold cross validation was used for validity measures. Key features factored into the algorithm included: diagnosis of dementia (ICD-9 code 290), medications furosemide and vitamins, and use of key word “obstruction” within the billing table. The validation measures with 95% confidence intervals are as follows: sensitivity of 28% (95% CI: 21% to 36%), specificity of 94% (95% CI: 93% to 96%), positive predictive value of 53% (95% CI: 42% to 64%), negative predictive value of 86% (95% CI: 83% to 88%). Conclusion/ImplicationsNo other primary care specific frailty screening tools have sufficient validity. These results suggest heterogeneous diseases require clearly defined features and potentially more sophisticated algorithms to account for heterogeneity. Further research utilizing continuous features and continuous frailty scores may be more suitable in the creation of a case detection algorithm.

Sign in / Sign up

Export Citation Format

Share Document