The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation

PeerJ Computer Science ◽

10.7717/peerj-cs.623 ◽

2021 ◽

Vol 7 ◽

pp. e623

Author(s):

Davide Chicco ◽

Matthijs J. Warrens ◽

Giuseppe Jurman

Keyword(s):

Machine Learning ◽

Regression Analysis ◽

Binary Classification ◽

Ground Truth ◽

Absolute Error ◽

Supervised Machine Learning ◽

Coefficient Of Determination ◽

Target Range ◽

Percentage Error ◽

The Mean

Regression analysis makes up a large part of supervised machine learning, and consists of the prediction of a continuous independent target from a set of other predictor variables. The difference between binary classification and regression is in the target range: in binary classification, the target can have only two values (usually encoded as 0 and 1), while in regression the target can have multiple values. Even if regression analysis has been employed in a huge number of machine learning studies, no consensus has been reached on a single, unified, standard metric to assess the results of the regression itself. Many studies employ the mean square error (MSE) and its rooted variant (RMSE), or the mean absolute error (MAE) and its percentage variant (MAPE). Although useful, these rates share a common drawback: since their values can range between zero and +infinity, a single value of them does not say much about the performance of the regression with respect to the distribution of the ground truth elements. In this study, we focus on two rates that actually generate a high score only if the majority of the elements of a ground truth group has been correctly predicted: the coefficient of determination (also known as R-squared or R2) and the symmetric mean absolute percentage error (SMAPE). After showing their mathematical properties, we report a comparison between R2 and SMAPE in several use cases and in two real medical scenarios. Our results demonstrate that the coefficient of determination (R-squared) is more informative and truthful than SMAPE, and does not have the interpretability limitations of MSE, RMSE, MAE and MAPE. We therefore suggest the usage of R-squared as standard metric to evaluate regression analyses in any scientific domain.

Download Full-text

Nanosecond Photodynamics Simulations of a Cis-Trans Isomerization Are Enabled by Machine Learning

10.26434/chemrxiv.13047863 ◽

2020 ◽

Author(s):

Jingbai Li ◽

Patrick Reiser ◽

André Eberhard ◽

Pascal Friederich ◽

Steven Lopez

Keyword(s):

Machine Learning ◽

Neural Networks ◽

Excited State ◽

Adaptive Sampling ◽

Computational Cost ◽

Ground Truth ◽

Absolute Error ◽

Photochemical Reactions ◽

Computational Techniques ◽

Full Potential

Photochemical reactions are being increasingly used to construct complex molecular architectures with mild and straightforward reaction conditions. Computational techniques are increasingly important to understand the reactivities and chemoselectivities of photochemical isomerization reactions because they offer molecular bonding information along the excited-state(s) of photodynamics. These photodynamics simulations are resource-intensive and are typically limited to 1–10 picoseconds and 1,000 trajectories due to high computational cost. Most organic photochemical reactions have excited-state lifetimes exceeding 1 picosecond, which places them outside possible computational studies. Westermeyr et al. demonstrated that a machine learning approach could significantly lengthen photodynamics simulation times for a model system, methylenimmonium cation (CH2NH2+).We have developed a Python-based code, Python Rapid Artificial Intelligence Ab Initio Molecular Dynamics (PyRAI2MD), to accomplish the unprecedented 10 ns cis-trans photodynamics of trans-hexafluoro-2-butene (CF3–CH=CH–CF3) in 3.5 days. The same simulation would take approximately 58 years with ground-truth multiconfigurational dynamics. We proposed an innovative scheme combining Wigner sampling, geometrical interpolations, and short-time quantum chemical trajectories to effectively sample the initial data, facilitating the adaptive sampling to generate an informative and data-efficient training set with 6,232 data points. Our neural networks achieved chemical accuracy (mean absolute error of 0.032 eV). Our 4,814 trajectories reproduced the S1 half-life (60.5 fs), the photochemical product ratio (trans: cis = 2.3: 1), and autonomously discovered a pathway towards a carbene. The neural networks have also shown the capability of generalizing the full potential energy surface with chemically incomplete data (trans → cis but not cis → trans pathways) that may offer future automated photochemical reaction discoveries.

Download Full-text

Machine Learning for Sensorless Temperature Estimation of a BLDC Motor

Sensors ◽

10.3390/s21144655 ◽

2021 ◽

Vol 21 (14) ◽

pp. 4655

Author(s):

Dariusz Czerwinski ◽

Jakub Gęca ◽

Krzysztof Kolano

Keyword(s):

Machine Learning ◽

Temperature Measurement ◽

Stochastic Gradient Descent ◽

Estimation Accuracy ◽

Coefficient Of Determination ◽

Percentage Error ◽

Support Vector ◽

Bldc Motor ◽

Temperature Estimation ◽

Motor Operation

In this article, the authors propose two models for BLDC motor winding temperature estimation using machine learning methods. For the purposes of the research, measurements were made for over 160 h of motor operation, and then, they were preprocessed. The algorithms of linear regression, ElasticNet, stochastic gradient descent regressor, support vector machines, decision trees, and AdaBoost were used for predictive modeling. The ability of the models to generalize was achieved by hyperparameter tuning with the use of cross-validation. The conducted research led to promising results of the winding temperature estimation accuracy. In the case of sensorless temperature prediction (model 1), the mean absolute percentage error MAPE was below 4.5% and the coefficient of determination R2 was above 0.909. In addition, the extension of the model with the temperature measurement on the casing (model 2) allowed reducing the error value to about 1% and increasing R2 to 0.990. The results obtained for the first proposed model show that the overheating protection of the motor can be ensured without direct temperature measurement. In addition, the introduction of a simple casing temperature measurement system allows for an estimation with accuracy suitable for compensating the motor output torque changes related to temperature.

Download Full-text

Alometric Model for Estimation of Leaf Area of Garcinia brasiliensis Mart. Through Non-destructive Method

Journal of Agricultural Science ◽

10.5539/jas.v11n10p154 ◽

2019 ◽

Vol 11 (10) ◽

pp. 154

Author(s):

Vinicius de Souza Oliveira ◽

Cássio Francisco Moreira de Carvalho ◽

Juliany Morosini França ◽

Flávia Barreto Pinto ◽

Karina Tiemi Hassuda dos Santos ◽

...

Keyword(s):

Leaf Area ◽

Linear Equations ◽

Absolute Error ◽

Coefficient Of Determination ◽

Maximum Width ◽

Linear Dimensions ◽

Length Width ◽

Independent Variable ◽

The Mean ◽

Non Destructive

The objective of the present study was to test and establish mathematical models to estimate the leaf area of Garcinia brasiliensis Mart. through linear dimensions of the length, width and product of both measurements. In this way, 500 leaves of trees with age between 4 and 6 years were collected from all the cardinal points of the plant in the municipality of São Mateus, North of the State of Espírito Santo, Brazil. The length (L) along the main midrib, the maximum width (W), the product of the length with the width (LW) and the observed leaf area (OLA) were obtained for all leaves. From these measurements were adjusted linear equations of first degree, quadratic and power, in which OLA was used as dependent variable as function of L, W and LW as independent variable. For the validation, the values of L, W and LW of 100 random leaves were substituted in the equations generated in the modeling, thus obtaining the estimated leaf area (ELA). The values of the means of ELA and OLA were tested by Student’s t test 5% of probability. The mean absolute error (MAE), root mean square error (RMSE) and Willmott’s index d for all proposed models were also determined. The choice of the best model was based on the non significant values in the comparison of the means of ELA and OLA, values of MAE and RMSE closer to zero and value of the index d and coefficient of determination (R2) close to unity. The equation that best estimates leaf area of Garcinia brasiliensis Mart. in a way non-destructive is the power model represented by por ELA = 0.7470(LW)0.9842 and R2 = 0.9949.

Download Full-text

Novel Spatiotemporal Feature Extraction Parallel Deep Neural Network for Forecasting Confirmed Cases of Coronavirus Disease 2019

10.1101/2020.04.30.20086538 ◽

2020 ◽

Author(s):

Chiou-Jye Huang ◽

Yamin Shen ◽

Ping-Huan Kuo ◽

Yung-Hsiang Chen

Keyword(s):

Neural Network ◽

Deep Neural Network ◽

Absolute Error ◽

Mean Absolute Percentage Error ◽

The Other ◽

Percentage Error ◽

Model Assessment ◽

Absolute Percentage Error ◽

Spatiotemporal Feature ◽

The Mean

AbstractThe coronavirus disease 2019 pandemic continues as of March 26 and spread to Europe on approximately February 24. A report from April 29 revealed 1.26 million confirmed cases and 125 928 deaths in Europe. This study proposed a novel deep neural network framework, COVID-19Net, which parallelly combines a convolutional neural network (CNN) and bidirectional gated recurrent units (GRUs). Three European countries with severe outbreaks were studied—Germany, Italy, and Spain—to extract spatiotemporal feature and predict the number of confirmed cases. The prediction results acquired from COVID-19Net were compared to those obtained using a CNN, GRU, and CNN-GRU. The mean absolute error, mean absolute percentage error, and root mean square error, which are commonly used model assessment indices, were used to compare the accuracy of the models. The results verified that COVID-19Net was notably more accurate than the other models. The mean absolute percentage error generated by COVID-19Net was 1.447 for Germany, 1.801 for Italy, and 2.828 for Spain, which were considerably lower than those of the other models. This indicated that the proposed framework can accurately predict the accumulated number of confirmed cases in the three countries and serve as a crucial reference for devising public health strategies.

Download Full-text

Programmatic Labeling of Dark Data for Artificial Intelligence in Spatial Informatics

10.5194/egusphere-egu21-16326 ◽

2021 ◽

Author(s):

Jason Meil

Keyword(s):

Machine Learning ◽

Binary Classification ◽

Joint Probability ◽

Ground Truth ◽

Training Data ◽

Data Sources ◽

Joint Probability Distribution ◽

Weak Supervision ◽

Response Variable ◽

High Degree

Data preparation process generally consumes up to 80% of the Data Scientists time, with 60% of that being attributed to cleaning and labeling data.[1]&#160; Our solution is to use automated pipelines to prepare, annotate, and catalog data. The first step upon ingestion, especially in the case of real world&#8212;unstructured and unlabeled datasets&#8212;is to leverage Snorkel, a tool specifically designed around a paradigm to rapidly create, manage, and model training data. Configured properly, Snorkel can be leveraged to temper this labeling bottle-neck through a process called weak supervision. Weak supervision uses programmatic labeling functions&#8212;heuristics, distant supervision, SME or knowledge base&#8212;scripted in python to generate &#8220;noisy labels&#8221;. The function traverses the entirety of the dataset and feeds the labeled data into a generative&#8212;conditionally probabilistic&#8212;model. The function of this model is to output the distribution of each response variable and predict the conditional probability based on a joint probability distribution algorithm. This is done by comparing the various labeling functions and the degree to which their outputs are congruent to each other. A single labeling function that has a high degree of congruence with other labeling functions will have a high degree of learned accuracy, that is, the fraction of predictions that the model got right. Conversely, single labeling functions that have a low degree of congruence with other functions will have low learned accuracy. Each prediction is then combined by the estimated weighted accuracy, whereby the predictions of the higher learned functions are counted multiple times. The result yields a transformation from a binary classification of 0 or 1 to a fuzzy label between 0 and 1&#8212; there is &#8220;x&#8221; probability that based on heuristic &#8220;n&#8221;, the response variable is &#8220;y&#8221;. The addition of data to this generative model multi-class inference will be made on the response variables positive, negative, or abstain, assigning probabilistic labels to potentially millions of data points. Thus, we have generated a discriminative ground truth for all further labeling efforts and have improved the scalability of our models. Labeling functions can be applied to unlabeled data to further machine learning efforts. &#160; Once our datasets are labeled and a ground truth is established, we need to persist the data into our delta lake since it combines the most performant aspects of a warehouse with the low-cost storage for data lakes. In addition, the lake can accept unstructured, semi structured, or structured data sources, and those sources can be further aggregated into raw ingestion, cleaned, and feature engineered data layers.&#160; By sectioning off the data sources into these &#8220;layers&#8221;, the data engineering portion is abstracted away from the data scientist, who can access model ready data at any time.&#160; Data can be ingested via batch or stream.&#160; &#160; The design of the entire ecosystem is to eliminate as much technical debt in machine learning paradigms as possible in terms of configuration, data collection, verification, governance, extraction, analytics, process management, resource management, infrastructure, monitoring, and post verification.&#160;

Download Full-text

Machine learning and Grad-Cam based vascular aging assessment using photoplethysmogram (Preprint)

10.2196/preprints.31709 ◽

2021 ◽

Author(s):

Hangsik Shin

Keyword(s):

Machine Learning ◽

Correlation Coefficient ◽

Age Estimation ◽

Mean Squared Error ◽

Mean Absolute Error ◽

Absolute Error ◽

Coefficient Of Determination ◽

Vascular Aging ◽

Squared Error ◽

Vascular Age

BACKGROUND Arterial stiffness due to vascular aging is a major indicator for evaluating cardiovascular risk. OBJECTIVE In this study, we propose a method of estimating age by applying machine learning to photoplethysmogram for non-invasive vascular age assessment. METHODS The machine learning-based age estimation model that consists of three convolutional layers and two-layer fully connected layers, was developed using segmented photoplethysmogram by pulse from a total of 752 adults aged 19–87 years. The performance of the developed model was quantitatively evaluated using mean absolute error, root-mean-squared-error, Pearson’s correlation coefficient, coefficient of determination. The Grad-Cam was used to explain the contribution of photoplethysmogram waveform characteristic in vascular age estimation. RESULTS Mean absolute error of 8.03, root mean squared error of 9.96, 0.62 of correlation coefficient, and 0.38 of coefficient of determination were shown through 10-fold cross validation. Grad-Cam, used to determine the weight that the input signal contributes to the result, confirmed that the contribution to the age estimation of the photoplethysmogram segment was high around the systolic peak. CONCLUSIONS The machine learning-based vascular aging analysis method using the PPG waveform showed comparable or superior performance compared to previous studies without complex feature detection in evaluating vascular aging. CLINICALTRIAL 2015-0104

Download Full-text

Predicting Convective Storm Characteristics using Machine Learning from Hi-Resolution NWP Forecasts

10.5194/egusphere-egu21-7516 ◽

2021 ◽

Author(s):

Aniel Jardines

Keyword(s):

Machine Learning ◽

Weather Forecasting ◽

Binary Classification ◽

Air Traffic ◽

Mitigation Strategies ◽

Supervised Machine Learning ◽

Machine Learning Techniques ◽

Traffic Flow Management ◽

Storm Characteristics ◽

Convective Weather

Convective weather&#160;represents a significant disruption to air traffic flow management (ATFM) operations. Thunderstorms&#160;are the cause for a substantial amount of delay in&#160;both the en-route and airport&#160;environment. Before the day of operations, poor prediction capability of convective weather prohibits traffic managers from considering weather mitigation strategies during the pre-tactical phase of ATFM planning. As a result, convective weather is mitigated tactically, possibly leading to excessive delays. &#160;The skill of weather forecasting has greatly improved in recent years. Hi-resolution weather models can predict the future state of the atmosphere for some weather parameters. However, incorporating the output from these sophisticated weather products into an ATFM solution that provides easily interpreted information by the air traffic managers remains a challenge.&#160;This paper combines data from high-resolution&#160;numerical&#160;weather&#160;predictions&#160;with actual storm observations from lightning detecting and satellite images. It applies supervised machine learning techniques such as binary classification, multiclass classification, and regression to train neural networks to predict the occurrence, severity, and altitude of thunderstorms. The model predictions are given up to 36hr in advance, within timeframes necessary for pre-tactical planning of ATFM, providing traffic managers with valuable information for developing weather mitigation plans.&#160;

Download Full-text

Prediction of Compound-Protein Interactions with Machine Learning Methods

Chemoinformatics and Advanced Machine Learning Perspectives ◽

10.4018/978-1-61520-911-8.ch016 ◽

2011 ◽

pp. 304-317

Author(s):

Yoshihiro Yamanishi ◽

Hisashi Kashima

Keyword(s):

Machine Learning ◽

Protein Interactions ◽

Chemical Structure ◽

Genomic Sequence ◽

Sequence Data ◽

Binary Classification ◽

Biological Data ◽

Supervised Machine Learning ◽

Learning Methods ◽

Machine Learning Methods

In silico prediction of compound-protein interactions from heterogeneous biological data is critical in the process of drug development. In this chapter the authors review several supervised machine learning methods to predict unknown compound-protein interactions from chemical structure and genomic sequence information simultaneously. The authors review several kernel-based algorithms from two different viewpoints: binary classification and dimension reduction. In the results, they demonstrate the usefulness of the methods on the prediction of drug-target interactions and ligand-protein interactions from chemical structure data and genomic sequence data.

Download Full-text

Retrieving individual tree heights from a point cloud generated with optical imagery from an unmanned aerial vehicle (UAV)

Canadian Journal of Forest Research ◽

10.1139/cjfr-2019-0418 ◽

2020 ◽

Vol 50 (10) ◽

pp. 1012-1024

Author(s):

Meimei Wang ◽

Jiayuan Lin

Keyword(s):

Unmanned Aerial Vehicle ◽

Point Cloud ◽

Laser Scanning ◽

Tree Height ◽

Absolute Error ◽

Coefficient Of Determination ◽

Surface Model ◽

Individual Tree ◽

Aerial Vehicle ◽

The Mean

Individual tree height (ITH) is one of the most important vertical structure parameters of a forest. Field measurement and laser scanning are very expensive for large forests. In this paper, we propose a cost-effective method to acquire ITHs in a forest using the optical overlapping images captured by an unmanned aerial vehicle (UAV). The data sets, including a point cloud, a digital surface model (DSM), and a digital orthorectified map (DOM), were produced from the UAV imagery. The canopy height model (CHM) was obtained by subtracting the digital elevation model (DEM) from the DSM removed of low vegetation. Object-based image analysis was used to extract individual tree crowns (ITCs) from the DOM, and ITHs were initially extracted by overlaying ITC outlines on the CHM. As the extracted ITHs were generally slightly shorter than the measured ITHs, a linear relationship was established between them. The final ITHs of the test site were retrieved by inputting extracted ITHs into the linear regression model. As a result, the coefficient of determination (R2), the root mean square error (RMSE), the mean absolute error (MAE), and the mean relative error (MRE) of the retrieved ITHs against the measured ITHs were 0.92, 1.08 m, 0.76 m, and 0.08, respectively.

Download Full-text

Automatic recognition of self-acknowledged limitations in clinical research literature

Journal of the American Medical Informatics Association ◽

10.1093/jamia/ocy038 ◽

2018 ◽

Vol 25 (7) ◽

pp. 855-861 ◽

Cited By ~ 4

Author(s):

Halil Kilicoglu ◽

Graciela Rosemblat ◽

Mario Malički ◽

Gerben ter Riet

Keyword(s):

Machine Learning ◽

Clinical Research ◽

Binary Classification ◽

Classification Performance ◽

Research Literature ◽

Machine Learning Algorithms ◽

Supervised Machine Learning ◽

Support Vector ◽

Rule Based ◽

Research Transparency

Abstract Objective To automatically recognize self-acknowledged limitations in clinical research publications to support efforts in improving research transparency. Methods To develop our recognition methods, we used a set of 8431 sentences from 1197 PubMed Central articles. A subset of these sentences was manually annotated for training/testing, and inter-annotator agreement was calculated. We cast the recognition problem as a binary classification task, in which we determine whether a given sentence from a publication discusses self-acknowledged limitations or not. We experimented with three methods: a rule-based approach based on document structure, supervised machine learning, and a semi-supervised method that uses self-training to expand the training set in order to improve classification performance. The machine learning algorithms used were logistic regression (LR) and support vector machines (SVM). Results Annotators had good agreement in labeling limitation sentences (Krippendorff’s α = 0.781). Of the three methods used, the rule-based method yielded the best performance with 91.5% accuracy (95% CI [90.1-92.9]), while self-training with SVM led to a small improvement over fully supervised learning (89.9%, 95% CI [88.4-91.4] vs 89.6%, 95% CI [88.1-91.1]). Conclusions The approach presented can be incorporated into the workflows of stakeholders focusing on research transparency to improve reporting of limitations in clinical studies.

Download Full-text