scholarly journals A Pointwise Evaluation Metric to Visualize Errors in Machine Learning Surrogate Models

2021 ◽  
Author(s):  
Seyed Shayan Sajjadinia ◽  
Bruno Carpentieri ◽  
Gerhard A. Holzapfel
Keyword(s):  
Machine Learning ◽  
Numerical Model ◽  
High Performance ◽  
Mean Squared Error ◽  
Performance Model ◽  
Squared Error ◽  
The Mean ◽  
Evaluation Metric ◽  
Definition Of ◽  
Insight Into

Numerical simulation is widely used to study physical systems, although it can be computationally too expensive. To counter this limitation, a surrogate may be used, which is a high-performance model that replaces the main numerical model by using, e.g., a machine learning (ML) regressor that is trained on a previously generated subset of possible inputs and outputs of the numerical model. In this context, inspired by the definition of the mean squared error (MSE) metric, we introduce the pointwise MSE (PMSE) metric, which can give a better insight into the performance of such ML models over the test set, by focusing on every point that forms the physical system. To show the merits of the metric, we will create a dataset of a physics problem that will be used to train an ML surrogate, which will then be evaluated by the metrics. In our experiment, the PMSE contour demonstrates how the model learns the physics in different model regions and, in particular, the correlation between the characteristics of the numerical model and the learning progress can be observed. We therefore conclude that this simple and efficient metric can provide complementary and potentially interpretable information regarding the performance and functionality of the surrogate.

scholarly journals Estimation of Mean Squared Error of X-11-ARIMA and Other Estimators of Time Series Components

2014 ◽  
Vol 30 (4) ◽  
pp. 811-838 ◽  
Author(s):  
Danny Pfeffermann ◽  
Michail Sverchkov
Keyword(s):  
Mean Squared Error ◽  
Decomposition Methods ◽  
Seasonal Effects ◽  
Sampling Errors ◽  
Squared Error ◽  
The Mean ◽  
Definition Of ◽  
Establishment Survey ◽  
Time Invariant ◽  
Bureau Of Labor Statistics

Abstract This article considers the familiar but very important problem of how to estimate the mean squared error (MSE) of seasonally adjusted and trend estimators produced by X-11-ARIMA or other decomposition methods. The MSE estimators are obtained by defining the unknown target components such as the trend and seasonal effects to be the hypothetical X-11 estimates of them that would be obtained if there were no sampling errors and the series were sufficiently long to allow the use of the symmetric filters embedded in the programme, which are time invariant. This definition of the component series conforms to the classical definition of the target parameters in design-based survey sampling theory, so that users should find it comfortable to adjust to this definition. The performance of the MSE estimators is assessed by a simulation study and by application to real series obtained from an establishment survey carried out by the Bureau of Labor Statistics in the U.S.A.

scholarly journals Bitcoin Price Prediction using SVM and ARIMA Model

2021 ◽  
pp. 1094-1096
Author(s):  
Gausiya Momin ◽  
Trupti Ingle ◽  
Vaishnavi Mirajkar ◽  
A. A. Magar
Keyword(s):  
Machine Learning ◽  
Regression Models ◽  
Mean Squared Error ◽  
Learning Algorithms ◽  
Arima Model ◽  
Machine Learning Algorithms ◽  
Price Prediction ◽  
Squared Error ◽  
Minute Interval ◽  
The Mean

Bitcoin is the most profitable in the cryptocurrency market. However, the prices of Bitcoin have highly fluctuated which makes them very difficult to predict. This research aims to discover the most efficient accuracy model to predict Bitcoin prices from various machine learning algorithms. Using one-minute interval trading data on the exchange website name is bit stamp from January 1, 2012, to January 8, 2018, some different regression models with sci-kit- learn and Keras libraries had experimented. The best results showed that the Mean Squared Error (MSE) was as low as 0.00002 and the R-Square (R2) was as high as 99.2 Percentage.

scholarly journals Machine Learning Technique to Improve an Impedance Matching Characteristic of a Bent Monopole Antenna

2021 ◽  
Vol 11 (22) ◽  
pp. 10829
Author(s):  
Jaeyul Choo ◽  
Pho Thi Ha Anh ◽  
Yong-Hwa Kim
Keyword(s):  
Machine Learning ◽  
Mean Squared Error ◽  
Impedance Matching ◽  
Learning Model ◽  
Monopole Antenna ◽  
Machine Learning Technique ◽  
Squared Error ◽  
Machine Learning Model ◽  
Learning Technique ◽  
The Mean

We designed the wire monopole antenna bent at three points by applying a machine learning technique to achieve a good impedance matching characteristic. After performing the deep neural network (DNN)-based training, we validated our machine learning model by evaluating mean squared error and R-squared score. Considering the mean squared error of about zero and R-squared score of about one, the performance prediction by the resulting machine learning model showed a high accuracy compared with that by the numerical electromagnetic simulation. Finally, we interpreted the operating principle of the antennas with a good impedance matching characteristic by analyzing equivalent circuits corresponding to their structures. The accomplished works in this research provide us with the possibility to use the machine learning technique in the antenna design.

scholarly journals Experimental Evaluation of Computer Vision and Machine Learning-Based UAV Detection and Ranging

Drones ◽  
2021 ◽  
Vol 5 (2) ◽  
pp. 37
Author(s):  
Bingsheng Wei ◽  
Martin Barczyk
Keyword(s):  
Machine Learning ◽  
Mean Squared Error ◽  
Tracking System ◽  
Ground Truth ◽  
White Background ◽  
Cascade Classifier ◽  
Detection Algorithms ◽  
Squared Error ◽  
Test Conditions ◽  
Video Feed

We consider the problem of vision-based detection and ranging of a target UAV using the video feed from a monocular camera onboard a pursuer UAV. Our previously published work in this area employed a cascade classifier algorithm to locate the target UAV, which was found to perform poorly in complex background scenes. We thus study the replacement of the cascade classifier algorithm with newer machine learning-based object detection algorithms. Five candidate algorithms are implemented and quantitatively tested in terms of their efficiency (measured as frames per second processing rate), accuracy (measured as the root mean squared error between ground truth and detected location), and consistency (measured as mean average precision) in a variety of flight patterns, backgrounds, and test conditions. Assigning relative weights of 20%, 40% and 40% to these three criteria, we find that when flying over a white background, the top three performers are YOLO v2 (76.73 out of 100), Faster RCNN v2 (63.65 out of 100), and Tiny YOLO (59.50 out of 100), while over a realistic background, the top three performers are Faster RCNN v2 (54.35 out of 100, SSD MobileNet v1 (51.68 out of 100) and SSD Inception v2 (50.72 out of 100), leading us to recommend Faster RCNN v2 as the recommended solution. We then provide a roadmap for further work in integrating the object detector into our vision-based UAV tracking system.

Computing trade-offs in robust design: Perspectives of the mean squared error

2011 ◽  
Vol 60 (2) ◽  
pp. 248-255 ◽  
Author(s):  
Sangmun Shin ◽  
Funda Samanlioglu ◽  
Byung Rae Cho ◽  
Margaret M. Wiecek
Keyword(s):  
Robust Design ◽  
Mean Squared Error ◽  
Squared Error ◽  
Trade Offs ◽  
The Mean

A new generic method to improve machine learning applications in official statistics

2021 ◽  
pp. 1-16
Author(s):  
Kevin Kloos
Keyword(s):  
Machine Learning ◽  
Mean Squared Error ◽  
Statistical Properties ◽  
Machine Learning Algorithms ◽  
Official Statistics ◽  
Academic Literature ◽  
Misclassification Bias ◽  
Squared Error ◽  
Machine Learning Applications ◽  
Applications Of Machine Learning

The use of machine learning algorithms at national statistical institutes has increased significantly over the past few years. Applications range from new imputation schemes to new statistical output based entirely on machine learning. The results are promising, but recent studies have shown that the use of machine learning in official statistics always introduces a bias, known as misclassification bias. Misclassification bias does not occur in traditional applications of machine learning and therefore it has received little attention in the academic literature. In earlier work, we have collected existing methods that are able to correct misclassification bias. We have compared their statistical properties, including bias, variance and mean squared error. In this paper, we present a new generic method to correct misclassification bias for time series and we derive its statistical properties. Moreover, we show numerically that it has a lower mean squared error than the existing alternatives in a wide variety of settings. We believe that our new method may improve machine learning applications in official statistics and we aspire that our work will stimulate further methodological research in this area.

scholarly journals Day-Ahead Forecasting of Hourly Photovoltaic Power Based on Robust Multilayer Perception

2018 ◽  
Vol 10 (12) ◽  
pp. 4863 ◽  
Author(s):  
Chao Huang ◽  
Longpeng Cao ◽  
Nanxin Peng ◽  
Sijia Li ◽  
Jing Zhang ◽  
...  
Keyword(s):  
Power Plants ◽  
Mean Squared Error ◽  
Absolute Error ◽  
Multilayer Perception ◽  
Squared Error ◽  
The Mean ◽  
Effectiveness And Efficiency ◽  
Mlp Network ◽  
Grid Operation ◽  
Better Than

Photovoltaic (PV) modules convert renewable and sustainable solar energy into electricity. However, the uncertainty of PV power production brings challenges for the grid operation. To facilitate the management and scheduling of PV power plants, forecasting is an essential technique. In this paper, a robust multilayer perception (MLP) neural network was developed for day-ahead forecasting of hourly PV power. A generic MLP is usually trained by minimizing the mean squared loss. The mean squared error is sensitive to a few particularly large errors that can lead to a poor estimator. To tackle the problem, the pseudo-Huber loss function, which combines the best properties of squared loss and absolute loss, was adopted in this paper. The effectiveness and efficiency of the proposed method was verified by benchmarking against a generic MLP network with real PV data. Numerical experiments illustrated that the proposed method performed better than the generic MLP network in terms of root mean squared error (RMSE) and mean absolute error (MAE).

scholarly journals On double stage minimax-shrinkage estimator for generalized Rayleigh model

2016 ◽  
Vol 5 (1) ◽  
pp. 39 ◽  
Author(s):  
Abbas Najim Salman ◽  
Maymona Ameen
Keyword(s):  
Sample Size ◽  
Shape Parameter ◽  
Mean Squared Error ◽  
Scale Parameter ◽  
Rayleigh Distribution ◽  
Shrinkage Estimator ◽  
Squared Error ◽  
Expected Sample Size ◽  
Generalized Rayleigh Distribution ◽  
The Mean

<p>This paper is concerned with minimax shrinkage estimator using double stage shrinkage technique for lowering the mean squared error, intended for estimate the shape parameter (a) of Generalized Rayleigh distribution in a region (R) around available prior knowledge (a<sub>0</sub>) about the actual value (a) as initial estimate in case when the scale parameter (l) is known .</p><p>In situation where the experimentations are time consuming or very costly, a double stage procedure can be used to reduce the expected sample size needed to obtain the estimator.</p><p>The proposed estimator is shown to have smaller mean squared error for certain choice of the shrinkage weight factor y(<strong>×</strong>) and suitable region R.</p><p>Expressions for Bias, Mean squared error (MSE), Expected sample size [E (n/a, R)], Expected sample size proportion [E(n/a,R)/n], probability for avoiding the second sample and percentage of overall sample saved  for the proposed estimator are derived.</p><p>Numerical results and conclusions for the expressions mentioned above were displayed when the consider estimator are testimator of level of significanceD.</p><p>Comparisons with the minimax estimator and with the most recent studies were made to shown the effectiveness of the proposed estimator.</p>

scholarly journals Performance Analysis of AOA-Based Localization Using the LS Approach: Explicit Expression of Mean-Squared Error

2020 ◽  
Vol 2020 ◽  
pp. 1-22
Author(s):  
Byung-Kwon Son ◽  
Do-Jin An ◽  
Joon-Ho Lee
Keyword(s):  
Explicit Expression ◽  
Mean Squared Error ◽  
Weighted Least Squares ◽  
Localization Algorithm ◽  
Angle Of Arrival ◽  
Squared Error ◽  
Distance Weighted ◽  
The Mean ◽  
Passive Localization ◽  
Location Estimate

In this paper, a passive localization of the emitter using noisy angle-of-arrival (AOA) measurements, called Brown DWLS (Distance Weighted Least Squares) algorithm, is considered. The accuracy of AOA-based localization is quantified by the mean-squared error. Various estimates of the AOA-localization algorithm have been derived (Doğançay and Hmam, 2008). Explicit expression of the location estimate of the previous study is used to get an analytic expression of the mean-squared error (MSE) of one of the various estimates. To validate the derived expression, we compare the MSE from the Monte Carlo simulation with the analytically derived MSE.

Sign in / Sign up

Export Citation Format

Share Document