Using Multiple Surrogates for Minimization of the RMS Error in Meta-Modeling

Volume 1: 34th Design Automation Conference, Parts A and B ◽

10.1115/detc2008-49240 ◽

2008 ◽

Cited By ~ 4

Author(s):

Felipe A. C. Viana ◽

Raphael T. Haftka

Keyword(s):

Cross Validation ◽

Weighted Average ◽

Large Set ◽

High Dimensions ◽

Engineering Problems ◽

Cross Validation Error ◽

Data Points ◽

Meta Modeling ◽

Rms Error ◽

Rms Errors

Surrogate models are commonly used to replace expensive simulations of engineering problems. Frequently, a single surrogate is chosen based on past experience. Previous work has shown that fitting multiple surrogates and picking one based on cross-validation errors (PRESS in particular) is a good strategy, and that cross validation errors may also be used to create a weighted surrogate. In this paper, we discuss whether to use the best PRESS solution or a weighted surrogate when a single surrogate is needed. We propose the minimization of the integrated square error as a way to compute the weights of the weighted average surrogate. We find that it pays to generate a large set of different surrogates and then use PRESS as a criterion for selection. We find that the cross validation error vectors provide an excellent estimate of the RMS errors when the number of data points is high. Hence the use of cross validation errors for choosing a surrogate and for calculating the weights of weighted surrogates becomes more attractive in high dimensions. However, it appears that the potential gains from using weighted surrogates diminish substantially in high dimensions.

Download Full-text

Ensemble of surrogates and cross-validation for rapid and accurate predictions using small data sets

Artificial intelligence for engineering design analysis and manufacturing ◽

10.1017/s089006041900026x ◽

2019 ◽

Vol 33 (4) ◽

pp. 484-501 ◽

Cited By ~ 4

Author(s):

Reza Alizadeh ◽

Liangyue Jia ◽

Anand Balu Nellippallil ◽

Guoxin Wang ◽

Jia Hao ◽

...

Keyword(s):

Cross Validation ◽

Weighted Average ◽

Computational Time ◽

Small Data ◽

Data Sets ◽

Computationally Efficient ◽

Cross Validation Error ◽

Rod Rolling ◽

Ensemble Of Surrogates ◽

Time Required

AbstractIn engineering design, surrogate models are often used instead of costly computer simulations. Typically, a single surrogate model is selected based on the previous experience. We observe, based on an analysis of the published literature, that fitting an ensemble of surrogates (EoS) based on cross-validation errors is more accurate but requires more computational time. In this paper, we propose a method to build an EoS that is both accurate and less computationally expensive. In the proposed method, the EoS is a weighted average surrogate of response surface models, kriging, and radial basis functions based on overall cross-validation error. We demonstrate that created EoS is accurate than individual surrogates even when fewer data points are used, so computationally efficient with relatively insensitive predictions. We demonstrate the use of an EoS using hot rod rolling as an example. Finally, we include a rule-based template which can be used for other problems with similar requirements, for example, the computational time, required accuracy, and the size of the data.

Download Full-text

A test of various computational solvation models on a set of “difficult” organic compounds

Canadian Journal of Chemistry ◽

10.1139/v09-071 ◽

2009 ◽

Vol 87 (8) ◽

pp. 1154-1162 ◽

Cited By ~ 18

Author(s):

J. Peter Guthrie ◽

Igor Povar

Keyword(s):

Large Set ◽

Aromatic Rings ◽

Halogen Compounds ◽

Solvation Energies ◽

Polyfunctional Compounds ◽

Solvation Models ◽

Rms Error ◽

Definition Of ◽

Dielectric Continuum ◽

Rms Errors

Various dielectric continuum models in Gaussian 03, based on the SCRF approach, PCM, CPCM, DPCM, IEFPCM, IPCM, and SCIPCM, have been tested on a set of 54 highly polar, generally polyfunctional compounds for which experimental solvation energies are available. These compounds span a range of 13 kcal/mol in ΔGt. The root-mean-square (RMS) errors for the full set of compounds range from 2.48 for DPCM to 1.77 for IPCM. For each method, classes of compounds which were not handled well could be identified. If these classes of compounds were omitted, the performance improved, and ranged from 1.58 (PCM, 39 compounds) to 1.02 (IPCM, 42 compounds). Models in the PCM family (PCM, CPCM, DPCM, and IEFPCM) with the recommended UAHF or UAKS sets of radii rely on a highly parameterized definition of the solvent cavity. Where this parameterization was inadequate, the calculated solvation energies were less reliable. This has been demonstrated by devising a new parameterization for PCM and halogen compounds, which markedly improves performance for polyhalogen compounds. The effective radius for the portion of the cavity centered on a halogen atom was assumed to be linear in the electron-withdrawing or -donating properties of the rest of the molecule as measured by Hammett σ (for halogens on aromatic rings) or Taft σ* (for halogens on aliphatic carbons). This new parameterization for PCM was tested on a set of 45 aliphatic and 22 aromatic polyhalogen compounds and shown to do well. IPCM, which was already the best of the methods in Gaussian, can be considerably improved by a parameterization to allow for cavitation, dispersion, and hydrogen bonding. A large set of compounds was used for the parameterization to have multiple examples for each parameter and as far as possible to have molecules with multiple instances of each structural feature. In the end, 15 parameters were found to be defined by the data for 241 compounds. With this parameter set, the RMS error for the set used for fitting was 0.81 kcal/mol, and the RMS error for the original set of 54 compounds was 0.85. With this new parameterization, IPCM is clearly the best of the methods available in Gaussian 03.

Download Full-text

Discrete natural neighbour interpolation with uncertainty using cross-validation error-distance fields

PeerJ Computer Science ◽

10.7717/peerj-cs.282 ◽

2020 ◽

Vol 6 ◽

pp. e282

Author(s):

Thomas R. Etherington

Keyword(s):

Cross Validation ◽

Interpolation Method ◽

Local Method ◽

Distance Fields ◽

Spatially Adaptive ◽

Cross Validation Error ◽

Data Points ◽

Point Data ◽

Continuous Field ◽

Error Distance

Interpolation techniques provide a method to convert point data of a geographic phenomenon into a continuous field estimate of that phenomenon, and have become a fundamental geocomputational technique of spatial and geographical analysts. Natural neighbour interpolation is one method of interpolation that has several useful properties: it is an exact interpolator, it creates a smooth surface free of any discontinuities, it is a local method, is spatially adaptive, requires no statistical assumptions, can be applied to small datasets, and is parameter free. However, as with any interpolation method, there will be uncertainty in how well the interpolated field values reflect actual phenomenon values. Using a method based on natural neighbour distance based rates of error calculated for data points via cross-validation, a cross-validation error-distance field can be produced to associate uncertainty with the interpolation. Virtual geography experiments demonstrate that given an appropriate number of data points and spatial-autocorrelation of the phenomenon being interpolated, the natural neighbour interpolation and cross-validation error-distance fields provide reliable estimates of value and error within the convex hull of the data points. While this method does not replace the need for analysts to use sound judgement in their interpolations, for those researchers for whom natural neighbour interpolation is the best interpolation option the method presented provides a way to assess the uncertainty associated with natural neighbour interpolations.

Download Full-text

A multilayer multimodal detection and prediction model based on explainable artificial intelligence for Alzheimer’s disease

Scientific Reports ◽

10.1038/s41598-021-82098-3 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Shaker El-Sappagh ◽

Jose M. Alonso ◽

S. M. Riazul Islam ◽

Ahmad M. Sultan ◽

Kyung Sup Kwak

Keyword(s):

Alzheimer’S Disease ◽

Alzheimer's Disease ◽

Cross Validation ◽

Disease Risk ◽

Binary Classification ◽

Fuzzy Rule ◽

Large Set ◽

Detection Model ◽

Multi Class Classification ◽

Clinical Measures

AbstractAlzheimer’s disease (AD) is the most common type of dementia. Its diagnosis and progression detection have been intensively studied. Nevertheless, research studies often have little effect on clinical practice mainly due to the following reasons: (1) Most studies depend mainly on a single modality, especially neuroimaging; (2) diagnosis and progression detection are usually studied separately as two independent problems; and (3) current studies concentrate mainly on optimizing the performance of complex machine learning models, while disregarding their explainability. As a result, physicians struggle to interpret these models, and feel it is hard to trust them. In this paper, we carefully develop an accurate and interpretable AD diagnosis and progression detection model. This model provides physicians with accurate decisions along with a set of explanations for every decision. Specifically, the model integrates 11 modalities of 1048 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) real-world dataset: 294 cognitively normal, 254 stable mild cognitive impairment (MCI), 232 progressive MCI, and 268 AD. It is actually a two-layer model with random forest (RF) as classifier algorithm. In the first layer, the model carries out a multi-class classification for the early diagnosis of AD patients. In the second layer, the model applies binary classification to detect possible MCI-to-AD progression within three years from a baseline diagnosis. The performance of the model is optimized with key markers selected from a large set of biological and clinical measures. Regarding explainability, we provide, for each layer, global and instance-based explanations of the RF classifier by using the SHapley Additive exPlanations (SHAP) feature attribution framework. In addition, we implement 22 explainers based on decision trees and fuzzy rule-based systems to provide complementary justifications for every RF decision in each layer. Furthermore, these explanations are represented in natural language form to help physicians understand the predictions. The designed model achieves a cross-validation accuracy of 93.95% and an F1-score of 93.94% in the first layer, while it achieves a cross-validation accuracy of 87.08% and an F1-Score of 87.09% in the second layer. The resulting system is not only accurate, but also trustworthy, accountable, and medically applicable, thanks to the provided explanations which are broadly consistent with each other and with the AD medical literature. The proposed system can help to enhance the clinical understanding of AD diagnosis and progression processes by providing detailed insights into the effect of different modalities on the disease risk.

Download Full-text

Evaluation and Recommendations for Improving the Accuracy of an Inexpensive Water Temperature Logger

Journal of Atmospheric and Oceanic Technology ◽

10.1175/jtech-d-12-00204.1 ◽

2013 ◽

Vol 30 (7) ◽

pp. 1576-1582 ◽

Cited By ~ 7

Author(s):

S. J. Lentz ◽

J. H. Churchill ◽

C. Marquette ◽

J. Smith

Keyword(s):

Surface Layer ◽

Solar Radiation ◽

Water Temperature ◽

Solar Heating ◽

Bias Error ◽

Mean Square ◽

Temperature Logger ◽

Accurate Temperature ◽

Rms Error ◽

Rms Errors

Abstract Onset's HOBO U22 Water Temp Pros are small, reliable, relatively inexpensive, self-contained temperature loggers that are widely used in studies of oceans, lakes, and streams. An in-house temperature bath calibration of 158 Temp Pros indicated root-mean-square (RMS) errors ranging from 0.01° to 0.14°C, with one value of 0.23°C, consistent with the factory specifications. Application of a quadratic calibration correction substantially reduced the RMS error to less than 0.009°C in all cases. The primary correction was a bias error typically between −0.1° and 0.15°C. Comparison of water temperature measurements from Temp Pros and more accurate temperature loggers during two oceanographic studies indicates that calibrated Temp Pros have an RMS error of ~0.02°C throughout the water column at night and beneath the surface layer influenced by penetrating solar radiation during the day. Larger RMS errors (up to 0.08°C) are observed near the surface during the day due to solar heating of the black Temp Pro housing. Errors due to solar heating are significantly reduced by wrapping the housing with white electrical tape.

Download Full-text

Heat Transfer Correlation for Supercritical Carbon Dioxide Flowing in Vertical Bare Tubes

Volume 6: Beyond Design Basis Events; Student Paper Competition ◽

10.1115/icone21-16453 ◽

2013 ◽

Cited By ~ 2

Author(s):

Sahil Gupta ◽

Eugene Saltanov ◽

Igor Pioro

Keyword(s):

Heat Transfer ◽

Error Analysis ◽

Nuclear Power ◽

Nuclear Reactor ◽

Large Set ◽

Transfer Coefficients ◽

Data Points ◽

Institute Of Technology ◽

Using Data ◽

Gen Iv

Canada among many other countries is in pursuit of developing next generation (Generation IV) nuclear-reactor concepts. One of the main objectives of Generation-IV concepts is to achieve high thermal efficiencies (45–50%). It has been proposed to make use of SuperCritical Fluids (SCFs) as the heat-transfer medium in such Gen IV reactor design concepts such as SuperCritical Water-cooled Reactor (SCWR). An important aspect towards development of SCF applications in novel Gen IV Nuclear Power Plant (NPP) designs is to understand the thermodynamic behavior and prediction of Heat Transfer Coefficients (HTCs) at supercritical (SC) conditions. To calculate forced convection HTCs for simple geometries, a number of empirical 1-D correlations have been proposed using dimensional analysis. These 1-D HTC correlations are developed by applying data-fitting techniques to a model equation with dimensionless terms and can be used for rudimentary calculations. Using similar statistical techniques three correlations were proposed by Gupta et al. [1] for Heat Transfer (HT) in SCCO2. These SCCO2 correlations were developed at the University of Ontario Institute of Technology (Canada) by using a large set of experimental SCCO2 data (∼4,000 data-points) obtained at the Chalk River Laboratories (CRL) AECL. These correlations predict HTC values with an accuracy of ±30% and wall temperatures with an accuracy of ±20% for the analyzed dataset. Since these correlations were developed using data from a single source - CRL (AECL), they can be limited in their range of applicability. To investigate the tangible applicability of these SCCO2 correlations it was imperative to perform a thorough error analysis by checking their results against a set of independent SCCO2 tube data. In this paper SCCO2 data are compiled from various sources and within various experimental flow conditions. HTC and wall-temperature values for these data points are calculated using updated correlations presented in [1] and compared to the experimental values. Error analysis is then shown for these datasets to obtain a sense of the applicability of these updated SCCO2 correlations.

Download Full-text

NURBS Surface Reconstruction From a Large Set of Image and World Data Points

19th Design Automation Conference: Volume 2 — Design Optimization; Geometric Modeling and Tolerance Analysis; Mechanism Synthesis and Analysis; Decomposition and Design Optimization ◽

10.1115/detc1993-0373 ◽

1993 ◽

Author(s):

Vassilios E. Theodoracatos ◽

Vasudeva Bobba

Keyword(s):

Least Squares ◽

Vision System ◽

Physical Object ◽

Free Form ◽

Approximation Technique ◽

Global Coordinate System ◽

Large Set ◽

Affine Invariant ◽

Nurbs Surface ◽

Data Points

Abstract In this paper an approach is presented for the generation of a NURBS (Non-Uniform Rational B-splines) surface from a large set of 3D data points. The main advantage of NURBS surface representation is the ability to analytically describe both, precise quadratic primitives and free-form curves and surfaces. An existing three dimensional laser-based vision system is used to obtain the spatial point coordinates of an object surface with respect to a global coordinate system. The least-squares approximation technique is applied in both the image and world space of the digitized physical object to calculate the homogeneous vector and the control net of the NURBS surface. A new non-uniform knot vectorization process is developed based on five data parametrization techniques including four existing techniques, viz., uniform, chord length, centripetal, and affine invariant angle and a new technique based on surface area developed in this study. Least-squares error distribution and surface interrogation are used to evaluate the quality of surface fairness for a minimum number of NURBS control points.

Download Full-text

The Existence of A Priori Distinctions Between Learning Algorithms

Neural Computation ◽

10.1162/neco.1996.8.7.1391 ◽

1996 ◽

Vol 8 (7) ◽

pp. 1391-1420 ◽

Cited By ~ 48

Author(s):

David H. Wolpert

Keyword(s):

Cross Validation ◽

Learning Algorithm ◽

A Priori ◽

Learning Algorithms ◽

Loss Functions ◽

Average Error ◽

Quadratic Loss ◽

Training Set ◽

Natural Restriction ◽

Cross Validation Error

This is the second of two papers that use off-training set (OTS) error to investigate the assumption-free relationship between learning algorithms. The first paper discusses a particular set of ways to compare learning algorithms, according to which there are no distinctions between learning algorithms. This second paper concentrates on different ways of comparing learning algorithms from those used in the first paper. In particular this second paper discusses the associated a priori distinctions that do exist between learning algorithms. In this second paper it is shown, loosely speaking, that for loss functions other than zero-one (e.g., quadratic loss), there are a priori distinctions between algorithms. However, even for such loss functions, it is shown here that any algorithm is equivalent on average to its “randomized” version, and in this still has no first principles justification in terms of average error. Nonetheless, as this paper discusses, it may be that (for example) cross-validation has better head-to-head minimax properties than “anti-cross-validation” (choose the learning algorithm with the largest cross-validation error). This may be true even for zero-one loss, a loss function for which the notion of “randomization” would not be relevant. This paper also analyzes averages over hypotheses rather than targets. Such analyses hold for all possible priors over targets. Accordingly they prove, as a particular example, that cross-validation cannot be justified as a Bayesian procedure. In fact, for a very natural restriction of the class of learning algorithms, one should use anti-cross-validation rather than cross-validation (!).

Download Full-text