scholarly journals Development of a hyperparameter optimization method for recommendatory models based on matrix factorization

2021 ◽  
Vol 5 (4 (113)) ◽  
pp. 45-54
Author(s):  
Alexander Nechaev ◽  
Vasily Meltsov ◽  
Dmitry Strabykin
Keyword(s):  
Gaussian Process ◽  
Loss Function ◽  
Matrix Factorization ◽  
Optimization Method ◽  
Bayesian Optimization ◽  
Optimization Process ◽  
Absolute Difference ◽  
Characteristic Features ◽  
Factorization Algorithms ◽  
Time Required

Many advanced recommendatory models are implemented using matrix factorization algorithms. Experiments show that the quality of their performance depends significantly on the selected hyperparameters. Analysis of the effectiveness of using various methods for solving this problem of optimizing hyperparameters was made. It has shown that the use of classical Bayesian optimization which treats the model as a «black box» remains the standard solution. However, the models based on matrix factorization have a number of characteristic features. Their use makes it possible to introduce changes in the optimization process leading to a decrease in the time required to find the sought points without losing quality. Modification of the Gaussian process core which is used as a surrogate model for the loss function when performing the Bayesian optimization was proposed. The described modification at first iterations increases the variance of the values predicted by the Gaussian process over a given region of the hyperparameter space. In some cases, this makes it possible to obtain more information about the real form of the investigated loss function in less time. Experiments were carried out using well-known data sets for recommendatory systems. Total optimization time when applying the modification was reduced by 16 % (or 263 seconds) at best and remained the same at worst (less than 1-second difference). In this case, the expected error of the recommendatory model did not change (the absolute difference in values is two orders of magnitude lower than the value of error reduction in the optimization process). Thus, the use of the proposed modification contributes to finding a better set of hyperparameters in less time without loss of quality

scholarly journals Sample-efficient Optimization Using Neural Networks

2021 ◽  
Author(s):  
◽  
Mashall Aryan
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Monte Carlo ◽  
Gaussian Process ◽  
Optimization Methods ◽  
Optimization Method ◽  
Monte Carlo Sampling ◽  
Bayesian Optimization ◽  
Expected Improvement ◽  
Bayesian Neural Network

<p>The solution to many science and engineering problems includes identifying the minimum or maximum of an unknown continuous function whose evaluation inflicts non-negligible costs in terms of resources such as money, time, human attention or computational processing. In such a case, the choice of new points to evaluate is critical. A successful approach has been to choose these points by considering a distribution over plausible surfaces, conditioned on all previous points and their evaluations. In this sequential bi-step strategy, also known as Bayesian Optimization, first a prior is defined over possible functions and updated to a posterior in the light of available observations. Then using this posterior, namely the surrogate model, an infill criterion is formed and utilized to find the next location to sample from. By far the most common prior distribution and infill criterion are Gaussian Process and Expected Improvement, respectively.    The popularity of Gaussian Processes in Bayesian optimization is partially due to their ability to represent the posterior in closed form. Nevertheless, the Gaussian Process is afflicted with several shortcomings that directly affect its performance. For example, inference scales poorly with the amount of data, numerical stability degrades with the number of data points, and strong assumptions about the observation model are required, which might not be consistent with reality. These drawbacks encourage us to seek better alternatives. This thesis studies the application of Neural Networks to enhance Bayesian Optimization. It proposes several Bayesian optimization methods that use neural networks either as their surrogates or in the infill criterion.    This thesis introduces a novel Bayesian Optimization method in which Bayesian Neural Networks are used as a surrogate. This has reduced the computational complexity of inference in surrogate from cubic (on the number of observation) in GP to linear. Different variations of Bayesian Neural Networks (BNN) are put into practice and inferred using a Monte Carlo sampling. The results show that Monte Carlo Bayesian Neural Network surrogate could performed better than, or at least comparably to the Gaussian Process-based Bayesian optimization methods on a set of benchmark problems.  This work develops a fast Bayesian Optimization method with an efficient surrogate building process. This new Bayesian Optimization algorithm utilizes Bayesian Random-Vector Functional Link Networks as surrogate. In this family of models the inference is only performed on a small subset of the entire model parameters and the rest are randomly drawn from a prior. The proposed methods are tested on a set of benchmark continuous functions and hyperparameter optimization problems and the results show the proposed methods are competitive with state-of-the-art Bayesian Optimization methods.  This study proposes a novel Neural network-based infill criterion. In this method locations to sample from are found by minimizing the joint conditional likelihood of the new point and parameters of a neural network. The results show that in Bayesian Optimization methods with Bayesian Neural Network surrogates, this new infill criterion outperforms the expected improvement.   Finally, this thesis presents order-preserving generative models and uses it in a variational Bayesian context to infer Implicit Variational Bayesian Neural Network (IVBNN) surrogates for a new Bayesian Optimization. This new inference mechanism is more efficient and scalable than Monte Carlo sampling. The results show that IVBNN could outperform Monte Carlo BNN in Bayesian optimization of hyperparameters of machine learning models.</p>

scholarly journals Controller parameter optimization for complex industrial system with uncertainties

2019 ◽  
Vol 52 (7-8) ◽  
pp. 888-895
Author(s):  
Heping Chen ◽  
Seth Bowels ◽  
Biao Zhang ◽  
Thomas Fuhlbrigge
Keyword(s):  
Gaussian Process ◽  
Parameter Optimization ◽  
Optimization Algorithm ◽  
Gaussian Process Regression ◽  
Optimization Method ◽  
Industrial Applications ◽  
Bayesian Optimization ◽  
Potential Candidate ◽  
Bayesian Optimization Algorithm ◽  
Controller Parameter

Proportional–integral–derivative control system has been widely used in industrial applications. For complex systems, tuning controller parameters to satisfy the process requirements is very challenging. Different methods have been proposed to solve the problem. However these methods suffer several problems, such as dealing with system complexity, minimizing tuning effort and balancing different performance indices including rise time, settling time, steady-state error and overshoot. In this paper, we develop an automatic controller parameter optimization method based on Gaussian process regression Bayesian optimization algorithm. A non-parametric model is constructed using Gaussian process regression. By combining Gaussian process regression with Bayesian optimization algorithm, potential candidate can be predicted and applied to guide the optimization process. Both experiments and simulation were performed to demonstrate the effectiveness of the proposed method.

scholarly journals Sample-efficient Optimization Using Neural Networks

2021 ◽  
Author(s):  
◽  
Mashall Aryan
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Monte Carlo ◽  
Gaussian Process ◽  
Optimization Methods ◽  
Optimization Method ◽  
Monte Carlo Sampling ◽  
Bayesian Optimization ◽  
Expected Improvement ◽  
Bayesian Neural Network

<p>The solution to many science and engineering problems includes identifying the minimum or maximum of an unknown continuous function whose evaluation inflicts non-negligible costs in terms of resources such as money, time, human attention or computational processing. In such a case, the choice of new points to evaluate is critical. A successful approach has been to choose these points by considering a distribution over plausible surfaces, conditioned on all previous points and their evaluations. In this sequential bi-step strategy, also known as Bayesian Optimization, first a prior is defined over possible functions and updated to a posterior in the light of available observations. Then using this posterior, namely the surrogate model, an infill criterion is formed and utilized to find the next location to sample from. By far the most common prior distribution and infill criterion are Gaussian Process and Expected Improvement, respectively.    The popularity of Gaussian Processes in Bayesian optimization is partially due to their ability to represent the posterior in closed form. Nevertheless, the Gaussian Process is afflicted with several shortcomings that directly affect its performance. For example, inference scales poorly with the amount of data, numerical stability degrades with the number of data points, and strong assumptions about the observation model are required, which might not be consistent with reality. These drawbacks encourage us to seek better alternatives. This thesis studies the application of Neural Networks to enhance Bayesian Optimization. It proposes several Bayesian optimization methods that use neural networks either as their surrogates or in the infill criterion.    This thesis introduces a novel Bayesian Optimization method in which Bayesian Neural Networks are used as a surrogate. This has reduced the computational complexity of inference in surrogate from cubic (on the number of observation) in GP to linear. Different variations of Bayesian Neural Networks (BNN) are put into practice and inferred using a Monte Carlo sampling. The results show that Monte Carlo Bayesian Neural Network surrogate could performed better than, or at least comparably to the Gaussian Process-based Bayesian optimization methods on a set of benchmark problems.  This work develops a fast Bayesian Optimization method with an efficient surrogate building process. This new Bayesian Optimization algorithm utilizes Bayesian Random-Vector Functional Link Networks as surrogate. In this family of models the inference is only performed on a small subset of the entire model parameters and the rest are randomly drawn from a prior. The proposed methods are tested on a set of benchmark continuous functions and hyperparameter optimization problems and the results show the proposed methods are competitive with state-of-the-art Bayesian Optimization methods.  This study proposes a novel Neural network-based infill criterion. In this method locations to sample from are found by minimizing the joint conditional likelihood of the new point and parameters of a neural network. The results show that in Bayesian Optimization methods with Bayesian Neural Network surrogates, this new infill criterion outperforms the expected improvement.   Finally, this thesis presents order-preserving generative models and uses it in a variational Bayesian context to infer Implicit Variational Bayesian Neural Network (IVBNN) surrogates for a new Bayesian Optimization. This new inference mechanism is more efficient and scalable than Monte Carlo sampling. The results show that IVBNN could outperform Monte Carlo BNN in Bayesian optimization of hyperparameters of machine learning models.</p>

scholarly journals Antenna Optimization Design Based on Deep Gaussian Process Model

2020 ◽  
Vol 2020 ◽  
pp. 1-10
Author(s):  
Xin-Yu Zhang ◽  
Yu-Bo Tian ◽  
Xie Zheng
Keyword(s):  
Gaussian Process ◽  
Process Model ◽  
Optimization Design ◽  
Optimization Method ◽  
Simulation Software ◽  
Modeling Method ◽  
Electromagnetic Problems ◽  
Training Samples ◽  
Low Efficiency ◽  
Time Required

When using Gaussian process (GP) machine learning as a surrogate model combined with the global optimization method for rapid optimization design of electromagnetic problems, a large number of covariance calculations are required, resulting in a calculation volume which is cube of the number of samples and low efficiency. In order to solve this problem, this study constructs a deep GP (DGP) model by using the structural form of convolutional neural network (CNN) and combining it with GP. In this network, GP is used to replace the fully connected layer of the CNN, the convolutional layer and the pooling layer of the CNN are used to reduce the dimension of the input parameters and GP is used to predict output, while particle swarm optimization (PSO) is used algorithm to optimize network structure parameters. The modeling method proposed in this paper can compress the dimensions of the problem to reduce the demand of training samples and effectively improve the modeling efficiency while ensuring the modeling accuracy. In our study, we used the proposed modeling method to optimize the design of a multiband microstrip antenna (MSA) for mobile terminals and obtained good optimization results. The optimized antenna can work in the frequency range of 0.69–0.96 GHz and 1.7–2.76 GHz, covering the wireless LTE 700, GSM 850, GSM 900, DCS 1800, PCS1900, UMTS 2100, LTE 2300, and LTE 2500 frequency bands. It is shown that the DGP network model proposed in this paper can replace the electromagnetic simulation software in the optimization process, so as to reduce the time required for optimization while ensuring the design accuracy.

scholarly journals A New Hyper-Parameter Optimization Method for Power Load Forecast Based on Recurrent Neural Networks

Algorithms ◽  
2021 ◽  
Vol 14 (6) ◽  
pp. 163
Author(s):  
Yaru Li ◽  
Yulai Zhang ◽  
Yongping Cai
Keyword(s):  
Neural Networks ◽  
Parameter Optimization ◽  
Recurrent Neural Networks ◽  
Critical Role ◽  
Optimization Method ◽  
Bayesian Optimization ◽  
Power Load ◽  
Gradient Calculation ◽  
Truncated Newton ◽  
Selection Of

The selection of the hyper-parameters plays a critical role in the task of prediction based on the recurrent neural networks (RNN). Traditionally, the hyper-parameters of the machine learning models are selected by simulations as well as human experiences. In recent years, multiple algorithms based on Bayesian optimization (BO) are developed to determine the optimal values of the hyper-parameters. In most of these methods, gradients are required to be calculated. In this work, the particle swarm optimization (PSO) is used under the BO framework to develop a new method for hyper-parameter optimization. The proposed algorithm (BO-PSO) is free of gradient calculation and the particles can be optimized in parallel naturally. So the computational complexity can be effectively reduced which means better hyper-parameters can be obtained under the same amount of calculation. Experiments are done on real world power load data,where the proposed method outperforms the existing state-of-the-art algorithms,BO with limit-BFGS-bound (BO-L-BFGS-B) and BO with truncated-newton (BO-TNC),in terms of the prediction accuracy. The errors of the prediction result in different models show that BO-PSO is an effective hyper-parameter optimization method.

Shape-design optimization of hull structures considering thermal deformation

2013 ◽  
Vol 228 (13) ◽  
pp. 2266-2277
Author(s):  
Myung-Jin Choi ◽  
Min-Geun Kim ◽  
Seonho Cho
Keyword(s):  
Optimal Design ◽  
Design Optimization ◽  
Thermal Deformation ◽  
Parametric Design ◽  
Optimization Method ◽  
Optimization Process ◽  
Design Parameters ◽  
Shape Design ◽  
Shape Design Optimization ◽  
Modified Method

We developed a shape-design optimization method for the thermo-elastoplasticity problems that are applicable to the welding or thermal deformation of hull structures. The point is to determine the shape-design parameters such that the deformed shape after welding fits very well to a desired design. The geometric parameters of curved surfaces are selected as the design parameters. The shell finite elements, forward finite difference sensitivity, modified method of feasible direction algorithm and a programming language ANSYS Parametric Design Language in the established code ANSYS are employed in the shape optimization. The objective function is the weighted summation of differences between the deformed and the target geometries. The proposed method is effective even though new design variables are added to the design space during the optimization process since the multiple steps of design optimization are used during the whole optimization process. To obtain the better optimal design, the weights are determined for the next design optimization, based on the previous optimal results. Numerical examples demonstrate that the localized severe deviations from the target design are effectively prevented in the optimal design.

Simulation-Based Hybrid Optimization Method for the Digital Twin of Garment Production Lines

2021 ◽  
Vol 21 (3) ◽  
Author(s):  
Woo-Kyun Jung ◽  
Young-Chul Park ◽  
Jae-Won Lee ◽  
Eun Suk Suh
Keyword(s):  
Expert Knowledge ◽  
Structural Characteristics ◽  
Optimization Method ◽  
Hybrid Optimization ◽  
Optimization Process ◽  
Production Lines ◽  
Distribution Method ◽  
Workload Analysis ◽  
Hybrid Optimization Method ◽  
Simulation Based

AbstractImplementing digital transformation in the garment industry is very difficult, owing to its labor-intensive structural characteristics. Further, the productivity of a garment production system is considerably influenced by a combination of processes and operators. This study proposes a simulation-based hybrid optimization method to maximize the productivity of a garment production line. The simulation reflects the actual site characteristics, i.e., process and operator level indices, and the optimization process reflects constraints based on expert knowledge. The optimization process derives an optimal operator sequence through a genetic algorithm (GA) and sequentially removes bottlenecks through workload analysis based on the results. The proposed simulation optimization (SO) method improved productivity by ∼67.4%, which is 52.3% higher than that obtained by the existing meta-heuristic algorithm. The correlation between workload and production was verified by analyzing the workload change trends. This study holds significance because it presents a new simulation-based optimization model that further applies the workload distribution method by eliminating bottlenecks and digitizing garment production lines.

Stochastic Bayesian optimization for predicting borderline knock

2021 ◽  
pp. 146808742110652
Author(s):  
Jian Tang ◽  
Anuj Pal ◽  
Wen Dai ◽  
Chad Archer ◽  
James Yi ◽  
...  
Keyword(s):  
Fuel Economy ◽  
Learning Algorithm ◽  
Feedforward Control ◽  
Model Development ◽  
Kriging Model ◽  
Optimization Method ◽  
Surrogate Models ◽  
Bayesian Optimization ◽  
Control Parameters ◽  
Operation Conditions

Engine knock is an undesirable combustion that could damage the engine mechanically. On the other hand, it is often desired to operate the engine close to its borderline knock limit to optimize combustion efficiency. Traditionally, borderline knock limit is detected by sweeping tests of related control parameters for the worst knock, which is expensive and time consuming, and also, the detected borderline knock limit is often used as a feedforward control without considering its stochastic characteristics without compensating current engine operational condition and type of fuel used. In this paper, stochastic Bayesian optimization method is used to obtain a tradeoff between stochastic knock intensity and fuel economy. The log-nominal distribution of knock intensity signal is converted to Gaussian one using a proposed map to satisfy the assumption for Kriging model development. Both deterministic and stochastic Kriging surrogate models are developed based on test data using the Bayesian iterative optimization process. This study focuses on optimizing two competing objectives, knock intensity and indicated specific fuel consumption using two control parameters: spark and intake valve timings. Test results at two different operation conditions show that the proposed learning algorithm not only reduces required time and cost for predicting knock borderline but also provides control parameters, based on trained surrogate models and the corresponding Pareto front, with the best fuel economy possible.

Gaussian Process Based Model to Optimize Additively Manufactured Powder Microstructures From Phase Field Modeling

2021 ◽  
Author(s):  
Arunabha Batabyal ◽  
Sugrim Sagar ◽  
Jian Zhang ◽  
Tejesh Dube ◽  
Xuehui Yang ◽  
...  
Keyword(s):  
Gaussian Process ◽  
Phase Field ◽  
Bayesian Optimization ◽  
Expected Improvement ◽  
Neck Size ◽  
Sources Of Uncertainty ◽  
Surface Diffusivity ◽  
Input Parameters ◽  
Optimal Values ◽  
Probability Of Improvement

Abstract A persistent problem in the selective laser sintering process is to maintain the quality of additively manufactured parts, which can be attributed to the various sources of uncertainty. In this work, a two-particle phase-field microstructure model has been analyzed. The sources of uncertainty as the two input parameters were surface diffusivity and inter-particle distance. The response quantity of interest (QOI) was selected as the size of the neck region that develops between the two particles. Two different cases with equal and unequal sized particles were studied. It was observed that the neck size increased with increasing surface diffusivity and decreased with increasing inter-particle distance irrespective of particle size. Sensitivity analysis found that the inter-particle distance has more influence on variation in neck size than that of surface diffusivity. The machine learning algorithm Gaussian Process Regression was used to create the surrogate model of the QOI. Bayesian Optimization method was used to find optimal values of the input parameters. For equal-sized particles, optimization using Probability of Improvement provided optimal values of surface diffusivity and inter-particle distance as 23.8268 and 40.0001, respectively. The Expected Improvement as an acquisition function gave optimal values 23.9874 and 40.7428, respectively. For unequal sized particles, optimal design values from Probability of Improvement were 23.9700 and 33.3005, respectively, while those from Expected Improvement were 23.9893 and 33.9627, respectively. The optimization results from the two different acquisition functions seemed to be in good agreement.

scholarly journals Hierarchical Analysis of Thorax Models to Measure Tidal Volume

2018 ◽  
Vol 4 (1) ◽  
pp. 429-432
Author(s):  
Bernhard Laufer ◽  
Sabine Krueger-Ziolek ◽  
Knut Moeller ◽  
Paul David Docherty ◽  
Fabian Hoeflinger ◽  
...  
Keyword(s):  
Tidal Volume ◽  
Motion Tracking ◽  
Optimization Method ◽  
Optimal Combination ◽  
Hierarchical Optimization ◽  
Hierarchical Analysis ◽  
Hierarchical Method ◽  
Computational Costs ◽  
Sensor Configuration ◽  
Time Required

AbstractMotion tracking of thorax kinematics can be used to determine respiration. However, determining a minimal sensor configuration from 64 candidate sensor locations is associated with high computational costs. Hence, a hierarchical optimization method was proposed to determine the optimal combination of sensors. The hierarchical method was assessed by its ability to quickly determine the sensor combination that will yield optimal modelled tidal volume compared to body plethysmograph measurements. This method was able to find the optimal sensor combinations, in approximately 2% of the estimated time required by an exhaustive search.

Sign in / Sign up

Export Citation Format

Share Document