scholarly journals Derivative-free optimization methods

Acta Numerica ◽  
2019 ◽  
Vol 28 ◽  
pp. 287-404 ◽  
Author(s):  
Jeffrey Larson ◽  
Matt Menickelly ◽  
Stefan M. Wild
Keyword(s):  
Optimization Problems ◽  
Linear Optimization ◽  
Optimization Methods ◽  
Black Box ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Recent Developments ◽  
Randomized Methods ◽  
Derivative Information ◽  
Scientific Engineering

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

Benchmarking and Field-Testing of the Distributed Quasi-Newton Derivative-Free Optimization Method for Field Development Optimization

2021 ◽  
Author(s):  
Faruk Alpak ◽  
Yixuan Wang ◽  
Guohua Gao ◽  
Vivek Jain
Keyword(s):  
Optimization Problems ◽  
Field Testing ◽  
Optimization Method ◽  
Training Data ◽  
Local Optima ◽  
Field Development ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Data Points ◽  
Quasi Newton

Abstract Recently, a novel distributed quasi-Newton (DQN) derivative-free optimization (DFO) method was developed for generic reservoir performance optimization problems including well-location optimization (WLO) and well-control optimization (WCO). DQN is designed to effectively locate multiple local optima of highly nonlinear optimization problems. However, its performance has neither been validated by realistic applications nor compared to other DFO methods. We have integrated DQN into a versatile field-development optimization platform designed specifically for iterative workflows enabled through distributed-parallel flow simulations. DQN is benchmarked against alternative DFO techniques, namely, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method hybridized with Direct Pattern Search (BFGS-DPS), Mesh Adaptive Direct Search (MADS), Particle Swarm Optimization (PSO), and Genetic Algorithm (GA). DQN is a multi-thread optimization method that distributes an ensemble of optimization tasks among multiple high-performance-computing nodes. Thus, it can locate multiple optima of the objective function in parallel within a single run. Simulation results computed from one DQN optimization thread are shared with others by updating a unified set of training data points composed of responses (implicit variables) of all successful simulation jobs. The sensitivity matrix at the current best solution of each optimization thread is approximated by a linear-interpolation technique using all or a subset of training-data points. The gradient of the objective function is analytically computed using the estimated sensitivities of implicit variables with respect to explicit variables. The Hessian matrix is then updated using the quasi-Newton method. A new search point for each thread is solved from a trust-region subproblem for the next iteration. In contrast, other DFO methods rely on a single-thread optimization paradigm that can only locate a single optimum. To locate multiple optima, one must repeat the same optimization process multiple times starting from different initial guesses for such methods. Moreover, simulation results generated from a single-thread optimization task cannot be shared with other tasks. Benchmarking results are presented for synthetic yet challenging WLO and WCO problems. Finally, DQN method is field-tested on two realistic applications. DQN identifies the global optimum with the least number of simulations and the shortest run time on a synthetic problem with known solution. On other benchmarking problems without a known solution, DQN identified compatible local optima with reasonably smaller numbers of simulations compared to alternative techniques. Field-testing results reinforce the auspicious computational attributes of DQN. Overall, the results indicate that DQN is a novel and effective parallel algorithm for field-scale development optimization problems.

A Note on Wedge Trust Region Radius Update

2011 ◽  
Vol 52-54 ◽  
pp. 926-931
Author(s):  
Qing Hua Zhou ◽  
Feng Xia Xu ◽  
Yan Geng ◽  
Ya Rui Zhang
Keyword(s):  
Optimization Problems ◽  
Trust Region Method ◽  
Trust Region ◽  
Test Problems ◽  
Region Method ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Updating Rule

Wedge trust region method based on traditional trust region is designed for derivative free optimization problems. This method adds a constraint to the trust region problem, which is called “wedge method”. The problem is that the updating strategy of wedge trust region radius is somewhat simple. In this paper, we develop and combine a new radius updating rule with this method. For most test problems, the number of function evaluations is reduced significantly. The experiments demonstrate the effectiveness of the improvement through our algorithm.

Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems

2008 ◽  
Vol 31 (5) ◽  
pp. 743-757 ◽  
Author(s):  
K.R. Fowler ◽  
J.P. Reese ◽  
C.E. Kees ◽  
J.E. Dennis ◽  
C.T. Kelley ◽  
...  
Keyword(s):  
Optimization Methods ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Groundwater Supply

A New Global Optimization Method for Simultaneous Computation on Expensive Black-Box Functions

2003 ◽  
Author(s):  
Liqun Wang ◽  
Songqing Shan ◽  
G. Gary Wang
Keyword(s):  
Global Optimization ◽  
Optimization Problems ◽  
Optimization Methods ◽  
Search Space ◽  
Optimization Method ◽  
Global Optimum ◽  
Black Box ◽  
Efficient Global Optimization ◽  
Global Optimization Method ◽  
Global Optimization Methods

The presence of black-box functions in engineering design, which are usually computation-intensive, demands efficient global optimization methods. This work proposes a new global optimization method for black-box functions. The global optimization method is based on a novel mode-pursuing sampling (MPS) method which systematically generates more sample points in the neighborhood of the function mode while statistically covers the entire search space. Quadratic regression is performed to detect the region containing the global optimum. The sampling and detection process iterates until the global optimum is obtained. Through intensive testing, this method is found to be effective, efficient, robust, and applicable to both continuous and discontinuous functions. It supports simultaneous computation and applies to both unconstrained and constrained optimization problems. Because it does not call any existing global optimization tool, it can be used as a standalone global optimization method for inexpensive problems as well. Limitation of the method is also identified and discussed.

Multilevel Strategies and Geological Parameterizations for History Matching Complex Reservoir Models

SPE Journal ◽  
2019 ◽  
Vol 25 (01) ◽  
pp. 081-104 ◽  
Author(s):  
Yimin Liu ◽  
Louis J. Durlofsky
Keyword(s):  
History Matching ◽  
Ensemble Methods ◽  
Principal Component ◽  
Optimization Methods ◽  
Training Image ◽  
Pattern Search ◽  
Hard Data ◽  
Derivative Free Optimization ◽  
Derivative Free ◽  
Wide Range

Summary In this study, we explore using multilevel derivative-free optimization (DFO) for history matching, with model properties described using principal-component-analysis (PCA) -based parameterization techniques. The parameterizations applied in this work are optimization-based PCA (O-PCA) and convolutional-neural-network (CNN) -based PCA (CNN-PCA). The latter, which derives from recent developments in deep learning, is able to accurately represent models characterized by multipoint spatial statistics. Mesh adaptive direct search (MADS), a pattern-search method that parallelizes naturally, is applied for the optimizations required to generate posterior (history-matched) models. The use of PCA-based parameterization considerably reduces the number of variables that must be determined during history matching (because the dimension of the parameterization is much smaller than the number of gridblocks in the model), but the optimization problem can still be computationally demanding. The multilevel strategy introduced here addresses this issue by reducing the number of simulations that must be performed at each MADS iteration. Specifically, the PCA coefficients (which are the optimization variables after parameterization) are determined in groups, at multiple levels, rather than all at once. Numerical results are presented for 2D cases, involving channelized systems (with binary and bimodal permeability distributions) and a deltaic-fan system using O-PCA and CNN-PCA parameterizations. O-PCA is effective when sufficient conditioning (hard) data are available, but it can lead to geomodels that are inconsistent with the training image when these data are scarce or nonexistent. CNN-PCA, by contrast, can provide accurate geomodels that contain realistic features even in the absence of hard data. History-matching results demonstrate that substantial uncertainty reduction is achieved in all cases considered, and that the multilevel strategy is effective in reducing the number of simulations required. It is important to note that the parameterizations discussed here can be used with a wide range of history-matching procedures (including ensemble methods), and that other derivative-free optimization methods can be readily applied within the multilevel framework.

scholarly journals Investigating the Generalization Ability of Parameterized Quantum Circuits with Hierarchical Structures

2021 ◽  
pp. 11-22
Author(s):  
Runheng Ran ◽  
Haozhen Situ
Keyword(s):  
Machine Learning ◽  
Hierarchical Structures ◽  
Optimization Methods ◽  
Quantum Circuits ◽  
Generalization Ability ◽  
Covariance Matrix Adaptation ◽  
Derivative Free Optimization ◽  
Machine Learning Model ◽  
Derivative Free ◽  
Adaptation Evolution

Quantum computing provides prospects for improving machine learning, which are mainly achieved through two aspects, one is to accelerate the calculation, and the other is to improve the performance of the model. As an important feature of machine learning models, generalization ability characterizes models' ability to predict unknown data. Aiming at the question of whether the quantum machine learning model provides reliable generalization ability, quantum circuits with hierarchical structures are explored to classify classical data as well as quantum state data. We also compare three different derivative-free optimization methods, i.e., Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Constrained Optimization by Linear Approximation (COBYLA) and Powell. Numerical results show that these quantum circuits have good performance in terms of trainability and generalization ability.

Sign in / Sign up

Export Citation Format

Share Document