Model-Based Derivative-Free Optimization Methods and Software

This thesis studies derivative-free optimization (DFO), particularly model-based methods and software. These methods are motivated by optimization problems for which it is impossible or prohibitively expensive to access the first-order information of the objective function and possibly the constraint functions. In particular, this thesis presents PDFO, a package we develop to provide both MATLAB and Python interfaces to Powell's model-based DFO solvers, namely COBYLA, UOBYQA, NEWUOA, BOBYQA, and LINCOA. Moreover, a significant part of this thesis is devoted to developing a new DFO method based on the sequential quadratic programming (SQP) method. Therefore, we present an overview of the SQP method and provide some perspectives on its theory and practice. In particular, we show that the objective function of the SQP subproblem is a natural quadratic approximation of the original objective function in the tangent space of a surface. Finally, we elaborate on developing our new DFO method, named COBYQA after Constrained Optimization BY Quadratic Approximations. This derivative-free trust-region SQP method is designed to tackle nonlinearly constrained optimization problems that admit equality and inequality constraints. An important feature of COBYQA is that it always respects bound constraints, if any, which is motivated by applications where the objective function is undefined when bounds are violated. We expose extensive numerical experiments of COBYQA, showing evident advantages of COBYQA compared with Powell's DFO solvers. These experiments demonstrate that COBYQA is an excellent successor to COBYLA as a general-purpose DFO solver. This is the Ph.D. thesis finished under the supervision of Dr. Zaikun Zhang and Prof. Xiaojun Chen at The Hong Kong Polytechnic University. Financial support was provided by the UGC of Hong Kong under the Hong Kong Ph.D. Fellowship Scheme.