pith. sign in

arxiv: 2506.00648 · v2 · pith:3JJVL3NSnew · submitted 2025-05-31 · 🧮 math.OC

A Framework for Nonlinearly-Constrained Gradient-Enhanced Local Bayesian Optimization with Comparisons to Quasi-Newton Optimizers

Andr\'e L. Marchildon , David W. Zingg This is my paper

Pith reviewed 2026-05-19 11:52 UTC · model grok-4.3

classification 🧮 math.OC
keywords Bayesian optimizationnonlinear constraintsgradient enhancementaugmented Lagrangianlocal optimizationquasi-Newton comparison
0
0 comments X

The pith

Two new methods let gradient-enhanced Bayesian optimization handle nonlinear equality constraints and converge with fewer evaluations than quasi-Newton solvers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops two approaches for nonlinearly-constrained local Bayesian optimization using a gradient-enhanced surrogate. The first incorporates an exact augmented Lagrangian, while the second adds the nonlinear equality constraints directly to the minimization of the acquisition function. Both methods are tested on three unimodal problems with 2 to 30 variables. They enable the Bayesian optimizer to reach a desired tolerance using fewer function evaluations than popular quasi-Newton optimizers from SciPy and MATLAB, and they achieve deeper convergence than earlier constrained Bayesian methods.

Core claim

By applying either an exact augmented Lagrangian or by directly augmenting the acquisition function with nonlinear equality constraints, a gradient-enhanced Bayesian optimizer can perform effective local optimization on nonlinearly-constrained unimodal problems, outperforming both prior constrained Bayesian approaches in convergence depth and quasi-Newton methods in the number of function evaluations required.

What carries the argument

Exact augmented Lagrangian formulation and direct constraint augmentation within the acquisition function minimization, applied to a gradient-enhanced Gaussian process surrogate for local Bayesian optimization.

If this is right

  • Both proposed methods allow deeper convergence on the test problems than previously developed constrained Bayesian optimizers.
  • The Bayesian optimizer reaches the target tolerance with fewer function evaluations than SciPy and MATLAB quasi-Newton optimizers on unimodal problems with 2 to 30 variables.
  • Similar performance is observed with both new methods.
  • The second method, which adds constraints to the acquisition function, is recommended because its parameters are more intuitive to tune.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework could be extended to problems with inequality constraints by modifying the constraint handling accordingly.
  • Performance might improve further if combined with techniques for handling multimodality in larger search spaces.
  • Since the methods rely on accurate gradient-enhanced surrogates, they are best suited for problems where gradients can be computed reliably alongside function values.

Load-bearing premise

The three test problems are unimodal and the gradient-enhanced surrogate model stays accurate enough when nonlinear equality constraints are added to the acquisition function minimization.

What would settle it

Running the optimizer on a multimodal nonlinearly-constrained problem and checking whether it still reaches tolerance with fewer function evaluations than the quasi-Newton methods.

read the original abstract

Bayesian optimization is a popular and versatile approach that is well suited to solve challenging optimization problems. Their popularity comes from their effective minimization of expensive function evaluations, their capability to leverage gradients, and their efficient use of noisy data. Bayesian optimizers have commonly been applied to global unconstrained problems, with limited development for many other classes of problems. In this paper, two alternative methods are developed that enable rapid and deep convergence of nonlinearly-constrained local optimization problems using a Bayesian optimizer. The first method uses an exact augmented Lagrangian and the second augments the minimization of the acquisition function to contain additional constraints. Both of these methods can be applied to nonlinear equality constraints, unlike most previous methods developed for constrained Bayesian optimizers. The new methods are applied with a gradient-enhanced Bayesian optimizer and enable deeper convergence for three nonlinearly-constrained unimodal optimization problems than previously developed methods for constrained Bayesian optimization. In addition, both new methods enable the Bayesian optimizer to reach a desired tolerance with fewer function evaluations than popular quasi-Newton optimizers from SciPy and MATLAB for unimodal problems with 2 to 30 variables. The Bayesian optimizer had similar results using both methods. It is recommended that users first try using the second method, which adds constraints to the acquisition function minimization, since its parameters are more intuitive to tune for new problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript develops two methods for nonlinearly-constrained local Bayesian optimization with gradient-enhanced Gaussian processes: (1) an exact augmented Lagrangian formulation and (2) direct augmentation of the acquisition function minimization with the nonlinear equality constraints. These are applied to three unimodal test problems (2–30 variables), claiming deeper convergence than prior constrained BO methods and fewer function evaluations to tolerance than SciPy and MATLAB quasi-Newton optimizers. The second method is recommended for its more intuitive parameters.

Significance. If the results hold after addressing validation gaps, the work provides practical frameworks for local constrained BO with nonlinear equalities, an underexplored area. Direct comparisons to widely available external optimizers (SciPy, MATLAB) add practical relevance and benchmark value. The gradient-enhanced surrogate and dual-method approach for equality constraints are clear strengths that could influence optimization practice for expensive black-box problems.

major comments (2)
  1. [Numerical experiments] Numerical experiments: The claim that both methods reach the desired tolerance with fewer function evaluations than SciPy and MATLAB quasi-Newton optimizers lacks error bars, the number of independent runs, or any statistical tests. This directly weakens the quantitative superiority assertion for unimodal problems with 2 to 30 variables.
  2. [Paragraph describing the two methods and the numerical experiments] Paragraph describing the two methods and the numerical experiments: No surrogate validation error, cross-validation scores, or constraint-violation statistics are reported for the three test problems under the second method (acquisition augmentation with nonlinear constraints). This is load-bearing for the central claim, since degraded GP posterior accuracy along the feasible manifold could eliminate any reported reduction in true objective evaluations.
minor comments (2)
  1. [Abstract] The abstract states that the Bayesian optimizer had similar results using both methods but does not quantify or illustrate this similarity, which would clarify the practical equivalence.
  2. [Methods] The augmented Lagrangian penalty parameters are identified as free parameters; a brief sensitivity discussion or default selection guideline in the methods section would improve usability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed review of our manuscript. We address each of the major comments below and describe the revisions we will make to strengthen the presentation of the numerical results and surrogate validation.

read point-by-point responses

  1. Referee: [Numerical experiments] Numerical experiments: The claim that both methods reach the desired tolerance with fewer function evaluations than SciPy and MATLAB quasi-Newton optimizers lacks error bars, the number of independent runs, or any statistical tests. This directly weakens the quantitative superiority assertion for unimodal problems with 2 to 30 variables.

    Authors: We agree that reporting error bars, the number of independent runs, and basic statistical comparisons would strengthen the quantitative claims. In the revised manuscript we will include results aggregated over multiple independent runs (specifying the exact number) with mean and standard deviation shown as error bars for the number of function evaluations required to reach tolerance on each test problem. Where appropriate we will also add simple statistical tests to support the reported differences versus the SciPy and MATLAB quasi-Newton baselines. revision: yes

  2. Referee: [Paragraph describing the two methods and the numerical experiments] Paragraph describing the two methods and the numerical experiments: No surrogate validation error, cross-validation scores, or constraint-violation statistics are reported for the three test problems under the second method (acquisition augmentation with nonlinear constraints). This is load-bearing for the central claim, since degraded GP posterior accuracy along the feasible manifold could eliminate any reported reduction in true objective evaluations.

    Authors: We concur that explicit surrogate validation and constraint-violation metrics are important to substantiate the reliability of the gradient-enhanced GP models, particularly for the acquisition-augmentation approach. In the revision we will add cross-validation scores (or equivalent validation error metrics) for the GPs on each of the three test problems and will report average constraint-violation values at the reported solutions to confirm that the observed reduction in true objective evaluations occurs while maintaining feasible points. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical performance claims rest on external benchmarks

full rationale

The paper presents two algorithmic frameworks for nonlinearly-constrained gradient-enhanced Bayesian optimization and reports empirical results on three unimodal test problems. All performance metrics (function evaluations to tolerance) are obtained by direct comparison against independent, publicly available implementations of quasi-Newton methods in SciPy and MATLAB. No derivation, equation, or result in the manuscript reduces a reported outcome to a parameter fitted inside the same paper, a self-citation chain, or a renaming of an input quantity. The central claims therefore remain externally falsifiable and do not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central contribution consists of two algorithmic frameworks rather than new mathematical axioms or postulated entities; the main unstated premise is that a standard gradient-enhanced Gaussian-process surrogate can be reused without modification once the constraints are incorporated.

free parameters (1)
  • Augmented Lagrangian penalty parameters
    Likely tuned per problem to balance objective and constraint violation, though exact values and tuning procedure are not stated in the abstract.
axioms (1)
  • domain assumption A gradient-enhanced Bayesian optimizer can be directly adapted to local nonlinear equality-constrained problems by either Lagrangian augmentation or acquisition-function constraints.
    This assumption underpins both proposed methods and is invoked when the abstract claims the new approaches work with the existing gradient-enhanced framework.

pith-pipeline@v0.9.0 · 5776 in / 1314 out tokens · 68084 ms · 2026-05-19T11:52:46.771054+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 1 internal anchor

  1. [1]

    Aircraft Wing Optimization based on Computationally Efficient Gradient-Enhanced Ordinary Kriging Metamodel Building

    Mortished C, Ollar J, Toropov V , Sienz J. Aircraft Wing Optimization based on Computationally Efficient Gradient-Enhanced Ordinary Kriging Metamodel Building. In: 2016; San Diego, California, USA

    work page 2016

  2. [2]

    Gradient-based multifidelity optimisation for aircraft design using Bayesian model calibration.The Aeronautical Journal

    March A, Willcox K, Wang Q. Gradient-based multifidelity optimisation for aircraft design using Bayesian model calibration.The Aeronautical Journal. 2011;115(1174):729–738. doi: 10.1017/S0001924000006473

    work page doi:10.1017/s0001924000006473 2011

  3. [3]

    Bayesian Optimization of a Low-Boom Supersonic Wing Planform.AIAA Journal

    Jim TMS, Faza GA, Palar PS, Shimoyama K. Bayesian Optimization of a Low-Boom Supersonic Wing Planform.AIAA Journal. 2021;59(11):4514–

    work page 2021

  4. [4]

    doi: 10.2514/1.J060225 Nonlinearly-Constrained Gradient-Enhanced Local Bayesian Optimization 21

    work page doi:10.2514/1.j060225

  5. [5]

    Gaussian Processes for Machine Learning

    Rasmussen CE, Williams CKI. Gaussian Processes for Machine Learning. Adaptive computation and machine learningCambridge, Mass: MIT Press, 2006

    work page 2006

  6. [6]

    Noise Estimation in Gaussian Process Regression

    Ameli S, Shadden SC. Noise Estimation in Gaussian Process Regression. 2022

    work page 2022

  7. [7]

    Dennis Cook

    Morris MD, Mitchell TJ, Ylvisaker D. Bayesian Design and Analysis of Computer Experiments: Use of Derivatives in Surface Prediction. Technometrics. 1993;35(3):243–255. doi: 10.1080/00401706.1993.10485320

    work page doi:10.1080/00401706.1993.10485320 1993

  8. [8]

    Performance study of gradient-enhanced Kriging

    Ulaganathan S, Couckuyt I, Dhaene T, Degroote J, Laermans E. Performance study of gradient-enhanced Kriging. Engineering with Computers. 2016;32(1):15–34. doi: 10.1007/s00366-015-0397-y

    work page doi:10.1007/s00366-015-0397-y 2016

  9. [9]

    Adams, and Nando de Freitas

    Shahriari B, Swersky K, Wang Z, Adams RP, Freitas dN. Taking the Human Out of the Loop: A Review of Bayesian Optimization.Proceedings of the IEEE. 2016;104(1):148–175. doi: 10.1109/JPROC.2015.2494218

    work page doi:10.1109/jproc.2015.2494218 2016

  10. [10]

    Expected improvement for expensive optimization: a review

    Zhan D, Xing H. Expected improvement for expensive optimization: a review. Journal of Global Optimization. 2020;78(3):507–544. doi: 10.1007/s10898-020-00923-x

    work page doi:10.1007/s10898-020-00923-x 2020

  11. [11]

    Scalable Global Optimization via Local Bayesian Optimization

    Eriksson D, Pearce M, Gardner J, Turner RD, Poloczek M. Scalable Global Optimization via Local Bayesian Optimization. In: Advances in Neural Information Processing Systems 2019; Vancouver, Canada:12

    work page 2019

  12. [12]

    Bayesian Optimization with Inequality Constraints

    Gardner JR, Kusner MJ, Xu Z, Weinberger KQ, Cunningham JP. Bayesian Optimization with Inequality Constraints. ICML. 2014:937–945

    work page 2014

  13. [13]

    Constrained Bayesian Optimization with Noisy Experiments

    Letham B, Karrer B, Ottoni G, Bakshy E. Constrained Bayesian Optimization with Noisy Experiments. Bayesian Analysis. 2019;14(2). doi: 10.1214/18-BA1110

    work page doi:10.1214/18-ba1110 2019

  14. [14]

    Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function

    Han ZH, Görtz S, Zimmermann R. Improving variable-fidelity surrogate modeling via gradient-enhanced kriging and a generalized hybrid bridge function. Aerospace Science and Technology.2013;25(1):177–189. doi: 10.1016/j.ast.2012.01.006

    work page doi:10.1016/j.ast.2012.01.006 2013

  15. [15]

    On optimum design in fluid mechanics

    Pironneau O. On optimum design in fluid mechanics. Journal of Fluid Mechanics. 1974;64(1):97–110. doi: 10.1017/S0022112074002023

    work page doi:10.1017/s0022112074002023 1974

  16. [16]

    Aerodynamic design via control theory

    Jameson A. Aerodynamic design via control theory. Journal of Scientific Computing. 1988;3(3):233–260. doi: 10.1007/BF01061285

    work page doi:10.1007/bf01061285 1988

  17. [17]

    Efficient and robust gradient enhanced Kriging emulators

    Dalbey K. Efficient and robust gradient enhanced Kriging emulators.. Tech. Rep. SAND2013-7022, 1096451, Sandia National Laboratories; 2013

    work page 2013

  18. [18]

    A Non-intrusive Solution to the Ill-Conditioning Problem of the Gradient-Enhanced Gaussian Covariance Matrix for Gaussian Processes

    Marchildon AL, Zingg DW. A Non-intrusive Solution to the Ill-Conditioning Problem of the Gradient-Enhanced Gaussian Covariance Matrix for Gaussian Processes. Journal of Scientific Computing. 2023;95(3). doi: 10.1007/s10915-023-02190-w

    work page doi:10.1007/s10915-023-02190-w 2023

  19. [19]

    A solution to the ill-conditioning of gradient-enhanced covariance matrices for Gaussian processes.International Journal for Numerical Methods in Engineering

    Marchildon AL, Zingg DW. A solution to the ill-conditioning of gradient-enhanced covariance matrices for Gaussian processes.International Journal for Numerical Methods in Engineering. 2024. doi: 10.1002/nme.7498

    work page doi:10.1002/nme.7498 2024

  20. [20]

    Efficient Global Optimization of Expensive Black-Box Functions.Journal of Global Optimization.1998;13:455–

    Jones DR, Schonlau M, Welch WJ. Efficient Global Optimization of Expensive Black-Box Functions.Journal of Global Optimization.1998;13:455–

    work page 1998

  21. [21]

    doi: https://doi.org/10.1023/A:1008306431147

    work page doi:10.1023/a:1008306431147

  22. [22]

    Numerical Optimization

    Nocedal J, Wright SJ. Numerical Optimization. Springer series in operation research and financial engineering, New York, NY: Springer. second edition ed., 2006

    work page 2006

  23. [23]

    Global Convergence of a Class of Quasi-Newton Methods on Convex Problems.SIAM Journal on Numerical Analysis

    Byrd RH, Nocedal J, Yuan YX. Global Convergence of a Class of Quasi-Newton Methods on Convex Problems.SIAM Journal on Numerical Analysis. 1987;24(5):1171–1190

    work page 1987

  24. [24]

    Efficient Gradient-Enhanced Bayesian Optimizer with Comparisons to Conjugate-Gradient and Quasi-Newton Optimizers for Unconstrained Local Optimization

    Marchildon AL, Zingg DW. Efficient Gradient-Enhanced Bayesian Optimizer with Comparisons to Quasi-Newton Optimizers for Unconstrained Local Optimization. 2025. arXiv:2504.09375 [math]

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    Importance sampling-based transport map

    Pourmohamad T, Lee HKH. Bayesian Optimization Via Barrier Functions.Journal of Computational and Graphical Statistics. 2021:1–10. doi: 10.1080/10618600.2021.1935270

    work page doi:10.1080/10618600.2021.1935270 2021

  26. [26]

    Aerodynamic Optimization and Fuel Burn Evaluation of a Transonic Strut-Braced-Wing Single-Aisle Aircraft.Journal of Aircraft

    Chau T, Zingg DW. Aerodynamic Optimization and Fuel Burn Evaluation of a Transonic Strut-Braced-Wing Single-Aisle Aircraft.Journal of Aircraft. 2023;60(5):1638–1658. doi: 10.2514/1.C037158

    work page doi:10.2514/1.c037158 2023

  27. [27]

    Gradient-Enhanced Bayesian Optimization With Application to Aerodynamic Shape Optimization

    Marchildon AL, Zingg DW. Gradient-Enhanced Bayesian Optimization With Application to Aerodynamic Shape Optimization. In: AIAA 2024-4405. American Institute of Aeronautics and Astronautics 2024; Las Vegas, Nevada

    work page 2024

  28. [28]

    Modeling an Augmented Lagrangian for Blackbox Constrained Optimization

    Gramacy RB, Gray GA, Le Digabel S, et al. Modeling an Augmented Lagrangian for Blackbox Constrained Optimization. Technometrics. 2016;58(1):1–11. doi: 10.1080/00401706.2015.1014065

    work page doi:10.1080/00401706.2015.1014065 2016

  29. [29]

    Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian

    Picheny V , Gramacy RB, Wild S, Le Digabel S. Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian. Advances in Neural Information Processing Systems. 2016;29:9

    work page 2016

  30. [30]

    Exact Penalty Methods

    Di Pillo G. Exact Penalty Methods. In: Spedicato E., ed.Algorithms for Continuous Optimization, , Dordrecht: Springer Netherlands, 1994:209–253

    work page 1994

  31. [31]

    A Unified Approach to the Global Exactness of Penalty and Augmented Lagrangian Functions II: Extended Exactness.Journal of Optimization Theory and Applications

    Dolgopolik MV . A Unified Approach to the Global Exactness of Penalty and Augmented Lagrangian Functions II: Extended Exactness.Journal of Optimization Theory and Applications. 2018;176(3):745–762. doi: 10.1007/s10957-018-1239-z 22 Marchildon and Zingg

    work page doi:10.1007/s10957-018-1239-z 2018

  32. [32]

    A Class of Methods for Nonlinear Programming with Termination and Convergence Properties.Integer and Nonlinear Programming

    Fletcher R. A Class of Methods for Nonlinear Programming with Termination and Convergence Properties.Integer and Nonlinear Programming. 1970:157–173

    work page 1970

  33. [33]

    A multiplier method with automatic limitation of penalty growth

    Glad T, Polak E. A multiplier method with automatic limitation of penalty growth. Mathematical Programming. 1979;17(1):140–155. doi: 10.1007/BF01588240

    work page doi:10.1007/bf01588240 1979

  34. [34]

    Global Optimization of Costly Nonconvex Functions Using Radial Basis Functions

    Björkman M, Holmström K. Global Optimization of Costly Nonconvex Functions Using Radial Basis Functions. 2000;1:373–397. doi: 10.1023/A:1011584207202

    work page doi:10.1023/a:1011584207202 2000

  35. [35]

    Global versus local search in constrained optimization of computer models

    Schonlau M, Welch WJ, Jones DR. Global versus local search in constrained optimization of computer models. In: , , Hayward, CA: Institute of Mathematical Statistics, 1998:11–25

    work page 1998

  36. [36]

    Analysis of multi-objective Kriging-based methods for constrained global optimization

    Durantin C, Marzat J, Balesdent M. Analysis of multi-objective Kriging-based methods for constrained global optimization. Computational Optimization and Applications. 2016;63(3):903–926. doi: 10.1007/s10589-015-9789-6 APPENDIX A CLOSED FORM SOLUTION FOR LAGRANGE MULTIPLIERS In this appendix we present the closed form solution for the Lagrange multipliers ...

    work page doi:10.1007/s10589-015-9789-6 2016