REVIEW 2 major objections 5 minor 1 cited by
A trust-region framework for optimization using Hermite kernel surrogate models
T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper proves that a trust-region method using Hermite kernel surrogates converges to stationary points, and shows efficiency gains on PDE-constrained optimization problems.
desk verdict A genuinely new trust-region idea with a convergence proof that, as written, does not cover the BFGS implementation; the gap is real but reparable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Hermite kernel interpolation error bound of (2.5), converted into the trust-region constraint $c^{(i)}(\mu)\ge 0$ of (3.7). The named objects are the (Hermite) power function $P_{M^{(i)}}(\mu)$, the RKHS norm $\|J\|_{H_k(P)}$, and the approximate generalized Cauchy point $\mu_{\mathrm{AGC}}$, defined as the first Armijo-backtracking step of the inner gradient descent. The proof runs on two regularity facts: Theorem 2.2 shows the power function is Hölder continuous with exponent $1/2$, and Theorem 2.3 shows the surrogate gradient is uniformly Lipschitz, with constants independent of the iteration because they depend only on the kernel and on $\|J\|_{H_k(P)}$. These facts yield the decrease estimate (3.15), which Theorem 3.7 sums along the iterations to force the gradients of the objective at the iterates to zero.
What would settle it
Test the algorithm on the paper's 12D heat-distribution problem and, at each accepted iterate, compute the true interpolation error $|J(\mu^{(i+1)}) - \hat{J}^{(i)}(\mu^{(i+1)})|$ with a high-accuracy full-order solve, comparing it to the claimed bound $P_{M^{(i)}}(\mu^{(i+1)})\cdot\|J\|_{H_k(P)}$ using a much larger global interpolant to estimate the norm. If the true error exceeds the bound at any accepted iterate, the trust-region definition that the convergence proof relies on is violated in practice; alternatively, run the algorithm on a smooth objective that is provably outside the kernel's RKHS and see whether the iterates still converge.
Extended reading notes
Core claim
The central claim is Theorem 3.7: if the sufficient decrease condition (3.12) holds at every iteration, then the gradient norms of the objective vanish, $\lim_{i\to\infty}\|\nabla J(\mu^{(i)})\| = 0$. The mechanism is a trust-region constraint of the form $c^{(i)}(\mu) = \delta^{(i)} - P_{M^{(i)}}(\mu)\|J\|_{H_k(P)}/\hat{J}^{(i)}(\mu) \ge 0$, which replaces the usual ball constraint and admits exactly those parameters where the guaranteed Hermite interpolation error stays small relative to the surrogate value. Under mild kernel assumptions the power function $P_{M^{(i)}}$ is Hölder continuous and the surrogate gradient is uniformly Lipschitz, and these regularity properties yield a lower bound on the decrease achieved at the approximate generalized Cauchy point, which sums into gradient convergence. In the numerical sections the authors estimate the unknown RKHS norm from the surrogate itself via (2.6)–(2.7) and report that the projected variant (PHKTR) handles box constraints.
Load-bearing premise
The entire trust-region guarantee depends on the objective actually belonging to the kernel's reproducing kernel Hilbert space; in the numerical tests that space's norm is only estimated, and if the objective is not in that space the error bound that defines the search region is invalid.
Editorial extensions
If this is right
- Surrogate-based optimization of expensive objectives can carry a convergence guarantee without physics-based reduced models, provided the objective lies in the chosen kernel's RKHS.
- Trust regions defined by error bounds can be used instead of Euclidean balls, allowing the step size to adapt to where the surrogate is provably accurate.
- The convergence proof extends to box-constrained optimization through projection, giving first-order criticality in the projected-gradient sense.
- In the 12D heat-distribution example, the projected variant uses roughly 20% fewer full-order evaluations than the limited-memory BFGS solver it is compared against, with a relative error in the objective of order $10^{-5}$.
- The method treats the surrogate as purely data-driven, so it applies to nonlinear or non-PDE objectives where physics-based reduced models are harder to construct.
Reading between the lines
- Because Theorem 3.7 assumes the sufficient decrease condition (3.12) at every iteration while the implementation usually tests the cheaper bound (3.13) or the necessary condition (3.14), the alignment between the proven statement and the executed algorithm is looser than the theorem's hypothesis; checking (3.12) only in the ambiguous case means some accepted iterates may fall outside the proof's c
- The paper estimates the RKHS norm from the surrogate itself in the higher-dimensional experiments, and it excludes those estimation costs from the reported full-order-model counts, so in practice the error bound defining the trust region is approximate even when the theory's assumptions are satisfied.
- The box-constrained projected variant is stated to converge by following the Section 3.4 argument with the projection's Lipschitz continuity, but that transfer is referenced rather than demonstrated in detail, so the projected-gradient conclusion depends on conditions left implicit.
- The paper sketches adaptive kernel shape parameters but does not analyze them; because per-iterate shape parameters would break the symmetry of the kernel matrices, an immediate theoretical question is whether the Hermite interpolation systems remain solvable and whether the convergence bounds survive.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a trust-region optimization framework (HKTR/PHKTR, Algorithm 2) in which the surrogate model is a Hermite kernel interpolant and the trust region is defined through the Hermite power-function error bound rather than a Euclidean ball. The authors prove, in Theorem 3.7, that if the sufficient decrease condition (3.12) holds at every iteration and the assumptions of Theorem 3.6 are satisfied, then the sequence of iterates satisfies lim ||∇J(µ(i))|| = 0. The numerical section applies a box-constrained projected version (PHKTR) to a 1D toy problem, a 2D PDE-constrained problem, and a 12D PDE-constrained problem, comparing FOM evaluations, accuracy, and termination conditions against scipy's L-BFGS-B and trust-constr.
Significance. If the convergence proof is correct, the paper provides a nontrivial convergence guarantee for kernel-surrogate trust-region methods with error-based trust regions, complementing the mostly empirical literature on kernel surrogates in optimization. The central mechanism — using the Hermite interpolation error bound to define the trust region and an AGC point to certify decrease — is conceptually interesting and goes beyond the standard quadratic-model trust-region analysis. The manuscript is also transparent about implementation, ships code on GitHub, and explicitly identifies the RKHS-assumption and the kernel-shape-parameter dependence as limitations. The theoretical claims are, however, conditional on a missing step-length normalization in the AGC analysis, so the main convergence theorem does not yet cover the BFGS-based algorithm actually implemented.
major comments (2)
- [Appendix A, Eq. (A.11); Theorem 3.6] The lower bound (A.10) relies on the equality ||p(i)|| = ||∇Ĵ(i)(µ(i))|| at (A.11). No assumption in Theorem 3.6 or Algorithm 2 states that p(i) is the steepest descent direction; in fact, Algorithm 2 line 3 and the surrounding text (§3.3) specify BFGS directions p(i) = -H_i ∇Ĵ(i)(µ(i)) with an unconstrained positive-definite H_i. If H_i has a small eigenvalue along ∇Ĵ(i), the first backtracking trial can satisfy (A.1) immediately, and the AGC displacement is of order ||p(i)|| ≪ ||∇Ĵ(i)(µ(i))||, so the decrease bound (3.15) does not follow. Since (3.15) drives the telescoping sum in Theorem 3.7, the central convergence claim is not proven for the algorithm as implemented. This is repairable by defining p(i) = -∇Ĵ(i)(µ(i)) for the AGC computation, or by adding an explicit hypothesis such as ||p(i)|| ≥ c_g ||∇Ĵ(i)(µ(i))|| with a uniform c_g > 0 and checking the proof under that condition.
- [Section 3.5 and Appendix A] The paper's numerical experiments use the projected PHKTR algorithm, but the convergence proof in Section 3.4 and Appendix A is written entirely for the unconstrained case. Section 3.5 states that the convergence proof of the projected version "follows identical" and refers to [8], yet the AGC argument in Appendix A is based on line-search points of the form µ = µ(i) + α p(i), whereas the projected version replaces these with ΠP(µ(i,l) + κ^j_bt p(i,l)). The Hölder-continuity and Armijo arguments in Appendix A do not automatically carry over to projected steps, and no theorem for PHKTR is stated or proved. Since the abstract and numerical claims concern the algorithm as implemented, this gap should be closed or the scope of the convergence theorem explicitly restricted to the unconstrained case.
minor comments (5)
- [Eq. (2.6)] The displayed formula for ||s_n^f||^2_{H_k(Ω)} should read [αᵀ βᵀ] M_{X_n} [α; β] (with a transpose and consistent block ordering); as written the row/column dimensions do not match.
- [Theorem 3.6 and Eq. (3.9)] The Armijo constant κ_arm is introduced in (3.9) as an element of (0, 0.5), but Theorem 3.6 assumes κ_arm ∈ (0,1). The weaker assumption is fine for the proof, but the two ranges should be reconciled for consistency.
- [Tables 3 and 5] The RKHS-norm estimates used to define η(i) are computed with additional FOM solves that are not counted in the reported FOM budgets. The authors explain this in §4.3 (parallelization, coarser mesh), but the same caveat is not restated in §4.4, making the efficiency comparison in Table 6 somewhat optimistic on its face.
- [Theorem 3.7 proof] In the display after the telescoping sum, the term ||Ĵ(i)(µ(i))|| should be ||∇Ĵ(i)(µ(i))||; this is a typographical slip, but it makes the displayed inequality confusing on first reading.
- [Section 2, projection argument] The proof of the minimal-norm property of Π_{V(X_n)}f contains an orthogonality argument that is correct in substance, but the line "−⟨∂_1^a k(x,·), (I−Π_{V(X_n)})f⟩ = 0" is written incorrectly (the negative sign and the equality target are misaligned). This should be cleaned up.
Circularity Check
No significant circularity: the convergence proof is self-contained given standard kernel-interpolation facts and explicit assumptions; the numerical RKHS-norm estimation affects cost accounting but does not make the theorem equivalent to its inputs.
full rationale
The derivation chain is not circular. The central theorem (Theorem 3.7) is a standard trust-region Armijo-Goldstein-Cauchy argument: under Assumption 3.1, including J in the RKHS, positive bounded surrogates, and Lipschitz gradients, Theorem 3.6 supplies a lower decrease bound for the AGC point, and the telescoping-sum argument converts that per-iteration decrease into convergence to a stationary point. No fitted parameter is renamed as a prediction, and the conclusion lim ||grad J(mu_i)|| = 0 is not used as an input. The citations to the same group's prior work, [19], are technical support (positive definiteness of the Hermite Gram matrix and existence of positivity-preserving constructions) that is external to the convergence claim and is not equivalent to the target result. The questionable step in the manuscript is in Appendix A, where (A.11) asserts ||p(i)|| = ||grad Jhat_i(mu_i)|| for an unspecified BFGS direction; this is an unproven assumption and a correctness gap, not a circular identification of output with input. Finally, the numerical experiments estimate ||J||_H via (2.6)-(2.7) and exclude the cost of those estimates from the FOM budgets; that is a benchmarking limitation, but the estimated norm is an input to the implementation, not a predicted output of the theory.
Assumptions & free parameters
free parameters (1)
- kernel shape parameter epsilon =
epsilon = 0.725 (1D), 0.4 (2D), 0.0008 (12D)
assumptions (5)
- domain assumption J in H_k(P), the objective lies in the RKHS of the chosen kernel
- domain assumption The surrogate Jhat(i) is uniformly bounded away from zero
- domain assumption The kernel k and its mixed second derivatives are uniformly Lipschitz continuous
- domain assumption The objective J is bounded away from zero
- ad hoc to paper The first search direction for the AGC point is the steepest descent direction, or an equivalent gradient-related condition holds
Cite this review
Pith. "Pith review of A trust-region framework for optimization using Hermite kernel surrogate models." pith.science (2026). https://pith.science/paper/IXCVO3TX
@misc{pith2026250701729,
author = {Pith},
title = {Pith review of: A trust-region framework for optimization using Hermite kernel surrogate models},
year = {2026},
howpublished = {\url{https://pith.science/paper/IXCVO3TX}},
note = {Machine review of arXiv:2507.01729}
}
read the original abstract
In this work, we present a trust-region optimization framework that employs Hermite kernel surrogate models. The method targets optimization problems with computationally demanding objective functions, for which direct optimization is often impractical due to expensive function evaluations. To address these challenges, we leverage a trust-region strategy, where the objective function is approximated by an efficient surrogate model within a local neighborhood of the current iterate. In particular, we construct the surrogate using Hermite kernel interpolation and define the trust-region based on bounds for the interpolation error. As mesh-free techniques, kernel-based methods are naturally suited for medium- to high-dimensional problems. Furthermore, the Hermite formulation incorporates gradient information, enabling precise gradient estimates that are crucial for many optimization algorithms. We prove that the proposed algorithm converges to a stationary point, and we demonstrate its effectiveness through numerical experiments, which illustrate the convergence behavior as well as the efficiency gains compared to direct optimization.
Figures
Forward citations
Cited by 1 Pith paper
-
Adaptive Reduced Basis Trust Region Methods for Parabolic Inverse Problems
An adaptive reduced-basis trust-region Gauss-Newton method is extended to parabolic parameter identification, with POD-based enrichment, achieving 5 to 18x speedups in four reaction-diffusion tests.
Reference graph
Works this paper leans on
-
[8]
Tim Keil, Luca Mechelli, Mario Ohlberger, Felix Schindler, and Stefan Volkwein. A non-conforming dual approach for adaptive trust-region reduced basis approximation of PDE-constrained parameter optimization. ESAIM: Mathematical Modelling and Numerical Analysis, 55(3):1239–1269, May 2021
work page 2021
-
[1]
Hon Ho Kwok, Manohar P. Kamat, and Layne T. Watson. Location of stable and unstable equilibrium configurations using a model trust region quasi-newton method and tunnelling. Comput. & Struct., 21:909–916, 1985
work page 1985
-
[2]
Richard Barakat and Barbara H. Sandler. Determination of the wave-front aberration function from measured values of the point-spread function: a two-dimensional phase retrieval problem.J. Opt. Soc. Am. A, 9(10):1715–1723, Oct 1992
work page 1992
-
[3]
Electron Correlation in Molecules Using Direct Second Order MCSCF, pages 179–206
Hans Jørgen Aagaard Jensen. Electron Correlation in Molecules Using Direct Second Order MCSCF, pages 179–206. Springer US, Boston, MA, 1994
work page 1994
-
[4]
Maximum loss for risk measurement of portfolios
Gabriel Studer and Hans-Jakob Lüthi. Maximum loss for risk measurement of portfolios. In Operations Research Proceedings 1996, pages 386–391, Berlin, Heidelberg, 1997. Springer Berlin Heidelberg
work page 1996
-
[5]
Matthias Heinkenschloss and Luis N. Vicente. Analysis of inexact trust-region SQP algorithms. SIAM J. Optim., 12(2):283–302, 2002
work page 2002
-
[6]
Natalia M. Alexandrov, J. E. Dennis, R. Michael Lewis, and Virginia Torczon. A trust-region framework for managing the use of approximation models in optimization.Structural Optimization, 15(1):16–23, 1998
work page 1998
-
[7]
Andrew R. Conn, Nicholas I. M. Gould, and Philippe L. Toint.Trust Region Methods. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2000
work page 2000
Show all 31 references
-
[9]
A certified trust region reduced basis approach to PDE-constrained optimization.SIAM J
Elizabeth Qian, Martin Grepl, Karen Veroy, and Karen Willcox. A certified trust region reduced basis approach to PDE-constrained optimization.SIAM J. Sci. Comput., 39(5):S434–S460, 2017
2017
-
[10]
Tianyang Wen and Matthew J. Zahr. An augmented Lagrangian trust-region method with inexact gradient evaluations to accelerate constrained optimization problems using model hyperreduction. Int. J. Numer. Methods Fluids, 97(3):621–645, 2025
2025
-
[11]
Accelerating optimization of parametric linear systems by model order reduction
Yao Yue and Karl Meerbergen. Accelerating optimization of parametric linear systems by model order reduction. SIAM J. Optim., 23(2):1344–1370, 2013. 24
2013
-
[12]
Adaptive reduced basis trust region methods for parameter identification problems.Comput
Michael Kartmann, Tim Keil, Mario Ohlberger, Stefan Volkwein, and Barbara Kaltenbacher. Adaptive reduced basis trust region methods for parameter identification problems.Comput. Sci. Eng., 1(1):3, 2024
2024
-
[13]
Scattered Data Approximation
Holger Wendland. Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, Cambridge, UK, 2004
2004
-
[14]
Carlberg, Antony Jameson, Mykel J
Kevin T. Carlberg, Antony Jameson, Mykel J. Kochenderfer, Jeremy Morton, Liqian Peng, and Freddie D Witherden. Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning.J. Comput. Phys., 395:105–124, 2019
2019
-
[15]
Gaussian Process Regression for minimum energy path optimization and transition state search.J
Andreas Denzel, Bernard Haasdonk, and Johannes Kästner. Gaussian Process Regression for minimum energy path optimization and transition state search.J. Phys. Chem. A, 123(44):9600– 9611, Nov 2019
2019
-
[16]
Goal- Oriented Two-Layered Kernel Models as Automated Surrogates for Surface Kinetics in Reactor Simulations
Felix Döppel, Tizian Wenzel, Robin Herkert, Bernard Haasdonk, and Martin Votsmeier. Goal- Oriented Two-Layered Kernel Models as Automated Surrogates for Surface Kinetics in Reactor Simulations. Chemie Ingenieur Technik, 96:759–768, 2024
2024
-
[17]
Smola.Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond
Bernhard Schölkopf and Alexander J. Smola.Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, Cambridge, MA, 12 2001
2001
-
[18]
Springer, New York, NY, 2008
Ingo Steinwart and Andreas Christmann.Support Vector Machines. Springer, New York, NY, 2008
2008
-
[19]
Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems.Adv
Tobias Ehring and Bernard Haasdonk. Hermite kernel surrogates for the value function of high-dimensional nonlinear optimal control problems.Adv. Comput. Math., 50(3):36, 2024
2024
-
[20]
Fasshauer and Qi Ye
Gregory E. Fasshauer and Qi Ye. Reproducing kernels of generalized Sobolev spaces via a Green function approach with distributional operators.Numer. Math., 119(3):585–611, Jun 2011
2011
-
[21]
C. T. Kelley.Iterative Methods for Optimization. Society for Industrial and Applied Mathematics, Philadelphia, PA, 1999
1999
-
[22]
Charles G. Broyden. The convergence of a class of double-rank minimization algorithms 1. General considerations. IMA J. Appl. Math., 6(1):76–90, 1970
1970
-
[23]
A new approach to variable metric algorithms.Comput
Roger Fletcher. A new approach to variable metric algorithms.Comput. J., 13(3):317–322, 01 1970
1970
-
[24]
A family of variable-metric methods derived by variational means.Math
Donald Goldfarb. A family of variable-metric methods derived by variational means.Math. Comput., 24:23–26, 1970
1970
-
[25]
David F. Shanno. Conditioning of quasi-Newton methods for function minimization. Math. Comput., 24(111):647–656, 1970
1970
-
[26]
Byrd, Mary E
Richard H. Byrd, Mary E. Hribar, and Jorge Nocedal. An interior point algorithm for large-scale nonlinear programming. SIAM J. Optim., 9(4):877–900, 1999
1999
-
[27]
A software package for sequential quadratic programming
Dieter Kraft. A software package for sequential quadratic programming. Tech. Rep. DFVLR-FB 88-28, DLR German Aerospace Center – Institute for Flight Mechanics, Köln, Germany, 1988
1988
-
[28]
Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu
Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization.SIAM J. Sci. Comput., 16:1190–1208, Sep 1995. 25
1995
-
[29]
Byrd, Peihuang Lu, and Jorge Nocedal
Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization.ACM Trans. Math. Softw., 23(4):550–560, Dec 1997
1997
-
[30]
pyMOR – Generic algorithms and interfaces for Model Order Reduction.SIAM J
René Milk, Stephan Rave, and Felix Schindler. pyMOR – Generic algorithms and interfaces for Model Order Reduction.SIAM J. Sci. Comput., 38(5):S194–S216, 2016
2016
-
[31]
A new certified hierarchical and adaptive RB-ML-ROM surrogate model for parametrized PDEs
Bernard Haasdonk, Hendrik Kleikamp, Mario Ohlberger, Felix Schindler, and Tizian Wenzel. A new certified hierarchical and adaptive RB-ML-ROM surrogate model for parametrized PDEs. SIAM J. Sci. Comput., 45(3):A1039–A1065, 2023. A Proof of Theorem 3.6 Proof. We start by proving ...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.