Pith. sign in

REVIEW 5 major objections 4 minor 1 cited by

Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that two recursive AdaGrad variants, one multilevel and one domain-decomposition, reach an $\epsilon$-approximate first-order critical point of a bound-constrained noisy problem in $O(\epsilon^{-2})$ iterations and…

desk verdict Solid extension of bound-constrained AdaGrad to multilevel and domain-decomposition settings; the unified O(eps^-2) bound is plausible but not fully self-contained, and the stochastic theory leans on an unverified coarse-model assumption. read the letter →

arxiv 2507.11513 v1 pith:5YLQLIWL submitted 2025-07-15 math.OC cs.AIcs.NAmath.NA

Serge Gratton , Alena Kopaničáková , Philippe Toint This is my paper
classification math.OCcs.AIcs.NAmath.NA MSC 90C2690C3065K0590C15
keywords bound-constrainedoptimizationAdaGradobjective-function-freemultilevelmethodsdomaindecompositionstochasticgradientnoisenonconvexevaluationcomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to extend the OFFO (objective-function-free) AdaGrad framework to problems with both variable bounds and noisy gradients while exploiting problem structure. It proposes ML-ADAGB2 for hierarchical multilevel problems and DD-ADAGB2 for additive Schwarz domain-decomposition problems, and shows that both fit one shared convergence theory. The main claim is an $O(\epsilon^{-2})$ high-probability evaluation complexity bound for reaching an $\epsilon$-approximate first-order critical point. If correct, this makes structure-exploiting, noise-tolerant optimization with bounds available in PDE-based and deep-learning training settings at the same asymptotic rate as single-level AdaGrad.

What carries the argument

The load-bearing mechanism is the recursive step: at level $\ell+1$, the algorithm calls level $\ell$ with thresholds $\theta_{1,\ell}=\kappa_{1st}|d^T_{\ell,k}\Delta_{\ell,k}|$ and $\theta_{2,\ell}=\kappa_{2nd}\|s^L_{\ell,k}\|$, and accepts the returned step only while condition (14) preserves first-order progress. Lemma 3.3 propagates the linear-descent inequality upward through the hierarchy, updating coefficients via the recurrence (39), namely $\beta_{1,\ell+1}=\kappa_{1st}\beta_{1,\ell}$ and $\beta_{2,\ell+1}=\kappa_{2nd}^2(\kappa_{gs}\beta_{2,\ell}+\kappa_\tau\sqrt{\alpha_\ell}+\kappa_s\kappa_{Gg})$. AdaGrad-style weights grow unboundedly if progress stalls, forcing the first-order term to dominate, while AS.6 is the only assumption tying coarse-level gradients to fine-level gradients. The named algorithms are ML-ADAGB2 and DD-ADAGB2.

What would settle it

Run a two-level test with deliberately inconsistent coarse gradients whose expected mismatch exceeds the bound in AS.6 and compare the measured probability of reaching a small $\|\Xi_{r,j}\|$ with the $O(\epsilon^{-2})$ schedule predicted by Corollary 3.9; if the schedule degrades, AS.6 is the broken link. A sharper calculation is to track the ratio $E[\|P^T G_{\ell+1,k}-g_{\ell,0}\|^2]/E[\theta_{2,\ell}^2]$ as step sizes shrink, since unbounded growth of this ratio makes the constant $\kappa_\tau\sqrt{\alpha_\ell}$ in the recurrence diverge.

Watch

Extended reading notes

Core claim

The central discovery is that recursive lower-level steps can be absorbed into the AdaGrad analysis without changing the asymptotic complexity. The paper shows that the expected squared projected-gradient measure at the top level satisfies $\mathrm{average}_{j\in\{0,\ldots,k\}} E[\|\Xi_{r,j}\|^2] \le \kappa_{\mathrm{conv}}/(k+1)$, with $\kappa_{\mathrm{conv}}$ built by the two-parameter recurrence (39). Corollary 3.9 converts this into $P(\min_{0\le j\le k}\|\Xi_{r,j}\| \le \epsilon) \ge 1-\delta$ for $k \ge p_E\kappa_{\mathrm{conv}}/((p_E-(1-\delta))\epsilon^2)$. In the deterministic case, the tau-correction makes the cross-level coherence assumption AS.6 hold exactly, so the same bound gives deterministic convergence. The domain-decomposition variant inherits the theorem because the standard additive Schwarz prolongations and restrictions satisfy the same nonnegativity and row-sum conditions as the multilevel transfer operators.

Load-bearing premise

The whole bound rests on AS.6: at every recursive call, the coarse-level gradient estimate must match the restricted fine-level gradient in expectation, with error at most a constant times the expected squared second-order threshold; if this coherence fails, the multilevel descent recurrence breaks.

Editorial extensions

If this is right

  • ML-ADAGB2 and DD-ADAGB2 each need at most $O(\epsilon^{-2})$ iterations and noisy gradient evaluations, with no logarithmic term, to find an $\epsilon$-approximate first-order critical point with probability at least $1-\delta$.
  • The domain-decomposition variant inherits the same bound for additive Schwarz, restricted additive Schwarz, weighted variants, and their harmonic counterparts, because those transfer operators satisfy the assumptions of the common framework.
  • In the deterministic case, applying the tau-correction enforces the cross-level coherence condition, yielding the same complexity without stochastic assumptions.
  • When the gradient noise is coherently distributed, the bound transfers from the noisy projected gradient to the true first-order criticality measure of the original problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If AS.6 holds more generally, the same recursion may cover other decompositions beyond grids and Schwarz subdomains, including block-coordinate or randomized decompositions, as long as the transfer operators have nonnegative entries and bounded row sums.
  • The theory suggests that using structure does not worsen the stochastic first-order rate; an implicit practical prediction is that multilevel and domain-decomposition variants stay competitive with single-level AdaGrad even with small gradient batches.
  • A testable extension is to vary the overlap or weighting in the additive Schwarz operators and check whether the predicted constant degradation matches the observed change in the number of decomposition iterations.
  • Because the tau-correction is only shown to enforce AS.6 deterministically, extending it to stochastic coarse gradients through control variates would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes two recursive, bound-constrained, objective-function-free AdaGrad variants: ML-ADAGB2 for hierarchical/multilevel problems and DD-ADAGB2 for additive Schwarz domain decompositions. Both allow noisy or inexact gradients and optional second-order information. The theoretical core is a recursive descent lemma (Lemma 3.3) that propagates a linear-descent inequality from lower to higher levels via recurrence (39), leading to Theorem 3.5 (an O(1/k) bound on the expected squared projected gradient) and Corollary 3.9 (an O(epsilon^{-2}) high-probability complexity bound). Numerical experiments on PDE-based problems, ResNet training, and DeepONet training report substantial speedups over single-level ADAGB2.

Significance. If the central claims hold, the paper extends the single-level stochastic AdaGrad analysis of [6] to a genuinely multilevel and domain-decomposition setting, giving the first high-probability O(epsilon^{-2}) complexity guarantee for bound-constrained OFFO methods that exploit hierarchical or overlapping structure. The new lemmas (3.2--3.4) are proved in detail, the feasibility argument (Lemma 2.1) is careful, and the numerical study is broad, covering deterministic PDE problems and stochastic deep-learning tasks. The work is clearly significant for the optimization community, provided the load-bearing stochastic-coherence assumption AS.6 and the final complexity transfer in Theorem 3.8 are made rigorous.

major comments (5)
  1. [§3, Lemma 3.3, equation (41)] The tower-property step E_{ℓ+1,k}[E_{ℓ,0}[||P^T G_{ℓ+1,k} - g_{ℓ,0}||^2]] = E_{ℓ+1,k}[||P^T G_{ℓ+1,k} - g_{ℓ,0}||^2] is used to combine AS.6 with Lemma 3.2, but the paper does not specify the underlying filtration or state the measurability/conditional-independence properties of g_{ℓ,0}. The lower-level draw g_{ℓ,0} occurs after the parent iteration (ℓ+1,k), and AS.6's inner expectation E_{ℓ,0} conditions on lower-level history; without an explicit assumption that this tower step is valid, the cross term controlling the lower-level bias is not justified. This gap is load-bearing because it is exactly what yields the κτ√α_ℓ term in recurrence (39) and hence in Corollary 3.9.
  2. [§3, Theorem 3.8, equation (53)] Assumption (53) states E_{r,k}[||g_{r,k}-G_{r,k}||] ≤ κerr E_{r,k}[||d_{r,k}||^2], which mixes a first moment with a squared quantity, whereas Lemma 3.7 and the constant (1+κerr) in (55) correspond to the first-moment bound E[||g-G||] ≤ κerr E[||d||]. As written, Theorem 3.8's bound on E[||Ξ_{r,j}||^2] does not follow from the displayed assumptions and Lemma 3.7. This inconsistency directly affects Corollary 3.9 and must be repaired by either restating (53), or by deriving the squared-error conclusion from an appropriate second-moment assumption, or by restricting the claim to the coherent-distribution alternative (50).
  3. [§3, Theorem 3.5] The central O(1/k) estimate (45) is stated without proof and deferred to [34] and [6], both of which have overlapping authorship with the present paper. The new multilevel content is in Lemmas 3.2--3.4, but the final step from the recursive inequality (42) to the averaged bound (45) is not shown. Since the assumptions here include the new AS.6, the paper should either provide the full proof or identify the exact theorem in [34] or [6] that applies, with a verification that all hypotheses match. This is necessary for the paper to be self-contained on its main complexity claim.
  4. [§3, AS.6 and §5.2] AS.6 is the only assumption relating fine and coarse objective functions and is used at every recursive call. The deterministic tau-correction (48) makes the left side of (25) zero only at x_{ℓ,0}, and it is not applied in the stochastic DNN experiments. No stochastic construction or verification of AS.6 is provided, so the high-probability O(epsilon^{-2}) bound for stochastic ML-ADAGB2 and DD-ADAGB2 remains conditional on an unverified modeling hypothesis. The paper should either give a verifiable sufficient condition for AS.6 in the stochastic case (for example, an unbiased coarse gradient oracle with variance bounded by κτ² E[θ_{2,ℓ}²]) or explicitly delimit the stochastic claims.
  5. [§3, Lemma 3.4 and discussion after Theorem 3.5] The recurrence (39) gives β_{1,ℓ+1} = κ_{1st} β_{1,ℓ}. With κ_{1st}=0.95 as used in all experiments, β_{1,r} decays exponentially in the number of levels r, and κ_W in (47) is inversely proportional to β_{1,r}. The text's statement that κ_{1st}≤1 'ensures' β_{1,t+1}≤β_{1,t} and that the decrease is 'not too quick' is not quantified; it does not rule out an exponential dependence of the complexity constant on r. If r is fixed, this is acceptable, but the paper should state the dependence on r explicitly, especially because multilevel methods are intended for hierarchies with many levels.
minor comments (4)
  1. [Abstract and Section 5.2] There are typos such as 'addditive' in the abstract, 'descritized' in the NeoHook example, 'Jounal' in reference [18], and 'DDN' instead of 'DNN' in the caption of Figure 5.
  2. [§3, proof of Lemma 3.3] In the chain of equalities after equation (41), the term 'κ2_Gg' appears where 'κsκGg' is used in the final definition (39); this notational slip should be corrected.
  3. [§3, Theorem 3.8] The coherent-distribution condition (50) uses a constant κopt ∈ (0,1], but the conclusion (55) involves (1+κerr) without explaining how κopt or κerr enters; the relationship between these constants should be clarified.
  4. [§4, after Algorithm 4.2] The sentence 'we may apply Theorems 3.8 and Corollaries 3.6 and 3.9' is grammatically inconsistent; it should read 'Theorem 3.8 and Corollaries 3.6 and 3.9'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multilevel lemmas are proved in the text and the O(epsilon^-2) bound is a genuine extension of prior single-level results.

full rationale

The derivation chain is not circular. The multilevel-specific content (Lemma 3.2 controlling lower-level step lengths, Lemma 3.3 propagating the descent constants one level up, and Lemma 3.4 inducting over the hierarchy) is proved in the text from assumptions AS.1-AS.6, and the recurrence (39) is computed rather than fitted. The single-level lemmas and the final high-probability corollary are delegated to references [6] and [34], which share authors with this paper, but those references concern the single-level or parametric AdaGrad case and are not equivalent to the multilevel or domain-decomposition claims derived here; the dependence is reliance on prior work, not a reduction of the new result to its own inputs. AS.6 is a genuine modeling assumption relating coarse and fine gradients: in the deterministic case the tau-correction (48) enforces it, while in the stochastic case it remains an unverified hypothesis, which is a correctness or robustness risk rather than a circular step. No fitted parameter is relabeled as a prediction, and no uniqueness theorem is imported to force the authors' choice. The central O(epsilon^-2) bound therefore has independent content beyond its cited inputs.

Assumptions & free parameters 6 free parameters · 9 assumptions · 0 invented entities

The central complexity claim rests on six stated assumptions (AS.1-AS.6) plus an additional coherence condition for the exact-gradient result. AS.6 is the only link between fine and coarse models and is the most fragile. The algorithmic constants ς, κ1st, κ2nd are user-chosen; DNN learning rates are tuned to the data. No new physical or mathematical entities are introduced.

free parameters (6)
  • zeta (ς) = 0.01 in experiments
    Threshold in the event E and in the bound-constrained step normalization; appears in complexity constants (e.g., 1/ς in Lemma 3.2). Chosen by hand, not derived.
  • kappa_1st (κ1st) = 0.95
    Controls the minimal first-order progress passed to lower levels in recursive iterations; chosen by hand.
  • kappa_2nd (κ2nd) = 10
    Controls the maximal second-order effect allowed at lower levels; chosen by hand.
  • learning rate (IndPines) = 1e-2
    Tuned to the IndPines benchmark; affects reported DNN accuracy and C_ML.
  • learning rate (Aniso) = 5e-2
    Tuned to the DeepONet benchmark; affects reported validation loss and speedups.
  • recursive iteration counts (m_l) = 3 Taylor pre/post smoothing; 5 bottom steps; 10 or 25 decomposition iterations
    Algorithmic choices made per example; affect practical speedups but not asymptotic complexity.
assumptions (9)
  • domain assumption AS.1: each f_l is twice continuously differentiable
    Smoothness of all level objectives; used for Taylor expansions and Lipschitz arguments.
  • domain assumption AS.2: gradients of f_l are Lipschitz continuous with constant L_l
    Used in Lemma 3.2 to control the growth of lower-level iterates.
  • domain assumption AS.3: curvature approximations satisfy ||B_l,k|| <= kappa_B
    Bounded second-order information; needed for the linear descent bound (27).
  • standard math AS.4: objective is bounded below on the feasible set
    Standard assumption for global convergence; also needed for the Lambert-W bound.
  • domain assumption AS.5: gradient noise variance is bounded by kappa_Gg^2 E[min(||s||^2, theta_2^2)]
    Couples noise to step size; used in Lemma 3.1 and throughout.
  • domain assumption AS.6: model coherence E||P^T G_{l+1,k} - g_{l,0}||^2 <= kappa_tau^2 E[theta_{2,l}^2]
    The only assumption linking different levels; load-bearing for Lemmas 3.2 and 3.3.
  • domain assumption Event E: ||d_{r,0}||^2 >= ς has probability p_E > 1-delta
    All stochastic bounds are conditioned on this event; if the initial projected step is too small, the analysis does not apply.
  • domain assumption Coherently distributed gradient assumption (50) or the (misprinted) condition (53)
    Needed to convert convergence of the approximate criticality measure d to the exact measure Xi in Theorem 3.8.
  • domain assumption Transfer operators P_l, R_l have non-negative entries and row-sums sigma defined as in (2); empty columns handled by convention
    Feasibility of prolongated lower-level steps (Lemma 2.1) requires componentwise non-negativity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization." pith.science (2026). https://pith.science/paper/5YLQLIWL

@misc{pith2026250711513,
  author       = {Pith},
  title        = {Pith review of: Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5YLQLIWL}},
  note         = {Machine review of arXiv:2507.11513}
}
abstract

Two OFFO (Objective-Function Free Optimization) noise tolerant algorithms are presented that handle bound constraints, inexact gradients and use second-order information when available.The first is a multi-level method exploiting a hierarchical description of the problem and the second is a domain-decomposition method covering the standard addditive Schwarz decompositions. Both are generalizations of the first-order AdaGrad algorithm for unconstrained optimization. Because these algorithms share a common theoretical framework, a single convergence/complexity theory is provided which covers them both. Its main result is that, with high probability, both methods need at most $O(\epsilon^{-2})$ iterations and noisy gradient evaluations to compute an $\epsilon$-approximate first-order critical point of the bound-constrained problem. Extensive numerical experiments are discussed on applications ranging from PDE-based problems to deep neural network training, illustrating their remarkable computational efficiency.

Figures

Figures reproduced from arXiv: 2507.11513 by the authors.

Figure 1
Figure 1. Simulation results (solution z) for Minsurf and NeoHook examples. 1. Membrane: Following [25], let Ω := (0, 1)2 be a computational domain with boundary Γ = ∂Ω, decomposed into three parts: Γl = {0} × (0, 1), Γr = {1} × (0, 1), and Γf = (0, 1) × {0, 1}. The minimization problem is given as min z∈Z f(z) := 1 2 Z Ω ∥∇z(x)∥ 2 dz + Z Ω z(x) dz, subject to l(x) ≤ z, on Γr, (72) 4Because the requested decrease is now on a … view at source ↗
Figure 2
Figure 2. Left: An example of a multilevel hierarchy of ResNets, with the upper level depicted [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. The comparison of the convergence of the [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Convergence behavior of ML-ADAGB2 for Membrane (Top) and MinSurf (Bottom) exam￾ples with three levels. Left and Middle: Convergence as a function of V-cycles and computational cost for different choices of γℓ,k. The values of γℓ,k, which remain constant across all leve…
Figure 5
Figure 5. Figure 5: Top Left: Convergence of the ML-ADAGB2 without noise (red), with constant noise (blue, σ 2 = 10−7 ) and the noise reduced using the exponential scheduler (brown) for MinSurf (two levels). Bottom Left: Convergence of the ML-ADAGB2 with noise due to gradients subsampled …
Figure 6
Figure 6. Figure 6: The parallel computational cost CDD of ADAGB2(# subdomains=1) and DD-ADAGB2 for Aniso with varying number of subdomains (no overlap). Left: The validation loss fval. Right: The relative validation error Eval = 1/ns Pns s=1(y D s − ys)/∥ys∥, where y D s is the DeepONet-…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Method

    math.NA 2025-07 conditional novelty 6.0 of 10

    Deflation vectors taken from a trained DeepONet, via trunk functions or predicted solutions, reduce PCG iteration counts across varied parametric PDE benchmarks.

Reference graph

Works this paper leans on

78 extracted references · 74 canonical work pages · cited by 1 Pith paper

  1. [34]

    Gratton, S

    S. Gratton, S. Jerad, and Ph. L. Toint. Parametric complexity analysis for a class of first-order Adagrad-like algorithms. Optimization Methods and Software, (to appear), 2025

    work page 2025

  2. [6]

    Cosmological Consequences of Unconstrained Gravity and Electromagnetism

    S. Bellavia, G. Gratton, B. Morini, and Ph. L. Toint. Fast stochastic Adagrad for nonconvex bound-constrained optimization. arXiv:2405.06374, 2025

    work page Pith review arXiv 2025

  3. [1]

    Abreu, Z

    R. Abreu, Z. Su, J. Kamm, and J. Gao. On the accuracy of the complex-step-finite-difference method. Journal of Computational and Applied Mathematics, 340:390–403, 2018

    work page 2018

  4. [2]

    Badea and R

    L. Badea and R. Krause. One-and two-level Schwarz methods for variational inequalities of the second kind and their application to frictional contact. Numerische Mathematik, 120:573–599, 2012

    work page 2012

  5. [3]

    Badea, X.-C

    L. Badea, X.-C. Tai, and J. Wang. Convergence rate analysis of a multiplicative Schwarz method for variational inequalities. SIAM Journal on Numerical Analysis, pages 1052–1073, 2004

    work page 2004

  6. [4]

    Badea and J

    L. Badea and J. Wang. An additive Schwarz method for variational inequalities. Mathematics of Computation, 69(232):1341–1354, 2000. Gratton, Kopaniˇ c´ akov´ a, Toint: Recursive AdaGrad with Bound Constraints 31

    work page 2000

  7. [5]

    M. F. Baumgardner, L. L. Biehl, and D. A. Landgrebe. 220 Band A VIRIS Hyperspectral Image Data Set: June 12, 1992 Indian Pine Test Site 3. Purdue University Research Repository, 10(7):991, 2015

    work page 1992

  8. [7]

    Bellavia, G

    S. Bellavia, G. Gurioli, B. Morini, and Ph. L. Toint. Trust-region algorithms: probabilistic complexity and intrinsic noise with applications to subsampling techniques. EURO Journal on Computational Optimization, 20(100043), 2022

    work page 2022

Show all 78 references
  1. [8]

    Bellavia, G

    S. Bellavia, G. Gurioli, B. Morini, and Ph. L. Toint. The impact of noise on evaluation complexity: The deterministic trust-region case. Journal of Optimization Theory and Applications, 196(2):700–729, 2023

    2023

  2. [9]

    Bellavia, N

    S. Bellavia, N. Kreji´ c, and N. Krklec Jerinki´ c. Subsampled inexact Newton methods for minimizing large sums of convex functions. arXiv:1811.05730, 2018

    2018 arXiv

  3. [10]

    Berahas, L

    A. Berahas, L. Cao, and K. Scheinberg. Global convergence rate analysis of a generic line search algorithm with noise. SIAM Journal on Optimization, 31:1489–1518, 2021

    2021

  4. [11]

    Blanchet, C

    J. Blanchet, C. Cartis, M. Menickelly, and K. Scheinberg. Convergence rate analysis of a stochastic trust region method via supermartingales. INFORMS Journal on Optimization, 1(2):92–119, 2019

    2019

  5. [12]

    Bottou, F

    G. Bottou, F. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223–311, 2018

    2018

  6. [13]

    Brandt and C

    A. Brandt and C. W. Cryer. Multigrid algorithms for the solution of linear complementarity problems arising from free boundary problems. SIAM Journal on Scientific and Statistical Computing, 4(4):655–684, 1983

    1983

  7. [14]

    W. L. Briggs, V. E. Henson, and S. F. McCormick. A Multigrid Tutorial. SIAM, Philadelphia, USA, 2nd edition, 2000

    2000

  8. [15]

    X. C. Cai and D. E. Keyes. Nonlinearly preconditioned inexact Newton algorithms. SIAM Journal on Scientific Computing, 24(1):183–200, 2002

    2002

  9. [16]

    Cai and M

    X.-C. Cai and M. Sarkis. A restricted additive Schwarz preconditioner for general sparse linear systems. SIAM Journal on Scientific Computing, 21(2):792–797, 1999

    1999

  10. [17]

    Calandra, S

    H. Calandra, S. Gratton, E. Riccietti, and X. Vasseur. On high-order multilevel optimization strategies. SIAM Journal on Optimization, 31(1):307–330, 2021

    2021

  11. [18]

    R. G. Carter. On the global convergence of trust region methods using inexact gradient information. SIAM Jounal on Numerical Analysis, 28(1):251–265, 1991

    1991

  12. [19]

    Cartis and K

    C. Cartis and K. Scheinberg. Global convergence rate analysis of unconstrained optimization methods based on probabilistic models. Mathematical Programming A, 159(2):337–375, 2018

    2018

  13. [20]

    Chang, L

    B. Chang, L. Meng, E. Haber, F. Tung, and D. Begert. Multi-level residual networks from dynamical systems view. arXiv:1710.10348, 2017

    2017 arXiv

  14. [21]

    Chaouqui, M

    F. Chaouqui, M. J. Gander, P. M. Kumbhar, and T. Vanzan. Linear and nonlinear substructured restricted additive Schwarz iterations and preconditioning. Numerical Algorithms, 91(1):81–107, 2022

    2022

  15. [22]

    Ciaramella, F

    G. Ciaramella, F. Nobile, and T. Vanzan. A multigrid solver for PDE-constrained optimization with uncertain inputs. Journal of Scientific Computing, 101(1):13, 2024

    2024

  16. [23]

    Dolean, M

    V. Dolean, M. J. Gander, W. Kheriji, F. Kwok, and R. Masson. Nonlinear preconditioning: How to use a nonlinear Schwarz method to precondition Newton’s method. SIAM Journal on Scientific Computing, 38(6):A3357–A3380, 2016

    2016

  17. [24]

    Dolean, P

    V. Dolean, P. Jolivet, and F. Nataf. An introduction to domain decomposition methods: algorithms, theory, and parallel implementation. SIAM, 2015

    2015

  18. [25]

    Domor´ adov´ a and Z

    M. Domor´ adov´ a and Z. Dost´ al. Projector preconditioning for partially bound-constrained quadratic optimiza- tion. Numerical Linear Algebra with Applications, 14(10):791–806, 2007

    2007

  19. [26]

    Duchi, E

    J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimiza- tion. Journal of Machine Learning Research, 12, July 2011

    2011

  20. [27]

    M. C. Ferris and O. L. Mangasarian. Parallel variable distribution. SIAM Journal on Optimization, 4(4):815– 832, 1994

    1994

  21. [28]

    Frommer and D

    A. Frommer and D. B. Szyld. An algebraic convergence theory for restricted additive Schwarz methods using weighted max norms. SIAM Journal on Numerical Analysis, 39:463–479, 2002

    2002

  22. [29]

    Gaedke-Merzh¨ auser, A

    L. Gaedke-Merzh¨ auser, A. Kopaniˇ c´ akov´ a, and R. Krause. Multilevel minimization for deep residual networks. In Proceedings of French-German-Swiss Optimization Conference (FGS’2019), 2021

    2019

  23. [30]

    Gelman and J

    E. Gelman and J. Mandel. On multilevel iterative methods for optimization problems. Mathematical Programming, 48(1-3):1–17, 1990

    1990

  24. [31]

    Glorot and Y

    X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research, 9:249–256, 2010

    2010

  25. [32]

    Goodfellow, Y

    I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016. Gratton, Kopaniˇ c´ akov´ a, Toint: Recursive AdaGrad with Bound Constraints 32

    2016

  26. [33]

    Gratton, S

    S. Gratton, S. Jerad, and Ph. L. Toint. Complexity of Adagrad and other first-order methods for nonconvex optimization problems with bounds constraints. arXiv:2406.15793, 2024

    2024 arXiv

  27. [35]

    Gratton, A

    S. Gratton, A. Kopaniˇ c´ akov´ a, and Ph. L. Toint. Multilevel objective-function-free optimization with an application to neural networks training. SIAM Journal on Optimization, 33(4):2772–2800, 2023

    2023

  28. [36]

    Gratton, V

    S. Gratton, V. Mercier, E. Riccietti, and Ph. L. Toint. A block-coordinate approach of multi-level optimization with an application to physics-informed neural networks. Computational Optimization and Applications, (to appear), 2025

    2025

  29. [37]

    Gratton, M

    S. Gratton, M. Mouffe, Ph. L. Toint, and M. Weber-Mendon¸ ca. A recursive trust-region method in infinity norm for bound-constrained nonlinear optimization. IMA Jounal of Numerical Analysis, 28(4):827–861, 2008

    2008

  30. [38]

    Gratton, A

    S. Gratton, A. Sartenaer, and Ph. L. Toint. Recursive trust-region methods for multiscale nonlinear optimiza- tion. SIAM Journal on Optimization, 19(1):414–444, 2008

    2008

  31. [39]

    Gratton and Ph

    S. Gratton and Ph. L. Toint. A note on solving nonlinear optimization problems in variable precision. Computational Optimization and Applications, 76(3):917–933, 2020

    2020

  32. [40]

    Gratton, L.N

    S. Gratton, L.N. Vicente, and Z. Zhang. Optimization by space transformation and decomposition. Technical report, Polytechnic University of Hong Kong, Hong Kong, 2019

    2019

  33. [41]

    Ch. Groß. A unifying theory for nonlinear additively and multiplicatively preconditioned globalization strategies: convergence results and examples from the field of nonlinear elastostatics and elastodynamics. PhD thesis, Bonn University, Bonn, Germany, 2009

    2009

  34. [42]

    Groß and R

    Ch. Groß and R. Krause. A new class of non-linear additively preconditioned trust-region strategies: Con- vergence results and applications to non-linear mechanics. Technical Report 904, Institute for Numerical Simulation, University of Bonn, INS preprint, 2009

    2009

  35. [43]

    Groß and R

    Ch. Groß and R. Krause. On the globalization of ASPIN employing trust-region control strategies - convergence analysis and numerical examples. Technical Report 2011-03, Universita della Svizzera Italiana, Lugano, CH,

    2011

  36. [44]

    Haber and L

    E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017

    2017

  37. [45]

    Hackbusch and H

    W. Hackbusch and H. D. Mittelmann. On multi-grid methods for variational inequalities. Numerische Mathematik, 42(1):65–76, 1983

    1983

  38. [46]

    M. Holst. Algebraic Schwarz theory. Technical report, Department of Applied Mathematics and CRPC, California Institute of Technology, California, USA, 1994

    1994

  39. [47]

    R. H. W. Hoppe and R. Kornhuber. Adaptive multilevel methods for obstacle problems. SIAM journal on numerical analysis, 31(2):301–323, 1994

    1994

  40. [48]

    Imambi, K

    S. Imambi, K. B. Prakash, and G. R. Kanagachidambaresan. Pytorch. Programming with TensorFlow: solution for edge computing applications, pages 87–104, 2021

    2021

  41. [49]

    Karypis and V

    G. Karypis and V. Kumar. Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. Retrieved from the University Digital Con- servancy, https://hdl.handle.net/11299/215346, 1997

    1997

  42. [50]

    Kingma and J

    D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings in the International Conference on Learning Representations (ICLR), 2015

    2015

  43. [51]

    Koˇ cvara and S

    M. Koˇ cvara and S. Mohammed. A first-order multigrid method for bound-constrained convex optimization. Optimization Methods and Software, 31(3):622–644, 2016

    2016

  44. [52]

    Kopaniˇ c´ akov´ a, H

    A. Kopaniˇ c´ akov´ a, H. Kothari, G. E. Karniadakis, and R. Krause. Enhancing training of physics-informed neural networks using domain decomposition–based preconditioning strategies. SIAM Journal on Scientific Computing, 46(5):S46–S67, 2024

    2024

  45. [53]

    Kopaniˇ c´ akov´ a and R

    A. Kopaniˇ c´ akov´ a and R. Krause. A recursive multilevel trust region method with application to fully monolithic phase-field models of brittle fracture. Computer Methods in Applied Mechanics and Engineering, 360:112720, 2020

    2020

  46. [54]

    Kopaniˇ c´ akov´ a and R

    A. Kopaniˇ c´ akov´ a and R. Krause. A Multilevel Active-Set Trust-Region (MASTR) Method for Bound Con- strained Minimization. In Domain Decomposition Methods in Science and Engineering XXVI, pages 355–363. Springer, 2023

    2023

  47. [55]

    Kopaniˇ c´ akov´ a

    A. Kopaniˇ c´ akov´ a. On the use of hybrid coarse-level models in multilevel minimization methods. InInternational Conference on Domain Decomposition Methods, pages 303–310. Springer, 2022

    2022

  48. [56]

    Kopaniˇ c´ akov´ a and R

    A. Kopaniˇ c´ akov´ a and R. Krause. Globally convergent multilevel training of deep residual networks. SIAM Journal on Scientific Computing, 45(0):S254–S280, 2022

    2022

  49. [57]

    Kornhuber

    R. Kornhuber. Monotone multigrid methods for elliptic variational inequalities I. Numerische Mathematik, 69(2):167–184, 1994. Gratton, Kopaniˇ c´ akov´ a, Toint: Recursive AdaGrad with Bound Constraints 33

    1994

  50. [58]

    Kornhuber and R

    R. Kornhuber and R. Krause. Adaptive multigrid methods for Signorini’s problem in linear elasticity. Computing and Visualization in Science, 4(1):9–20, 2001

    2001

  51. [59]

    Kothari, A

    H. Kothari, A. Kopaniˇ c´ akov´ a, and R. Krause. Nonlinear Schwarz preconditioning for nonlinear optimization problems with bound constraints. In International Conference on Domain Decomposition Methods, pages 319–326. Springer, 2022

    2022

  52. [60]

    R. Krause. A nonsmooth multiscale method for solving frictional two-body contact problems in 2D and 3D with multigrid efficiency. SIAM Journal on Scientific Computing, 31(2):1399–1423, 2009

    2009

  53. [61]

    Y. Lee, A. Kopaniˇ c´ akov´ a, and G. E. Karniadakis. Two-level overlapping additive schwarz preconditioner for training scientific machine learning applications. arXiv:2406.10997, 2024

    2024

  54. [62]

    Leimkuhler, T

    B. Leimkuhler, T. Vlaar, T. Pouchon, and A. Storkey. Better training using weight-constrained stochastic dynamics. arXiv:2106.10704, 2021

    2021 arXiv

  55. [63]

    L. Lu, P. Jin, G. Pang, Z. Zhang, and G. Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021

    2021

  56. [64]

    L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, and S. G. Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021

    2021

  57. [65]

    J. Mandel. A multilevel iterative method for symmetric, positive definite linear complementarity problems. Applied Mathematics and Optimization, 11(1):77–95, 1984

    1984

  58. [66]

    O. L. Mangasarian. Parallel gradient distribution in unconstrained optimization. SIAM Journal on Control and Optimization, 33(6):1916–1925, 1995

    1916

  59. [67]

    S. G. Nash. A multigrid approach to discretized optimization problems. Optimization Methods and Software, 14:99–116, 2000

    2000

  60. [68]

    J. Park. Additive Schwarz methods for convex optimization as gradient methods. SIAM Journal on Numerical Analysis, 58(3):1495–1530, 2020

    2020

  61. [69]

    J. Park. Accelerated additive Schwarz methods for convex optimization with adaptive restart. Journal of Scientific Computing, 89(3):58, 2021

    2021

  62. [70]

    J. Park. Additive Schwarz methods for convex optimization with backtracking. Computers & Mathematics with Applications, 113:332–344, 2022

    2022

  63. [71]

    A. F. Queiruga, N. B. Erichson, D. Taylor, and M. W. Mahoney. Continuous-in-depth neural networks. arXiv:2008.02389, 2020

    2008 arXiv

  64. [72]

    Rathgeber, D

    F. Rathgeber, D. A. Ham, L. Mitchell, M. Lange, F. Luporiniand A. T. T. McRae, G.-T. Bercea, G. R. Markall, and P. H. J. Kelly. Firedrake: automating the finite element method by composing abstractions. ACM Transactions on Mathematical Software (TOMS), 43(3):1–27, 2016

    2016

  65. [73]

    Robbins and S

    H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statistics, 22(3):400––407, 1951

    1951

  66. [74]

    Trottenberg, C

    U. Trottenberg, C. W. Oosterlee, and A. Schuller. Multigrid methods. Academic press, 2001

    2001

  67. [75]

    Trotti, S

    K. Trotti, S. A. Cruz, A. Kopaniˇ c´ akov´ a, and R. Krause. Parallel trust-region approaches in neural network training. Mathematical Optimization for Machine Learning: Proceedings of the MATH+ Thematic Einstein Semester 2023, page 107, 2025

    2023

  68. [76]

    Vallejos

    M. Vallejos. MGOPT with gradient projection method for solving bilinear elliptic optimal control problems. Computing, 87(1-2):21–33, 2010

    2010

  69. [77]

    Youett, O

    J. Youett, O. Sander, and R. Kornhuber. A globally convergent filter-trust-region method for large deformation contact problems. SIAM Journal on Scientific Computing, 41(1):B114–B138, 2019

    2019

  70. [2011]

    Also available as arXiv:2104.05672v1

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.