Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

Bartolomeo Stellato; Rajiv Sambharya

REVIEW 4 major objections 4 minor 3 cited by

Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that learned per-iteration hyperparameter sequences can accelerate first-order solvers on parametric convex optimization problems, using only 10 training instances while preserving guaranteed convergence.

desk verdict A genuinely useful learning-to-optimize framework with sound closed-form results; the headline convergence guarantee for varying-instance problems like logistic regression rests on an unverified uniform Lipschitz bound and a biased safeguard. read the letter →

arxiv 2411.15717 v1 pith:RRJ7USGZ submitted 2024-11-24 math.OC cs.LG

Rajiv Sambharya , Bartolomeo Stellato This is my paper · ORCID

classification math.OCcs.LG MSC 90C2590C0668T05

keywords learningtooptimizeparametricconvexoptimizationhyperparameterfirst-ordermethodsfixed-pointiterationsoperatorsplittinggeneralizationguaranteesADMM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the hyperparameter sequence of a first-order optimization algorithm—such as step sizes in gradient descent—can be learned from data to solve parametric convex problems much faster than hand-tuned schedules. The framework, called LAH, runs fixed-point iterations in which the same hyperparameters are shared across all problem instances, with a step-varying phase followed by a steady-state phase that guarantees eventual convergence to an optimal solution. Training minimizes the mean squared error to a ground-truth solution, and in several cases the training subproblems admit closed-form solutions. The paper reports that only 10 training instances are needed across all experiments, and that the learned schedules outperform neural-network warm starts and learned metrics on control, signal processing, and machine learning problems while carrying high-probability performance guarantees on unseen data.

What carries the argument

The central object is the two-phase fixed-point iteration $z_{k+1} = T_{\theta_k}(z_k, x)$ for $k < H$ and $z_{k+1} = T_{\theta_H}(z_k, x)$ for $k \geq H$, where $T$ is the fixed-point map of the algorithm (gradient descent, proximal gradient descent, OSQP, or SCS) and $\theta_k$ are the learned hyperparameters shared across all problem instances. The step-varying phase is what purchases speed; the steady-state phase is what purchases convergence, since standard theory applies once the hyperparameters are constant and within the usual ranges. The training machinery is progressive $B$-step lookahead: at each stage only $B$ hyperparameters are optimized, and for gradient descent the one-step problem reduces to a least squares problem while the two- and three-step quadratic problems reduce to finding roots of a quadratic and a cubic. The generalization machinery is the KL bound in Theorem 8, inverted over a grid of tolerances with a union bound to give quantile bounds on the performance metric.

What would settle it

Run LAH on a fixed-matrix parametric family with 10 training instances and measure the 97.5th percentile upper bound on the residual after a fixed step budget; if the bound is vacuous or the learned schedule is slower than vanilla gradient descent with step size $1/L$, the claim of data-efficient acceleration is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a two-phase hyperparameter schedule shared across parametric instances makes a fixed-point solver both fast and safe: the step-varying phase is free to use aggressive, iteration-specific hyperparameters, while the steady-state phase returns to a provably convergent constant hyperparameter, so the whole trajectory converges to an optimal solution no matter what the learned phase did. Because the hyperparameters are shared, the expensive matrix factorizations in solvers such as OSQP and SCS can be computed offline once and reused for every instance. The paper further claims that the one-step gradient-descent training problem is a least squares problem, that the two- and three-step problems for unconstrained quadratic minimization have closed-form global solutions, and that when the parameter distribution is Gaussian the training problem can be solved without any sampled instances. Finally, it claims that KL-based sample convergence bounds, computed on a validation set of 1000 instances, provide non-vacuous upper and lower quantile guarantees for the learned optimizer's performance on unseen data.

Load-bearing premise

The problem-defining matrices $P$ and $A$ must be the same for all parametric instances, so that one shared hyperparameter sequence transfers across instances and the cached matrix factorizations remain valid.

Editorial extensions

If this is right

First-order solvers on parametric convex problems can be substantially accelerated using only 10 training instances, turning the solve-time bottleneck into a one-time offline training cost.
For parametric OSQP and SCS, all matrix factorizations can be done offline, so each new instance only requires cheap linear-system solves with cached factorizations.
Because the steady-state phase uses fixed hyperparameters, the learned optimizer can be evaluated on any number of iterations and inherits the underlying solver's convergence guarantee.
The learned schedules routinely use step sizes far outside the classical convergence range during the step-varying phase, suggesting that large-step strategies can be effective when balanced by a safe steady-state phase.
High-probability upper and lower quantile bounds on performance can be computed for unseen data, using only 1000 validation instances.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The fixed-matrix assumption is the most natural boundary of the method: if $P$ and $A$ vary across instances, the shared hyperparameter sequence will not transfer and the offline factorization caching ceases to apply, so extending LAH would require learning a mapping from matrix features to schedules.
The closed-form results for quadratic minimization suggest a general principle: the step-size schedule that minimizes a finite-horizon objective over a distribution is available in closed form whenever the residual covariance can be tracked, which may extend to other linear fixed-point iterations such as Anderson acceleration.
The data efficiency of using 10 instances indicates that the effective sample complexity is tied to the number of hyperparameters being learned (dozens of scalars), not the problem dimension, so the method should scale to larger problems without needing more training data.
One could test whether the learned schedules transfer across distributions with the same fixed matrices but different parameter distributions; the paper's convergence guarantees hold for any parameter, but the speed gains might degrade if the test distribution shifts far from the training distribution.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, this is a real contribution, not a repackaging: shared hyperparameter sequences, a two-phase architecture with a provably convergent steady state, closed-form two- and three-step lookahead for quadratic minimization, and KL-based generalization bounds are combined in a way I have not seen. The math in Theorems 1–7 looks correct, the proofs are detailed, and the numerical scope is broad—gradient descent, proximal gradient descent, OSQP, SCS, with code released. The data efficiency claim also holds up: ten training instances is genuinely surprising.

Second, the abstract's guarantee that the learned optimizer is 'guaranteed to converge to an optimal solution' is not actually established for all the experiments as written. The steady-state phase only ensures convergence if the fixed hyperparameters satisfy the classical range condition, e.g., 0 < theta_H < 2/L for gradient descent. In ridge regression, lasso, image deblurring, Kalman filtering, and maxcut, the problem data that determines L (or the analog for ADMM) is indeed fixed across instances, so the guarantee is legitimate there. But in logistic regression, the parameter x is the whole dataset and L(x) varies across instances. A single learned theta_H cannot satisfy 0 < theta_H < 2/L(x) for every unseen x unless the authors enforce it against a uniform upper bound on L over the instance distribution. They do not report such a bound or say how theta_H was chosen in that example.

The safeguarding mechanism is supposed to rescue this, but it has its own issue. Appendix B states the trigger uses f* defined as the average optimal value over the training instances. For a test instance whose optimal value differs from that average, the estimated suboptimality is biased, so the safeguard can either fail to trigger during divergence or fire on healthy iterates. The stress-test note lands here. This is the paper's softest spot, and it is a load-bearing one for the logistic regression result. The fix is straightforward: enforce a certified uniform bound on L (or a per-instance safeguard that uses an unbiased estimate of f*(x)), and report safeguard trigger rates across the test set.

Minor issues: Tables 3–8 report mean iterations without standard deviations, and the generalization bounds for logistic regression are wide—the strong test performance is not fully certified. Also, the fixed-matrix assumption stated in Section 4 is clearly acknowledged but limits the scope; the factorization caching claim depends on it.

Overall, the central framework is sound and the flaws are overclaiming and missing calibration details, not fundamental unsoundness. The paper deserves a serious referee. I would send it to review and ask for a major revision that tightens the convergence claim, fixes the safeguard calibration, and adds error bars and trigger statistics.

Referee Report

4 major / 4 minor

Summary. This paper proposes LAH, a framework for learning a shared hyperparameter sequence for parametric fixed-point and convex optimization solvers. The architecture consists of a step-varying phase of H iterations followed by a steady-state phase whose hyperparameters are chosen so that classical convergence theory applies. Training minimizes the mean squared error to ground-truth solutions using progressive B-step lookahead, with closed-form one-, two-, and three-step solutions for gradient descent and unconstrained quadratic minimization, and backpropagation for other algorithms. The paper also constructs validation-set generalization bounds and quantile bounds, and reports experiments on ridge and logistic regression, lasso, image deblurring, robust Kalman filtering, and max cut, using only 10 training instances in most examples.

Significance. If the claims hold, the paper makes a useful contribution: it shows that a very small number of shared hyperparameters can be learned with little data while retaining an asymptotic convergence guarantee, and it provides closed-form lookahead solutions plus numerically meaningful generalization bounds. The full proofs in the appendix and the public code repository are strengths, as is the breadth of the numerical study across GD, proximal GD, OSQP, and SCS. The main weaknesses are that the headline convergence guarantee is not actually established for the logistic regression experiment, the safeguarding mechanism is underspecified and biased as written, and the empirical tables report no measure of dispersion.

major comments (4)

[Section 7.1.2, Table 1, and Appendix B] The asymptotic convergence guarantee claimed in the abstract is not established for the logistic regression experiment. Table 1 states that the steady-state gradient-descent step size must satisfy 0 < theta_H < 2/L, but in Section 7.1.2 the parameter x is the full dataset (v_j, y_j), so the smoothness constant L = L(x) varies across instances. Since LAH learns a single theta_H shared by all instances, the condition 0 < theta_H < 2/L cannot be verified for unseen x unless the paper gives a uniform upper bound L_max over the parameter distribution and explicitly enforces theta_H < 2/L_max. No such bound or enforcement is reported. The safeguarding mechanism in Appendix B does not fill this gap because its trigger uses f*, the average optimal value over the training instances, which is not a valid per-instance optimality certificate for test instances. Please provide either a rigorous per-instance convergence argument, a repaired fallback with per-instance certificates, or a restriction of the 'guaranteed to converge' claim to the settings where the bound is actually enforced.
[Section 5.5 and Appendix B] The safeguarding mechanism is load-bearing for the logistic and lasso experiments, but as written it is not fully specified. The trigger condition in Appendix B, 'f(z^k(x), x) - f* > 10(f(z^k(x), x) - f*)', is self-referential and presumably a typo for comparing consecutive iterates. If the intended condition is f(z^k(x), x) - f* > 10(f(z^{k-1}(x), x) - f*), then replacing the per-instance optimal value with the training-average f* biases the test-time criterion: for instances with atypical optimal values, the safeguard can either fail to trigger during divergence or trigger on healthy iterates. The paper should state the corrected condition, justify the factor of 10, and report how often the safeguard triggers in every experiment where it is used; currently only the lasso example gives a trigger count.
[Section 7, Tables 3-8] The empirical claims are reported as a single mean iteration count per tolerance with no measure of dispersion. Given that the training set has only 10 instances and both training and test instances are randomly generated, the reported speedups in Tables 3-8 could be sensitive to the particular random draw. Please report standard deviations or quantiles over multiple independent training seeds, or at least the raw per-instance distributions, for LAH and the main baselines. This is needed to support the data-efficiency claim rather than a claim about one favorable draw.
[Section 5.3, Theorem 6] Theorem 6 as stated contains a mathematical typo in the central formula for the stochastic lookahead problem. It defines \bar z^k_j = (a^k_j)^T mu + (a^k_j)^T Sigma a^k_j, and the proof writes E[(a_j^T x)^2] = mu^T a_j + a_j^T Sigma a_j. For x ~ N(mu, Sigma), the correct identity is E[(a_j^T x)^2] = (a_j^T mu)^2 + a_j^T Sigma a_j, so the displayed formula is missing a square on the first term and is not a valid second moment. Because this formula underlies the claim that no training instances are needed for the ridge regression example, please correct it and re-verify the corresponding numerical results.

minor comments (4)

[Abstract and Section 7.1.1] The statement that 'we only use 10 problem instances to train the hyperparameters in all of our examples' is not literal, since the ridge regression example uses zero training instances and solves the stochastic problem directly; please rephrase to indicate that at most 10 instances are used.
[Notation and typos] There are several typos: 'identiy matrix' in the notation paragraph, 'Kullback-Liebler' should be 'Kullback-Leibler', 'we firsting generating H different trajectories' in Section 7.4.1, and 'outpeforms' in Section 7.3.1. The cross-reference to 'Problem (7.1.1)' in Section 7.1.1 is not meaningful because that problem is not numbered.
[Section 4] The fixed P and A assumption is stated only in Section 4; since it is central to the factorization-caching efficiency claim and to the transfer of hyperparameters across instances, it should be stated as a scope limitation in the introduction or abstract.
[Section 7.1.2] The smoothness parameter L is used to define the comparison algorithms and the steady-state constraint, but L is not defined for the logistic regression problem; please state explicitly how L is computed for each instance or how a single value is used.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; the learned-hyperparameter derivation is self-contained, and self-citations appear only as baselines or as an adaptation of an external sample-convergence bound.

full rationale

The paper's core derivation chain is self-contained. The asymptotic convergence guarantee follows from the steady-state constraints in Table 1 (e.g., 0 < theta_H < 2/L), which are classical sufficient conditions from first-order method theory rather than consequences of the learned objective. The one-step lookahead step size is derived as the closed-form solution of a least-squares problem (Equations 5-6), and the two- and three-step quadratic lookahead problems are solved by explicit critical-point analysis (Theorems 4-5); none of these derivations presupposes that the learned schedule will be fast on unseen instances. The generalization guarantees in Section 6 rest on the external KL sample-convergence bound of Langford and Caruana (Theorem 8, reference [49]); the authors' self-citation [62] is only an adaptation of that method and is not load-bearing. Other self-citations, such as [60] and [61], are baselines and comparison points, while [56] is related context, so the central claims do not reduce to a self-citation chain. The numerical claims are evaluated on hold-out test instances, and the data-efficiency observation is empirical rather than derived from the training loss. Separately, there are correctness risks that are not circularity: the paper itself notes in Section 7.4.1 that no generalization guarantees are provided when parameters are not i.i.d.; Appendix B's safeguard trigger is written with identical expressions on both sides and uses the training-average optimal value f* for what should be a per-instance suboptimality estimate; and for logistic regression the smoothness constant L depends on the instance, so a single learned theta_H satisfying 0 < theta_H < 2/L for every test instance is not demonstrated. These issues weaken the out-of-sample guarantee, but they do not make any derivation equivalent to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No new theoretical entities are introduced. The free parameters are training setup choices (H, K, B, safeguarding threshold, validation sizes). The axioms are standard domain assumptions and one ad-hoc assumption about the safeguarding mechanism.

free parameters (5)

H (number of step-varying iterations) = 50 (100 for logistic regression)
Number of learned varying hyperparameters; chosen by hand for all experiments.
K (total training iterations) = 60
Training horizon after which the MSE to ground truth is minimized; chosen by hand.
B (lookahead steps in progressive training) = 1, 2, 3, or 10
Number of steps optimized simultaneously; the default is 10, while 1-3 are used for closed-form cases.
Safeguarding threshold factor = 10
If the objective increases by more than a factor of 10 in a single iteration, the algorithm reverts to a safe update. Hand-chosen in Section 5.5.
Validation hyperparameters (N_val, delta, N_tol) = 1000, 1e-5, 151
Used to construct quantile bounds; chosen in Appendix B.

assumptions (5)

domain assumption The fixed-point operator T for the considered convex problems is nonexpansive or averaged, so running the steady-state phase with safe hyperparameters guarantees convergence to an optimal solution.
Invoked in Section 3 to justify convergence of the steady-state phase; relies on standard monotone operator theory.
domain assumption The problem matrices P and A are fixed across parametric instances, so the smoothness constants and matrix factorizations are shared.
Stated in Section 4; enables factorization caching and justifies a single hyperparameter sequence.
domain assumption Ground-truth solutions z*(x) are available for the training instances.
The training loss in Eq. (3) is the MSE to z*(x_i); this is a supervised learning setting.
domain assumption The validation set is drawn i.i.d. from the same distribution as the test set.
Generalization bounds in Section 6 rely on the Langford-Caruana sample convergence bound, which assumes i.i.d. samples.
ad hoc to paper The safeguarding mechanism does not trigger frequently enough to dominate the reported results.
The paper reports trigger counts for lasso (2 out of 1000 test problems) but not for logistic regression, where learned step sizes are orders of magnitude above the stability limit.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization." pith.science (2026). https://pith.science/paper/RRJ7USGZ

@misc{pith2026241115717,
  author       = {Pith},
  title        = {Pith review of: Learning Algorithm Hyperparameters for Fast Parametric Convex Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRJ7USGZ}},
  note         = {Machine review of arXiv:2411.15717}
}

abstract

We introduce a machine-learning framework to learn the hyperparameter sequence of first-order methods (e.g., the step sizes in gradient descent) to quickly solve parametric convex optimization problems. Our computational architecture amounts to running fixed-point iterations where the hyperparameters are the same across all parametric instances and consists of two phases. In the first step-varying phase the hyperparameters vary across iterations, while in the second steady-state phase the hyperparameters are constant across iterations. Our learned optimizer is flexible in that it can be evaluated on any number of iterations and is guaranteed to converge to an optimal solution. To train, we minimize the mean square error to a ground truth solution. In the case of gradient descent, the one-step optimal step size is the solution to a least squares problem, and in the case of unconstrained quadratic minimization, we can compute the two and three-step optimal solutions in closed-form. In other cases, we backpropagate through the algorithm steps to minimize the training objective after a given number of steps. We show how to learn hyperparameters for several popular algorithms: gradient descent, proximal gradient descent, and two ADMM-based solvers: OSQP and SCS. We use a sample convergence bound to obtain generalization guarantees for the performance of our learned algorithm for unseen data, providing both lower and upper bounds. We showcase the effectiveness of our method with many examples, including ones from control, signal processing, and machine learning. Remarkably, our approach is highly data-efficient in that we only use $10$ problem instances to train the hyperparameters in all of our examples.

Figures

Figures reproduced from arXiv: 2411.15717 by the authors.

**Figure 2.** Ridge regression results. The conjugate gradient method performs the best out of all [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

**Figure 3.** Step sizes in gradient descent to solve the ridge regression problem. First on the left: [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Logistic regression results. Progressive training 10 steps at a time reaches a geometric [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

**Figure 5.** Figure 5: Logistic regression step sizes. Many of the step sizes learned with LAH in the step [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

**Figure 6.** Figure 6: Left: Lasso results. LAH, along with its upper and lower bounds, performs the best here [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

**Figure 7.** Figure 7: Image deblurring results. LAH performs the best for both the primal and dual residuals. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

**Figure 8.** Figure 8: Image deblurring visuals. Each row corresponds to an unseen digit from the EMNIST [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

**Figure 9.** Figure 9: Robust Kalman filtering results. LAH performs the best, with the LM approach coming [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

**Figure 10.** Figure 10: Robust Kalman filtering visualizations. The parameter for these unseen instances is the [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

**Figure 11.** Figure 11: Maxcut results. In this example, only LAH siginificantly improves the performance over [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Accelerating Proximal Gradient Descent via Silver Stepsizes
math.OC 2024-12 conditional novelty 8.0 of 10

Proximal and projected gradient descent using the silver stepsize schedule achieve the silver convergence rate O(ε^{-log_ρ 2}) for composite convex optimization, matching the rate known only for unconstrained smooth g...
Learning Adaptive Solvers for Distributed Factor Graph Optimization on Matrix Lie Groups
cs.RO 2026-07 conditional novelty 6.0 of 10

A learned feedback policy replaces manual parameter tuning in distributed Riemannian optimization over matrix Lie groups, achieving lower objective values on multi-robot mapping benchmarks.
Deep Distributed Optimization for Large-Scale Quadratic Programming
math.OC 2024-12 reject novelty 6.0 of 10

A deep-unfolded distributed ADMM/OSQP solver learns penalty parameters on small QPs and solves much larger ones with large wall-clock speedups, with PAC-Bayes bounds on relative progress.

Reference graph

Works this paper leans on

71 extracted references · 56 canonical work pages · cited by 3 Pith papers

[1]

J. M. Altschuler and P. A. Parrilo, Acceleration by stepsize hedging i: Multi-step descent and the silver stepsize schedule , arXiv preprint arXiv:2309.07879, (2023)

arXiv 2023
[2]

J. M. Altschuler and P. A. Parrilo , Acceleration by stepsize hedging ii: Silver stepsize schedule for smooth convex optimization , arXiv preprint arXiv:2309.16530, (2023)

arXiv 2023
[3]

Amos, Tutorial on amortized optimization , Foundations and Trends in Machine Learning, 16 (2023), pp

B. Amos, Tutorial on amortized optimization , Foundations and Trends in Machine Learning, 16 (2023), pp. 592–732

work page 2023
[4]

Baker , Learning warm-start points for ac optimal power flow , in IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2019

K. Baker , Learning warm-start points for ac optimal power flow , in IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2019

work page 2019
[5]

Balatsoukas-Stimming and C

A. Balatsoukas-Stimming and C. Studer , Deep unfolding for communications systems: A survey and some new directions , in 2019 IEEE International Workshop on Signal Processing Systems (SiPS), 2019, pp. 266–271

work page 2019
[6]

Balcan, Data-driven algorithm design , arXiv preprint arXiv:2011.07177, (2020)

M.-F. Balcan, Data-driven algorithm design , arXiv preprint arXiv:2011.07177, (2020)

arXiv 2020
[7]

Balcan, T

M.-F. Balcan, T. Dick, T. Sandholm, and E. Vitercik, Learning to branch, in Proceed- ings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 344–353

work page 2018
[8]

Accelerated Forward-Backward Optimization using Deep Learning

S. Banert, J. Rudzusika, O. ¨Oktem, and J. Adler , Accelerated forward-backward opti- mization using deep learning , arXiv preprint arXiv:2105.05210, (2021)

work page Pith review arXiv 2021

Show all 71 references

[9]

A. G. Baydin, B. A. Pearlmutter, A. Radul, and J. M. Siskind , Automatic differen- tiation in machine learning: a survey , J. Mach. Learn. Res., 18 (2017), pp. 153:1–153:43

2017
[10]

Beck , First-Order Methods in Optimization , Society for Industrial and Applied Mathe- matics, 2017

A. Beck , First-Order Methods in Optimization , Society for Industrial and Applied Mathe- matics, 2017

2017
[11]

Beck and M

A. Beck and M. Teboulle , A fast iterative shrinkage-thresholding algorithm with applica- tion to wavelet-based image deblurring , in 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 2009, pp. 693–696. 26

2009
[12]

D. P. Bertsekas, Nonlinear Programming, Athena Scientific, 2nd ed., 1999

1999
[13]

Borrelli, A

F. Borrelli, A. Bemporad, and M. Morari , Predictive Control for Linear and Hybrid Systems, Cambridge University Press, 2017

2017
[14]

Boyd and L

S. Boyd and L. V andenberghe, Convex Optimization , Cambridge University Press, 2004

2004
[15]

Boyd and L

S. Boyd and L. V andenberghe, Introduction to applied linear algebra: vectors, matrices, and least squares , Cambridge university press, 2018

2018
[16]

Briden, C

J. Briden, C. Choi, K. Yun, R. Linares, and A. Cauligi , Constraint-informed learning for warm starting trajectory optimization , arXiv preprint arXiv:2312.14336, (2023)

2023 arXiv
[17]

Burton, The History of Mathematics: An Introduction , 7th ed., 2010

D. Burton, The History of Mathematics: An Introduction , 7th ed., 2010

2010
[18]

H. Cai, J. Liu, and W. Yin , Learned robust pca: A scalable deep unfolding approach for high-dimensional outlier detection , Advances in Neural Information Processing Systems, 34 (2021), pp. 16977–16989

2021
[19]

Callebaut , Generalization of the cauchy-schwarz inequality , Journal of Mathematical Analysis and Applications, 12 (1965), pp

D. Callebaut , Generalization of the cauchy-schwarz inequality , Journal of Mathematical Analysis and Applications, 12 (1965), pp. 491–494

1965
[20]

S. Chen, K. Saulnier, N. Atanasov, D. D. Lee, V. Kumar, G. J. Pappas, and M. Morari , Approximating explicit model predictive control using constrained neural net- works, in 2018 Annual American Control Conference (ACC), 2018, pp. 1520–1527

2018
[21]

S. W. Chen, T. W ang, N. Atanasov, V. Kumar, and M. Morari, Large scale model pre- dictive control with neural networks and primal active sets , Automatica, 135 (2022), p. 109947

2022
[22]

T. Chen, X. Chen, W. Chen, H. Heaton, J. Liu, Z. W ang, and W. Yin , Learning to optimize: A primer and a benchmark , Journal of Machine Learning Research, 23 (2022), pp. 1–59

2022
[23]

X. Chen, J. Liu, Z. W ang, and W. Yin , Theoretical linear convergence of unfolded ista and its practical weights and thresholds , Advances in Neural Information Processing Systems, 31 (2018)

2018
[24]

Cohen, S

G. Cohen, S. Afshar, J. Tapson, and A. van Schaik , Emnist: an extension of mnist to handwritten letters , 2017, https://arxiv.org/abs/1702.05373

2017 arXiv
[25]

Das Gupta, B

S. Das Gupta, B. P. V an Parys, and E. Ryu, Branch-and-bound performance estimation programming: A unified methodology for constructing optimal optimization methods , Mathe- matical Programming, 204 (2024), pp. 567–639

2024
[26]

d’Aspremont, D

A. d’Aspremont, D. Scieur, A. Taylor, et al. , Acceleration methods, Foundations and Trends in Optimization, 5 (2021), pp. 1–245

2021
[27]

Diamond, V

S. Diamond, V. Sitzmann, F. Heide, and G. Wetzstein, Unrolled optimization with deep priors, arXiv preprint arXiv:1705.08041, (2017)

2017 arXiv
[28]

Diehl, H

M. Diehl, H. J. Ferreau, and N. Haverbeke, Efficient Numerical Methods for Nonlinear MPC and Moving Horizon Estimation , 2009. 27

2009
[29]

Douglas and H

J. Douglas and H. H. Rachford, On the numerical solution of heat conduction problems in two and three space variables , Transactions of the American Mathematical Society, 82 (1956), pp. 421–439

1956
[30]

Drori and M

Y. Drori and M. Teboulle , Performance of first-order methods for smooth convex mini- mization: a novel approach , Mathematical Programming, 145 (2014), pp. 451–482

2014
[31]

Erdos and A

P. Erdos and A. Renyi , On random graphs i , Publicationes Mathematicae Debrecen, 6 (1959), p. 18

1959
[32]

C. Finn, P. Abbeel, and S. Levine , Model-agnostic meta-learning for fast adaptation of deep networks, in International Conference on Machine Learning, 2017

2017
[33]

Giselsson and S

P. Giselsson and S. Boyd , Linear convergence and metric selection for douglas-rachford splitting and admm , IEEE Transactions on Automatic Control, 62 (2017), pp. 532–544

2017
[34]

M. X. Goemans and D. P. Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, Journal of the ACM, 42 (1995), pp. 1115–1145

1995
[35]

Gregor and Y

K. Gregor and Y. LeCun , Learning fast approximations of sparse coding , in International Conference on Machine Learning, Madison, WI, USA, 2010, Omnipress

2010
[36]

Grimmer, Provably faster gradient descent via long steps , SIAM Journal on Optimization, 34 (2024), pp

B. Grimmer, Provably faster gradient descent via long steps , SIAM Journal on Optimization, 34 (2024), pp. 2588–2608

2024
[37]

Grimmer, K

B. Grimmer, K. Shu, and A. L. W ang, Accelerated gradient descent via long steps , arXiv preprint arXiv:2309.09961, (2023)

2023 arXiv
[38]

Heaton, X

H. Heaton, X. Chen, Z. W ang, and W. Yin, Safeguarded learned convex optimization, in Proceedings of the AAAI Conference on Artificial Intelligence, 2023

2023
[39]

P. V. Hentenryck, Machine Learning for Optimal Power Flows , 2021, ch. 3, pp. 62–82

2021
[40]

M. R. Hestenes, E. Stiefel, et al. , Methods of conjugate gradients for solving linear systems, vol. 49, 1952

1952
[41]

P. J. Huber, Robust estimation of a location parameter , The Annals of Mathematical Statis- tics, 35 (1964), pp. 73–101

1964
[42]

Ichnowski, P

J. Ichnowski, P. Jain, B. Stellato, G. Banjac, M. Luo, F. Borrelli, J. E. Gonza- les, I. Stoica, and K. Goldberg , Accelerating quadratic optimization with reinforcement learning, in Advances in Neural Information Processing Systems 35, 2021

2021
[43]

R. E. Kalman , A new approach to linear filtering and prediction problems , Transactions of the ASME–Journal of Basic Engineering, 82 (1960), pp. 35–45

1960
[44]

Karg and S

B. Karg and S. Lucia, Efficient representation and approximation of model predictive control laws via deep learning , IEEE Transactions on Cybernetics, PP (2020)

2020
[45]

Kawaguchi, L

K. Kawaguchi, L. P. Kaelbling, and Y. Bengio , Generalization in deep learning , arXiv preprint arXiv:1710.05468, 1 (2017). 28

2017 arXiv
[46]

E. King, J. Kotary, F. Fioretto, and J. Drgona , Metric learning to accelerate convergence of operator splitting methods for differentiable parametric programming , CoRR, abs/2404.00882 (2024)

2024 arXiv
[47]

Kotary, F

J. Kotary, F. Fioretto, P. V an Hentenryck, and B. Wilder, End-to-end constrained optimization learning: A survey , in International Joint Conference on Artificial Intelligence, IJCAI-21, 2021

2021
[48]

Kullback and R

S. Kullback and R. A. Leibler , On Information and Sufficiency , The Annals of Mathe- matical Statistics, 22 (1951), pp. 79 – 86

1951
[49]

Langford and R

J. Langford and R. Caruana, (not) bounding the true error , in Advances in Neural Infor- mation Processing Systems, vol. 14, MIT Press, 2001

2001
[50]

A. Li, Z. Ding, A. B. Dieng, and R. Beeson, Efficient and guaranteed-safe non-convex tra- jectory optimization with constrained diffusion model, arXiv preprint arXiv:2403.05571, (2024)

2024 arXiv
[51]

J. Liu, X. Chen, Z. W ang, and W. Yin, ALISTA: Analytic weights are as good as learned weights in LISTA , in International Conference on Learning Representations, 2019

2019
[52]

Nesterov , A method for unconstrained convex minimization problem with the rate of convergence o(1/k2), 1983

Y. Nesterov , A method for unconstrained convex minimization problem with the rate of convergence o(1/k2), 1983

1983
[53]

Nocedal and S

J. Nocedal and S. J. Wright , Numerical Optimization , Springer, New York, NY, USA, 2006

2006
[54]

O’Donoghue, Operator splitting for a homogeneous embedding of the linear complemen- tarity problem, SIAM Journal on Optimization, 31 (2021), pp

B. O’Donoghue, Operator splitting for a homogeneous embedding of the linear complemen- tarity problem, SIAM Journal on Optimization, 31 (2021), pp. 1999–2023

2021
[55]

Parikh and S

N. Parikh and S. Boyd , Proximal algorithms, Foundations and Trends in Optimization, 1 (2014)

2014
[56]

Ranjan and B

V. Ranjan and B. Stellato , Verification of first-order methods for parametric quadratic optimization, arXiv preprint arXiv:2403.03331, (2024)

2024 arXiv
[57]

Ryu and S

E. Ryu and S. P. Boyd , A primer on monotone operator methods , 2015

2015
[58]

E. Ryu, J. Liu, S. W ang, X. Chen, Z. W ang, and W. Yin , Plug-and-play methods provably converge with properly trained denoisers , in International Conference on Machine Learning, 2019

2019
[59]

Ryu and W

E. Ryu and W. Yin, Large-Scale Convex Optimization: Algorithms amp; Analyses via Mono- tone Operators, Cambridge University Press, 2022

2022
[60]

Sambharya, G

R. Sambharya, G. Hall, B. Amos, and B. Stellato , End-to-End Learning to Warm- Start for Real-Time Quadratic Optimization , in Proceedings of the Learning for Dynamics and Control, 2023

2023
[61]

Sambharya, G

R. Sambharya, G. Hall, B. Amos, and B. Stellato , Learning to warm-start fixed-point optimization algorithms , Journal of Machine Learning Research, 25 (2024), pp. 1–46. 29

2024
[62]

Sambharya and B

R. Sambharya and B. Stellato , Data-driven performance guarantees for classical and learned optimizers, arXiv preprint arXiv:2404.13831, (2024)

2024
[63]

Shapiro, D

A. Shapiro, D. Dentcheva, and A. Ruszczy ´nski, Lectures on Stochastic Programming: Modeling and Theory, vol. 9, Society for Industrial and Applied Mathematics (SIAM), 2nd ed., 2009

2009
[64]

Sharony, H

E. Sharony, H. Yang, T. Che, M. Pavone, S. Mannor, and P. Karkus , Learning multiple initial solutions to optimization problems , arXiv preprint arXiv:2411.02158, (2024)

2024 arXiv
[65]

Stellato, G

B. Stellato, G. Banjac, P. Goulart, A. Bemporad, and B. Stephen , OSQP: An Operator Splitting Solver for Quadratic Programs , Mathematical Programming Computation, 12 (2020), pp. 637–672

2020
[66]

Sucker, J

M. Sucker, J. F adili, and P. Ochs, Learning-to-optimize with PAC-Bayesian guarantees: Theoretical considerations and practical implementation , arXiv preprint arXiv:2404.03290, (2024)

2024 arXiv
[67]

Venkataraman and B

S. Venkataraman and B. Amos , Neural fixed-point acceleration for convex optimization , arXiv preprint arXiv:2107.10254, (2021)

2021 arXiv
[68]

H. F. W alker and P. Ni, Anderson acceleration for fixed-point iterations, SIAM Journal on Numerical Analysis, 49 (2011), pp. 1715–1735

2011
[69]

D. M. Young , On richardson ’s method for solving linear systems with positive definite ma- trices, Journal of Mathematics and Physics, 32 (1953), pp. 243–255

1953
[70]

Zhang, B

J. Zhang, B. O’Donoghue, and S. Boyd, Globally convergent type-I anderson acceleration for nonsmooth fixed-point iterations, SIAM Journal on Optimization, 30 (2020), pp. 3170–3197. A First-order methods Gradient descent. Here, z ∈ Rn is the decision variable, and f : Rn × Rd →...

2020
[71]

(29) This comes from treating zk(B−1) θ (x) as the initial points, and applying the worst-case rate of convergence for Young’s Chebyshev step sizes. We finish the proof as follows: E x∼X ∥zkB θ (x) − z⋆(x)∥2 2 ≤ E x∼X ∥(I − βk+B−1P ) · · ·(I − βkP )(z(k−1)B θ (x) − z⋆(x))∥2 2 ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

J. M. Altschuler and P. A. Parrilo, Acceleration by stepsize hedging i: Multi-step descent and the silver stepsize schedule , arXiv preprint arXiv:2309.07879, (2023)

arXiv 2023

[2] [2]

J. M. Altschuler and P. A. Parrilo , Acceleration by stepsize hedging ii: Silver stepsize schedule for smooth convex optimization , arXiv preprint arXiv:2309.16530, (2023)

arXiv 2023

[3] [3]

Amos, Tutorial on amortized optimization , Foundations and Trends in Machine Learning, 16 (2023), pp

B. Amos, Tutorial on amortized optimization , Foundations and Trends in Machine Learning, 16 (2023), pp. 592–732

work page 2023

[4] [4]

Baker , Learning warm-start points for ac optimal power flow , in IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2019

K. Baker , Learning warm-start points for ac optimal power flow , in IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2019

work page 2019

[5] [5]

Balatsoukas-Stimming and C

A. Balatsoukas-Stimming and C. Studer , Deep unfolding for communications systems: A survey and some new directions , in 2019 IEEE International Workshop on Signal Processing Systems (SiPS), 2019, pp. 266–271

work page 2019

[6] [6]

Balcan, Data-driven algorithm design , arXiv preprint arXiv:2011.07177, (2020)

M.-F. Balcan, Data-driven algorithm design , arXiv preprint arXiv:2011.07177, (2020)

arXiv 2020

[7] [7]

Balcan, T

M.-F. Balcan, T. Dick, T. Sandholm, and E. Vitercik, Learning to branch, in Proceed- ings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 344–353

work page 2018

[8] [8]

Accelerated Forward-Backward Optimization using Deep Learning

S. Banert, J. Rudzusika, O. ¨Oktem, and J. Adler , Accelerated forward-backward opti- mization using deep learning , arXiv preprint arXiv:2105.05210, (2021)

work page Pith review arXiv 2021