Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

This tutorial argues that differentiable programming turns first-order optimization algorithms into trainable, differentiable modules whose parameters can be learned end-to-end.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:32 UTC pith:VPOGYORW

load-bearing objection A broad but shallow tutorial that rehashes standard material and overclaims 'learning' — the actual implementations are fixed first-order methods in PyTorch, and the one numerical validation is circular. the 5 major comments →

arxiv 2601.16510 v4 pith:VPOGYORW submitted 2026-01-23 cs.MS cs.LGmath.OC

Learning to Optimize by Differentiable Programming

Liping Tao , Xindi Tong , Chee Wei Tan This is my paper
classification cs.MS cs.LGmath.OC MSC 90C2590C4665K0568T07
keywords differentiable programmingfirst-order optimizationFenchel-Rockafellar dualityADMMprimal-dual hybrid gradientcone programminglearning to optimizenonnegative least squares
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that differentiable programming—building programs that are themselves differentiable—lets optimization algorithms be treated as trainable components rather than fixed solvers. By embedding classic first-order methods like ADMM and PDHG in automatic-differentiation frameworks, it claims, the algorithm's parameters and even its update structure can be learned from data, improving convergence and solution quality. Fenchel–Rockafellar duality provides the saddle-point structure that makes this possible: primal and dual variables are updated by differentiable steps, and the dual problem supplies a certificate of optimality. The tutorial demonstrates this recipe on the running nonnegative least-squares example and on case studies in linear programming, optimal power flow, neural-network verification, and Laplacian-regularized minimization. A sympathetic reader would take away that optimization algorithms are no longer black boxes but differentiable components that can adapt to problem structure and be tuned end-to-end.

Core claim

The central claim is that a first-order optimization algorithm can be re-expressed as a differentiable program, and that enclosing it in an automatic-differentiation framework creates a mechanism for learning its design. The theoretical engine is duality: Fenchel–Rockafellar duality converts a convex constrained problem into a min-max saddle-point form whose iterates are proximal or gradient steps on primal and dual variables. Because each step is differentiable, reverse-mode automatic differentiation can propagate gradients backward through the entire iterative loop, allowing step sizes, penalty parameters, and dual variables to be adjusted by training. The authors establish this through de

What carries the argument

The central object is the primal–dual saddle-point reformulation of a convex cone program, typically obtained through Fenchel–Rockafellar duality. This reformulation turns the original constrained problem into an unconstrained min-max problem whose updates are proximal or gradient steps on primal and dual variables—ADMM, PDHG, or projected primal–dual gradient. Inside an automatic-differentiation framework, this iterative loop is a differentiable program: reverse-mode automatic differentiation propagates gradients backward through the iterates, so that free parameters such as step sizes, the ADMM penalty ρ, and the dual variables themselves can be learned or adjusted end-to-end. The Fenchel

Load-bearing premise

The Laplacian-regularized case study assumes that the scalar constant in the dual-to-primal recovery formula is uniquely fixed by primal feasibility and complementary slackness, but no derivation is given and the only numerical check fits that constant using the known answer.

What would settle it

Solve an LR-NNLS instance with a non-trivial active set, attempt to determine the scalar constant from complementary slackness and feasibility alone (without peeking at the primal solution), and compare the reconstructed x to a reference solver's output; if the constant is underdetermined, the reconstruction will fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The same solver code can double as a differentiable layer inside a neural network, so downstream learning objectives can directly tune optimization hyperparameters without manual gradient derivations.
  • Duality provides a certificate of solution quality during training, enabling principled stopping criteria based on duality gap rather than iteration count.
  • The framework supports parallel and GPU-accelerated execution of decomposed dual subproblems, as illustrated in the distributed nonnegative least-squares example, potentially scaling to very large cone programs.
  • The recipe extends across problem families—LPs, QPs, SDP relaxations—meaning a single differentiable programming approach can replace bespoke solver wrappers in end-to-end pipelines.
  • Training signals can flow through both primal and dual variables, allowing a dual solution to be refined iteratively until it is certified, which is useful when high-accuracy primal solutions are needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's demonstrations implement standard algorithms (ADMM, PDHG, projected gradient) and do not yet train any of their parameters from data; the central 'learning to design' claim would be tested by experiments showing learned step sizes or penalties outperforming fixed ones on unseen problem instances.
  • The dual-to-primal recovery formula in the Laplacian-regularized case study is validated by fitting the scalar constant to the known solution; a non-circular check would compute that constant from complementary slackness and active-set information alone.
  • If the recovery formula is correct, a practical method could solve Laplacian-regularized problems entirely in the dual space, avoiding costly pseudoinverse factorizations; this is worth testing on large graphs where the pseudoinverse is infeasible.
  • The tutorial's framework suggests that automatic differentiation through iterative solvers could serve as a generic mechanism for algorithm discovery, but supporting that broader claim will require evidence of learned algorithms, not just reimplementations of classical ones.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This tutorial argues that differentiable programming frameworks (PyTorch, JAX, TensorFlow) can be used not only to execute first-order optimization algorithms but also to "learn how to design" them. It reviews automatic differentiation, cone programming, Lagrangian and Fenchel duality, ADMM, PDHG, and presents PyTorch implementations for NNLS and for case studies spanning the Stigler diet problem, neural network verification, optimal power flow, and Laplacian-regularized NNLS. The paper's stated thesis is that embedding first-order methods in automatic-differentiation frameworks allows end-to-end training that improves convergence and solution quality, and that the case studies demonstrate these gains. As submitted, however, the manuscript does not actually learn any algorithm: the only "learning" routine treats fixed Lagrangian variables as PyTorch parameters and runs Adam, while the case studies execute hand-specified iterations. In addition, a core Fenchel-conjugate derivation is mathematically incorrect, and the one numerical validation of a duality-based recovery formula is circular.

Significance. If the claims were established, this would be a useful tutorial bridging differentiable programming and convex optimization, with reproducible PyTorch code. The paper has genuine strengths: a broad survey of differentiable-programming frameworks and optimization packages, a correct Lagrangian derivation of the NNLS dual, and concrete ADMM/PDHG implementations with accompanying source code. However, the central claim about learning to design algorithms is not demonstrated, a key conjugate computation is invalid, and the only recovery validation is circular. The current value of the manuscript is therefore primarily expository rather than as evidence for the learning-to-optimize thesis.

major comments (5)
  1. [§3.3.2, Eq. (36)] The computation of the Fenchel conjugate g*(y) drops the term -1/2||Ax-(y+b)||^2 inside a supremum over x≥0. This is valid only if inf_{x≥0}||Ax-(y+b)||^2=0 for every y, i.e., y+b∈range(A) with attainment in the nonnegative orthant. No such range condition is stated. Consequently, the displayed dual max_y -g*(y) is not generally equal to the Lagrangian dual (NNLS-Lagrangian-Dual); the essential constraint A^T y≥0 is missing. This error propagates to the Fenchel saddle-point representation and to the PDHG update in Example 3.5. The conjugate computation and the resulting dual must be corrected.
  2. [§4.4, Eq. (92)] The representation x* = L†(μ* - A^T λ*) + c·1 is asserted, with c "uniquely determined by primal feasibility and complementary slackness," but no derivation is given. The numerical validation fits c from the known CVXPY solution x* as c=(1/n)1^T(x* - z); z is not even defined in this section. The reported reconstruction error of 10^-6 therefore only shows that x* - L†(μ* - A^T λ*) is nearly constant, not that c can be recovered from the KKT conditions alone. The uniqueness/recoverability claim needs a proof and a validation that does not use the true primal solution.
  3. [Abstract and §3.5–§4] The paper's central thesis is that differentiable programming can "learn how to design" optimization algorithms and that end-to-end training improves convergence and solution quality. The only training routine, learn_nnls_dual_by_backprop (§3.5), initializes λ, μ, z and runs Adam on the fixed Lagrangian of one NNLS instance; no update rule, step size, preconditioner, or proximal operator is learned. The Section 4 case studies run fixed ADMM/PDHG/projected-gradient iterations with hand-chosen parameters, e.g., Eqs. (83)–(86) in §4.3. There is no training set, no held-out evaluation, and no comparison against an untrained baseline. The stated thesis is therefore not demonstrated. The paper should either be reframed as a tutorial on implementing known algorithms or include genuine learned-algorithm experiments.
  4. [§3.4.1, Eq. (42)] In Example 3.3 the Lagrangian is ℒ = f(x) - μ^T x with μ≥0, so ∇_μ ℒ = -x. The dual problem is a maximization, so the projected dual update should be μ^{k+1} = Π(μ^k + σ x^{k+1}). Equation (42) instead uses μ^{k+1} = Π(μ^k - σ x^{k+1}), which is a descent step for the dual and is incorrect as written. This sign error affects the PDG algorithm described in the tutorial.
  5. [§4.1 and §4.2] The Stigler Diet and Neural Network Verification case studies contain only problem formulations and references to external tools; no differentiable-programming implementation, convergence plot, or numerical result is reported for either. Table 2 nevertheless claims that these case studies illustrate the benefits of the framework. At minimum, the authors should either provide actual experiments or state explicitly that these are purely illustrative formulations with no empirical evaluation.
minor comments (5)
  1. [Throughout] Fenchel–Rockafellar is consistently misspelled as "Fenchel–Rockafeller." Also, "computation graphs, also as known Kantorovich graphs" is ungrammatical.
  2. [§3.5 code block] The function recover_x_from_mu_active_set is called with parameters tau and max_refine that are not defined in the displayed code or the surrounding text. The reader cannot reproduce the primal recovery step without additional detail.
  3. [§4.4] In the validation sentence, z is used in c=(1/n)1^T(x* - z) but z is never defined. It should be z = L†(μ* - A^T λ*).
  4. [References] There are duplicate references for PyTorch ([96] and [136]) and for CVX ([77] and [78]). The reference list should be consolidated.
  5. [§3.3.2, Eq. (33)] The line inf_{x≥0}{y^T A x} is not well defined for general y: if any component of A^T y is negative, the infimum is -∞. This ambiguity is part of the problem that makes Eq. (36) invalid and should be clarified in the revision.

Circularity Check

1 steps flagged

Dual-to-primal recovery in §4.4 is validated with a constant fitted to the target solution; the rest of the tutorial's mathematics is standard and not circular.

specific steps
  1. fitted input called prediction [Section 4.4, after Eq. (92)]
    "where μ⋆ and λ⋆ denote the optimal dual solutions, and the scalar c is uniquely determined by primal feasibility and the complementary slackness conditions. To numerically validate (92), we solve an LR-NNLS instance with(m, n) = (50, 30) using CVXPY and reconstruct x⋆ from the dual solutions via a least-squares estimate of c, namely c = 1/n 1⊤(x⋆ − z). The resulting reconstruction error is on the order of 10−6, confirming the theoretical consistency."

    Eq. (92) claims x* = L†(μ*−Aᵀλ*) + c1 with c determined by KKT conditions. The numerical validation instead computes c = (1/n)1ᵀ(x*−z), i.e., c is fitted from the CVXPY solution x* that the formula is supposed to recover from dual variables alone. The 10⁻⁶ error therefore only shows that x*−L†(μ*−Aᵀλ*) is nearly constant; it does not test whether primal feasibility and complementary slackness determine c. The 'reconstruction' is a fit to the target, so the reported validation is forced by construction.

full rationale

The core tutorial mathematics—computation-graph backpropagation, Lagrangian/Fenchel duals of NNLS, ADMM and PDHG updates, and the OPF/NNV formulations—is standard textbook material with external grounding and is not circular. The only concrete circular step is the §4.4 validation: the constant c in the closed-form primal recovery is estimated from the true CVXPY solution, turning the claimed KKT-based recovery into a fitted reconstruction. Self-citations ([155], [156], [173]) are present but not load-bearing: they are cited as examples/background, and the paper's derivations do not rest on them. Separately, the headline claim that the paper 'learns how to design' optimization algorithms is not demonstrated—the only 'learning' routine (learn_nnls_dual_by_backprop) runs Adam on the Lagrangian of one fixed NNLS instance and no update rule or step-size policy is learned—but this is a claim-evidence gap, not a circular derivation, so it does not change the circularity score beyond the one fitted validation.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. Its main free parameter is the null-space constant c in the LR-NNLS validation, which is fitted to the known solution. The key axioms are standard strong-duality, graph-connectivity, and modeling assumptions, plus an unflagged range assumption in the Fenchel conjugate derivation. The invented-entities list is empty because the paper reuses existing concepts (dual variables, Laplacian pseudoinverse, etc.).

free parameters (2)
  • c (LR-NNLS null-space constant) = c = (1/n) 1^T (x* - z), computed from the CVXPY solution
    In §4.4, Eq. (92) introduces x* = L†(μ* - A^T λ*) + c1. The text claims c is uniquely determined by KKT conditions, but the numerical validation estimates c from the known true solution x* using least squares. This makes the reported 10^-6 reconstruction error a fit, not a prediction.
  • ADMM/PDHG hyperparameters (ρ, ρ_c, τ, σ, θ, learning rate, iteration counts) = ρ=1.0, ρ_c=1.0, lr=1e-2, iters=500–3000, etc.
    These are chosen by hand in the code blocks without sensitivity analysis. The convergence plots and claimed performance depend on these unstated default choices, and no tuning procedure is described.
axioms (5)
  • domain assumption Slater's condition holds for NNLS and the cone programs considered, so strong duality holds and the primal optimum is recoverable from the dual.
    Invoked in §3.2.3 and §3.5.1 to justify solving the dual and recovering primal solutions. This is standard for feasible convex cone programs but is not verified for the specific instances.
  • domain assumption The graph Laplacian L is connected, so its null space is span{1} and the solution form x* = -L† g + c1 is valid.
    Used in §4.4, Eq. (89). Connectivity of the graph is not stated, and the derivation of the solution decomposition depends on it.
  • domain assumption The dual of the nonconvex neural network verification problem provides valid certificates with no duality gap.
    In §4.2, the Lagrangian dual (NNV-Dual) is used to certify properties of ReLU networks. NNV is nonconvex, so the dual may be weak; the paper does not discuss regularity conditions under which the lower bound is tight.
  • ad hoc to paper The OPF case study is well modeled by the resistive-network surrogate with nodal power p_i(v) = v_i(Gv)_i and a quadratic loss v^T G v.
    In §4.3, the paper replaces AC optimal power flow with a convex Laplacian-conductance model and explicitly 'temporarily disregards' voltage bounds (Constraint 77). This is a modeling choice that is not validated against standard OPF benchmarks.
  • ad hoc to paper In Example 3.2, inf_{x≥0} 1/2 ||Ax - (y+b)||² = 0 for every y, i.e., y+b ∈ range(A) is actually attained over the nonnegative orthant.
    Eq. (36) computes the Fenchel conjugate g*(y) by setting the infimum over x≥0 to zero, which assumes a range condition that is not stated and does not hold in general. This invalidates the displayed Fenchel dual max_y -g*(y).

pith-pipeline@v1.3.0-alltime-deepseek · 35623 in / 14221 out tokens · 141145 ms · 2026-08-03T08:32:07.720771+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Learning to Optimize by Differentiable Programming." pith.science (2026). https://pith.science/paper/VPOGYORW

@misc{pith2026260116510,
  author       = {Pith},
  title        = {Pith review of: Learning to Optimize by Differentiable Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VPOGYORW}},
  note         = {Machine review of arXiv:2601.16510}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as the alternating direction method of multipliers, and the primal-dual hybrid gradient can be learned and adapted through representative case studies.

Figures

Figures reproduced from arXiv: 2601.16510 by Chee Wei Tan, Liping Tao, Xindi Tong.

Figure 1
Figure 1. Figure 1: Learning to Optimize via Differentiable Programming by Combining Duality and First-order Methods. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Backpropagation in PyTorch via the Chain Rule: The forward pass computes [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Convergence comparison of PDG, ADMM, and PDHG on the NNLS problem [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ADMM with PyTorch. The primal problem is reformulated into a decomposed dual problem, whose [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Learning with PyTorch. The dual problem is reformulated as minimizing its Lagrangian, treated as a [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Convergence of the learning-based method on the NNLS problem [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Multi-GPU ADMM implementation in PyTorch. The dual problem is decomposed into multiple sub [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Controlled Periodic Synchronization for Efficient Data-Parallel Training

    cs.DC 2026-07 conditional novelty 4.0

    Periodic gradient+parameter synchronization with SlowMo beats DDP by 2.44 pp (K=4) on a WAN while cutting average wall-clock time by 13.8%, but only under a fixed LR=0.1 protocol.

  2. Learning-Based Spectrum Cartography in Low Earth Orbit Satellite Networks: An Overview

    cs.NI 2026-05 unverdicted novelty 3.0

    The paper overviews attention-based learning methods for spectrum cartography in LEO satellite networks to enable adaptive fusion of heterogeneous measurements for inference and resource allocation.

Reference graph

Works this paper leans on

188 extracted references · 20 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283

    2016

  2. [2]

    Martín Abadi and Gordon D Plotkin. 2019. A simple differentiable programming language.Proceedings of the ACM on Programming Languages4, POPL (2019), 1–28. 8Source code is available athttps://github.com/convexsoft/diffprog. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Learning to Optimize by Differentiable Programming 111:31

    2019

  3. [3]

    Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. 2019. Differen- tiable convex optimization layers.Advances in Neural Information Processing Systems32 (2019)

    2019

  4. [4]

    Akshay Agrawal and Stephen Boyd. 2020. Differentiating through log-log convex programs. arXiv preprint arXiv:2004.12553 (2020)

    Pith/arXiv arXiv 2020

  5. [5]

    Akshay Agrawal, Stephen Boyd, Deepak Narayanan, Fiodar Kazhamiaka, and Matei Zaharia. 2022. Allocation of fungible resources via a fast, scalable price discovery method.Mathematical Programming Computation14, 3 (2022), 593–622

    2022

  6. [6]

    Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. 2018. A rewriting system for convex optimization problems.Journal of Control and Decision5, 1 (2018), 42–60

    2018

  7. [7]

    Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. 2023. Optimus: Optimization modeling using mip solvers and large language models.arXiv preprint arXiv:2310.06116(2023)

    Pith/arXiv arXiv 2023

  8. [8]

    Aws Albarghouthi et al. 2021. Introduction to neural network verification.Foundations and Trends® in Programming Languages 7, 1–2 (2021), 1–157

    2021

  9. [9]

    Rita Ammanouil, André Ferrari, and Cédric Richard. 2015. A graph Laplacian regularization for hyperspectral data unmixing.In 2015IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing(ICASSP) .IEEE,1637–1641

    2015

  10. [10]

    Brandon Amos. 2019. Differentiable optimization-based modeling for machine learning. (2019)

    2019

  11. [11]

    Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. 2018. Differentiable mpc for end-to-end planning and control.Advances in Neural Information Processing Systems31 (2018)

    2018

  12. [12]

    Brandon Amos and J Zico Kolter. 2017. Optnet: Differentiable optimization as a layer in neural networks. InInter- national Conference on Machine Learning. PMLR, 136–145

    2017

  13. [13]

    Erling D Andersen and Knud D Andersen. 2000. The MOSEK interior point optimizer for linear programming: an implementation of the homogeneous algorithm. InHigh Performance Optimization. Springer, 197–232

    2000

  14. [14]

    MarcinAndrychowicz,MishaDenil,SergioGomez,MatthewWHoffman,DavidPfau,TomSchaul,BrendanShilling- ford, and Nando De Freitas. 2016. Learning to learn by gradient descent by gradient descent.Advances in Neural Information Processing Systems29 (2016)

    2016

  15. [16]

    David Applegate, Mateo Díaz, Oliver Hinder, Haihao Lu, Miles Lubin, Brendan O’Donoghue, and Warren Schudy

  16. [17]

    AnewinsightonaugmentedLagrangianmethodwithapplications in machine learning.Journal of Scientific Computing99, 2 (2024), 53

    JianchaoBai,LinyuanJia,andZhengPeng.2024. AnewinsightonaugmentedLagrangianmethodwithapplications in machine learning.Journal of Scientific Computing99, 2 (2024), 53

    2024

  17. [18]

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2019. Deep equilibrium models.Advances in Neural Information Processing Systems32 (2019)

    2019

  18. [19]

    Friedrich L Bauer. 1974. Computational graphs and rounding error.SIAM J. Numer. Anal.11, 1 (1974), 87–96

    1974

  19. [20]

    Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. Automatic differentiation in machine learning: A survey.Journal of Machine Learning Research18, 153 (2018), 1–43

    2018

  20. [21]

    2017.First-order methods in optimization

    Amir Beck. 2017.First-order methods in optimization. SIAM

    2017

  21. [22]

    Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. 2021. Machine learning for combinatorial optimization: a methodological tour d’horizon.European Journal of Operational Research290, 2 (2021), 405–421

    2021

  22. [23]

    Bentkamp, R

    A. Bentkamp, R. Fernández Mir, and J. Avigad. 2023. Verified reductions for optimization. InTools and Algorithms for the Construction and Analysis of Systems, Vol. 13994. Springer, 74–92

    2023

  23. [24]

    Quentin Bertrand, Quentin Klopfenstein, Mathurin Massias, Mathieu Blondel, Samuel Vaiter, Alexandre Gramfort, and Joseph Salmon. 2022. Implicit differentiation for fast hyperparameter selection in non-smooth convex learning. Journal of Machine Learning Research23, 149 (2022), 1–43

    2022

  24. [25]

    2014.Constrained optimization and Lagrange multiplier methods

    Dimitri P Bertsekas. 2014.Constrained optimization and Lagrange multiplier methods. Academic press

    2014

  25. [26]

    Mathieu Blondel and Vincent Roulet. 2024. The elements of differentiable programming. arXiv preprint arXiv:2403.14606 (2024)

    Pith/arXiv arXiv 2024

  26. [27]

    Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. 2020. Fast differentiable sorting and ranking. In International Conference on Machine Learning. PMLR, 950–959

    2020

  27. [28]

    Jonathan Borwein and Adrian Lewis. 2006. Convex Analysis and Nonlinear Optimization: Theoryand Examples. Springer

    2006

  28. [29]

    Nacime Bouziani, David A Ham, and Ado Farsi. 2024. Differentiable programming across the PDE and machine learning barrier.arXiv preprint arXiv:2409.06085(2024)

    Pith/arXiv arXiv 2024

  29. [30]

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers.Foundations and Trends® in Machine learning 3, 1 (2011), 1–122. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:32 Tao, Tong and Tan

    2011

  30. [31]

    2004.Convex optimization

    Stephen P Boyd and Lieven Vandenberghe. 2004.Convex optimization. Cambridge university press

    2004

  31. [32]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. 2018. JAX: composable transformations of Python+ NumPy programs. (2018)

    2018

  32. [33]

    Robin A Brown, Edward Schmerling, Navid Azizan, and Marco Pavone. 2022. A unified view of SDP-based neural network verification through completely positive programming. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 9334–9355

    2022

  33. [34]

    Deng Cai, Xiaofei He, Jiawei Han, and Thomas S Huang. 2010. Graph regularized nonnegative matrix factorization for data representation.IEEE Transactions on Pattern Analysis and Machine Intelligence33, 8 (2010), 1548–1560

    2010

  34. [35]

    Raafi Careem, G Johar, and Ali Khatibi. 2024. Deep neural networks optimization for resource-constrained envi- ronments: Techniques and models.Indonesian Journal of Electrical Engineering and Computer Science33, 3 (2024), 1843–1854

    2024

  35. [36]

    Antonin Chambolle and Thomas Pock. 2011. A first-order primal-dual algorithm for convex problems with applica- tions to imaging.Journal of Mathematical Imaging and Vision40, 1 (2011), 120–145

    2011

  36. [37]

    Tsung-Hui Chang, Mingyi Hong, Wei-Cheng Liao, and Xiangfeng Wang. 2016. Asynchronous distributed ADMM for large-scale optimization—Part I: Algorithm and convergence analysis.IEEE Transactions on Signal Processing64, 12 (2016), 3118–3130

    2016

  37. [38]

    Fei Chen, Gene Cheung, and Xue Zhang. 2024. Manifold graph signal restoration using gradient graph Laplacian regularizer. IEEE Transactions on Signal Processing72 (2024), 744–761

    2024

  38. [39]

    Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. 2022. Learning to optimize: A primer and a benchmark.Journal of Machine Learning Research23, 189 (2022), 1–59

    2022

  39. [40]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsba...

    2018

  40. [41]

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient primitives for deep learning.arXiv preprint arXiv:1410.0759(2014)

    Pith/arXiv arXiv 2014

  41. [42]

    Hong-Ming Chiu, Hao Chen, Huan Zhang, and Richard Y Zhang. 2025. SDP-CROWN: Efficient bound propagation for neural network verification with tightness of semidefinite programming.arXiv preprint arXiv:2506.06665(2025)

    Pith/arXiv arXiv 2025

  42. [43]

    Ontheconvergenceanalysisofthedecentralizedprojectedgradientdescent method

    WoocheolChoiandJimyeongKim.2025. Ontheconvergenceanalysisofthedecentralizedprojectedgradientdescent method. SIAM Journal on Optimization35, 3 (2025), 1673–1702

    2025

  43. [44]

    1990.Optimization and nonsmooth analysis

    Frank H Clarke. 1990.Optimization and nonsmooth analysis. SIAM

    1990

  44. [45]

    Gonzalo E Constante-Flores, Hao Chen, and Can Li. 2025. Enforcing hard linear constraints in deep learning models with decision rules.arXiv preprint arXiv:2505.13858(2025)

    arXiv 2025

  45. [46]

    Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016)

    Pith/arXiv arXiv 2016

  46. [47]

    Ron S Dembo, Stanley C Eisenstat, and Trond Steihaug. 1982. Inexact newton methods.SIAM J. Numer. Anal.19, 2 (1982), 400–408

    1982

  47. [48]

    Jelena Diakonikolas, Chenghui Li, Swati Padmanabhan, and Chaobing Song. 2022. A fast scale-invariant algorithm for non-negative least squares with non-negative data.Advances in Neural Information Processing Systems35 (2022), 6264–6277

    2022

  48. [49]

    Steven Diamond and Stephen Boyd. 2016. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research17, 83 (2016), 1–5

    2016

  49. [50]

    Jean C Digitale, Jeffrey N Martin, and Medellena Maria Glymour. 2022. Tutorial on directed acyclic graphs.Journal of Clinical Epidemiology142 (2022), 264–267

    2022

  50. [51]

    Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. 2017. Tensorflow distributions.arXiv preprint arXiv:1711.10604(2017)

    Pith/arXiv arXiv 2017

  51. [52]

    Alexander Domahidi, Eric Chu, and Stephen Boyd. 2013. ECOS: An SOCP solver for embedded systems. In2013 European control conference (ECC). IEEE, 3071–3076

    2013

  52. [53]

    Hermann W Dommel and William F Tinney. 2007. Optimal power flow solutions. IEEE Transactions on Power Apparatus and Systems10 (2007), 1866–1876

    2007

  53. [54]

    Simon S Du and Wei Hu. 2019. Linear convergence of the primal-dual gradient method for convex-concave saddle point problems without strong convexity. InThe 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 196–205

    2019

  54. [55]

    Mirjam Dür and Franz Rendl. 2021. Conic optimization: A survey with special focus on copositive optimization and binary quadratic problems. (2021). J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. Learning to Optimize by Differentiable Programming 111:33

    2021

  55. [56]

    Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Ue- sato, and Pushmeet Kohli. 2018. Training verified learners with learned verifiers.arXiv preprint arXiv:1805.10265 (2018)

    Pith/arXiv arXiv 2018

  56. [57]

    First-ordermethodsforconvexoptimization

    PavelDvurechensky,ShimritShtern,andMathiasStaudigl.2021. First-ordermethodsforconvexoptimization. EURO Journal on Computational Optimization9 (2021), 100015

    2021

  57. [58]

    Zeinab Ebrahimi, Gustavo Batista, and Mohammad Deghat. 2024. AA-DLADMM: An accelerated ADMM-based framework for training deep neural networks.arXiv preprint arXiv:2401.03619(2024)

    Pith/arXiv arXiv 2024

  58. [59]

    DualitybetweenLagrangianandLegendrianinvariants

    TobiasEkholmandYankıLekili.2023. DualitybetweenLagrangianandLegendrianinvariants. Geometry & Topology 27, 6 (2023), 2049–2179

    2023

  59. [60]

    Ernie Esser, Yifei Lou, and Jack Xin. 2013. A method for finding structured sparse solutions to nonnegative least squares problems with applications.SIAM Journal on Imaging Sciences6, 4 (2013), 2010–2046

    2013

  60. [61]

    Ernie Esser, Xiaoqun Zhang, and Tony F Chan. 2010. A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science.SIAM Journal on Imaging Sciences3, 4 (2010), 1015–1046

    2010

  61. [62]

    Werner Fenchel. 2013. On conjugate convex functions. InTraces and Emergence of Nonlinear Programming. Springer, 125–129

    2013

  62. [63]

    FerdinandoFioretto,PascalVanHentenryck,TerrenceWKMak,CuongTran,FedericoBaldo,andMicheleLombardi

  63. [64]

    Robert Fourer, Chandrakant Maheshwari, Arnold Neumaier, Dominique Orban, and Hermann Schichl. 2010. Con- vexity and concavity detection in computational graphs: Tree walks for convexity assessment.INFORMS Journal on Computing 22, 1 (2010), 26–43

    2010

  64. [65]

    Marguerite Frank, Philip Wolfe, et al. 1956. An algorithm for quadratic programming.Naval Research Logistics Quarterly 3, 1-2 (1956), 95–110

    1956

  65. [66]

    Roy Frostig, Matthew James Johnson, and Chris Leary. 2019. Compiling machine learning programs via high-level tracing. InSysML Conference 2018. Stanford, United States

    2019

  66. [67]

    Bin Gao, Tie-Yan Liu, Wei Wei, Taifeng Wang, and Hang Li. 2011. Semi-supervised ranking on very large graphs with rich metadata. InProceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 96–104

    2011

  67. [68]

    Yuan Gao and Christian Kroer. 2020. First-order methods for large-scale market equilibrium computation.Advances in Neural Information Processing Systems33 (2020), 21738–21750

    2020

  68. [69]

    Susan Garner Garille and Saul I Gass. 2001. Stigler’s diet problem revisited.Operations Research49, 1 (2001), 1–13

    2001

  69. [70]

    Maximilian Gelbrecht, Alistair White, Sebastian Bathiany, and Niklas Boers. 2023. Differentiable programming for Earth system modeling.Geoscientific Model Development16, 11 (2023), 3123–3135

    2023

  70. [71]

    Google. 2024. OR-Tools: Google optimization tools.https://github.com/google/or-tools.git

    2024

  71. [72]

    Google. 2026. Google OR-Tools.https://developers.google.com/optimization

    2026

  72. [73]

    Stephen Gould. 2025. Lecture notes on differentiable optimisation in deep learning

    2025

  73. [74]

    Stephen Gould, Richard Hartley, and Dylan Campbell. 2019. Deep declarative networks: A new hope.arXiv e-prints (2019), arXiv–1909

    2019

  74. [75]

    Deepdeclarativenetworks

    StephenGould,RichardHartley,andDylanCampbell.2021. Deepdeclarativenetworks. IEEE Transactions on Pattern Analysis and Machine Intelligence44, 8 (2021), 3988–4004

    2021

  75. [76]

    Graphimplementationsfornonsmoothconvexprograms

    MichaelGrantandStephenBoyd.2008. Graphimplementationsfornonsmoothconvexprograms. In RecentAdvances in Learning and Control. Springer-Verlag Limited, 95–110

    2008

  76. [77]

    Michael Grant and Stephen Boyd. 2020. CVX: Matlab software for disciplined vonvex programming, version 2.2. https://cvxr.com/cvx/

    2020

  77. [78]

    Michael Grant, Stephen Boyd, and Yinyu Ye. 2008. CVX: Matlab software for disciplined convex programming

    2008

  78. [79]

    Harshit Gupta, Kyong Hwan Jin, Ha Q Nguyen, Michael T McCann, and Michael Unser. 2018. CNN-based projected gradient descent for consistent CT image reconstruction.IEEE Transactions on Medical Imaging37, 6 (2018), 1440– 1453

    2018

  79. [80]

    Mohamad Azrin Syafiq Halim and Yeak Su Hoe. 2024. Introduction to automatic differentiation and neural differen- tiation equation.Proceedings of the Scientific Mathematics Journal24 (2024), 81–89

    2024

  80. [81]

    Nguyen TV Hang, Boris S Mordukhovich, and M Ebrahim Sarabi. 2020. Second-order variational analysis in second- order cone programming.Mathematical Programming180, 1 (2020), 75–116

    2020

Showing first 80 references.