Pith. sign in

REVIEW 2 major objections 5 minor 299 references

This paper claims that exact convergence to the true optimum is achievable despite persistent state-dependent update bias, and that a bilevel 'residual learning' reformulation does it with only a polynomial price in the hardware condition n

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:47 UTC pith:QCQ7GCRN

load-bearing objection The Residual Learning idea is genuinely interesting, but the central equivalence theorem is false under the stated assumptions and the convergence proof analyzes the wrong P*; this needs major revision before it can be trusted. the 2 major comments →

arxiv 2607.26032 v1 pith:QCQ7GCRN submitted 2026-07-28 math.OC

Optimization under Persistent State-Dependent Bias: Gradient-based Method and Complexity Analysis

Zhaoxian Wu , Quan Xiao , Tayfun Gokmen , Tianyi Chen This is my paper
classification math.OC MSC 90C2690C1590C0690C6049M3768Q25
keywords stochastic optimizationstate-dependent biasanalog in-memory computingbilevel optimizationimplicit penalizationresidual learninghardware condition numberconvergence complexity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper studies stochastic gradient descent on hardware where every desired update is scaled component-wise by state-dependent response functions, so positive and negative updates are systematically asymmetric. It first shows that this persistent bias is not harmless: naive analog SGD implicitly minimizes a penalized objective whose solution sits between the true optimum and the hardware's symmetric point, and the gap grows with gradient noise. The main constructive claim is that exact convergence to the original optimum is still possible, by reformulating the task as a bilevel problem: an inner variable learns the residual to the optimum, and an outer variable minimizes that residual. The proposed Residual Learning algorithm provably reaches the true minimizer despite the bias, at a last-iterate function gap of order O~(kappa1*kappa2^4*sigma^2/(mu*K)). A lower-bound construction shows a polynomial dependence on the hardware condition number kappa2 is unavoidable.

Core claim

The paper establishes that Analog SGD does not optimize the original problem: its stationary points are near the minimizer of f(W)+<Sigma,Rc(W)>, an implicit penalty that drags iterates toward the hardware symmetry point W_diamond. To fix this, the authors construct a bilevel problem BP whose upper-level objective is the squared residual ||P*(W)-W_diamond||^2, where P*(W) is the minimizer of a shifted lower-level objective. They prove that BP, and even its implicitly penalized version IBP, has the same unique global minimizer W* as the original problem. Consequently Residual Learning—updating the inner variable with distorted gradients and the outer variable with a deterministic residual ste

What carries the argument

The central object is the asymmetry ratio R(W)=G(W)/F(W), where F and G are the average and half-difference of the positive/negative response functions, and its componentwise antiderivative Rc(W). These define the implicit penalty <Sigma,Rc(W)> that explains why naive SGD fails. The constructive machinery is the bilevel reformulation BP: minimize gamma^2/2 ||P*(W)-W_diamond||^2 subject to P*(W)=argmin_P f(W+gamma(P-W_diamond)). Shifting by the symmetric point W_diamond ensures that, when W=W*, the inner minimizer coincides with W_diamond, so the penalty term becomes harmless. The convergence proof combines sufficient descent in both levels through a Lyapunov function and uses the hardware co

Load-bearing premise

The proof that the implicitly penalized bilevel problem has the same unique minimizer as the original assumes the penalty term <Sigma,Rc(W)> is nonnegative for all W, a condition not implied by the stated regularity assumptions on the response functions and false for some natural non-monotone responses.

What would settle it

Run Residual Learning in one dimension with f(W)=mu/2*(W-W*)^2, response functions q+(w)=4+2tanh(w), q-(w)=4-2tanh(w), and symmetric point W_diamond=0. For these responses F=4, R(w)=-0.5 tanh(w), and Rc(w)=-0.5 log(cosh(w))<0 for w != 0. If the stochastic noise level Sigma satisfies Sigma > 2*mu*gamma^2, the penalized lower-level objective has a local maximum at P=W_diamond, so P*(W*) != W_diamond. If Residual Learning's iterates do not converge to W*, Theorem 3's claimed equivalence is disproved.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Naive analog SGD cannot reach the true optimum; its limit is biased toward the symmetry point by an amount proportional to the stochastic-gradient noise at the optimum.
  • Residual Learning restores exact convergence: the last-iterate function gap decays as O~(kappa1*kappa2^4*sigma^2/(mu*K)), preserving the classical log K/K SGD rate up to logarithmic factors.
  • A polynomial dependence on the hardware condition number is fundamental: there exists a hard instance forcing at least Omega~(kappa2^2*sigma^2/(mu*K)).
  • In simulations on image-classification tasks, Residual Learning matches digital SGD accuracy while Analog SGD lags, confirming the theoretical claims on realistic response models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the implicit penalty can be negative—for example with q+(w)=4+2tanh(w), q-(w)=4-2tanh(w), where Rc(w)=-0.5 log(cosh(w))<0—the equivalence between IBP and P is not guaranteed; testing Residual Learning on such non-monotone responses would delimit the theorem's true domain.
  • The gap between the kappa2^4 upper bound and the kappa2^2 lower bound suggests a more refined algorithm, perhaps exploiting the harmonic-mean effective response, could improve the worst-case exponent.
  • The sign-dependent scaling model is not specific to memory hardware: any actuator or quantizer with asymmetric positive/negative response fits the same dynamics, so the residual-learning construction could transfer to control or communication settings with sign-asymmetric distortions.
  • The kappa1 factor in the upper bound might be removable by iterate averaging, but the averaging variable would itself suffer the same state-dependent bias; designing a bias-immune averaging scheme is a natural next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper analyzes stochastic gradient methods on hardware where every update is scaled component-wise by state-dependent response functions q_+(·), q_-(·). It first shows that the resulting 'Analog SGD' recursion implicitly optimizes a penalized objective rather than the original problem P. To recover exact convergence, the authors introduce a bilevel reformulation BP and a two-timescale algorithm Residual Learning; they claim that the algorithm's implicit penalized bilevel problem IBP is equivalent to P, so Residual Learning converges exactly to the minimizer of P despite the persistent bias. The paper also gives an O~(κ1κ2^4σ^2/(μK)) upper bound and an Ω~(κ2^2σ^2/(μK)) lower bound, with κ2 the hardware condition number q_max/q_min.

Significance. If the results were correct, the paper would make a valuable contribution to optimization on analog in-memory hardware: it would provide the first exact-convergence guarantee for a gradient method under asymmetric, state-dependent response functions, and a complexity analysis in terms of a hardware condition number. The algorithm is simple, the convergence-rate comparison with digital SGD is informative, and the hard-instance lower bound is a useful step. The paper also includes reproducible simulation code and experiments on real datasets, which are strengths. However, the central equivalence theorem that underpins the paper's interpretation of exact convergence is not valid under the stated assumptions, and the proof of a key technical lemma contains a sign error; the main claims need substantial revision.

major comments (2)
  1. [§3, Theorem 3] The proof of Theorem 3 starts 'Since f(·) ≥ f* and ⟨Σ,Rc(·)⟩ ≥ 0', but the second inequality is not a consequence of Definition 1. Definition 2 only guarantees Rc≥0 when R is strictly monotone, and no such assumption appears in Theorem 3. A concrete allowed instance is q±(w)=4±2tanh(w), for which F=4, R(w)=−0.5tanh(w), and Rc(w)=−0.5log cosh(w)<0 for w≠0. For scalar f(w)=μ/2(w−w*)² and Σ>2μγ², the lower-level IBP objective at W=W* is μγ²P²/2+ΣRc(P), whose second derivative at P=W⋄=0 is μγ²−Σ/2<0; hence P*(W*)≠W⋄ and the proof's asserted equality is false. One can even construct a Lipschitz R with a double-well Rc so that W* is not the global minimizer of IBP. Thus Theorem 3 is false as stated, and the equivalence between IBP and P used in the abstract and §3 is unsupported.
  2. [Appendix A, Lemma 8(iii), Eq. (64)] The displayed chain in the proof of Lemma 8(iii) bounds E⟨∆Z,(|∆Z+δZ|−|∆Z|)⊙G(Z)⟩ from below, but the term appearing in (63) is ⟨∆Z,(|∆Z|−|∆Z+δZ|)⊙G(Z)⟩; the required estimate is an upper bound on E⟨∆Z,(|∆Z+δZ|−|∆Z|)⊙G(Z)⟩. The inequality therefore has the wrong direction. The lemma statement itself appears true — it can be proved directly via E⟨a,A(a+δ)⟩ = F||a||² − ⟨a,G⟩E|a+δ| and Young's inequality — but the written proof is invalid. Since Lemma 1 and hence Theorem 4 rely on Lemma 8(iii), the proof needs a corrected argument.
minor comments (5)
  1. [§3 vs §4] The exact-convergence claim should be disentangled from Theorem 3. Theorem 4's Lyapunov proof bounds f(¯W_K)−f* + C||P*(W_K)−W⋄||² directly and does not invoke IBP equivalence. The text should state that exact convergence is established by Theorem 4 itself, and present Theorem 3 only as a separate (corrected) interpretation. As written, the failure of Theorem 3 makes the main convergence claim appear unsupported even if the direct proof can be repaired.
  2. [Eq. (17) and proof of Theorem 4] The theorem statement writes a bound on +C||P*(W_K)−W⋄||², while the proof uses the Lyapunov function with Cγ²/2||P*(W_K)−W⋄||². Please align the notation so the reader can verify the constant dependence.
  3. [Definition 2 / Theorem 3] If the authors intend to keep Theorem 3, they must add an explicit assumption such as monotonicity of R(·) (or nonnegativity of Rc(·)) and state it in the theorem. The current text says only 'if R is moreover strictly monotone' in Definition 2, which is not part of the theorem's hypotheses.
  4. [Theorem 5 and surrounding text] The phrase 'a polynomial dependence on κ2 is unavoidable in general' overstates the result: Theorem 5 gives a lower bound for Residual Learning on a hard instance, not an information-theoretic lower bound for all algorithms. Please qualify the claim accordingly.
  5. [Appendix A, Eq. (64)] There are typos in the displayed chain, including 'E[∥|δZ∥2]' with a missing bracket and an apparent misplaced factor. These should be corrected in a revision.

Circularity Check

0 steps flagged

No significant circularity: the derivation chain is self-contained; the questionable Rc≥0 step in Theorem 3 is a correctness gap, not a reduction of the result to its inputs.

full rationale

The paper's derivation is mathematically self-contained rather than circular. Theorem 1 constructs the penalty Rc as the antiderivative of R=G/F, so the gradient identity ∇<Σ,Rc>=Σ⊙R holds by definition; the theorem's quantitative statements are then proved directly in Appendix B, with no fitted parameters. The bilevel reformulations BP and IBP are algebraic restatements of the same optimization problem: P*(W)=W⋄+(W*−W)/γ makes the upper-level objective equal ||W−W*||²/(2γ²), so Theorem 2 is a definitional equivalence, and the nontrivial content is the algorithmic convergence in Lemmas 1–3 and Theorem 4, which contract a Lyapunov function under the biased updates (14)–(15). Theorem 5 is an explicit adversarial construction, not a lower bound reverse-engineered from the upper bound. Self-citations (Wu et al. 2024/2025, Kim et al. 2019) are used only for hardware motivation and for the practical assumption of a known symmetric point; none supplies a load-bearing mathematical premise. The most notable weakness—the proof of Theorem 3 asserts ⟨Σ,Rc(·)⟩≥0, which is not implied by Definition 1—is an unstated-assumption/correctness issue, not a circularity: the equivalence claim would be restored by explicitly assuming monotonicity of R or nonnegativity of Rc, and the rest of the analysis does not reduce to that assumption. No fitted input is later reported as a prediction, and no uniqueness theorem is imported from the authors' prior work to force the conclusion.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claims rely on standard smoothness/strong convexity and bounded-variance noise, plus hardware-specific domain assumptions: positive bounded Lipschitz response functions and a known symmetric point. The Theorem 3 proof additionally needs an unstated nonnegativity condition on the anti-derivative of the asymmetry ratio. The step sizes alpha, beta, gamma are proof-chosen and experiment-tuned, not fitted to a target result.

free parameters (3)
  • alpha (lower-level stepsize) = Theta(qmax log K / (gamma mu qmin^2 K))
    Algorithmic step size chosen in Lemma 3 to make the Lyapunov contraction work; not fitted to data, but the rate depends on this choice.
  • beta (upper-level stepsize) = Theta(alpha gamma mu qmin / qmax)
    Algorithmic step size chosen in Lemma 3; not fitted to data, but required for the descent coupling in Lemma 2.
  • gamma (mixing coefficient) = Omega(LS sigma sqrt(qmax) / (mu qmin^{3/2}))
    Hyper-parameter in the shifted problem and in Residual Learning updates. The proof requires a lower bound; in experiments it is swept by hand (Fig. 7), not fitted to reproduce the theorems.
axioms (6)
  • domain assumption Response functions are positive, uniformly bounded, differentiable, and LS-Lipschitz (Definition 1).
    This models hardware operation away from saturation. It is stated in Definition 1 and used throughout.
  • domain assumption The symmetric point W_diamond is known (Assumption 1).
    Residual Learning requires shifting the problem around W_diamond. If W_diamond is unknown, the method cannot be implemented as described.
  • standard math Objective is L-smooth and mu-strongly convex (Assumption 2).
    Standard strong convexity and smoothness used for the descent lemmas and the equivalence theorems.
  • standard math Stochastic gradients are unbiased with bounded variance (Assumption 3).
    Standard SGD noise model; also used to define Sigma = E|grad f(W*; xi)| in the implicit penalty.
  • domain assumption For Theorem 1, q+'(W) < -c0 and q-'(W) > c0 on S.
    This derivative condition implies R is strictly increasing, making the Jacobian positive definite. It is stated in Theorem 1 but not in Definition 1.
  • ad hoc to paper For Theorem 3, <Sigma, Rc(W)> >= 0 for all W.
    The proof of Theorem 3 uses this inequality, but it is not stated as an assumption and does not follow from Definition 1. It fails for q+(w)=4+2 tanh(w), q-(w)=4-2 tanh(w).

pith-pipeline@v1.3.0-alltime-deepseek · 33905 in / 20658 out tokens · 186380 ms · 2026-08-01T00:47:53.616153+00:00 · methodology

0 comments
read the original abstract

This paper studies the convergence of stochastic gradient descent (SGD) when the implemented updates are subject to a persistent and state-dependent bias, in which the desired update is scaled by response functions component-wise. Our first contribution is to demonstrate that SGD in this setting implicitly optimizes a penalized problem whose minimizer does not coincide with the true minimizer. To mitigate this convergence failure, we reformulate the original task as an equivalent bilevel optimization problem and propose a gradient-based algorithm, termed Residual Learning. Theoretical analysis shows that Residual Learning finds a solution to the original, unbiased optimization problem despite the hardware imperfections. Beyond exact convergence, we quantify how the response functions affect convergence complexity via the hardware condition number and show that a polynomial dependence on it is unavoidable in general, via a construction of a hard instance. The theoretical results are supported by numerical simulations that demonstrate the effectiveness of the proposed algorithm.

Figures

Figures reproduced from arXiv: 2607.26032 by Quan Xiao, Tayfun Gokmen, Tianyi Chen, Zhaoxian Wu.

Figure 1
Figure 1. Figure 1: Illustration of persistent state￾dependent bias in R 2 . Any desired update ∆Z, which is typically a de￾scent direction of the objective func￾tion, is scaled by state-dependent re￾sponse functions q+(·) and q−(·) component-wise. The bias is exacer￾bated when the hardware condition number increases. This paper investigates a distinct computational model involving persistent bias that arises in in-situ optim… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the relationship between [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples of response functions from Assumption [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the shifted problem in 2-dimensional space ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Numerical illustrations of the implicit penalty and the effect of the hardware condition number. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Test accuracy curves for the models with the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Test accuracy of ResNet models after 100 epochs trained by [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The weight distributions of ResNet models that are trained by [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

299 extracted references · 39 linked inside Pith

  1. [1]

    Transactions of the American Mathematical Society , volume =

    Federer, Herbert , title =. Transactions of the American Mathematical Society , volume =. 1959 , publisher =

    1959

  2. [2]

    arXiv preprint arXiv:1209.4340 , year=

    Moments and absolute moments of the normal distribution , author=. arXiv preprint arXiv:1209.4340 , year=

    Pith/arXiv arXiv

  3. [3]

    NIST Digital Library of Mathematical Functions

  4. [4]

    1948 , publisher=

    Handbook of mathematical functions with formulas, graphs, and mathematical tables , author=. 1948 , publisher=

    1948

  5. [5]

    2012 , publisher=

    Matrix Analysis , author=. 2012 , publisher=

    2012

  6. [6]

    2013 , publisher=

    Introductory Lectures on Convex Optimization: A Basic Course , author=. 2013 , publisher=

    2013

  7. [7]

    2018 , publisher=

    Reinforcement Learning: An Introduction , author=. 2018 , publisher=

    2018

  8. [8]

    2013 , publisher=

    Br. 2013 , publisher=

    2013

  9. [9]

    2004 , publisher=

    Kernel Methods for Pattern Analysis , author=. 2004 , publisher=

    2004

  10. [10]

    2006 , publisher=

    Nonlinear programming: theory and algorithms , author=. 2006 , publisher=

    2006

  11. [11]

    Artificial intelligence and statistics , pages=

    Communication-efficient learning of deep networks from decentralized data , author=. Artificial intelligence and statistics , pages=. 2017 , organization=

    2017

  12. [12]

    International Conference on Machine Learning , pages=

    Error feedback fixes signsgd and other gradient compression schemes , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  13. [13]

    International Conference on Learning Representations , year=

    Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training , author=. International Conference on Learning Representations , year=

  14. [14]

    IEEE Transactions on Signal Processing , volume=

    On maintaining linear convergence of distributed learning and optimization under limited communication , author=. IEEE Transactions on Signal Processing , volume=. 2020 , publisher=

    2020

  15. [15]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech

    Seide, Frank and Fu, Hao and Droppo, Jasha and Li, Gang and Yu, Dong , booktitle=. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech. 2014 , organization=

    2014

  16. [16]

    Advances in neural information processing systems , volume=

    Atomo: Communication-efficient learning via atomic sparsification , author=. Advances in neural information processing systems , volume=

  17. [17]

    Advances in neural information processing systems , volume=

    Terngrad: Ternary gradients to reduce communication in distributed deep learning , author=. Advances in neural information processing systems , volume=

  18. [18]

    arXiv preprint arXiv:1902.06101 , year=

    Local differential privacy in decentralized optimization , author=. arXiv preprint arXiv:1902.06101 , year=

    Pith/arXiv arXiv 1902

  19. [19]

    IEEE INFOCOM 2019-IEEE Conference on Computer Communications , pages=

    Compressed distributed gradient descent: Communication-efficient consensus over networks , author=. IEEE INFOCOM 2019-IEEE Conference on Computer Communications , pages=. 2019 , organization=

    2019

  20. [20]

    Qsparse-local-SGD: Distributed

    Basu, Debraj and Data, Deepesh and Karakus, Can and Diggavi, Suhas , journal=. Qsparse-local-SGD: Distributed

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    Robust and communication-efficient collaborative learning , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    IEEE Transactions on Smart Grid , volume=

    Real-time energy management in microgrids with reduced battery capacity requirements , author=. IEEE Transactions on Smart Grid , volume=. 2017 , publisher=

    2017

  23. [23]

    IEEE Journal on Selected Areas in Communications , volume=

    Dynamic energy management for smart-grid-powered coordinated multipoint systems , author=. IEEE Journal on Selected Areas in Communications , volume=. 2016 , publisher=

    2016

  24. [24]

    Proceedings of the IEEE international conference on computer vision , pages=

    Multi-view convolutional neural networks for 3d shape recognition , author=. Proceedings of the IEEE international conference on computer vision , pages=

  25. [25]

    IEEE Transactions on Wireless Communications , year=

    Multi-device task-oriented communication via maximal coding rate reduction , author=. IEEE Transactions on Wireless Communications , year=

  26. [26]

    IEEE Transactions on Circuits and Systems for Video Technology , year=

    Robust Multi-Agent Collaborative Perception via Spatio-Temporal Awareness , author=. IEEE Transactions on Circuits and Systems for Video Technology , year=

  27. [27]

    IEEE transactions on pattern analysis and machine intelligence , volume=

    Segmentation of multivariate mixed data via lossy data coding and compression , author=. IEEE transactions on pattern analysis and machine intelligence , volume=. 2007 , publisher=

    2007

  28. [28]

    Sparsified

    Stich, Sebastian U and Cordonnier, Jean-Baptiste and Jaggi, Martin , journal=. Sparsified

  29. [29]

    NeurIPS , year=

    Gradient sparsification for communication-efficient distributed optimization , author=. NeurIPS , year=

  30. [30]

    Information and Inference: A Journal of the IMA , volume=

    Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor , author=. Information and Inference: A Journal of the IMA , volume=. 2022 , publisher=

    2022

  31. [31]

    Alistarh, Dan and Grubic, Demjan and Li, Jerry and Tomioka, Ryota and Vojnovic, Milan , journal=

  32. [32]

    Adaptive Step-Size Methods for Compressed

    Subramaniam, Adarsh M and Magesh, Akshayaa and Veeravalli, Venugopal V , journal=. Adaptive Step-Size Methods for Compressed. 2024 , publisher=

    2024

  33. [33]

    Error compensated distributed

    Qian, Xun and Richt. Error compensated distributed. NeurIPS , year=

  34. [34]

    IEEE Transactions on Parallel and Distributed Systems , volume=

    Accelerating federated learning via momentum gradient descent , author=. IEEE Transactions on Parallel and Distributed Systems , volume=. 2020 , publisher=

    2020

  35. [35]

    2009 , publisher=

    Learning multiple layers of features from tiny images , author=. 2009 , publisher=

    2009

  36. [36]

    Computer Vision and Pattern Recognition , pages=

    Imagenet: A large-scale hierarchical image database , author=. Computer Vision and Pattern Recognition , pages=. 2009 , organization=

    2009

  37. [37]

    International Conference on Learning Representations , year=

    Mixed Precision Training , author=. International Conference on Learning Representations , year=

  38. [38]

    1951 , publisher=

    A Stochastic Approximation Method , author=. 1951 , publisher=

    1951

  39. [39]

    Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms , author=

  40. [40]

    Non-asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning , author=

  41. [41]

    2022 , publisher=

    Finite-Time Convergence Rates of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learning , author=. 2022 , publisher=

    2022

  42. [42]

    arXiv preprint arXiv:2111.12665 , year=

    Finite-time Error Bounds For Distributed Linear Stochastic Approximation , author=. arXiv preprint arXiv:2111.12665 , year=

    Pith/arXiv arXiv

  43. [43]

    Linear Stochastic Approximation: How Far does Constant Step-size and Iterate Averaging Go? , author=

  44. [44]

    Finite Time Analysis of Linear Two-timescale Stochastic Approximation with

    Kaledin, Maxim and Moulines, Eric and Naumov, Alexey and Tadic, Vladislav and Wai, Hoi-To , booktitle=COLT, year=. Finite Time Analysis of Linear Two-timescale Stochastic Approximation with

  45. [45]

    arXiv preprint arXiv:2112.12770 , year=

    Optimal and Instance-dependent Guarantees for Markovian Linear Stochastic Approximation , author=. arXiv preprint arXiv:2112.12770 , year=

    Pith/arXiv arXiv

  46. [46]

    The Annals of Statistics , pages=

    Nonparametric Stochastic Approximation with Large Step-sizes , author=. The Annals of Statistics , pages=. 2016 , publisher=

    2016

  47. [47]

    2005 , volume =

    Learning the Kernel with Hyperkernels , author=. 2005 , volume =

    2005

  48. [48]

    Advances in Neural Information Processing Systems , volume=

    A single-timescale analysis for stochastic approximation with multiple coupled sequences , author=. Advances in Neural Information Processing Systems , volume=

  49. [49]

    IEEE Transactions on Signal Processing , year=

    Single-Timescale Multi-Sequence Stochastic Approximation Without Fixed Point Smoothness: Theories and Applications , author=. IEEE Transactions on Signal Processing , year=

  50. [50]

    Luo, Zhi-Quan and Pang, Jong-Shi and Ralph, Daniel , year=

  51. [51]

    Mathematical programming , volume=

    Gradient methods for minimizing composite functions , author=. Mathematical programming , volume=

  52. [52]

    Computational Optimization and Applications , volume=

    Inexact proximal stochastic gradient method for convex composite optimization , author=. Computational Optimization and Applications , volume=

  53. [53]

    2014 , publisher=

    Implicit Functions and Solution Mappings: A View from Variational Analysis , author=. 2014 , publisher=

    2014

  54. [54]

    2022 , address=

    An Implicit Gradient-Type Method for Linearly Constrained Bilevel Problems , author=. 2022 , address=

    2022

  55. [55]

    Foundations and Trends

    Learning with Limited Samples: Meta-Learning and Applications to Communication Systems , author=. Foundations and Trends

  56. [56]

    Foundations and Trends

    Bilevel methods for image reconstruction , author=. Foundations and Trends

  57. [57]

    Mathematics of Operations Research , volume=

    Lipschitz continuity of solutions of variational inequalities with a parametric polyhedral constraint , author=. Mathematics of Operations Research , volume=. 1995 , publisher=

    1995

  58. [58]

    Applied Mathematics & Optimization , volume=

    On Lipschitz continuity of projections onto polyhedral moving sets , author=. Applied Mathematics & Optimization , volume=. 2021 , publisher=

    2021

  59. [59]

    Mathematical Programming , volume=

    Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning , author=. Mathematical Programming , volume=. 2021 , publisher=

    2021

  60. [60]

    Mathematical Programming , volume=

    Tame functions are semismooth , author=. Mathematical Programming , volume=. 2009 , publisher=

    2009

  61. [61]

    Optimization Letters , volume=

    Convergence of a stochastic subgradient method with averaging for nonsmooth nonconvex constrained optimization , author=. Optimization Letters , volume=. 2020 , publisher=

    2020

  62. [62]

    Set-Valued and Variational Analysis , pages=

    Conservative and semismooth derivatives are equivalent for semialgebraic maps , author=. Set-Valued and Variational Analysis , pages=. 2021 , publisher=

    2021

  63. [63]

    Bilevel optimization , author=

  64. [64]

    ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally! , author=

  65. [65]

    A single-timescale method for stochastic bilevel optimization , author=

  66. [66]

    New necessary optimality conditions for bilevel programs by combining the

    Ye, Jane J and Zhu, Daoli , journal=. New necessary optimality conditions for bilevel programs by combining the. 2010 , publisher=

    2010

  67. [67]

    Gradient-based hyperparameter optimization through reversible learning , author=

  68. [68]

    Forward and reverse gradient-based hyperparameter optimization , author=

  69. [69]

    Model-agnostic meta-learning for fast adaptation of deep networks , author=

  70. [70]

    Liu, Hanxiao and Simonyan, Karen and Yang, Yiming , booktitle=ICLR, year=

  71. [71]

    Tarzanagh, Davoud Ataee and Li, Mingchen and Thrampoulidis, Christos and Oymak, Samet , booktitle=ICML, year=

  72. [72]

    arXiv preprint arXiv:1607.05447 , year=

    On differentiating parameterized argmin and argmax problems with application to bi-level optimization , author=. arXiv preprint arXiv:1607.05447 , year=

    Pith/arXiv arXiv

  73. [73]

    The Mathematical Gazette , volume=

    The generalised inverse , author=. The Mathematical Gazette , volume=. 1978 , publisher=

    1978

  74. [74]

    SIAM Journal on Optimization , volume=

    Stochastic model-based minimization of weakly convex functions , author=. SIAM Journal on Optimization , volume=. 2019 , publisher=

    2019

  75. [75]

    Mathematical Programming , volume=

    Efficiency of minimizing compositions of convex functions and smooth maps , author=. Mathematical Programming , volume=. 2019 , publisher=

    2019

  76. [76]

    Bilevel Programming for Hyperparameter Optimization and Meta-Learning , author =

  77. [77]

    Barata, Jo. The. Brazilian Journal of Physics , volume=. 2012 , publisher=

    2012

  78. [78]

    Foundations and trends

    Proximal algorithms , author=. Foundations and trends. 2014 , publisher=

    2014

  79. [79]

    Communication-efficient learning of deep networks from decentralized data , author=

  80. [80]

    The error-feedback framework: Better rates for

    Stich, Sebastian U and Karimireddy, Sai Praneeth , journal=. The error-feedback framework: Better rates for

Showing first 80 references.