Pith. sign in

REVIEW 5 major objections 6 minor 39 references

DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DHO2 splits curvature computation across GPUs to scale a hybrid-order optimizer, reporting near-linear per-device memory reduction and 1.4–2.1× faster training than distributed first- and second-order baselines.

desk verdict A useful distributed training engineering contribution with real experiments, but the theory sections have load-bearing gaps that need fixing before the memory and correctness claims are accepted. read the letter →

arxiv 2505.00982 v2 pith:KWQJEHX2 submitted 2025-05-02 cs.LG cs.DC

Shunxian Gu , Chaoqun You , Bangbang Ren , Lailong Luo , Junxu Xia , Deke Guo This is my paper · ORCID
classification cs.LGcs.DC
keywords distributedtraininghybridorderoptimizerLanczosalgorithmADMMmodelparallelismcurvatureinformationmemoryreductionscalability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes DHO2, a distributed implementation of the hybrid-order optimizer FOSI that combines gradient and curvature information in each update. Its goal is to make FOSI practical on multiple GPUs: a distributed Lanczos algorithm computes the needed Hessian eigenvectors across devices via model parallelism, so each GPU stores only a slice of the large basis matrix. An ADMM-like update rule splits the model update into first- and second-order subproblems to keep convergence fast. If the design works as claimed, training large networks can be accelerated with limited per-device memory, a key concern for small compute clusters.

What carries the argument

The central mechanism is the distributed Lanczos algorithm (Algorithm 2), which shares the Lanczos basis matrix D row-wise across GPUs and uses all-reduce to compute the global Gram-Schmidt projections and norms. The second mechanism is the ADMM-like update rule (Algorithm 3), which introduces a consensus variable wa and a Lagrange multiplier π, then alternates a closed-form w-update with an inner loop that optimizes wa using the hybrid first/second-order step. These two parts together let curvature be computed in parallel while the model update is accelerated by ADMM's convergence properties.

What would settle it

On a small model (e.g., logistic regression), compute the full Hessian explicitly, run Algorithm 2 with C=2 GPUs, and compare the resulting B to the single-device Lanczos B; any discrepancy in the off-diagonal β entries indicates the distributed aggregation is incorrect. Also instrument peak per-GPU memory while increasing C; if it does not fall roughly as n/C, the central memory claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that FOSI's memory bottleneck, the n×m Lanczos basis matrix, can be partitioned across C devices so that each GPU holds an n/C×m slice, and the resulting curvature information matches the single-device result. Coupled with an ADMM-derived update that alternates between a consensus variable and a locally optimized model, DHO2 reports per-GPU memory that decreases roughly linearly with C and total training time 1.4–2.1× shorter than distributed K-FAC and distributed Shampoo baselines, with a 4–5% accuracy gain.

Load-bearing premise

The distributed Lanczos algorithm assumes each GPU computes the exact global Hessian-vector product by properly combining all GPUs' local results; if that aggregation is missing, the tridiagonal matrix is not the Lanczos reduction of the true Hessian.

Editorial extensions

If this is right

  • Per-GPU memory for the curvature basis drops from O(mn) to O(mn/C + m^2 + n/C), allowing larger models on fixed hardware.
  • Training time to reach a target accuracy is reduced by 1.4–2.1× relative to distributed K-FAC and Shampoo, and 20–40% relative to S-SGD in the reported settings.
  • The ADMM-like update contributes convergence speed: removing it (DHO2-WA) slows training and lowers final accuracy.
  • The framework scales sublinearly in time and memory on up to 32–64 GPUs, making second-order-style optimization accessible to resource-constrained users.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The memory bound assumes full-length vectors are not materialized on every GPU, but the all-gather and Hessian-vector product calls in Algorithm 2 materialize full vectors; a stricter analysis would include that cost.
  • The ADMM-like split suggests a generic recipe: any hybrid optimizer that splits into two subspaces could be wrapped with an ADMM consensus variable; one could test DHO2's update rule with a pure first-order base to isolate ADMM's contribution.
  • With many GPUs, communication of the all-gather may dominate; a natural extension is hierarchical all-reduce or layer-wise partitioning rather than row-wise splitting.
  • The 1.4–2.1× speedup is measured against specific baselines and hyperparameters; the method's advantage might diminish on models where Hessian structure is less low-rank.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DHO2, a distributed implementation of the hybrid-order optimizer FOSI. The main ideas are (i) a model-parallel distributed Lanczos algorithm that computes the low-rank eigendecomposition of the Hessian while splitting the n x m basis matrix D across GPUs, and (ii) an ADMM-like model update rule that alternates between a Gauss-Seidel style parameter update and inner FOSI updates on an augmented Lagrangian. The authors claim an approximately linear reduction of per-GPU memory with the number of devices, sublinear time-to-solution, and 1.4x to 2.1x training-time speedups over distributed first- and second-order baselines, supported by experiments on CIFAR-10/100 and tiny-imagenet with up to 64 GPUs. The manuscript also contains a source-code link.

Significance. If the technical claims were fully supported, the paper would be a valuable contribution: it is, to the best of my knowledge, the first distributed implementation of FOSI, and the combination of a model-parallel Lanczos iteration with an ADMM-style outer loop is a plausible way to reduce the memory bottleneck of curvature-based optimizers. The experimental setup is nontrivial, with results on up to 64 GPUs and comparisons against K-FAC, Shampoo, and S-SGD, and the public code link is a strength. However, the current manuscript has several load-bearing gaps: the linear-algebra derivation in Eq. (7) is incorrect, the distributed Lanczos pseudocode omits the global Hessian-vector-product aggregation and materializes full-length vectors, the memory analysis ignores the broadcast of the full V-hat matrix, and the ADMM-like update rule is not derived correctly and has no convergence analysis. The practical significance is therefore conditional on a corrected algorithm and analysis.

major comments (5)
  1. [Section II-B, Eq. (7)] The equation H_t = D B D^T is asserted to follow from D^T H_t D = B by inserting D D^T = I on both sides. This is incorrect: for the Lanczos matrix D in R^{n x m} with m < n, only D^T D = I_m holds, while D D^T is the orthogonal projector onto the m-dimensional Krylov subspace, not the identity. Consequently H_t = D B D^T is not an equality but a rank-m Rayleigh-Ritz approximation. The derivation of the extreme-eigenvalue extraction should be restated as a projection/approximation, since the current text claims an exact factorization that is false and is load-bearing for the curvature computation.
  2. [Algorithm 2, Section III-A] The distributed Lanczos pseudocode does not specify how the global Hessian-vector product hvpt(v_i) is formed. In the data-parallel setting each GPU has a different mini-batch, and the full-batch HVP is the average of the per-GPU HVPs; without an all-reduce over h (or an equivalent global aggregation), B[i,i] is not the Rayleigh quotient of the true Hessian and the tridiagonal matrix B is not the Lanczos reduction of H_t. The notation is also inconsistent: line 7 computes B[i,i] from h and v_i as if h were a full n-vector, while line 9 uses h[sc:ec] as a partial vector. If h is full, the claimed per-GPU memory is violated; if h is partial, line 7 needs an additional reduction. The pseudocode must be rewritten to make the distributed HVP and the vector layout explicit.
  3. [Section III-C, memory analysis] The claimed per-GPU peak memory O(mn/C + m^2 + n/C) omits at least three objects that the algorithm as written materializes: the full n-dimensional v_i created by the all-gather in line 5 of Algorithm 2, the full n-dimensional h if hvpt is applied globally, and the complete n x (k+l) matrix V-hat that Section III-A says is obtained by broadcasting every GPU's partial V-hat to all other GPUs. Including these terms gives a per-GPU memory of at least O(n + mn/C + m^2 + n(k+l)), so the central claim of approximately linear memory reduction with the device count does not follow from the presented algorithm. The authors need to either revise the algorithm to avoid these full-vector materializations or revise the memory bound to reflect them.
  4. [Section III-B, Eqs. (12)-(14)] The ADMM-like update rule is not a correct ADMM derivation as written. The augmented Lagrangian term sigma/2 * ||w_a - w||^2 is placed in L1, but the resulting sigma(w_a - w) gradient term is absent from the Newton step in Eq. (13). In addition, the stopping condition in Eq. (14) reduces to ||(1+sigma)(g_l + pi)||^2 < epsilon, which is not the ADMM stationarity condition ||nabla f(w_a) + pi|| = 0. Finally, Algorithm 3 runs a fixed number P of inner updates and never tests the stated condition. The paper's claim that ADMM enhances convergence is therefore not supported by the derivation; either the update must be derived correctly or it must be explicitly presented as an ADMM-inspired heuristic with empirical justification.
  5. [Section IV-B, Table I] The speedup comparison in Table I is partly against baselines that are reported as INF for the target accuracy, so the headline speedup figures do not apply uniformly. D-Shampoo has a shorter per-epoch time than DHO2 on CIFAR-10/100 but fails to reach 90%/67%, while D-KFAC reaches the target on CIFAR-10 but not on CIFAR-100. The text should state precisely for which target accuracies the 1.4x-2.1x speedup claim holds, and should discuss the accuracy-time tradeoff rather than treating INF entries as a comparison point.
minor comments (6)
  1. [Abstract vs. Conclusions] The abstract states a 1.4x to 2.1x speedup, while the contributions and conclusions state 1.4x to 2.0x; these numbers should be reconciled.
  2. [Algorithm 2, line 9] The variable m used for the Gram-Schmidt all-reduce result shadows the Lanczos iteration count m; a different name, such as t or q, would avoid the ambiguity.
  3. [Section III-C] The text says the extra computations in Algorithm 3 lines 5, 13, and 14 are trivial and 'will be proved in Section IV-B,' but Section IV-B contains experiments, not a proof; this claim should be removed or replaced with a concrete complexity estimate.
  4. [Section III-B, Eq. (14)] The symbol g_l is used in the stopping condition without a definition; it should be defined as the gradient of the augmented Lagrangian at the l-th inner iteration.
  5. [References] Reference [15] (Automon) appears unrelated to the second-order optimizer discussion in which it is cited; please verify whether it is the intended citation.
  6. [Figure 6] It would be clearer to state whether the reported peak memory usage is measured from the implementation or computed from the analytical bound, and to include error bars or run-to-run variability if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical speedups against external baselines plus a proposed distributed algorithm, not fitted outputs or self-citation chains.

full rationale

The paper's main claims are (i) a distributed Lanczos algorithm that partitions the Lanczos basis D across GPUs and (ii) an ADMM-like update rule, both evaluated by training-time and accuracy measurements against S-SGD, D-KFAC, and D-Shampoo. The distributed Lanczos procedure is presented as an algorithm with explicit all-gather/all-reduce steps; its correctness would follow from the standard Lanczos recurrence if the Hessian-vector product is global, not from assuming the conclusion. The ADMM-like update in Eq. 13 is a proposed heuristic that extends FOSI's Δ1 and Δ2 (Eqs. 3-4) by replacing g with g+π and A1 with A1+σI; it is not derived from the speedup numbers, and the speedup numbers are externally measured. There are no load-bearing self-citations, no uniqueness theorem imported from the authors' prior work, and no fitted parameter renamed as a prediction. The memory-complexity claim in Section III-C is questionable as written because Algorithm 2 all-gathers the full vector vi and leaves the scope of hvpt unspecified, but that is a soundness or implementation-level gap, not circular reasoning: the claimed O(mn/C + m^2 + n/C) bound is not the same object as the measured 50% memory reduction, which comes from the experiments in Fig. 6. Accordingly, no circular step can be quoted and no circularity score above 0 is warranted.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central design depends on two unstated assumptions about the distributed Lanczos implementation and the ADMM-like update; the only explicitly fitted hyperparameters are sigma and P, with k and l unreported. No new physical or algorithmic entities are introduced beyond the DHO2 framework itself.

free parameters (3)
  • sigma (ADMM penalty) = 5e-4 (ResNet-101), 5e-6 (VGG-16), 5e-7 (ResNet-152)
    Chosen empirically per model as the "best combination of hyperparameters" (Section IV-A). It weights the augmented Lagrangian penalty and affects the ADMM-like update trajectory and measured speedup.
  • P (number of inner FOSI loops) = 4
    Fixed at 4 for all experiments without sensitivity analysis (Section IV-A). It controls how accurately the inner w_a subproblem is solved before each outer ADMM update.
  • k and l (extreme eigenvalue counts for FOSI) = not reported
    The number of extreme Hessian eigenvalues used in the low-rank update is not given for any experiment, yet it directly sets the Lanczos iteration count m = max{4(k+l), 2 ln n} and the memory footprint of D.
assumptions (3)
  • domain assumption The augmented Lagrangian splitting into two orthogonal subspaces preserves the convergence-enhancing property of ADMM for nonconvex DNN training.
    Section III-B introduces the ADMM-like update and asserts it "enhances model convergence" citing [29], [30], but no convergence proof is given for this specific hybrid setting, and the inner stopping condition in Eq. (14) is not enforced in Algorithm 3.
  • ad hoc to paper The distributed Lanczos algorithm with local hvpt and all-reduce Gram-Schmidt yields the same B and D as the original Lanczos algorithm.
    Algorithm 2 lines 5-6 call hvpt(vi) without specifying how per-GPU Hessian-vector products are combined; without global aggregation the computed B is not the true Lanczos tridiagonal matrix.
  • ad hoc to paper Each GPU can afford the full-length vectors vi and h; these are not counted in the per-GPU memory bound.
    Section III-C memory analysis claims peak memory O(mn/C + m^2 + n/C), but the all-gather in line 5 and hvpt in line 6 materialize n-dimensional vectors on every GPU, adding an O(n) term per GPU.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM." pith.science (2026). https://pith.science/paper/KWQJEHX2

@misc{pith2026250500982,
  author       = {Pith},
  title        = {Pith review of: DHO$_2$: Accelerating Distributed Hybrid Order Optimization via Model Parallelism and ADMM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWQJEHX2}},
  note         = {Machine review of arXiv:2505.00982}
}
abstract

Scaling deep neural network (DNN) training to more devices can reduce time-to-solution. However, it is impractical for users with limited computing resources. FOSI, as a hybrid order optimizer, converges faster than conventional optimizers by taking advantage of both gradient information and curvature information when updating the DNN model. Therefore, it provides a new chance for accelerating DNN training in the resource-constrained setting. In this paper, we explore its distributed design, namely DHO$_2$, including distributed calculation of curvature information and model update with partial curvature information to accelerate DNN training with a low memory burden. To further reduce the training time, we design a novel strategy to parallelize the calculation of curvature information and the model update on different devices. Experimentally, our distributed design can achieve an approximate linear reduction of memory burden on each device with the increase of the device number. Meanwhile, it achieves $1.4\times\sim2.1\times$ speedup in the total training time compared with other distributed designs based on conventional first- and second-order optimizers.

Figures

Figures reproduced from arXiv: 2505.00982 by the authors.

Figure 1
Figure 1. Background and motivation. (a) presents the difference between our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of DHO2. get the required ˆa and Vˆ in the equation 2 by extracting the k largest and l smallest entries from u and selecting their corresponding eigenvectors from Z. H −1 1 is thus obtained. III. FRAMEWORK DESIGN In this section, we introduce the proposed DHO2 framework, which encompasses a distributed curvature information calcu￾lation algorithm and an ADMM-like model update rule for the hybrid order o… view at source ↗
Figure 5
Figure 5. Extended study on the SVHN dataset with 32 3090 GPUs. We fine-tune [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The test accuracy versus training epoch curves on the tiny-imagenet [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 26 canonical work pages

  1. [1]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  2. [2]

    Resfusion: Denoising diffusion probabilistic models for image restoration based on prior residual noise,

    Z. Shi, h. zheng, C. Xu, C. Dong, B. Pan, X. xueshuo, A. He, T. Li, and H. Fu, “Resfusion: Denoising diffusion probabilistic models for image restoration based on prior residual noise,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, ...

    work page 2024

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Imagenet training in minutes,

    Y . You, Z. Zhang, C.-J. Hsieh, J. Demmel, and K. Keutzer, “Imagenet training in minutes,” in Proceedings of the 47th international conference on parallel processing , 2018, pp. 1–10

    2018

  5. [5]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving,

    B. Liao, S. Chen, H. Yin, B. Jiang, C. Wang, S. Yan, X. Zhang, X. Li, Y . Zhang, Q. Zhanget al., “Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 12 037–12 047

    work page 2025

  6. [6]

    Towards federated customized neural architecture search for remote sensing scene classification,

    J. Li, S. Wang, R. Yang, M. Gong, Z. Hu, N. Zhang, K. Sheng, and Y . Zhou, “Towards federated customized neural architecture search for remote sensing scene classification,” IEEE Transactions on Geoscience and Remote Sensing , 2025

    work page 2025

  7. [7]

    Adam: A method for stochastic optimization,

    D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014

    arXiv 2014

  8. [8]

    Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,

    G. Hinton, N. Srivastava, and K. Swersky, “Neural networks for machine learning lecture 6a overview of mini-batch gradient descent,” Cited on , vol. 14, no. 8, p. 2, 2012

    work page 2012

Show all 39 references
  1. [9]

    Optimization methods for large- scale machine learning,

    L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large- scale machine learning,” SIAM review, vol. 60, no. 2, pp. 223–311, 2018

    2018

  2. [10]

    Large- batch training for lstm and beyond,

    Y . You, J. Hseu, C. Ying, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large- batch training for lstm and beyond,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–16

    2019

  3. [11]

    Deep neural network training with distributed k-fac,

    J. G. Pauloski, L. Huang, W. Xu, K. Chard, I. T. Foster, and Z. Zhang, “Deep neural network training with distributed k-fac,” IEEE Transactions on Parallel and Distributed Systems , vol. 33, no. 12, pp. 3616–3627, 2022

    2022

  4. [12]

    A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale,

    H.-J. M. Shi, T.-H. Lee, S. Iwasaki, J. Gallego-Posada, Z. Li, K. Ran- gadurai, D. Mudigere, and M. Rabbat, “A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale,” arXiv preprint arXiv:2309.06497 , 2023

    2023 arXiv

  5. [13]

    Optimizing neural networks with kronecker- factored approximate curvature,

    J. Martens and R. Grosse, “Optimizing neural networks with kronecker- factored approximate curvature,” in International conference on machine learning. PMLR, 2015, pp. 2408–2417

    2015

  6. [14]

    Shampoo: Preconditioned stochastic tensor optimization,

    V . Gupta, T. Koren, and Y . Singer, “Shampoo: Preconditioned stochastic tensor optimization,” in International Conference on Machine Learning . PMLR, 2018, pp. 1842–1850

    2018

  7. [15]

    Automon: Automatic distributed monitoring for arbitrary multivariate functions,

    H. Sivan, M. Gabel, and A. Schuster, “Automon: Automatic distributed monitoring for arbitrary multivariate functions,” in Proceedings of the 2022 International Conference on Management of Data , 2022, pp. 310– 324

    2022

  8. [16]

    Fosi: Hybrid first and second order optimization,

    S. Hadar, G. Moshe, and S. Assaf, “Fosi: Hybrid first and second order optimization,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  9. [17]

    On the limited memory bfgs method for large scale optimization,

    D. C. Liu and J. Nocedal, “On the limited memory bfgs method for large scale optimization,” Mathematical programming, vol. 45, no. 1, pp. 503–528, 1989

    1989

  10. [18]

    Towards practical second-order optimizers in deep learning: Insights from fisher information analysis,

    D. M. Gomes, “Towards practical second-order optimizers in deep learning: Insights from fisher information analysis,” arXiv preprint arXiv:2504.20096, 2025

    2025 arXiv

  11. [19]

    Nesterov momentum based optimization algorithm for deep learning,

    Y .-q. Zhao, Z.-Y . Qiu, L. Wang, C. Wang, and Z.-H. Guo, “Nesterov momentum based optimization algorithm for deep learning,” in 2024 10th International Conference on Computer and Communications (ICCC) . IEEE, 2024, pp. 127–131

    2024

  12. [20]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  13. [21]

    An iteration method for the solution of the eigenvalue problem of linear differential and integral operators,

    C. Lanczos, “An iteration method for the solution of the eigenvalue problem of linear differential and integral operators,” 1950

    1950

  14. [22]

    Federated learning via inexact admm,

    S. Zhou and G. Y . Li, “Federated learning via inexact admm,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9699–9708, 2023

    2023

  15. [23]

    An inexact admm for separable nonconvex and nonsmooth optimization,

    J. Bai, M. Zhang, and H. Zhang, “An inexact admm for separable nonconvex and nonsmooth optimization,” Computational Optimization and Applications, pp. 1–35, 2025

    2025

  16. [24]

    Newton raphson method,

    S. Akram and Q. U. Ann, “Newton raphson method,” International Journal of Scientific & Engineering Research , vol. 6, no. 7, pp. 1748– 1752, 2015

    2015

  17. [25]

    SOAP: Improving and stabilizing shampoo using adam for language modeling,

    N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Jan- son, and S. M. Kakade, “SOAP: Improving and stabilizing shampoo using adam for language modeling,” in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://o...

    2025

  18. [26]

    4-bit shampoo for memory- efficient network training,

    S. Wang, P. Zhou, J. Li, and H. Huang, “4-bit shampoo for memory- efficient network training,” Advances in Neural Information Processing Systems, vol. 37, pp. 126 997–127 029, 2024

    2024

  19. [27]

    Preconditioned stochastic gradient descent,

    X.-L. Li, “Preconditioned stochastic gradient descent,” IEEE transactions on neural networks and learning systems , vol. 29, no. 5, pp. 1454–1466, 2017

    2017

  20. [28]

    Fast exact multiplication by the hessian,

    B. A. Pearlmutter, “Fast exact multiplication by the hessian,” Neural computation, vol. 6, no. 1, pp. 147–160, 1994

    1994

  21. [29]

    Aa-dladmm: An acceler- ated admm-based framework for training deep neural networks,

    Z. Ebrahimi, G. Batista, and M. Deghat, “Aa-dladmm: An acceler- ated admm-based framework for training deep neural networks,” arXiv preprint arXiv:2401.03619, 2024

    2024 arXiv

  22. [30]

    Preconditioned inexact stochastic admm for deep model,

    S. Zhou, O. Wang, Z. Luo, Y . Zhu, and G. Y . Li, “Preconditioned inexact stochastic admm for deep model,” arXiv preprint arXiv:2502.10784 , 2025

    2025 arXiv

  23. [31]

    A dual algorithm for the solution of nonlinear variational problems via finite element approximation,

    D. Gabay and B. Mercier, “A dual algorithm for the solution of nonlinear variational problems via finite element approximation,” Computers & mathematics with applications , vol. 2, no. 1, pp. 17–40, 1976

    1976

  24. [32]

    Distributed optimization and statistical learning via the alternating direction method of multipliers,

    S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein et al. , “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning , vol. 3, no. 1, pp. 1–122, 2011

    2011

  25. [33]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    2014 arXiv

  26. [34]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  27. [35]

    Tiny imagenet,

    mnmoustafa and M. Ali, “Tiny imagenet,” https://kaggle.com/ competitions/tiny-imagenet, 2017, kaggle

    2017

  28. [36]

    Dct-snn: Using dct to distribute spatial information over time for low-latency spiking neural networks,

    I. Garg, S. S. Chowdhury, and K. Roy, “Dct-snn: Using dct to distribute spatial information over time for low-latency spiking neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4671–4680

    2021

  29. [37]

    Fedqclip: Accelerating federated learning via quantized clipped sgd,

    Z. Qu, N. Jia, B. Ye, S. Hu, and S. Guo, “Fedqclip: Accelerating federated learning via quantized clipped sgd,” IEEE Transactions on Computers , 2024

    2024

  30. [38]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  31. [39]

    One weird trick for parallelizing convolutional neural networks,

    A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” arXiv preprint arXiv:1404.5997 , 2014

    2014 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.