Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training

T0 review · 3 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that augmenting the left basis in a backward-corrected projector-splitting integrator yields a rank-adaptive low-rank training method with guaranteed loss descent, local convergence to locally optimal weights, and only…

desk verdict A useful rank-adaptive PSI variant with a correct descent bound, but the central convergence theorem rests on a false projection identity and is unproven as written. read the letter →

arxiv 2502.03006 v1 pith:EINXX7W5 submitted 2025-02-05 math.NA cs.LGcs.NA

Jonas Kusch , Steffen Schotthöfer , Alexandra Walter This is my paper · ORCID
classification math.NAcs.LGcs.NA MSC 68T0749Q1265L0565L2065L70
keywords dynamicallow-rankapproximationprojector-splittingintegratorneuralnetworktrainingrankadaptivitylocalconvergenceparameter-efficientfine-tuningstochasticgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces abc-PSI, an augmented backward-corrected projector-splitting integrator for training neural networks with low-rank weights. The authors claim that adding the previous left basis to the updated K-factor before the L-step makes the loss decrease provably and lets the iterates converge to weights satisfying the local optimality condition for the low-rank manifold. The same augmentation enables dynamic rank adjustment through a singular-value truncation, and the algorithm needs only one QR decomposition per step instead of two. If the claims hold, low-rank training becomes simultaneously more robust, cheaper, and rank-adaptive, with the same per-step cost structure as gradient descent on the factorized matrices.

What carries the argument

The load-bearing object is the left-basis augmentation step. At each step $\widehat U_{k+1}=\mathrm{ortho}([U_k, K_{k+1}])$, where $K_{k+1}$ is the result of the K-step. Because the augmented basis contains both the old left basis and the updated K factor, the projection $P_{\widehat U_{k+1}}=\widehat U_{k+1}\widehat U_{k+1}^\top$ satisfies $P_{\widehat U_{k+1}}U_k=U_k$ and $P_{\widehat U_{k+1}}K_{k+1}=K_{k+1}$, so the problematic residual $(I-U_1U_1^\top)Y_0$ from the bc-PSI bound vanishes. The same augmented basis is reused in the L-step initial condition $L(t_0)=V_0\bar S_1^\top$, and a truncated SVD then selects the next rank $r_{k+1}$ by asking that the omitted squared singular values stay below $\tau\|\Sigma\|$. The backward-corrected projection replaces the original PSI's backward-in-time S-step, so every substep moves forward in time.

What would settle it

For a small random low-rank matrix $Y=USV^\top$, define the augmented basis $\widehat U=\mathrm{ortho}([U,K_1])$ with $K_1=US - h\nabla\ell(USV^\top)V$, and compute $R=\widehat U\widehat U^\top(I-P(Y))\nabla\ell(Y)$. If $\|R\|>0$ for some gradient whose residual has a component in $\mathrm{span}(\nabla\ell(Y)V)$ orthogonal to $U$, the identity in the proof of Theorem 5.9 is false and the stated convergence guarantee needs an additional assumption.

Watch

Extended reading notes

Core claim

The paper's central claim is that the augmented backward-corrected PSI (Algorithm 4.1) is a projector-splitting integrator that combines rank adaptivity with guaranteed loss descent and local convergence. Its construction replaces the S-step of the PSI by the projection $\bar S_1 = \widehat U_1^\top U_0 S_0$ from the backward-corrected PSI, and then augments the left basis to $\widehat U_1 = \mathrm{ortho}([U_0, K(t_1)])$, the orthonormal span of the old basis and the updated K-factor. This removes the term $\|(I-U_1U_1^\top)Y_0\|$ that blocked descent guarantees for the non-augmented version; Theorem 5.8 then gives $\ell(\widehat Y_1)\le \ell(Y_0) - (1-h c_l/2) h \|P_{\widehat U_1}\nabla\ell(Y_0)\|^2$, and Theorem 5.9 concludes $\liminf_{T\to\infty}\mathbb{E}[\|P(Y_T)\nabla\ell(Y_T)\|^2]=0$ under Robbins-Monro learning rates. A truncation step based on an SVD with tolerance $\tau$ makes the rank dynamic, and each iteration costs one QR decomposition plus this SVD, versus two QR decompositions for BUG-type integrators.

Load-bearing premise

The convergence proof in Theorem 5.9 assumes the augmented basis makes $\widehat U_t\widehat U_t^\top(I-P(Y_{t-1}))\nabla\ell(Y_{t-1})=0$, meaning the part of the gradient left over after projecting onto the current tangent space is entirely annihilated by the augmented projection; if that identity fails, the proof's replacement of $\widehat U_t\widehat U_t^\top\nabla\ell$ by $P(Y_{t-1})\nabla\ell$ does not go through.

Editorial extensions

If this is right

  • Rank-adaptive low-rank training can be run with a monotone loss decrease for learning rates $h\le 2/c_l$, matching the descent behavior of BUG-based dynamical low-rank training.
  • Each training step costs one QR decomposition plus one SVD on matrices of rank at most $2r$, making abc-PSI cheaper per step than BUG-type integrators that need two QR decompositions.
  • The method applies to fine-tuning pretrained networks, with additive low-rank corrections evolved by the same integrator; the reported experiments reach the accuracy of established parameter-efficient adaptation baselines with fewer trainable parameters.
  • Local convergence to a point satisfying the manifold optimality condition $P(Y_\star)\nabla\ell(Y_\star)=0$ is asserted for stochastic gradients under Robbins-Monro step sizes and bounded cumulative truncation error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the asserted annihilation identity in Theorem 5.9 cannot be repaired, the practical algorithm may still work, but the local-convergence guarantee as stated would reduce to a conjecture; a numerical check on random instances would settle this quickly.
  • The same augmentation-then-truncate recipe could be carried over to other splitting integrators or to Tucker-format tensor training, where the analogous projection onto the span of old basis plus updated core would play the same role.
  • Because the algorithm touches only the K and L factorizations at any given time, with U, S, and V co-materialized only in the truncation step, an implementation that exploits this could reduce peak memory below what simultaneous-update low-rank methods achieve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes abc-PSI, an augmented backward-corrected projector-splitting integrator for dynamical low-rank training (DLRT). The method adds a rank augmentation step to the backward-corrected PSI of [2], claiming rank-adaptivity while preserving descent and local convergence guarantees, and reduces the number of QR decompositions per training step from two to one. The theoretical part contains a descent lemma (Theorem 5.8), a robust error bound (Theorem 5.6), and a convergence-to-local-optimum theorem (Theorem 5.9). Experiments on MNIST and on fine-tuning a vision transformer on CIFAR-10/100 are presented to support the method's practicality.

Significance. If the proven guarantees hold as stated, the contribution is valuable: abc-PSI would be the first PSI-type DLRT method with a local convergence guarantee and rank adaptivity, at reduced per-iteration cost compared to BUG-type integrators. The descent argument (Theorem 5.8) is elementary and correct, and the numerical experiments are consistent with the method being practically useful. The construction is not circular and does not rely on fitted constants. However, the proof of the central convergence theorem (Theorem 5.9) uses a false identity, so the claimed local convergence guarantee is not established in this manuscript.

major comments (3)
  1. [Section 5.7, proof of Theorem 5.9] The equation following (5.15) asserts that bU_t bU_t^T (I - P(Y_{t-1})) grad l(Y_{t-1}) = 0. This identity is false in general. The augmented basis bU_t contains U_{t-1} and grad l(Y_{t-1})V_{t-1}, but the residual (I - P(Y_{t-1}))grad l(Y_{t-1}) = (I - U_{t-1}U_{t-1}^T) grad l(Y_{t-1}) (I - V_{t-1}V_{t-1}^T) is right-orthogonal to V_{t-1} and in general has left components outside span(grad l V), so the projection onto span(bU_t) need not vanish. A concrete counterexample is Y0 = s e1 e1^T and grad l(Y0) = e2 e1^T + e2 e2^T. Then P(Y0)grad l = e2 e1^T, the residual is e2 e2^T, and for h>0 the augmented basis bU = orth([s e1, s e1 - h e2]) spans R^2, so bU bU^T residual = e2 e2^T ≠ 0. Consequently, the identity bU_t bU_t^T grad l = P(Y_{t-1})grad l used to pass from (5.15) to the limit statement fails, and Theorem 5.9 is not proven as written. A norm estimate controlling the extra term could repair the argument, but it is not present in the manuscript.
  2. [Section 5.5, proof of Theorem 5.6] The local error after truncation is stated as h*epsilon + 3*cl*h^2 + vartheta/h, but the truncation is defined by ||bY1 - Y1|| <= vartheta in the paragraph before Theorem 5.6. Therefore the local error bound should contain vartheta, not vartheta/h. The global bound can still acquire a vartheta/h contribution after summing over T/h steps, but the local error expression is incorrect as written and should be corrected.
  3. [Section 5.5, remark before Theorem 5.6] The remark states that the robust error bound, loss descent, and convergence results are proven for the discrete Algorithm 4.1, yet the proofs of Theorems 5.6, 5.8, and 5.9 operate with the continuous evolution equations (4.1), including integrals over the step. The manuscript should either state clearly that the theory applies to the continuous-in-time integrator (4.1) and add a separate discretization analysis for Algorithm 4.1, or modify the proofs to cover the explicit Euler update (4.3) directly.
minor comments (3)
  1. [Section 5.4, Lemma 5.4] The proof contains a typo: 'Cauchy-Schwartz' should be 'Cauchy-Schwarz'.
  2. [Theorem 5.9] The assumption sum_t E||Yt - bYt|| <= D is asserted with the gloss 'i.e., for sufficiently large t, the rank stabilizes', but no argument links the truncation rule of Section 4 to this boundedness. Since this assumption is not verified by the algorithm, the theorem's conclusion is conditional; the authors should either prove that the truncation rule enforces this condition or state explicitly that it is an additional regularity assumption.
  3. [Tables 1 and 2] In Table 1, several bc-PSI entries are marked with '-' without explanation; the caption should state that these correspond to failed training runs, and ideally the number of failures across seeds should be reported for all entries.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the descent and convergence analyses follow from the method's defining update, not from fitted inputs or self-citation chains; a gap in Theorem 5.9 is a correctness issue, not circularity.

full rationale

The paper's derivation chain is not circular. The descent bound (Theorem 5.8) follows from the explicit Euler update bY1 = Y0 - h P_bU1 grad l(Y0), which is algebraically derived from (4.1)/(4.3), and from the standard descent lemma (Lemma 5.7) whose proof is included in Appendix A. The convergence argument (Theorem 5.9) uses the same descent bound and standard Robbins-Monro summation; it does not fit any parameter to the claimed limit. The rank-adaptivity/truncation threshold tau is an algorithmic hyperparameter and is not used to manufacture the convergence conclusion; the theorem instead assumes a bounded cumulative truncation error. The method is validated on external benchmarks (MNIST and vision-transformer fine-tuning), not on values fitted in the theory. Self-citations to [21] and [40] are supporting rather than load-bearing: Lemma 5.2 of [21] is restated and proved in the appendix, and [40] is cited only for the multi-layer extension and as a comparison method, so the central single-layer result stands on the paper's own equations. One genuine concern should not be mistaken for circularity: in Section 5.7, the proof of Theorem 5.9 asserts that bU_t bU_t^T (I - P(Y_{t-1})) grad l(Y_{t-1}) = 0. This identity is not entailed by the augmentation step; the residual can have left components outside span(grad l V). This is a missing-support issue in the convergence proof, not a reduction of the theorem to its inputs. Because a few supporting references are to the authors' own concurrent or unpublished work, I assign score 2 rather than 0, but the central derivation is self-contained enough that the paper should not be scored as partially circular.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The method does not introduce physical entities or fitted constants; its free parameters are algorithmic hyperparameters. The main theoretical burden is carried by Assumptions A1-A4 and an unproven rank-stabilization assumption.

free parameters (3)
  • truncation tolerance tau = 0.005 to 0.2 in MNIST; 0.15 (CIFAR-10), 0.1 (CIFAR-100)
    Controls rank adaptivity and truncation error vartheta; convergence theorem requires the total truncation error to be bounded by D, which is not proven.
  • learning rate h = 0.001-0.01 on MNIST; 8e-4 and 1e-3 on ViT
    Learning rate sequence must satisfy Robbins-Monro conditions in Theorem 5.9; in experiments chosen by hand.
  • initial rank r0 = 32 for ViT; adaptive for MNIST
    Initial rank is an input to Algorithm 4.1; experiments select it heuristically.
assumptions (6)
  • domain assumption Stochastic gradient is Lipschitz continuous with constant c_l and bounded by B (Assumptions A2, A3).
    Used in every descent and error bound; not verified for neural network losses in practice.
  • domain assumption Residual R(Y) = grad l(Y) - P(Y) grad l(Y) is bounded by epsilon on the low-rank manifold (A4).
    Used in robust error bound Theorem 5.6; not verified empirically.
  • ad hoc to paper Rank stabilizes: sum_t E[||Y_t - hat Y_t||] <= D < infinity.
    Explicitly assumed in Theorem 5.9; the paper does not prove that the truncation criterion leads to rank stabilization.
  • domain assumption Robust error bound of the original PSI from Kieri, Lubich, Walach [27, Theorem 2.1] applies to the stochastic gradient setting.
    The bc-PSI robust bound in Theorem 5.3 relies on this cited theorem without adapting it to stochastic gradients.
  • domain assumption Multi-layer extension follows from Proposition 1 of [40].
    The theory is stated for single-layer networks and is claimed to transfer to multi-layer networks using this result.
  • ad hoc to paper The augmented basis hat U satisfies hat U hat U^T (I-P(Y)) grad l(Y) = 0.
    This identity is asserted in the proof of Theorem 5.9 and is false in general; it is a load-bearing step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training." pith.science (2026). https://pith.science/paper/EINXX7W5

@misc{pith2026250203006,
  author       = {Pith},
  title        = {Pith review of: An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EINXX7W5}},
  note         = {Machine review of arXiv:2502.03006}
}
read the original abstract

Layer factorization has emerged as a widely used technique for training memory-efficient neural networks. However, layer factorization methods face several challenges, particularly a lack of robustness during the training process. To overcome this limitation, dynamical low-rank training methods have been developed, utilizing robust time integration techniques for low-rank matrix differential equations. Although these approaches facilitate efficient training, they still depend on computationally intensive QR and singular value decompositions of matrices with small rank. In this work, we introduce a novel low-rank training method that reduces the number of required QR decompositions. Our approach integrates an augmentation step into a projector-splitting scheme, ensuring convergence to a locally optimal solution. We provide a rigorous theoretical analysis of the proposed method and demonstrate its effectiveness across multiple benchmarks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Galerkin Alternating Projection Method for Kinetic Equations in the Diffusive Limit

    math.NA 2025-05 conditional novelty 6.0 of 10

    The GAP scheme is a new dynamical low-rank integrator for the radiative transfer equation that provably preserves the diffusive limit and avoids CFL restrictions.

Reference graph

Works this paper leans on

49 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [21]

    Hnatiuk, J

    A. Hnatiuk, J. Kusch, L. Kusch, N. R. Gauger, and A. W alther , Stochastic aspects of dynamical low-rank approximation in the context of machine learning , TBA, (2024)

    work page 2024

  2. [40]

    GeoLoRA: Geometric integration for parameter efficient fine-tuning

    S. Schotth¨ofer, E. Zangrando, G. Ceruti, F. Tudisco, and J. Kusch, Geolora: Geometric integration for parameter efficient fine-tuning , 2024, https://arxiv.org/abs/2410.18720

    work page Pith review arXiv 2024

  3. [2]

    Bachmayr, H

    M. Bachmayr, H. Eisenmann, E. Kieri, and A. Uschmajew, Existence of dynamical low-rank approximations to parabolic problems, Mathematics of Computation, 90 (2021), pp. 1799– 1830

    2021

  4. [1]

    Alzubaidi, J

    L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santa- mar´ıa, M. A. F adhel, M. Al-Amidie, and L. F arhan, Review of deep learning: concepts, cnn architectures, challenges, applications, future directions, Journal of big Data, 8 (2021), pp. 1–74

    work page 2021

  5. [3]

    B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg , Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers , Information and Inference: A Journal of the IMA, 11 (2022), pp. 307–353

    work page 2022

  6. [4]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakan- tan, P. Shyam, G. Sastry, A. Askell, et al. , Language models are few-shot learners , Advances in neural information processing systems, 33 (2020), pp. 1877–1901

    work page 2020

  7. [5]

    Ceruti, L

    G. Ceruti, L. Einkemmer, J. Kusch, and C. Lubich , A robust second-order low-rank bug integrator based on the midpoint rule , BIT Numerical Mathematics, 64 (2024), p. 30

    work page 2024

  8. [6]

    Ceruti, J

    G. Ceruti, J. Kusch, and C. Lubich , A rank-adaptive robust integrator for dynamical low- rank approximation, BIT Numerical Mathematics, 62 (2022), pp. 1149–1174

    2022

Show all 49 references
  1. [7]

    Ceruti, J

    G. Ceruti, J. Kusch, and C. Lubich , A parallel rank-adaptive integrator for dynamical low- rank approximation, SIAM Journal on Scientific Computing, 46 (2024), pp. B205–B228

    2024

  2. [8]

    Ceruti and C

    G. Ceruti and C. Lubich , An unconventional robust integrator for dynamical low-rank ap- proximation, BIT Numerical Mathematics, 62 (2022), pp. 23–44

    2022

  3. [9]

    Courbariaux, I

    M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio , Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1, arXiv:1602.02830, (2016)

    2016 arXiv

  4. [10]

    T. A. D’Antonoli, L. K. Berger, A. K. Indrakanti, N. Vishwanathan, J. Weiß, M. Jung, Z. Berkarda, A. Rau, M. Reisert, T. K ¨ustner, et al. , Totalsegmentator mri: Sequence-independent segmentation of 59 anatomical structures in mr images , arXiv preprint arXiv:2405.19492, (2024)

    2024 arXiv

  5. [11]

    Denil, B

    M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas , Predicting parameters in deep learning, Advances in neural information processing systems, 26 (2013)

    2013

  6. [12]

    Einkemmer , Accelerating the simulation of kinetic shear alfv´ en waves with a dynamical low-rank approximation, Journal of Computational Physics, 501 (2024), p

    L. Einkemmer , Accelerating the simulation of kinetic shear alfv´ en waves with a dynamical low-rank approximation, Journal of Computational Physics, 501 (2024), p. 112757

    2024

  7. [13]

    Einkemmer and C

    L. Einkemmer and C. Lubich, A low-rank projector-splitting integrator for the vlasov–poisson AUGMENTED BACKW ARD-CORRECTED PSI FOR DLRT 23 equation, SIAM Journal on Scientific Computing, 40 (2018), pp. B1330–B1360

    2018

  8. [14]

    Frankle and M

    J. Frankle and M. Carbin , The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprint arXiv:1803.03635, (2018)

    2018 arXiv

  9. [15]

    Y. Guo, A. Yao, and Y. Chen , Dynamic network surgery for efficient dnns , Advances in neural information processing systems, 29 (2016)

    2016

  10. [16]

    Haegeman, C

    J. Haegeman, C. Lubich, I. Oseledets, B. V andereycken, and F. Verstraete , Unifying time evolution and optimization with matrix product states , Physical Review B, 94 (2016), p. 165116

    2016

  11. [17]

    Hassibi and D

    B. Hassibi and D. Stork , Second order derivatives for network pruning: Optimal brain sur- geon, Advances in neural information processing systems, 5 (1992)

    1992

  12. [18]

    Hatamizadeh, J

    A. Hatamizadeh, J. Song, G. Liu, J. Kautz, and A. V ahdat, Diffit: Diffusion vision trans- formers for image generation , in European Conference on Computer Vision, Springer, 2025, pp. 37–55

    2025

  13. [19]

    Hayou, N

    S. Hayou, N. Ghosh, and B. Yu , Lora+: Efficient low rank adaptation of large models , 2024, https://arxiv.org/abs/2402.12354

    2024 arXiv

  14. [20]

    Y. He, X. Zhang, and J. Sun , Channel pruning for accelerating very deep neural networks , in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1389–1397

    2017

  15. [22]

    Hochbruck, M

    M. Hochbruck, M. Neher, and S. Schrammer , Rank-adaptive dynamical low-rank integra- tors for first-order and second-order matrix differential equations , BIT Numerical Mathe- matics, 63 (2023), p. 9

    2023

  16. [23]

    E. J. Hu, Y. Shen, P. W allis, Z. Allen-Zhu, Y. Li, S. W ang, L. W ang, and W. Chen, Lora: Low-rank adaptation of large language models , arXiv preprint arXiv:2106.09685, (2021), https://arxiv.org/abs/2106.09685

    2021 arXiv

  17. [24]

    Khader, G

    F. Khader, G. M ¨uller-Franzes, S. Tayebi Arasteh, T. Han, C. Haarburger, M. Schulze-Hagen, P. Schad, S. Engelhardt, B. Baeßler, S. Foersch, et al. , De- noising diffusion probabilistic models for 3d medical image generation , Scientific Reports, 13 (2023), p. 7303

    2023

  18. [25]

    Khalitov, T

    R. Khalitov, T. Yu, L. Cheng, and Z. Yang, Chordmixer: A scalable neural attention model for sequences with different length , in The Eleventh International Conference on Learning Representations

  19. [26]

    Khodak, N

    M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi , Initialization and regularization of factorized neural layers , in International Conference on Learning Representations, 2021

    2021

  20. [27]

    Kieri, C

    E. Kieri, C. Lubich, and H. W alach , Discretized dynamical low-rank approximation in the presence of small singular values , SIAM Journal on Numerical Analysis, 54 (2016), pp. 1020–1038

    2016

  21. [28]

    Kieri and B

    E. Kieri and B. V andereycken, Projection methods for dynamical low-rank approximation of high-dimensional problems , Computational Methods in Applied Mathematics, 19 (2019), pp. 73–92

    2019

  22. [29]

    Koch and C

    O. Koch and C. Lubich , Dynamical low-rank approximation, SIAM Journal on Matrix Anal- ysis and Applications, 29 (2007), pp. 434–454

    2007

  23. [30]

    Kusch, Second-order robust parallel integrators for dynamical low-rank approximation, arXiv preprint arXiv:2403.02834, (2024)

    J. Kusch, Second-order robust parallel integrators for dynamical low-rank approximation, arXiv preprint arXiv:2403.02834, (2024)

    2024 arXiv

  24. [31]

    LeCun, J

    Y. LeCun, J. Denker, and S. Solla , Optimal brain damage , Advances in neural information processing systems, 2 (1989)

    1989

  25. [32]

    Z. Li, H. Li, and L. Meng, Model compression for deep neural networks: A survey, Computers, 12 (2023), p. 60

    2023

  26. [33]

    Lialin, N

    V. Lialin, N. Shivagunde, S. Muckatira, and A. Rumshisky , Relora: High-rank training through low-rank updates, 2023, https://arxiv.org/abs/2307.05695

    2023 arXiv

  27. [34]

    Lubich and I

    C. Lubich and I. V. Oseledets , A projector-splitting integrator for dynamical low-rank ap- proximation, BIT Numerical Mathematics, 54 (2014), pp. 171–188

    2014

  28. [35]

    Molchanov, S

    P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz , Pruning convolutional neural networks for resource efficient inference , in International Conference on Learning Repre- sentations, 2017

    2017

  29. [36]

    E. H. S. Norsett and G. W anner, Solving ordinary differential equations i: Nonsti problems , 1987

    1987

  30. [37]

    Sato, Riemannian optimization and its applications , vol

    H. Sato, Riemannian optimization and its applications , vol. 670, Springer, 2021

    2021

  31. [38]

    Schotth ¨ofer, E

    S. Schotth ¨ofer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco , Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations , Ad- vances in Neural Information Processing Systems, 35 (2022), pp. 20051–20063

    2022

  32. [39]

    Schotth ¨ofer and M

    S. Schotth ¨ofer and M. P. Laiu , Federated dynamical low-rank training with global loss 24 J. KUSCH, S. SCHOTTH ¨OFER, AND A. W ALTER convergence guarantees, 2024, https://arxiv.org/abs/2406.17887

    2024 arXiv

  33. [41]

    V alipour, M

    M. V alipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation , 2023, https: //arxiv.org/abs/2210.07558

    2023 arXiv

  34. [42]

    W ang, S

    H. W ang, S. Agarwal, and D. Papailiopoulos, Pufferfish: Communication-efficient models at no extra cost , Proceedings of Machine Learning and Systems, 3 (2021), pp. 365–386

    2021

  35. [43]

    Weissmann, Y

    T. Weissmann, Y. Huang, S. Fischer, J. Roesch, S. Mansoorian, H. Ayala Gaona, A.-O. Gostian, M. Hecht, S. Lettmaier, L. Deloch, et al. , Deep learning for automatic head and neck lymph node level delineation provides expert-level accuracy , Frontiers in Oncology, 13 (2023), p. 1115258

    2023

  36. [44]

    J. Wu, C. Leng, Y. W ang, Q. Hu, and J. Cheng , Quantized convolutional neural networks for mobile devices , in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4820–4828

    2016

  37. [45]

    Zangrando, S

    E. Zangrando, S. Schotth ¨ofer, G. Ceruti, J. Kusch, and F. Tudisco , Rank-adaptive spectral pruning of convolutional layers during training, in Advances in Neural Information Processing Systems, 2024

    2024

  38. [46]

    Zangrando, S

    E. Zangrando, S. Schotth ¨ofer, G. Ceruti, J. Kusch, and F. Tudisco , Geometry-aware training of factorized layers in tensor tucker format , 2024, https://arxiv.org/abs/2305. 19059

    2024

  39. [47]

    Zhang, M

    Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao , Adalora: Adaptive budget allocation for parameter-efficient fine-tuning , 2023, https://arxiv.org/abs/2303.10512

    2023 arXiv

  40. [48]

    J. Zhao, Z. Zhang, B. Chen, Z. W ang, A. Anandkumar, and Y. Tian , Galore: Memory- efficient llm training by gradient low-rank projection , 2024, https://arxiv.org/abs/2403. 03507

    2024

  41. [49]

    X. Zhao, L. W ang, Y. Zhang, X. Han, M. Deveci, and M. Parmar, A review of convolutional neural networks in computer vision , Artificial Intelligence Review, 57 (2024), p. 99. Author contribution statement (CRediT). Jonas Kusch: Conceptualiza- tion, Methodology, Formal analysi...

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.