Pith. sign in

REVIEW 3 major objections 7 minor 65 references

Approximate polar factors replace SVD for low-rank weight regularization

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 01:45 UTC pith:MCGDJNPV

load-bearing objection SLORR: Simple and Efficient In-Training Low-Rank Regularization the 3 major comments →

arxiv 2607.08754 v1 pith:MCGDJNPV submitted 2026-07-09 cs.LG cs.AI

SLORR: Simple and Efficient In-Training Low-Rank Regularization

David Gonz\'alez-Mart\'inez , Shiwei Liu This is my paper
classification cs.LG cs.AI
keywords low-rank regularizationpolar factor approximationneural network compressionsingular value decompositionnuclear normHoyer sparsityweight matrix factorizationtraining-time compression
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Neural network compression by low-rank factorization works best when weight matrices already have spectra concentrated in a few dominant directions. Training-time regularizers that push weights toward this structure exist, but they typically require computing singular value decompositions of every weight matrix at every training step, which is prohibitively expensive for modern models. This paper introduces SLORR, a framework that achieves the same goal by replacing exact SVD-dependent quantities with iterative approximations of the polar factor (the product U V^T from the SVD), computed via a GPU-efficient method called Polar Express. The key observation is that both the nuclear norm and the Hoyer sparsity metric on singular values can be evaluated and differentiated using only this polar factor and the Frobenius norm, neither of which requires an explicit SVD. The resulting regularizer is stateless (no cached factors to maintain), architecture-preserving (no new parameters, no factorized layers), and adds less than 8% overhead for vision models and less than 1% for language models at 135M and 560M scales. The paper provides worst-case approximation guarantees showing that the forward and backward errors of both SLORR variants vanish as the number of Polar Express iterations increases. Empirically, SLORR-trained models retain substantially more accuracy after SVD-based compression than unregularized models, across ResNet and Vision Transformer architectures on ImageNet and Llama-style language models on FineWeb-Edu.

Core claim

The central mechanism is that the polar factor U V^T of a weight matrix, which is the only SVD-dependent quantity needed to compute and differentiate both the nuclear norm and the squared Hoyer sparsity metric, can be approximated to provable accuracy by a small number of GPU-friendly matrix-polynomial iterations (Polar Express, 6 steps in practice). This replaces the expensive per-iteration SVD that has made direct spectral regularization impractical, while preserving the property that the regularizer operates on the original weight matrix without architectural changes or cached state.

What carries the argument

The method relies on three ingredients. First, the nuclear norm can be computed as the trace of W^T (U V^T), which equals the sum of singular values, and the Hoyer metric is the ratio of the squared nuclear norm to the squared Frobenius norm. Second, the gradient of each regularizer with respect to W involves U V^T and norms of W, but never requires the individual singular vectors U and V separately. Third, the polar factor U V^T is approximated by Polar Express, an iterative method that applies a fixed-degree polynomial to a normalized version of W, with a worst-case convergence rate of |1 - ℓ^2|^{(q+1)T} where ℓ is a lower bound on normalized nonzero singular values, q determines the polyn

Load-bearing premise

The Polar Express approximation with 6 iterations and a lower-bound parameter of one-thousandth is assumed to produce gradients accurate enough to be useful. The formal guarantee requires all nonzero singular values of the normalized matrix to exceed this lower bound, but the paper acknowledges this may not hold in practice and relies on empirical evidence rather than proof that the approximation remains adequate when it does not.

What would settle it

If, for a significant fraction of weight matrices encountered during training, the smallest nonzero normalized singular value falls well below ℓ = 10^{-3}, the Polar Express approximation of the polar factor could be poor enough that the regularizer gradient points in a substantially wrong direction, causing the method to either fail to induce compressibility or destabilize training. A concrete test would be to measure the actual distribution of normalized nonzero singular values across layers and training steps for the architectures tested, and to check whether the worst-case bound is ever接近

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the polar factor approximation is sufficiently accurate in practice, direct spectral regularization becomes a drop-in training modification for any architecture with matrix-shaped weights, removing the main computational barrier to training models that are compressible by design.
  • The stateless and architecture-preserving properties mean SLORR can be applied to pretrained checkpoints during fine-tuning or continued training without reparameterizing layers, which factorization-based methods cannot do without initialization tricks.
  • The framework extends to any regularizer whose forward and backward passes depend only on the polar factor and norms, suggesting a broader family of SVD-free spectral penalties beyond nuclear norm and Hoyer.
  • The observation that regularizing transformer block weights also shifts the spectral properties of the unregularized embedding matrix implies that spectral structure propagates through the model, which could inform which layers to target for regularization budget allocation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approximation quality depends on the spectral gap between zero and the smallest nonzero singular value; for matrices with many tiny but nonzero singular values, the lower-bound parameter ℓ = 10^{-3} may be violated, and the worst-case bound does not apply. The paper's empirical success suggests this is rarely catastrophic, but a formal characterization of when the approximation degrades would
  • Because SLORR does not backpropagate through the Polar Express iterations (it treats the approximate polar factor as a constant for gradient purposes), the effective gradient is an approximation not only in the forward value but also in the backward direction. This is analogous to straight-through estimators in quantization, and its interaction with adaptive optimizers like AdamW may differ from e
  • The finding that models become less compressible with longer training (the overtraining experiments) suggests that spectral concentration and task performance may be in tension during extended optimization, which has implications for the common practice of training far beyond compute-optimal budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces SLORR, a framework for in-training low-rank regularization of neural network weight matrices. The key idea is to approximate the spectral quantities needed for two standard low-rank penalties (the squared Hoyer sparsity metric and the nuclear norm) using the Polar Express algorithm, which computes an approximation of the generalized polar factor UV^T via iterative matrix multiplications. This avoids expensive per-step SVDs, requires no architectural modifications, and maintains no cached state. The authors provide approximation guarantees (Proposition 3.1) for both the regularizer values and their gradients, derived from the Polar Express convergence theorem. The method is evaluated extensively: over 150 runs on ImageNet-1K (ResNet-50, ViT-B/16, ViT-L/16 continued training; ResNet-18 pretraining) and LLM pretraining at 135M and 560M scales, showing improved post-training compressibility at low overhead (<8% for vision, <1% for LLMs).

Significance. The paper addresses a practical and well-motivated problem: making neural network weight matrices more amenable to low-rank factorization during training, without the prohibitive cost of per-iteration SVDs or the complications of architectural changes and stateful caches. The approach is technically clean: the gradient derivations (Appendix A) correctly use Clarke generalized gradients, and the approximation guarantees (Proposition 3.1) follow transparently from the Polar Express bound. The experimental evaluation is notably extensive for this area, spanning multiple architectures, training regimes, and scales. The release of code is a positive for reproducibility. The framework's versatility (applicable to any weight matrix, integrable as a loss term or decoupled regularizer) and the very low overhead at LLM scale are the main strengths.

major comments (3)
  1. Section 3, Proposition 3.1 and 'Practical remarks': The approximation guarantee requires nonzero singular values of the normalized matrix W/(||W||_F + ε) to lie in [ℓ, 1] with ℓ = 10^{-3}. In practice, weight matrices in large neural networks can have condition numbers far exceeding 10^3, meaning after Frobenius normalization, the smallest nonzero singular values can fall well below ℓ. When this happens, the worst-case bound δ = |1 - ℓ²|^{(q+1)T} no longer applies to those singular directions, and the approximate polar factor P̂ may deviate from the true UV^T in those directions. Since the regularizer gradient for both SLORR-Hoyer and SLORR-Nuc depends directly on P̂ (Eq. 1), poorly approximated small singular directions could produce misleading gradient signals. The paper acknowledges this in §3 ('Practical remarks') and cites Amsel et al. [38] that 'inaccurate guesses are typically not
  2. Section 4.1, Figure 1 and associated tables (Tables 6-9): The experimental comparison across methods shows that 'each exact setting appears to favor different methods, including different SLORR variants' (§4.1), suggesting no single variant dominates. While the paper is transparent about this, it raises a practical concern: a practitioner must select among SLORR-Hoyer, SLORR-Hoyer-D, SLORR-Nuc, and the regularization strength λ, and the paper does not provide clear guidance on which variant to choose for a new setting. Additionally, the hyperparameter selection is described as 'mainly followed a best-effort manual approach' (Appendix E), and Q3R has two interacting hyperparameters making it harder to tune. While the paper acknowledges this is not a definitive ranking, it would strengthen the contribution if the authors could identify settings or heuristics where SLORR is reliably the top
  3. Section 3.1, Propositions 3.2-3.3: The theoretical analysis of SLORR-Hoyer's effect on singular values is conducted in a regularizer-only gradient descent setting, without task loss or adaptive optimizers (Adam). The paper acknowledges this gap. While the analysis is informative for understanding the mechanism (concentrating spectral energy onto large singular values), the practical setting involves Adam with decoupled regularization (Algorithm 1), which may behave quite differently. The gap between theory and practice is not load-bearing for the central empirical claim, but it limits the theoretical contribution's practical relevance.
minor comments (7)
  1. Table 1: The 'Prior target rank' column for Q3R is marked 'Yes' while for SLORR it is 'No'. This is a useful distinction, but the text could clarify more explicitly that SLORR does not require specifying a target rank, which is a practical advantage.
  2. Section 4.2, Figure 3: The perplexity plots are clipped at 150 for visibility. While the full tabular results are in Appendix K, it would help to mention the clipping range in the figure caption.
  3. Appendix D.1, Table 3: The ablation on Polar Express iterations shows that 1 or 2 iterations produce very poor results (near-random accuracy at most compression ratios), while 6+ iterations work well. This is a sharp transition. It would be informative to understand why so few iterations fail so dramatically — is the polar factor approximation qualitatively wrong, or is it a numerical stability issue?
  4. Section 4.1.1, Figure 2: The overhead scaling plot uses interpolated ViT configurations (e.g., vit_256, vit_512). It would help to mark which configurations are standard (ViT-T/S/B/L/H) vs. interpolated in the figure or caption.
  5. Appendix I: The 135M×8 run with λ=10^{-5} encountered a numerical explosion and was rerun with a different seed. While the authors' handling of this is transparent, a brief discussion of potential instability mitigation (e.g., gradient clipping on the regularizer) would be useful.
  6. Listing 1: The code snippet is helpful but uses a function `polar_express` that is not defined in the listing. A brief note on where this function comes from (the Polar Express implementation) would make the listing more self-contained.
  7. Section 2: The related work discussion of Q3R [9] mentions that 'their main experiments are run with a refresh period of 5 iterations, which is expensive.' The footnote discusses using larger periods, but it would be clearer to state upfront in the main text that Q3R's cost depends heavily on this period parameter.

Circularity Check

0 steps flagged

No circularity: derivation chain is self-contained with external mathematical support

full rationale

The paper's derivation chain is non-circular. The regularizer gradients (Eq. 1) are derived from standard matrix calculus and Clarke generalized gradients (Appendix A), independent of any fitted parameters. The approximation guarantees (Proposition 3.1) follow from the Polar Express convergence theorem (Amsel et al. [38]), which is an external result by a different set of authors — not self-citation. Propositions 3.2–3.3 analyzing SLORR-Hoyer's effect on singular values are self-contained mathematical proofs (Appendix B) in a regularizer-only gradient descent setting. The experimental results compare against external baselines (Q3R, LoRITa, unregularized models) without renaming fitted parameters as predictions. No step in the derivation reduces to its own inputs by construction, and no load-bearing self-citation chain exists. The derivation is self-contained against external benchmarks and external mathematical results, warranting a score of 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

SLORR introduces no new mathematical entities, particles, or postulated objects. The method combines existing tools (Hoyer metric, nuclear norm, Polar Express) in a new way. All free parameters are standard hyperparameters for regularization methods or inherited from Polar Express recommendations. The axioms are either standard mathematical results or domain assumptions with external grounding.

free parameters (5)
  • λ (regularization strength) = varies: 0.000005–0.01 across experiments
    The primary hyperparameter controlling regularization strength. Swept manually for each setting. Not fitted to a theoretical prediction.
  • T (Polar Express iterations) = 6
    Number of Polar Express iterations. Set to 6 following Amsel et al. recommendations. Ablated in Table 3 showing diminishing returns after 6.
  • ℓ (lower bound parameter) = 0.001
    Lower bound on smallest nonzero singular value for Polar Express. Set to 10⁻³ following Amsel et al. recommendations. Affects approximation guarantee validity.
  • d (Polar Express polynomial degree) = 5
    Polynomial degree for Polar Express iteration. Set to 5 following Amsel et al. recommendations.
  • ε (numerical stabilization) = small positive constant
    Added to Frobenius norm in forward/backward to avoid division by zero. Standard numerical practice.
axioms (4)
  • domain assumption Polar Express convergence theorem (Amsel et al. 2026, Theorem 3.3): after T iterations, the approximation error is bounded by |1-ℓ²|^(q+1)^T when singular values lie in [ℓ, 1].
    Invoked in Proposition 3.1 to derive approximation guarantees for SLORR. External result, but the assumption that singular values lie in [ℓ, 1] may not hold in practice (acknowledged in §3 'Practical remarks').
  • standard math Clarke generalized gradient provides the correct backpropagation rule for non-smooth regularizers.
    Used in Appendix A to derive gradient rules for nuclear norm and Hoyer metric. Standard nonsmooth analysis (Clarke 1990).
  • domain assumption Low-rank weight structure (concentrated singular value spectra) leads to better post-training SVD compression.
    Implicit throughout: the paper regularizes to concentrate spectra, then compresses via SVD truncation. This is well-established (Eckart-Young-Mirsky theorem) but the link between training-time spectral concentration and downstream compression quality is an empirical assumption.
  • domain assumption The minimum-Frobenius-norm element of the Clarke generalized gradient is the appropriate gradient for optimization.
    Stated in §3 and Appendix A. This is a standard practical choice but other generalized gradient elements could be selected.

pith-pipeline@v1.1.0-glm · 55647 in / 3085 out tokens · 253452 ms · 2026-07-10T01:45:57.570465+00:00 · methodology

0 comments
read the original abstract

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

Figures

Figures reproduced from arXiv: 2607.08754 by David Gonz\'alez-Mart\'inez, Shiwei Liu.

Figure 1
Figure 1. Figure 1: Accuracy–compression curves of regularized ResNet-50, ViT-B/16, ViT-L/16, and ResNet-18 models on ImageNet-1K. The y-axis reports ImageNet-1K validation top-1 accuracy of the compressed model, and the x-axis reports either the retained parameter ratio or the retained FLOPs ratio, as indicated in each panel. Factorization is performed using the energy and uniform rank-selection criteria (see Section 4 for d… view at source ↗
Figure 2
Figure 2. Figure 2: Time and memory overhead of differ￾ent regularizers across ViT model scales. Run￾time (left) and peak memory (right) are normalized to unregularized training. For Q3R, the number in parentheses denotes the SVD refresh interval. The in￾set shows a zoomed view of the normalized-runtime curves near ViT-H scale. Model Method Time Mem. (GB) ViT-B/16 SLORR ×1.037 +0.34 LoRITa ×1.007 +0.57 Q3R(5) ×1.656 +1.14 Q3R… view at source ↗
Figure 3
Figure 3. Figure 3: LLM compression results with SLORR-Hoyer regularization. Colors indicate the SLORR-Hoyer regularization strength λ used during pretraining. λ = 0 corresponds to the unregu￾larized baseline. Top row: FineWeb-Edu validation perplexity for Llama 135M and Llama 560M after compression with either plain SVD or SVD-LLM whitening, as indicated in each panel title. Perplexity values are clipped at 150 for visibilit… view at source ↗
Figure 4
Figure 4. Figure 4: Llama models trained for 4× and 8× the compute-optimal token budget. Perplexity values are clipped to 150, as in our main text figures. For complete tabular results, see Appendix K. K Complete LLM Results In this section, we include tabular results for the LLM figures in the main text. We include one table for perplexity and, for the 560M variants, one table for each downstream task. Each row corresponds t… view at source ↗
Figure 5
Figure 5. Figure 5: Singular value spectra for all layers of Llama 135M. The different curves correspond to [PITH_FULL_IMAGE:figures/full_fig_p040_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 65 canonical work pages · 20 internal anchors

  1. [1]

    Byte pair encoding is suboptimal for language model pretraining

    Matan Ben Noach and Yoav Goldberg. Compressing Pre-trained Language Models by Matrix Decomposition. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 884–889, Suzhou,...

    work page doi:10.18653/v1/2020 2020

  2. [2]

    Exploiting linear structure within convolutional networks for efficient evaluation

    Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’14, pp. 1269–1277, Cambridge, MA, USA, 2014. MIT Press

    work page 2014

  3. [3]

    Speeding up Convolutional Neural Networks with Low Rank Expansions

    Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up Convolutional Neural Networks with Low Rank Expansions. In Proceedings of the British Machine Vision Conference. BMV A Press, 2014. doi: http://dx.doi.org/10.5244/C.28.88

    work page doi:10.5244/c.28.88 2014

  4. [4]

    DRONE: Data- aware Low-rank Compression for Large NLP Models

    Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. DRONE: Data- aware Low-rank Compression for Large NLP Models. In M. Ranzato, A. Beygelz- imer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems , volume 34, pp. 29321–29334. Curran Associates, Inc., 2021. URL https://proceedings.neur...

    work page 2021

  5. [5]

    Alvarez and Mathieu Salzmann

    Jose M. Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS’17, pp. 856–867, Red Hook, NY , USA, 2017. Curran Associates Inc. ISBN 9781510860964

    work page 2017

  6. [6]

    TRP: Trained Rank Pruning for Efficient Deep Neural Networks

    Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. TRP: Trained Rank Pruning for Efficient Deep Neural Networks. CoRR, abs/2004.14566, 2020. URL https://arxiv.org/abs/2004.14566

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  7. [7]

    Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification

    Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. CoRR, abs/2004.09031, 2020. URL https://arxiv.org/ abs/2004.09031

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  8. [8]

    Structure-Preserving Network Com- pression Via Low-Rank Induced Training Through Linear Layers Composition

    Ismail Alkhouri, Xitong Zhang, and Rongrong Wang. Structure-Preserving Network Com- pression Via Low-Rank Induced Training Through Linear Layers Composition. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/ forum?id=1KCrVMJoJ9

    work page 2024

  9. [9]

    Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

    Ipsita Ghosh, Ethan Nguyen, and Christian Kümmerle. Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id= ZtzWvNKOCr

    work page 2025

  10. [10]

    Patrik O. Hoyer. Non-negative Matrix Factorization with Sparseness Constraints. J. Mach. Learn. Res., 5:1457–1469, December 2004. ISSN 1532-4435

    work page 2004

  11. [11]

    ImageNet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848. 11

    work page doi:10.1109/cvpr.2009.5206848 2009

  12. [12]

    Deep Residual Learning for Image Recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016. doi: 10.1109/CVPR.2016.90

    work page doi:10.1109/cvpr.2016.90 2016

  13. [13]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021. URL https...

    work page 2021

  14. [14]

    From Low Rank Gradient Subspace Stabi- lization to Low-Rank Weights: Observations, Theories, and Applications

    Ajay Kumar Jaiswal, Yifan Wang, Lu Yin, Shiwei Liu, Runjin Chen, Jiawei Zhao, Ananth Grama, Yuandong Tian, and Zhangyang Wang. From Low Rank Gradient Subspace Stabi- lization to Low-Rank Weights: Observations, Theories, and Applications. In Forty-second International Conference on Machine Learning , 2025. URL https://openreview.net/ forum?id=fsf7LhbYdf

    work page 2025

  15. [15]

    Golub and Charles F

    Gene H. Golub and Charles F. Van Loan. Matrix Computations - 4th Edition. Johns Hopkins University Press, Philadelphia, PA, 2013. doi: 10.1137/1.9781421407944. URL https: //epubs.siam.org/doi/abs/10.1137/1.9781421407944

    work page doi:10.1137/1.9781421407944 2013

  16. [16]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278–2324, 1998. URL https://api. semanticscholar.org/CorpusID:14542261

    work page 1998

  17. [17]

    Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

    Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=kws76i5XB8

    work page 2025

  18. [18]

    SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=LNYIUouhdt

    work page 2025

  19. [19]

    SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression

    Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum...

    work page doi:10.18653/v1/2025.naacl-long 2025

  20. [20]

    URL https://aclanthology.org/2025.naacl-long.217/

    work page 2025

  21. [21]

    ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, and Guangyu Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, 2025. URL https://arxiv.org/abs/2312.05821

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    DARE: Data Augmented Relation Extraction with GPT-2

    M. Fazel, H. Hindi, and S. Boyd. Rank minimization and applications in system theory. In Proceedings of the 2004 American Control Conference, volume 4, pp. 3273–3278 vol.4, 2004. doi: 10.23919/ACC.2004.1384521

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  23. [23]

    Trace norm regularization and faster inference for embedded speech recognition RNNs

    Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, and Mohammad Shoeybi. Trace norm regularization and faster inference for embedded speech recognition RNNs, 2018. URL https://arxiv.org/abs/1710.09026

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  24. [24]

    Efficient Differentiable Approximation of Generalized Low-rank Regularization

    Naiqi Li, Yuqiu Xie, Peiyuan Liu, Tao Dai, Yong Jiang, and Shu-Tao Xia. Efficient Differentiable Approximation of Generalized Low-rank Regularization. In James Kwok (ed.), Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25 , pp. 5616–5624. International Joint Conferences on Artificial Intelligence Organiza...

    work page doi:10.24963/ijcai.2025/625 2025

  25. [25]

    Low-Rank Prehab: Preparing Neural Networks for SVD Compression, 2025

    Haoran Qin, Shansita Sharma, Ali Abbasi, Chayne Thrash, and Soheil Kolouri. Low-Rank Prehab: Preparing Neural Networks for SVD Compression, 2025. URL https://arxiv.org/ abs/2512.01980. 12

    work page arXiv 2025

  26. [26]

    Compression-aware Training of Neural Networks using Frank-Wolfe

    Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Compression-aware Training of Neural Networks using Frank-Wolfe, 2024. URL https://arxiv.org/abs/2205.11921

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training, 2026

    Hadi Mohaghegh Dolatabadi, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Chamin P Hewa Koneputugodage, Shamane Siriwardhana, Violetta Shevchenko, Karol Pajak, James Snewin, Gil Avraham, and Alexander Long. NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training, 2026. URL https://arxiv.org/abs/2603.03597

    work page arXiv 2026

  28. [28]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7

    work page 2019

  29. [29]

    ReLoRA: High-Rank Training Through Low-Rank Updates

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. ReLoRA: High-Rank Training Through Low-Rank Updates, 2023. URL https://arxiv.org/abs/ 2307.05695

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Tenenholtz, Lester Mackey, and Nicolo Fusi

    Mikhail Khodak, Neil A. Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and Regular- ization of Factorized Neural Layers. In International Conference on Learning Representations,

    work page

  31. [31]

    URL https://openreview.net/forum?id=KTlJT1nof6d

    work page

  32. [32]

    Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N. Gomez. Exploring Low Rank Training of Deep Neural Networks, 2022. URL https://arxiv.org/abs/2209.13569

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [33]

    Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers

    Xiuying Wei, Skander Moalla, Razvan Pascanu, and Caglar Gulcehre. Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers. In The Thirty- eighth Annual Conference on Neural Information Processing Systems , 2024. URL https: //openreview.net/forum?id=WxLVYZbIew

    work page 2024

  34. [34]

    SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining

    Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems,

    work page

  35. [35]

    URL https://openreview.net/forum?id=MXze4H7opg

    work page

  36. [36]

    LOST: Low-rank and Sparse Pre-training for Large Language Models

    Jiaxi Li, Lu Yin, Li Shen, Jinjin Xu, Liwu Xu, Tianjin Huang, Wenwu Wang, Shiwei Liu, and Xilu Wang. LOST: Low-rank and Sparse Pre-training for Large Language Models, 2025. URL https://arxiv.org/abs/2508.02668

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

    Huanrui Yang, Wei Wen, and Hai Li. DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rylBK34FDS

    work page 2020

  38. [38]

    Perturbation Bounds of Unitary and Subunitary Polar Factors

    Wen Li and Weiwei Sun. Perturbation Bounds of Unitary and Subunitary Polar Factors. SIAM Journal on Matrix Analysis and Applications , 23(4):1183–1193, 2002. doi: 10.1137/ S0895479801394623. URL https://doi.org/10.1137/S0895479801394623

    work page doi:10.1137/s0895479801394623 2002

  39. [39]

    Nicholas J. Higham. The matrix sign decomposition and its relation to the polar decomposition. Linear Algebra and its Applications, 212-213:3–20, 1994. ISSN 0024-3795. doi: https://doi. org/10.1016/0024-3795(94)90393-X. URL https://www.sciencedirect.com/science/ article/pii/002437959490393X

    work page doi:10.1016/0024-3795(94)90393-x 1994

  40. [40]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    work page 2024

  41. [41]

    Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=yRtgZ1K8hO

    work page 2026

  42. [42]

    Adam: A Method for Stochastic Optimization

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, 2017. URL https://arxiv.org/abs/1412.6980. 13

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  43. [43]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023. URL https://arxiv.org/abs/2302.13971

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024. URL https://arxiv.org/abs/2406.17557

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [45]

    Training Compute-Optimal Large Language Models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  46. [46]

    The Language Model Evaluation Harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The Languag...

    work page 2024

  47. [47]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018. URL https://arxiv.org/abs/1803.05457

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  48. [48]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics....

    work page doi:10.18653/v1/p19-1472 2019

  49. [49]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistic...

    work page doi:10.18653/v1/p16-1144 2016

  50. [50]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.),Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, Oct...

    work page doi:10.18653/v1/d18-1260 2018

  51. [51]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language, 2019. URL https://arxiv.org/abs/ 1911.11641

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  52. [52]

    Gram Newton-Schulz, 2026

    Jack Zhang, Noah Amsel, Berlin Chen, and Tri Dao. Gram Newton-Schulz, 2026. URL https://dao-ailab.github.io/blog/2026/gram-newton-schulz/

    work page 2026

  53. [53]

    Domain Generalization via Nuclear Norm Regularization

    Zhenmei Shi, Yifei Ming, Ying Fan, Frederic Sala, and Yingyu Liang. Domain Generalization via Nuclear Norm Regularization. In Conference on Parsimony and Learning (Proceedings Track), 2024. URL https://openreview.net/forum?id=hJd66ZzXEZ

    work page 2024

  54. [54]

    Frank H. Clarke. Optimization and Nonsmooth Analysis. Society for Industrial and Applied Mathematics, 1990. doi: 10.1137/1.9781611971309. URL https://epubs.siam.org/doi/ abs/10.1137/1.9781611971309. 14

    work page doi:10.1137/1.9781611971309 1990

  55. [55]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Perform...

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  56. [56]

    PyTorch Image Models, 2019

    Ross Wightman. PyTorch Image Models, 2019. URL https://github.com/huggingface/ pytorch-image-models

    work page 2019

  57. [57]

    TorchVision: PyTorch’s Computer Vision library, November 2016

    TorchVision maintainers and contributors. TorchVision: PyTorch’s Computer Vision library, November 2016. URL https://github.com/pytorch/vision

    work page 2016

  58. [58]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2018. URL https://arxiv.org/abs/1706.02677

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  59. [59]

    SGDR: Stochastic Gradient Descent with Warm Restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations, 2017. URL https://openreview. net/forum?id=Skq89Scxx

    work page 2017

  60. [60]

    BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

    David González-Martínez. BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression, 2025. URL https://arxiv.org/abs/2509.25136

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  61. [61]

    Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition

    Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition. In A. Beygelz- imer, Y . Dauphin, P. Liang, and J. Wortman Vaughan (eds.),Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=BvJkwMhyInm

    work page 2021

  62. [62]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Ma- lik, Willia...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [63]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074.html

    work page 2020

  64. [64]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale, 2019. URL https://arxiv.org/abs/ 1907.10641. 15 A Backpropagating Through the Regularizers As noted before, the SLORR regularizers are not differentiable everywhere. In practice, one can informally use “natural” backpropaga...

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  65. [65]

    rand-m9-mstd0.5-inc1

    With a batch size of 1024, LoRITa produced out-of-memory errors, even with our optimizations. We therefore use a batch size of 896 for all methods in this setting to ensure comparable conditions. We only used a learning rate of 1 × 10−5. Hyperparameters for ResNet-50. For ResNet-50, we use a learning rate of 5 × 10−5. We use the same scheduler as in the V...

    work page 2048