Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Training a language model so every weight matrix splits into a low-rank part plus a sparse part yields a single checkpoint that can be resized to any memory budget at deployment, without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:51 UTC pith:RPMZX5FT

load-bearing objection A serious training-time SLR framework with a genuinely new optimizer, but the elastic-deployment headline likely rests on per-budget oracle tuning of κ, and the main table compares float32 to bf16; it deserves peer review after those are fixed. the 4 major comments →

arxiv 2602.00942 v3 pith:RPMZX5FT submitted 2026-02-01 cs.LG

SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference

Hao Ma , Melis Ilayda Bal , Liang Zhang , Bingcong Li , Niao He , Melanie Zeilinger , Michael Muehlebach This is my paper
classification cs.LG
keywords sparse and low-rank decompositionADMMLLM pretrainingmodel compressionelastic deploymentintegral controlhomomorphic parameter allocationembedding layer structure
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that sparse-and-low-rank structure must be induced while a language model trains, not stamped on afterwards: if each weight matrix is trained to decompose into a low-rank part L plus a sparse part S, a single checkpoint can later be truncated to any memory budget with smooth, predictable quality loss. The proposed framework, SALAAD, does this with a two-stage ADMM loop layered on ordinary training, plus an integral controller that adaptively tunes rank and sparsity per block so only one global penalty coefficient ρ has to be set. Across 60M to 1B parameter models, the structured surrogate matches or beats prior sparse/low-rank pretraining methods at comparable parameter counts, and its performance-capacity curve stays smooth where post-hoc-compressed vanilla checkpoints degrade sharply. The paper also surfaces an asymmetry: embedding layers take on low-rank and sparse structure without hurting training, while the language-model head resists it. If the results hold, one training run replaces the current practice of training and releasing many fixed-size models.

Core claim

SALAAD imposes structure during training, not after: it trains dense weights X together with a structured surrogate X ≈ L + S, L low-rank and S sparse. Training alternates ordinary gradient steps on ℓ(X) + (ρ/2)‖X − L − S + Y/ρ‖² with closed-form proximal updates — singular-value soft-thresholding for L, elementwise soft-thresholding for S, plus a dual update. An integral controller adapts the two penalties blockwise so only the global coefficient ρ needs tuning, and the surrogate supports continuous post-hoc truncation by a homomorphic allocation rule. Across 60M–1B parameters the paper reports that the surrogate matches or beats prior sparse/low-rank pretraining baselines, degrades smoothl

What carries the argument

The engine is the ADMM decomposition X = L + S, whose two proximal operators do the structural work in closed form: singular-value soft-thresholding (the nuclear-norm proximal step, which shrinks small singular values of L toward zero) and elementwise soft-thresholding (the ℓ1 proximal step, which zeros small entries of S), with a dual variable Y pulling X and L+S together. Around that loop sit two named pieces: the I-controller, an integral controller that raises or lowers the rank and sparsity penalties per block so that only the global penalty ρ needs tuning, and HPA, a greedy budget-splitting rule that truncates rank and sparsity by uniform global ratios across blocks, giving continuous,

Load-bearing premise

Everything hinges on a single undisclosed number: the proportionality constant in the ρ ∝ 1/(N√(nm)) scaling law is fitted on the 60M and 130M models and then fixed for 350M and 1B, yet its value is never reported, and the 130M ablation shows perplexity swinging from 21.7 to 30.0 — roughly 40% — as ρ moves from 1e-8 to 1e-6.

What would settle it

Re-run the 1B pretraining twice, at ρ = 1e-8 and ρ = 1e-6 (the ends of the 130M ablation range in the paper's Table 8): if the single fixed ρ transfers, both runs should train stably with similar perplexity and compression; if the roughly 40% perplexity swing reproduces at 1B, the 'one hyperparameter across scales' premise collapses and with it the 350M/1B results that anchor the elastic-deployment claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single SALAAD checkpoint can be truncated at deployment to any parameter budget below nominal size with a smooth, predictable performance–capacity trade-off, whereas RPCA-compressed vanilla checkpoints degrade sharply.
  • The structured surrogate and its truncated variants match or beat existing sparse-and-low-rank pretraining methods at comparable parameter counts, without changing the Transformer architecture.
  • Post-hoc decomposition is not enough: RPCA applied to standard-trained weights recovers only weak structure (roughly 48–55% effective rank), so SLR compression needs the training-time induction SALAAD performs.
  • Embedding layers can be folded into SLR training at no perceptible loss cost, adding compression redundancy, while the LM head cannot — delimiting where structure can be extracted.
  • The I-controller collapses per-block structural hyperparameters into one global ρ, so the recipe transfers across model scales without hand-crafted per-layer rank or sparsity schedules.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The undisclosed proportionality constant in the ρ scaling law is the crux: the paper says it is fitted on 60M/130M models and fixed for 350M/1B, but never reports the number, so the across-scale claim is not yet fully reproducible from the text.
  • The headline table compares float32-trained SALAAD against bfloat16-trained baselines — a precision difference the paper discloses but never quantifies; rerunning baselines in float32 would separate structural gains from precision gains.
  • HPA assumes block homomorphism — uniform global truncation ratios ignore block-specific sensitivity; if the surrogate structure is genuinely heterogeneous, a sensitivity-weighted allocation should beat HPA, a testable extension the paper does not run.
  • The benign-embedding finding suggests an untested design rule: embedding widths and vocabulary sizes could be chosen for compressibility inside the same framework, without altering the Transformer skeleton.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SALAAD, a training-time framework that induces sparse and low-rank structure in LLM weights by solving a regularized decomposition X = L + S through a two-stage stochastic ADMM variant. An integral controller adaptively adjusts the nuclear-norm and ℓ1 penalties per block so that only one penalty coefficient ρ needs tuning; after training, a homomorphic parameter allocation (HPA) strategy removes parameters from the SLR surrogate to obtain a continuous capacity–perplexity trade-off from a single checkpoint. Experiments on LLaMA-style models from 60M to 1B report perplexity and parameter counts for the dense model X, the structured surrogate L+S, and HPA-compressed variants, and claim that SALAAD matches or beats prior compressed-pretraining methods while enabling elastic deployment.

Significance. If the claims are sustained, the paper would make a useful contribution: it offers a training-compatible, architecture-preserving way to obtain SLR structure, and the HPA idea of converting a single trained checkpoint into a continuous family of capacity levels is practically appealing. The manuscript also has strengths that should be acknowledged: a broad set of scales (60M–1B), explicit ablations of ρ, Δα, Δβ, and κ, and a transparency that the main results use float32 training while Appendix D reports bf16 training. The empirical finding about embedding layers is interesting and potentially valuable. However, the significance as currently framed is limited by two load-bearing issues: the headline comparison in Table 1 is at unequal numerical precision, and the elastic-deployment curves in Figure 2 appear to rely on per-budget oracle selection of κ. Both need to be resolved before the central claims can be accepted.

major comments (4)
  1. [Section 5.1; Appendix D, Table 4] The main comparison is not at equal numerical precision. Table 1 states that all SALAAD results are trained in float32, while all baselines are trained in bf16. Appendix D reports bf16 training: at 130M, SALAAD L+S gives PPL 24.18 with 129M parameters vs LOST 24.05 with 94M; at 350M, L+S gives 18.93/287M vs LOST 18.95/185M; and eL+eS at 130M gives 25.67/97M vs LOST 24.05/94M. Under fair precision, the advantage of Table 1 substantially vanishes or reverses. The §5.1 statement that SALAAD "consistently match or outperform" baselines must be re-supported using bf16 results as the primary comparison, or explicitly qualified as a float32-vs-bf16 comparison.
  2. [§4.3, Eq. (9); §5.2, Fig. 3; Table 1] The elastic-deployment curves in Figure 2 do not state how κ is selected. Section 5.2 (Figure 3) shows that κ is swept for each model size and budget, with the optimal value lying in a gray region, and Table 1 uses different κ values per scale (0.7, 0.6, 0.6, 0.8). If the Figure 2 points are obtained with the best κ on the evaluation data, the smooth capacity–perplexity curve is an upper envelope, not a deployable single-rule curve. A user facing a fixed memory budget has no stated way to choose κ without oracle tuning. This directly threatens the central claim that one checkpoint yields a continuous spectrum of capacities without retraining.
  3. [§4.3] HPA relies on two unvalidated assumptions: unit importance I(u) proportional to magnitude |u|, and structural homomorphism across blocks (shared global ratios). No experiment compares this proxy with exact or oracle importance, nor with alternative allocation rules. Since Figure 2 is the only evidence for the elastic-deployment claim, and Eq. (8) is explicitly intractable, the greedy magnitude-based rule needs at least a sanity-check experiment (e.g., leave-one-out importance on a small model, or comparison with one or more alternative κ rules) to be load-bearing.
  4. [§5.1, Eq. (7); Appendix G, Table 8] The ρ scaling law is fitted only on 60M and 130M models, and the proportionality constant is never reported. The sensitivity is substantial: at 130M, Table 8 shows PPL ranging from about 21.7 at ρ=1e-8 to about 30.0 at ρ=1e-6 for some (Δα,Δβ) settings — roughly a 40% degradation over two orders of magnitude. Without reporting the fitted constant and demonstrating that the 350M and 1B runs lie in a stable plateau, the claim that a single fixed ρ transfers across scales is not supported by the evidence.
minor comments (4)
  1. [Figures 1, 7, 8] The target density is written as \hat\Upsilon = 0.05 in the text, but several figure panels show "\hat\Upsilon = 0.95" (e.g., Figure 1b, Figure 7c). This inconsistency makes the convergence plots hard to interpret.
  2. [Table 1] The L+S rows in Table 1 print perplexity and parameter counts without separators (e.g., "31.2650 22.65126"), making the table difficult to read. Use explicit column separators.
  3. [All experiments] No seeds or error bars are reported for any run. Given the small parameter-count and perplexity differences in Appendix D, single-run numbers cannot be distinguished from noise.
  4. [Appendix E] The text says that |X − (L+S)|_F "remains bounded throughout training" and refers to Appendix E, but Appendix E contains only empirical learning-dynamics plots. A theoretical boundedness statement or a clear statement that this is empirical observation should be added.

Circularity Check

0 steps flagged

No load-bearing circularity: SALAAD's headline results are measured against external baselines; I-controller targets are explicit user-set controls, and remaining concerns (undisclosed rho constant, unspecified per-budget kappa, unproven boundedness claim) are transparency/robustness gaps, not circular steps.

full rationale

Walk of the derivation chain: the load-bearing outputs are (a) Table 1 perplexity/parameter counts vs external baselines (LOST, SLTrain, GaLore, LORO, CoLA) on C4, with LOST numbers quoted from Li et al. 2025; (b) Fig. 2 elastic-deployment curves; (c) the rho scaling law (Eq. 7) claimed to transfer from 60M/130M to 350M/1B; and (d) the embedding-layer SLR characterization. None reduces to its inputs by construction. The I-controller does drive alpha,beta via alpha <- alpha + rho(Gamma - Gamma_hat)Delta_alpha, beta <- beta + rho(Upsilon - Upsilon_hat)Delta_beta, so steady-state rank/density are pulled toward user-set targets Gamma_hat=0.15, Upsilon_hat=0.05, which the paper explicitly labels as user-designed deployment objectives ('designed by the user based on deployment requirements'). The L+S parameter counts are therefore controlled outcomes, honestly framed as explicit control ('enabling explicit control over the evolution of effective model capacity'), not disguised predictions; moreover achieved values deviate from setpoints (Fig. 1b: ~21% rank, ~12% density vs 15%/5% targets), so Table 1 L+S numbers are measured, not pure setpoint echoes. Perplexities are held-out measurements, and the baselines are external, so the central empirical claims are benchmarked rather than self-referential. The rho scaling law is an out-of-sample extrapolation test: the proportionality constant is tuned on 60M/130M and fixed for larger models, so 350M/1B results are genuine tests. Two flagged gaps: (i) 'the discrepancy |X - X_hat|_F remains bounded during training (see Appendix E)' is asserted as a guarantee but Appendix E only shows empirical curves — an omitted proof; (ii) the fitted rho constant is never reported while Table 8/Appendix G show PPL swinging from 21.7 to 30.0 over rho in [1e-8,1e-6] at 130M — a steep-sensitivity, reproducibility risk. Fig. 2 omits per-point kappa values and error bars, so the smooth curve may reflect oracle kappa selection (Fig. 3 sweeps kappa per budget); this is a methodology/selection-transparency concern about a measured curve, not a derivation that equals its input. The paper also discloses that SALAAD is trained in float32 while baselines use bfloat16, a fairness caveat it acknowledges. Self-citations (PoLAR with co-authors Zhang/Li/He; Ouyang et al. with He) appear only in background lists and are not load-bearing. No claimed step is equivalent by construction to its own input.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The ledger shows the paper's structural results are largely target-driven: the user sets Gamma_hat=0.15 and Upsilon_hat=0.05 and the I-controller forces the network toward them, so the 'compression' observed in L+S is partly a control outcome. The genuinely free scientific content is the empirical PPL behavior and the elastic-deployment comparison, which are governed by the undisclosed rho constant. No speculative entities are introduced.

free parameters (8)
  • rho proportionality constant (Eq. 7) = not reported
    'We tune rho on 60M and 130M models to determine the proportionality constant, and then fix it when scaling to larger models' (Section 5.1); its value never appears, yet 350M/1B results depend on it; Table 8 shows roughly 40% PPL sensitivity across rho in [1e-8, 1e-6].
  • Target effective rank ratio Gamma_hat (energy coverage gamma=0.999) = 0.15 for all blocks
    User-set global target (Section 5.1) that the I-controller drives Gamma^gamma_L toward; determines the reported L+S rank structure by construction.
  • Target density Upsilon_hat = 0.05 for all blocks
    User-set target driving sparsity of S (Section 5.1); Fig 1b shows achieved embedding density approx 0.12, deviating from target.
  • I-controller step sizes Delta_alpha, Delta_beta = Delta_alpha ~ 1e-1, Delta_beta ~ 1e-3 (orders of magnitude)
    Given only as orders of magnitude in Section 5.1; Table 2 shows PPL varies 18.25 to 18.97 (X) and 17.97 to 19.07 (L+S) as Delta_alpha moves 0.08 to 0.2.
  • gamma energy coverage = 0.999
    Chosen 'for numerical stability' in Definition 4.1; defines the effective rank ratio used by the controller and all reported rank ratios.
  • HPA allocation ratio kappa = 0.7 / 0.6 / 0.6 / 0.8 (60M/130M/350M/1B)
    Swept post-hoc per budget (Section 5.2, Fig 3) and the best value per scale is reported in Table 1; the reported eL+eS row is the selected optimum of this sweep.
  • ADMM iteration counts K, J = K=40, J=1
    Algorithm 1 / Appendix B; J=1 is justified heuristically; the boundedness and stability claims depend on these choices.
  • rho in ablation tables = 5e-8 to 1e-6
    The rho actually used per scale in Table 1 is implied by the fitted constant but never stated; only ablation rho values are given.
axioms (5)
  • ad hoc to paper Two-stage stochastic ADMM with J=1 second-stage update preserves the boundedness/proximity properties of classical ADMM, so |X-(L+S)|_F stays bounded throughout training.
    Invoked in Section 4.1 as a guarantee ('remains bounded during training (see Appendix E)') but Appendix E supports it only with empirical plots; no convergence theorem is stated for the stochastic, single-proximal-step variant used in Algorithm 1.
  • ad hoc to paper Unit importance I(u) is proportional to magnitude |u|, and SLR components across all blocks are structurally homomorphic (scaled by shared global ratios).
    Stated explicitly in Section 4.3 as the basis for HPA; the importance proxy is asserted because I(u) in Eq. (8) is 'not directly computable', and the homomorphism assumption is never validated against block-wise sensitivity.
  • domain assumption Effective rank under 99.9% energy coverage (gamma=0.999) is the operational definition of rank for the controller and for the reported compression ratios.
    Definition 4.1; gamma is chosen 'for numerical stability'; the target Gamma_hat=0.15 is defined on this metric, so reported rank ratios are not true ranks.
  • domain assumption The penalty scaling rho proportional to 1/(N*sqrt(nm)) with weight Frobenius norms scaling as O(sqrt(nm)) holds across architectures.
    Eq. (7), Section 4.2; motivated empirically; the proportionality constant is fitted on 60M/130M and the form is assumed to transfer to 350M/1B.
  • standard math Standard convex-analysis machinery: nuclear norm and l1 norm as tractable surrogates for rank/sparsity; SVD and soft-thresholding proximal operators; ADMM update structure.
    Sections 3 and 4.1, Eqs (1)-(5); classical results (Candes et al. 2011; Lin et al. 2010; Hong et al. 2015) invoked as background.

pith-pipeline@v1.3.0-alltime-deepseek · 20252 in / 25784 out tokens · 267293 ms · 2026-08-03T05:51:37.921124+00:00 · methodology

0 comments
read the original abstract

Modern large language models are increasingly deployed under compute and memory constraints, making flexible control of model capacity a central challenge. While sparse and low-rank structures naturally trade off capacity and performance, existing approaches often rely on heuristic designs that ignore layer and matrix heterogeneity or require model-specific architectural modifications. We propose SALAAD, a plug-and-play framework applicable to different model architectures that induces sparse and low-rank structures during training. By formulating structured weight learning under an augmented Lagrangian framework and introducing an adaptive controller that dynamically balances the training loss and structural constraints, SALAAD preserves the stability of standard training dynamics while enabling explicit control over the evolution of effective model capacity during training. Experiments across model scales show that SALAAD substantially reduces memory consumption during deployment while achieving performance comparable to ad-hoc methods. Moreover, a single training run yields a continuous spectrum of model capacities, enabling smooth and elastic deployment across diverse memory budgets without the need for retraining.

Figures

Figures reproduced from arXiv: 2602.00942 by Bingcong Li, Hao Ma, Liang Zhang, Melanie Zeilinger, Melis Ilayda Bal, Michael Muehlebach, Niao He.

Figure 1
Figure 1. Figure 1: Comparison of SALAAD training with and without embedding layer inclusion on a LLaMA-based 350M model. (a) Training loss trajectories. (b) Convergence of effective rank ratio and density in the embedding layer. (c) Convergence behavior of a randomly selected Transformer block. (d) Singular value spectra of the learned low-rank components. Overall, the results indicate that including the embedding layer does… view at source ↗
Figure 2
Figure 2. Figure 2: Perplexity versus parameter count for SALAAD models under different parameter budgets, compared with vanilla models. All results are obtained by applying HPA strategy. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of allocation ratio κ on model performance under different parameter budgets for LLaMA-based (a) 130M model, (b) 350M model, and (c) 1B model. The gray region indicates the relatively stable range of optimal allocation ratio κ ⋆ across different budgets. vanilla models in perplexity across a wide range of param￾eter budgets. More importantly, the performance-capacity curves reveal a qualitative diff… view at source ↗
Figure 4
Figure 4. Figure 4: Post-hoc RPCA results on standard-trained models. (a) Results for the LLaMA 1B model. (b) Results for the LLaMA 3.2 3B model. For each model, representative layers from shallow, middle, and deep regions are selected, and for each layer the effective rank ratios and sparsity levels obtained after RPCA decomposition are reported for different matrix types. Across both the 1B (see Figure 4a) and 3B (see Figur… view at source ↗
Figure 5
Figure 5. Figure 5: Post-hoc RPCA results on SALAAD-trained LLaMA-based 1B model. For several representative layers from shallow, middle, and deep regions, the effective rank ratios and sparsity levels obtained after RPCA decomposition are compared with the ground-truth values from the original SLR components learned by SALAAD. Black and gray boxes denote the recovered effective rank ratios and sparsity levels, respectively, … view at source ↗
Figure 6
Figure 6. Figure 6: provides a visualization of how the HPA strategy operates across Transformer blocks. The core idea of HPA is to Q K V O Gate Up Down Q K V O Gate Up Down Q K V O Gate Up Down 0 50 100 ϕL = 50% { ϕS = 30% } · · · Block 1 Block 2 Block N Ratio (%) Rank Density Removed part [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Learning dynamics of SALAAD training across model scales. Each row corresponds to a different model size, from top to bottom: 60M, 130M, 350M, and 1B. For each model scale, the four panels report the same set of training diagnostics: training loss (a, e, i, m), average reconstruction error across all blocks (b, f, j, n), sparsity and effective rank ratio evolution of a randomly selected block (c, g, k, o),… view at source ↗
Figure 8
Figure 8. Figure 8: Effect of embedding layer inclusion in SALAAD training across model scales. The first row corresponds to the 60M model and the second row to the 130M model. (a, e) Training loss trajectories with and without embedding layer inclusion remain highly overlapping throughout training, including the late stage. (b, f) The embedding layer exhibits smooth and well-behaved convergence of effective rank and density.… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Low-Rank Adaptation Redux for Large Models

    cs.LG 2026-04 unverdicted novelty 3.0

    An overview revisits LoRA variants by categorizing advances in architectural design, efficient optimization, and applications while linking them to classical signal processing tools for principled fine-tuning.

Reference graph

Works this paper leans on

26 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D. et al. Language Models are Few-Shot Learners. InAd- vances in Neural Information Processing Systems, pp. 1877–1901,

    1901

  2. [6]

    MiniLLM: Knowl- edge Distillation of Large Language Models.arXiv, cs/2306.08543:1–23,

    Gu, Y ., Dong, L., Wei, F., Huang, M. MiniLLM: Knowl- edge Distillation of Large Language Models.arXiv, cs/2306.08543:1–23,

    Pith/arXiv arXiv

  3. [7]

    Convergence Analy- sis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems.arXiv, math/1410.1390: 1–29,

    Hong, M., Luo, Z.-Q., Razaviyayn, M. Convergence Analy- sis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems.arXiv, math/1410.1390: 1–29,

    Pith/arXiv arXiv

  4. [9]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y . et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv, cs/2106.09685:1–26,

    Pith/arXiv arXiv

  5. [10]

    9 SALAAD: Sparse And Low-Rank Adaptation via ADMM Li, J., Yin, L., Shen, L., Xu, J., Xu, L. et al. LOST: Low- rank and Sparse Pre-training for Large Language Models. arXiv, cs/2508.02668:1–14,

    Pith/arXiv arXiv

  6. [12]

    The Augmented Lagrange Multi- plier Method for Exact Recovery of Corrupted Low-Rank Matrices.arXiv preprint arXiv:1009.5055,

    Lin, Z., Chen, M., Ma, Y . The Augmented Lagrange Multi- plier Method for Exact Recovery of Corrupted Low-Rank Matrices.arXiv preprint arXiv:1009.5055,

    Pith/arXiv arXiv

  7. [14]

    Liu, Z., Zhang, R., Wang, Z., Yan, M., Yang, Z. et al. CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation.arXiv, cs/2502.10940:1–19,

    arXiv

  8. [15]

    Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R. et al. Large Language Models: A Survey.arXiv, cs/2402.06196:1–44,

    Pith/arXiv arXiv

  9. [16]

    GLU Variants Improve Transformer.arXiv, cs/2002.05202:1–5,

    Shazeer, N. GLU Variants Improve Transformer.arXiv, cs/2002.05202:1–5,

    Pith/arXiv arXiv 2002

  10. [17]

    Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y . et al. Mobile- BERT: A Compact Task-Agnostic BERT for Resource- Limited Devices.arXiv, cs/2004.02984:1–13,

    Pith/arXiv arXiv 2004

  11. [18]

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupati- raju, S. et al. Gemma: Open Models Based on Gemini Research and Technology.arXiv, cs/2403.08295:1–17, 2024a. Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C. et al. Gemma 2: Improving Open Language Models at a Practical Size.arXiv, cs/2408.00118:1–21, 2024b. Touvron, H., Lavril, T., Izaca...

    Pith/arXiv arXiv

  12. [19]

    Tseng, A., Chee, J., Sun, Q., Kuleshov, V ., Sa, C. D. QuIP#: Even Better LLM Quantization with Hadamard Incoher- ence and Lattice Codebooks.arXiv, cs/2402.04396:1–27,

    Pith/arXiv arXiv

  13. [20]

    Convergence of Bregman alter- nating direction method with multipliers for nonconvex composite problems.arXiv, math/1410.8625:1–17,

    Wang, F., Xu, Z., Xu, H.-K. Convergence of Bregman alter- nating direction method with multipliers for nonconvex composite problems.arXiv, math/1410.8625:1–17,

    Pith/arXiv arXiv

  14. [21]

    Structured Pruning Learns Compact and Accurate Models.arXiv, cs/2204.00408: 1–16,

    10 SALAAD: Sparse And Low-Rank Adaptation via ADMM Xia, M., Zhong, Z., Chen, D. Structured Pruning Learns Compact and Accurate Models.arXiv, cs/2204.00408: 1–16,

    Pith/arXiv arXiv

  15. [22]

    Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.arXiv, cs/2310.06694:1–25,

    Xia, M., Gao, T., Zeng, Z., Chen, D. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning.arXiv, cs/2310.06694:1–25,

    Pith/arXiv arXiv

  16. [24]

    X., Zhou, K., Li, J., Tang, T., Wang, X

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X. et al. A Survey of Large Language Models.arXiv, cs/2303.18223: 1–144,

    Pith/arXiv arXiv

  17. [26]

    Limitations of Post-hoc Sparse and Low-Rank Decomposition In this section, we examine the limitations of post-hoc SLR decomposition via RPCA

    11 SALAAD: Sparse And Low-Rank Adaptation via ADMM A. Limitations of Post-hoc Sparse and Low-Rank Decomposition In this section, we examine the limitations of post-hoc SLR decomposition via RPCA. We show that, without SLR-aware training, weight matrices learned by standard optimization do not admit sufficiently structured SLR decompositions, rendering pos...

    2011

  18. [2010]

    PoLAR: Polar- Decomposed Low-Rank Adapter Representation.arXiv preprint arXiv:2506.03133,

    Lion, K., Zhang, L., Li, B., He, N. PoLAR: Polar- Decomposed Low-Rank Adapter Representation.arXiv preprint arXiv:2506.03133,

    arXiv

  19. [2015]

    Hsu, Y .-C., Hua, T., Chang, S., Lou, Q., Shen, Y . et al. Language model compression with weighted low-rank factorization.arXiv, cs/2207.00112:1–15,

    Pith/arXiv arXiv

  20. [2019]

    Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L. et al. LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. InFindings of the As- sociation for Computational Linguistics, pp. 3013–3026, 2024a. Zhang, Y ., Zhang, X., Ananthanarayanan, G., Iyer, A., Shu, Y . et al. Vulcan: Automatic Query Planning for Live ML Analytics. InUSENIX Sym...

    Pith/arXiv arXiv

  21. [2020]

    Cai, R., Muralidharan, S., Heinrich, G., Yin, H., Wang, Z. et al. Flextron: Many-in-One Flexible Large Language Model.arXiv, cs/2406.10260:1–15,

    Pith/arXiv arXiv

  22. [2021]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J. et al. DeepSeek-R1: Incentivizing Reasoning Capa- bility in LLMs via Reinforcement Learning.arXiv, cs/2501.12948:1–22, 2025a. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B. et al. DeepSeek-V3 Technical Report.arXiv, cs/2412.19437: 1–53, 2025b. Dettmers, T., Lewis, M., Belkada, Y ., Zettlemoyer, L. GPT...

    Pith/arXiv arXiv

  23. [2022]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A. et al. The Llama 3 Herd of Models.arXiv, cs/2407.21783: 1–92,

    Pith/arXiv arXiv

  24. [2023]

    ReLoRA: High-Rank Training Through Low-Rank Up- dates.arXiv, cs/2307.05695:1–16,

    Lialin, V ., Shivagunde, N., Muckatira, S., Rumshisky, A. ReLoRA: High-Rank Training Through Low-Rank Up- dates.arXiv, cs/2307.05695:1–16,

    Pith/arXiv arXiv

  25. [2024]

    Egiazarian, V ., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A. et al. Extreme Compression of Large Language Models via Additive Quantization.arXiv, cs/2401.06118:1–20,

    Pith/arXiv arXiv

  26. [2025]

    Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y . et al. A Comprehen- sive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT.arXiv, cs/2302.09419:1–99,

    Pith/arXiv arXiv