Pith. sign in

REVIEW 3 major objections 4 minor 27 references

This paper claims that LoRA fine-tuning can be made 50% sparse without losing accuracy by pruning only the frozen base weights and recovering the pruned information in a low-rank residual adapter.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:39 UTC pith:GHFTGZBV

load-bearing objection A useful sparse-LoRA engineering recipe is propped up by a false theorem; the empirical idea deserves a look, the theory does not. the 3 major comments →

arxiv 2601.16991 v3 pith:GHFTGZBV submitted 2026-01-08 cs.LG cs.AI

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models

Longteng Zhang , Sen Wu , Shuai Hou , Zhengyu Qing , Zhuo Zheng , Danning Ke , Qihong Lin , Qiang Wang
show 2 more authors
Shaohuai Shi Xiaowen Chu
This is my paper
classification cs.LG cs.AI
keywords SALRLoRApruningsparsitylow-rank adaptationlarge language modelstruncated SVDinference speedup
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's goal is to make LoRA fine-tuning of large language models genuinely sparse: not just pruning on paper, but a deployed model that is smaller and faster. Its central claim is that the right place to prune is the frozen base weight matrix, not the low-rank adapters and not the merged update—because a static mask on the base weights has the lowest mean-squared error among the options it analyzes. To keep accuracy, the pruned values are not thrown away; they are approximated by a low-rank residual adapter built from a truncated SVD, which provably reduces per-entry error by the factor (1 - r/min(d,k)). On Llama2-7B, Llama3-8B, and Mixtral-8x7B, the method reaches 50% sparsity while matching LoRA on GSM8K and MMLU, roughly halves the stored model, and delivers up to 1.7x faster inference. The practical story is that sparsity and low-rank adaptation can cooperate rather than compete.

Core claim

On its own terms, the paper establishes a design rule: for a LoRA-fine-tuned model, prune only the frozen base weights with a static magnitude mask, then use a rank-r truncated SVD of the sparse residual as an additional low-rank adapter that is fine-tuned together with the LoRA adapters. The paper proves (under Gaussian assumptions) that this static-base-only scheme has the smallest per-entry pruning MSE among static-base, dynamic-base, and dynamic-full-update schemes, and that adding the SVD residual lowers the per-entry MSE to (1 - r/min(d,k)) times the bare pruning error. It then shows how to make the sparse model actually compress: concatenate all low-rank adapters along the rank dimens

What carries the argument

The load-bearing mechanism is the sparsity-preservation residual adapter: pruned entries of the frozen base weight are collected in a sparse residual matrix E, and a truncated SVD of E gives a compact rank-r correction that is trained alongside the usual LoRA matrices. Because this residual adapter is concatenated with the LoRA adapters into single Acat and Bcat matrices, all adapter updates run as one fused GEMM. On the storage side, a bitmap marks kept entries, a 256-entry lookup table reconstructs byte-aligned sparse blocks, and a ring-buffered two-stage pipeline overlaps bitmap decoding with tensor-core GEMM, so the sparsity translates to real compression and speed rather than just theor

Load-bearing premise

The whole ranking argument assumes the entries of the optimal low-rank update are independent Gaussian and independent of the frozen base weights; in a real fine-tuned LoRA the update is correlated and non-Gaussian, so the proof that static base-weight pruning is best may not transfer to actual adapters.

What would settle it

Numerically compare the three pruning rules after an actual LoRA fine-tune: for a trained W0 and AB on a small model, mask the smallest |W0| entries (Method 1), mask where |W0+AB| is small then prune W0 (Method 2), and mask |W0+AB| directly (Method 3); if Method 2's per-entry MSE is not the worst of the three at p=0.5, Theorem 2's ranking fails. Alternatively, recompute E2-E3 symbolically: the paper derives it as 2 sigma^2 tau^2/(sigma^2+tau^2) tp phi(tp), which does not follow from the previous line; a reader can check whether the claimed E1<=E3<=E2 actually holds numerically for sigma=tau an

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At 50% static sparsity, SALR keeps GSM8K and MMLU accuracy within about one point of dense LoRA on Llama2-7B, Llama3-8B, and Mixtral-8x7B, while LoSA drops sharply at the same sparsity.
  • The stored model is roughly 2x smaller at 50% sparsity, and combining 20% sparsity with NF4 quantization gives about 5x size reduction with minimal accuracy loss on DeepSeek-V2-Lite and Mixtral-8x7B.
  • 2:4 semi-structured sparsity yields 1.7x inference speedup over dense LoRA on an RTX4090, at 78.9 vs 79.5 GSM8K accuracy.
  • Fusing all low-rank adapters into one concatenated GEMM reduces kernel-launch overhead; fine-tuning memory drops about 30% and sustained TFLOPS rise about 20% relative to the LoSA baseline.
  • Sparsity levels between 10% and 50% all match dense LoRA on GSM8K for Llama3-8B, with 30% sparsity slightly ahead, suggesting a regularizing effect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • SALR's recipe does not depend on a particular LoRA initialization or rank-allocation scheme, so it could be layered on adapters that adaptively grow or prune ranks; the residual SVD rank r could then be chosen by the residual spectrum rather than fixed.
  • The bitmap plus two-stage decode+GEMM pipeline is not specific to GPUs; testing it on CPU or NPU with lower GEMM throughput would show whether the decoding overlap stays hidden or becomes the bottleneck.
  • The per-entry MSE bound predicts that accuracy loss should shrink as r grows; a direct experiment varying r at fixed 50% sparsity would quantify how much of the residual's tail spectrum actually matters for downstream accuracy.
  • Because the paper's Theorem 2 comparison rests on a Gaussian model and a symbolic step in the E2-E3 calculation, the static-base-only ordering should be re-checked empirically before treating it as general; the method's empirical success does not require the theorem to hold in all real cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SALR, a fine-tuning method that applies a static magnitude-based mask to the frozen base weights of a LoRA-fine-tuned LLM and recovers the pruned residual with a trainable truncated-SVD low-rank adapter. It also contributes system-level designs: concatenation of multiple low-rank adapters into one GEMM and a bitmap-encoded two-stage decoding+GEMM pipeline for actual model-size compression. The authors claim a rigorous MSE framework (Theorems 1-4) showing that static pruning of W0 minimizes pruning error, that the SVD correction provably reduces per-entry MSE by (1-r/min(d,k)), and that their method matches LoRA accuracy at 50% sparsity with 2x compression and up to 1.7x inference speedup on several LLMs and benchmarks.

Significance. If the empirical claims hold, SALR is practically significant: it demonstrates LoRA-level accuracy at 50% sparsity with actual compression and speedup, and the system contributions (fused adapters, bitmap decoding, pipelined GEMM) are concrete and plausible. The empirical section covers multiple models and benchmarks with quantitative memory/throughput/accuracy numbers. However, the paper presents the theoretical MSE framework as its primary novelty, and that framework is currently unreliable: Theorem 2 is false as stated, and Theorem 4 overclaims optimality. The empirical contribution is still strong enough to warrant a carefully corrected revision, but the current theoretical foundation cannot be accepted without substantial rework.

major comments (3)
  1. [Preliminary, Theorem 2, Eqs. (1)-(3)] The claimed ordering E1≤E3≤E2 is false and the proof contains an algebraic error. With V^2=σ^2+τ^2, Eqs. (1)-(3) give E2−E3 = (σ^2τ^2/V^2)p + 2(σ^4/V^2 − V^2)Q(tp) = (τ^2/V^2)[σ^2 p − 2(2σ^2+τ^2)Q(tp)], not (σ^2τ^2/V^2)[p − 2Q(tp)] as claimed in the Comparison paragraph. For σ=τ=1, p=0.99 (tp=2.576, Q≈0.458), E2−E3≈−0.878, so E3≤E2 fails. Thus the theorem cannot serve as the stated rigorous foundation. It appears that E1≤E2 and E1≤E3 may still be provable; the authors should correct the theorem statement and proof accordingly.
  2. [Methodology, Theorem 4] The step-size η*=1/λmax(X^T X) is not the minimizer of the worst-case contraction factor. For the non-strongly-convex quadratic, the error recursion on ker(X^TX) has eigenvalue 1 for every η, so the worst-case contraction factor over the full space is 1 for all η∈[0,2/λmax]; over the range space the minimax step is 2/(λ_min^+ + λ_max), not 1/λmax unless X^TX is a scalar multiple of the identity. The proof itself acknowledges the first issue but then asserts an optimality that is not established. Please weaken the statement to a convergence guarantee and justify the heuristic choice separately.
  3. [Preliminary, Theorem 2 assumptions] Theorem 2 assumes Δ=(A*B*) entries are i.i.d. N(0,τ^2) and independent of W0. This is not validated for trained LoRA adapters: entries of products of optimized matrices are correlated and generally non-Gaussian, and independence from W0 is a strong structural assumption. Since this assumption underlies the design decision to prune only W0, the theory does not currently transfer to the empirical setting. Please either test the assumption on actual trained adapters (e.g., distribution and independence diagnostics) or explicitly present Theorem 2 as an idealized motivating model rather than as a proof of practical optimality.
minor comments (4)
  1. [Theorem 3 / Abstract] The inequality ∑_{i=r+1}^q σ_i^2 ≤ (1−r/q)∑σ_i^2 is a deterministic consequence of sorted singular values; the phrase 'worst case (uniformly distributed spectrum)' is misleading. Also, the abstract's 'provably reduces per-entry MSE by a factor' overstates what is an upper bound.
  2. [Table 2] On Llama3-8B, SALR reports MMLU 68.2 vs. LoRA 69.2; the text's 'matching' should be qualified. Similar wording appears in the conclusion.
  3. [Baselines / References] The main text lists 'DeepSparse (Kurtic et al. 2023)' while Table 2 and the references attribute DeepSparse to Agarwalla et al. 2024; the attribution should be made consistent.
  4. [Figure 3] The notation i_0.99^{LoSA} ≪ i_0.99^{SALR} and the phrase 'mark the cube' are unclear; please define the index and say 'curve'. No code release or reproducibility details are mentioned for the system pipeline.

Circularity Check

0 steps flagged

No significant circularity; one minor non-load-bearing self-citation.

full rationale

The derivation chain is self-contained rather than circular. Theorem 1 derives a closed-form Gaussian pruning MSE; Theorem 2 computes the three per-entry MSEs by direct integration under explicit normality assumptions (Eqs. 1-3); Theorem 3 bounds the truncated-SVD residual error using the external Eckart-Young theorem; Theorem 4 is a standard convexity/Lipschitz analysis. None of these steps fits a parameter to the quantity it later claims to predict: the theoretical quantities are derived from stated distributional assumptions, and the empirical GSM8K/MMLU results are evaluated on held-out external benchmarks against LoRA and other baselines. The only self-citation is Zhang et al. 2023a (LoRA-FA) in a background sentence about memory overhead; it is not load-bearing. The proof issue in Theorem 2 flagged by the reviewer is an algebraic-correctness concern, not a circularity concern: the claimed ranking E1<=E3<=E2, even if wrong, is not equivalent by construction to the model inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 1 invented entities

The theoretical claims depend on idealized Gaussian assumptions (i.i.d. entries for W0 and the low-rank update Δ), which are not true for real adapters. The central theorem also contains an algebraic error. The method introduces a new trainable low-rank residual component, which is empirically ablated but not theoretically guaranteed by the flawed proof.

free parameters (3)
  • adapter rank r (LoRA and SVD residual) = 64
    Manual hyperparameter; all experiments use rank 64. The claimed MSE reduction factor (1-r/min(d,k)) depends on r, and the reported accuracy match is conditional on this budget.
  • global sparsity p = 50% (2:4 structured), 20% in QSALR
    Target sparsity chosen for experiments. Theorem 2's ordering is stated for every p∈[0,1) but is false at p=0.99, so the theoretical guarantee does not cover all claimed operating points.
  • SVD residual learning rate η_SVD = ≈1/σ_max(X)²
    Estimated via power iterations on a mini-batch. This is an optimization schedule choice rather than a fitted accuracy constant, but it is required for stable residual updates and is not independently benchmarked.
axioms (5)
  • domain assumption W0 entries are i.i.d. N(0, σ²)
    Invoked in Theorems 1-3. Real trained LLM weight matrices have correlated, non-Gaussian entries, so the Gaussian MSE calculations are an idealized model.
  • ad hoc to paper Δ=(A*B*) entries are i.i.d. N(0, τ²) and independent of W0
    Invoked in Theorem 2. The product of two low-rank matrices has dependent, non-Gaussian entries, making this assumption unrealistic and load-bearing for the theorem's ranking.
  • standard math Eckart-Young theorem
    Used in Theorem 3 to express the optimal rank-r approximation error of the residual matrix E.
  • standard math Sorted-spectrum tail bound: sum_{i=r+1}^q σ_i² ≤ (1-r/q) sum_{i=1}^q σ_i² for nonincreasing σ_i
    Used in Theorem 3 to convert the Eckart-Young tail sum into the claimed multiplicative factor. The bound holds for sorted nonnegative sequences, though the paper mislabels it as a 'worst-case uniform spectrum' case.
  • ad hoc to paper η*=1/λ_max(X^T X) minimizes worst-case contraction for gradient descent on L(M)=½||XM-R||²_F
    Theorem 4 and its proof. For rank-deficient X, directions in the null space have contraction factor 1, so no uniform contraction exists; the paper's own proof acknowledges this but still claims an optimal step size.
invented entities (1)
  • trainable SVD residual adapter M independent evidence
    purpose: Captures information lost when frozen base weights are pruned; initialized from truncated SVD of the residual, then fine-tuned alongside LoRA adapters.
    Not a physical entity. Its contribution is empirically tested via the frozen-vs-trainable residual ablation (Table 5) and it is trained on downstream data, so it has a falsifiable performance handle.

pith-pipeline@v1.3.0-alltime-deepseek · 15726 in / 19994 out tokens · 188695 ms · 2026-08-03T11:39:46.492745+00:00 · methodology

0 comments
read the original abstract

Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs. Magnitude-based pruning can yield sparse models but typically degrades LoRA's performance when applied naively. In this paper, we introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework. We prove that statically pruning only the frozen base weights minimizes the pruning error bound, and we recover the discarded residual information via a truncated-SVD low-rank adapter, which provably reduces per-entry MSE by a factor of $(1 - r/\min(d,k))$. To maximize hardware efficiency, we fuse multiple low-rank adapters into a single concatenated GEMM, and we adopt a bitmap-based encoding with a two-stage pipelined decoding + GEMM design to achieve true model compression and speedup. Empirically, SALR attains 50\% sparsity on various LLMs while matching the performance of LoRA on GSM8K and MMLU, reduces model size by $2\times$, and delivers up to a $1.7\times$ inference speedup.

Figures

Figures reproduced from arXiv: 2601.16991 by Danning Ke, Longteng Zhang, Qiang Wang, Qihong Lin, Sen Wu, Shaohuai Shi, Shuai Hou, Xiaowen Chu, Zhengyu Qing, Zhuo Zheng.

Figure 1
Figure 1. Figure 1: Memory-accuracy trade-off on the GSM8K (Cobbe [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SALR. (a) SALR first prunes the base model, resulting in a pruned module and a corresponding residual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Normalized cumulative singular-value energy spec [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 26 linked inside Pith

  1. [2]

    arXiv:2406.01775

    OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models. arXiv:2406.01775. Cobbe, K.; Kosaraju, V .; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J

    Pith/arXiv arXiv

  2. [6]

    arXiv:2407.21783

    The Llama 3 Herd of Models. arXiv:2407.21783. Eckart, C.; and Young, G

    Pith/arXiv arXiv

  3. [7]

    arXiv preprint arXiv:2301.00774

    SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. arXiv preprint arXiv:2301.00774. Guo, S.; Xu, J.; Zhang, L. L.; and Yang, M

    Pith/arXiv arXiv

  4. [8]

    arXiv:2310.05015

    Com- presso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models. arXiv:2310.05015. Han, S.; Mao, H.; and Dally, W. J

    Pith/arXiv arXiv

  5. [10]

    Huang, W.; Zhang, Y .; Zheng, X.; Liu, Y .; Lin, J.; Yao, Y .; and Ji, R

    LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Lan- guage Models.arXiv preprint arXiv:2304.01933. Huang, W.; Zhang, Y .; Zheng, X.; Liu, Y .; Lin, J.; Yao, Y .; and Ji, R

    Pith/arXiv arXiv

  6. [11]

    arXiv:2502.14816

    Dynamic Low-Rank Sparse Adaptation for Large Language Models. arXiv:2502.14816. Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D. S.; de las Casas, D.; Hanna, E. B.; Bressand, F.; Lengyel, G.; Bour, G.; Lample, G.; Lavaud, L. R.; Saulnier, L.; Lachaux, M.-A.; Stock, P.; Sub- ramanian, S.; Yang, S.; Antoniak, S.; Sc...

    Pith/arXiv arXiv

  7. [12]

    arXiv:2401.04088

    Mixtral of Experts. arXiv:2401.04088. Khaki, S.; Li, X.; Guo, J.; Zhu, L.; Xu, C.; Plataniotis, K. N.; Yazdanbakhsh, A.; Keutzer, K.; Han, S.; and Liu, Z

    Pith/arXiv arXiv

  8. [13]

    arXiv:2506.16500

    SparseLoRA: Accelerating LLM Fine-Tuning with Contex- tual Sparsity. arXiv:2506.16500. Kurtic, E.; Kuznedelev, D.; Frantar, E.; Goin, M.; and Alis- tarh, D

    Pith/arXiv arXiv

  9. [14]

    arXiv:2310.06927

    Sparse Fine-tuning for Inference Acceleration of Large Language Models. arXiv:2310.06927. Lester, B.; Al-Rfou, R.; and Constant, N

    Pith/arXiv arXiv

  10. [15]

    arXiv:2104.08691

    The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691. Li, M.; Si, W. M.; Backes, M.; Zhang, Y .; and Wang, Y

    Pith/arXiv arXiv

  11. [16]

    arXiv:2501.01765

    SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation. arXiv:2501.01765. Li, X. L.; and Liang, P

    Pith/arXiv arXiv

  12. [17]

    Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y .; and Paul, S

    DoRA: Weight-Decomposed Low-Rank Adaptation.arXiv preprint arXiv:2402.09353. Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y .; and Paul, S

    Pith/arXiv arXiv

  13. [18]

    arXiv:2404.02948

    PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. arXiv:2404.02948. Meta

    Pith/arXiv arXiv

  14. [19]

    https://ai.meta.com/blog/ llama-4-multimodal-intelligence/ [Accessed: 2025-04-05]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/ [Accessed: 2025-04-05]. OpenAI

    2025

  15. [20]

    arXiv:2303.08774

    GPT-4 Technical Report. arXiv:2303.08774. Sun, M.; Liu, Z.; Bair, A.; and Kolter, J. Z

    Pith/arXiv arXiv

  16. [21]

    arXiv:2306.11695

    A Simple and Effective Pruning Approach for Large Language Models. arXiv:2306.11695. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y .; Bashlykov, N.; Batra, S.; Bhargava, P.; and et al., S. B

    Pith/arXiv arXiv

  17. [22]

    arXiv:2307.09288

    Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Wang, S.; Yu, L.; and Li, J

    Pith/arXiv arXiv

  18. [23]

    arXiv:2407.05000

    LoRA-GA: Low-Rank Adaptation with Gradient Approximation. arXiv:2407.05000. Wang, Z.; Liang, J.; He, R.; Wang, Z.; and Tan, T

    Pith/arXiv arXiv

  19. [24]

    Xia, H.; Zheng, Z.; Li, Y .; Zhuang, D.; Zhou, Z.; Qiu, X.; Li, Y .; Lin, W.; and Song, S

    LoRA-Pro: Are Low-Rank Adapters Properly Optimized? arXiv:2407.18242. Xia, H.; Zheng, Z.; Li, Y .; Zhuang, D.; Zhou, Z.; Qiu, X.; Li, Y .; Lin, W.; and Song, S. L

    Pith/arXiv arXiv

  20. [25]

    arXiv:2309.10285

    Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. arXiv:2309.10285. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; Zheng, C.; Liu, D.; Zhou, F.; Huang, F.; Hu, F.; Ge, H.; Wei, H.; Lin, H.; Tang, J.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yan...

    Pith/arXiv arXiv

  21. [26]

    Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y .; Kwok, J

    Qwen3 Technical Report.arXiv preprint arXiv:2505.09388. Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y .; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W

    Pith/arXiv arXiv

  22. [27]

    Zhang, L.; Zhang, L.; Shi, S.; Chu, X.; and Li, B

    MetaMath: Boot- strap Your Own Mathematical Questions for Large Language Models.arXiv preprint arXiv:2309.12284. Zhang, L.; Zhang, L.; Shi, S.; Chu, X.; and Li, B. 2023a. LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. arXiv:2308.03303. Zhang, Q.; Chen, M.; Bukharin, A.; Karampatziakis, N.; He, P.; Cheng, Y .; Chen, W....

    Pith/arXiv arXiv

  23. [2016]

    arXiv:1510.00149

    Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv:1510.00149. Hu, E. J.; Shen, Y .; Wallis, P.; Allen-Zhu, Z.; Li, Y .; Wang, S.; Wang, L.; and Chen, W

    Pith/arXiv arXiv

  24. [2021]

    DeepSeek-AI

    Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168. DeepSeek-AI

    Pith/arXiv arXiv

  25. [2023]

    arXiv preprint arXiv:2305.14314

    QLoRA: Efficient Finetuning of Quantized LLMs. arXiv preprint arXiv:2305.14314. Dubey, A.; and et al., A. J

    Pith/arXiv arXiv

  26. [2024]

    arXiv:2405.03594

    Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment. arXiv:2405.03594. B¨uy¨ukaky¨uz, K

    Pith/arXiv arXiv

  27. [2025]

    arXiv:2501.12948

    DeepSeek-R1: Incentivizing Rea- soning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L

    Pith/arXiv arXiv