Pith. sign in

REVIEW 5 major objections 6 minor 29 references

SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a differentiable soft-threshold on singular values lets each linear layer learn its own rank during fine-tuning, so that compressing a language model to half its parameters costs about a 1% average accuracy drop and…

desk verdict The adaptive-rank idea is real and the BERT results are credible, but the GPT-2/TinyLlama baseline omission makes the headline comparison unverifiable as written. read the letter →

arxiv 2411.10543 v1 pith:QDJQVOWO submitted 2024-11-15 cs.LG cs.CL

Priyansh Bhatnagar , Linfeng Wen , Mingu Kang This is my paper · ORCID
classification cs.LGcs.CL
keywords low-rankapproximationsingularvaluedecompositionlearnablethresholdsoftthresholdingmodelcompressionfine-tuninglanguagemodelsefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The method compresses a pre-trained language model by replacing each large linear weight with its SVD factorization and letting a single learnable threshold per layer decide which singular values survive. The threshold is optimized during fine-tuning through a differentiable soft-threshold function, so the network itself chooses each layer's rank under a compression loss. At a 50 percent reduction in total parameters, the paper reports 1.33 times to 1.72 times inference speed-ups, 33 to 44 percent memory savings, and roughly 1 percent average accuracy loss on GLUE for BERT. The central comparison is against static SVD with the same rank across layers: adaptive ranks give GPT-2 perplexity 19.6 versus 32.1 and TinyLlama 15.1 versus 20.4 at 50 percent compression. The paper's conclusion is that per-layer rank allocation, not the SVD itself, drives the gain.

What carries the argument

The machinery is the soft-thresholded SVD layer: each linear weight $W$ is decomposed once into $U\Sigma V^T$, and the singular-value matrix is passed through $\mathrm{Th}_s(x)$, which keeps values above the learnable cutoff roughly as $x\tanh(s(x-\alpha))$ and sends values below it to zero. A per-layer scalar $\alpha_i$ is trained together with the model under total loss $L_{\text{acc}}+\gamma\sum_i e^{-\alpha_i}$, so the exponential compression term gives a large push early and nearly vanishes once a threshold is large. When the parameter count reaches the target, $\alpha_i$ is frozen; at the end, $V\cdot \mathrm{Th}_s(\Sigma)$ is merged into one matrix, so inference stores two smaller matrices instead of three. This makes rank selection a continuous, learned part of fine-tuning rather than a post-hoc static choice.

What would settle it

Run the same fine-tuning on a small model with the compression loss omitted and the thresholds replaced by random fixed values; if the learned thresholds still match the best brute-force rank assignment at the same parameter budget, the gradient signal is not what drives the result. A more direct version is to enumerate integer ranks around each learned value on the validation set; if a nearby hand-picked assignment beats the learned model by a large margin across seeds, the soft threshold is not finding the true trade-off.

Watch

Extended reading notes

Core claim

The paper's central claim is that the optimal low-rank approximation of a language model is not a single rank applied to every layer but a per-layer rank chosen by the model itself during fine-tuning. By substituting each linear layer with $W \approx U\,\mathrm{Th}_s(\Sigma)\,V^T$, where $\mathrm{Th}_s$ smoothly clips singular values below a learnable threshold $\alpha_i$, the training loss can push each layer's threshold up until the target parameter budget is met. Because the threshold is differentiable, gradient descent replaces exhaustive search over rank combinations. The evidence is the consistent gap over static truncation: SoftBERT reaches 90.7 F1 on MRPC versus 86.4 for SVD-BERT at the same 50% encoder compression, and the generative models show perplexity reductions of 12.5 points (GPT-2) and 5.3 points (TinyLlama) at CR 0.50. The paper also reports near-parity for SoftMamba, with a 1.2% average task degradation at 50% compression of the projection layers.

Load-bearing premise

The load-bearing premise is that the smooth soft-threshold function provides gradients that faithfully represent the true accuracy-versus-compression trade-off of each layer; if those gradients do not track the effect of dropping singular values, the learned thresholds are arbitrary and the gain over static SVD would not reproduce.

Editorial extensions

If this is right

  • If the central claim holds, static SVD compression of transformers should be replaced by adaptive rank allocation whenever fine-tuning is possible, because the same parameter budget buys substantially lower perplexity.
  • The method extends beyond attention models: Mamba's dominant projection layers can be halved with only about 1.2% average task degradation, so state-space language models are viable targets for this compression.
  • The 1.33–1.72× speed-up and 33–44% memory savings at CR 0.50 mean models compressed this way fit into tighter device budgets without task-specific distillation or quantization.
  • The learned rank distributions show earlier layers keep higher ranks, implying layer position is a real factor in rank sensitivity, not just matrix shape.
  • At higher compression ratios the gap over static SVD widens, so adaptive rank allocation is especially valuable under aggressive compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper leaves the sharpness parameter $s$ unspecified and ties the compression loss only indirectly to the parameter count, one direct test would be to read off the final rank each threshold produces and compare it with a brute-force per-layer rank search at the same parameter budget.
  • The reported gains come from fine-tuning on each target task; a natural extension is task-agnostic or few-shot compression, where the thresholds might need a calibration set to transfer.
  • The SVD-at-initialization cost could be reduced with randomized or incremental SVD, and if the gains persist the approach becomes practical for billion-parameter models without the fine-tuning memory overhead the paper reports.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes SoftLM, a post-training compression method for transformer and state-space language models. Each linear layer is replaced by an SVD factorization UΣV^T, and a differentiable soft-threshold function with a per-layer learnable threshold α is applied to the diagonal entries of Σ; an exponential compression loss encourages α to grow so that the number of surviving singular values, and hence the rank, is chosen automatically during fine-tuning. The threshold is frozen once the target parameter budget is reached, and the compressed factors are merged at the end of fine-tuning. The method is evaluated on BERT (GLUE tasks), GPT-2 and TinyLlama (WikiText-2 perplexity), and Mamba (GLUE tasks), reporting that adaptive rank allocation outperforms static SVD at matched budgets and yields 1.33–1.72× inference speedups at 50% compression.

Significance. If the results hold, this is a useful contribution to low-rank model compression: layer-wise ranks are learned during fine-tuning rather than chosen by an external search, and the approach transfers across attention-based and state-space architectures. The consistent improvements over static SVD across four model families, the compression-ratio ablations, and the attempt to make thresholding differentiable are concrete strengths. The main caveats are that the comparison may conflate fine-tuning with adaptive ranking, that the SVD semantics are not preserved after unconstrained fine-tuning of U, V, and Σ, and that several hyperparameters controlling the threshold dynamics are not reported. With those points addressed, the work would be a solid systems-oriented compression result; as it stands, the central comparative claim is not fully established.

major comments (5)
  1. [§4.1 and §4.5, Eq. (5)] The fine-tuning loop optimizes U, Σ, and V without any orthogonality constraint. After the first AdamW step, U and V are no longer orthogonal, so the diagonal entries of Σ are not singular values of the current weight matrix; the soft-threshold operation is then a diagonal mask on a factorized matrix rather than a truncation of an SVD. Because the paper motivates the method as clipping singular values and compares against truncated SVD, this discrepancy should be resolved either by enforcing orthogonality (e.g., re-orthogonalizing or parameterizing on the Stiefel manifold) or by explicitly reframing the method as a gated low-rank factorization initialized from SVD.
  2. [§5.2, Tables 4 and 5] The static SVD baselines for GPT-2 and TinyLlama are not described as fine-tuned; the text states only that static SVD applies the same compression ratio across all blocks, whereas for BERT the SVD baseline is explicitly said to be fine-tuned. If SVD-GPT2 and SVD-TinyLlama are not fine-tuned, their worse perplexities (e.g., 32.1 versus 19.6 at CR 0.50 for GPT-2) reflect the lack of adaptation rather than the inferiority of a static rank assignment. Please specify the baseline protocol and, if possible, include a static SVD baseline that receives the same U, V, Σ fine-tuning budget as SoftGPT2 and SoftTinyLlama.
  3. [Eq. (6), §4.4, Table 6] The sharpness parameter s in Eq. (6) controls the gradient of the soft threshold with respect to α and hence the rank-selection dynamics, but its value or schedule is never reported. Relatedly, Table 6 lists ranges for the threshold learning rate and for γ (called "w") rather than the values used for each dataset and model. Since the learned thresholds are the core mechanism that determines each layer's rank, these omissions prevent reproduction and make it impossible to assess sensitivity of the reported results.
  4. [Abstract and §5.3] The abstract and execution-cost section claim a 50% reduction in total parameters, but the compression is applied only to the linear layers in the encoder/decoder (or to the in/out projections in Mamba), not to the whole model. For BERT the total size drops from 110M to 67M (about 39%), for Mamba from 130M to 87M (about 33%), and for GPT-2 from 345M to roughly 195M (about 43%); only Table 1's "w/o emb" row is a true 50% reduction. The reported speedups and memory savings should be restated on the same basis as the model totals.
  5. [Tables 1–5] All experimental tables report a single run per configuration, with no seeds, standard deviations, or statistical tests. In several places the reported advantage is small (e.g., SoftBERT G-Avg 82.6 versus DistilBERT 81.2, or SoftMamba 86.3 versus Mamba 87.5), so without variance information the reader cannot determine whether the observed differences are meaningful. Please report multiple seeds or otherwise justify that the comparisons are stable.
minor comments (6)
  1. [Throughout] There are typos and inconsistent notation: "1.33X" (Abstract) versus "1.33×" (Figure 8), "upto" (Section 1), "encoder/ decoder" (Abstract), "compression ration" (Section 5.2), and the regularizer is called w in Table 6 but γ in Eqs. (7) and (10).
  2. [Table 3] Table 3 compares against S4, but no S4 reference appears in the bibliography; please add it.
  3. [Algorithm 1] Algorithm 1 ends with "Ensure: #param in (U and V S) < #param in W", but the post-merge shapes of U and V S are not explained; clarify how the dimensions are derived from the thresholded Σ.
  4. [Figure 1] Figure 1(b) would benefit from a label explaining what "static" means (uniform rank across blocks) and whether the ranks shown are before or after fine-tuning.
  5. [§5.2, Table 1] The fine-tuned SVD-BERT baseline is not described in terms of training budget; please state whether it uses the same number of epochs, learning rate, and fine-tuning procedure as SoftBERT.
  6. [Appendix A.2] The appendix lists batch sizes and learning rates but not the number of training epochs or the stopping criterion for the compression phase; please add these details so the procedure is fully specified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SoftLM's rank-adaptive compression is an empirical method evaluated against external baselines, with no fitted quantity masquerading as a prediction.

full rationale

The paper's central claim is that a differentiable soft-thresholding mechanism learns per-layer ranks during fine-tuning and that the resulting compressed models outperform static-SVD baselines at matched parameter budgets. This claim is supported by experiments on BERT, GPT-2, TinyLlama, and Mamba against external references (DistilBERT, MiniLMv2, static SVD baselines, etc.) rather than by a derivation that assumes the conclusion. The learnable threshold alpha_i is optimized with a loss combining task accuracy and a compression regularizer, and the final performance is measured on the same benchmark tasks; this is standard fine-tuning practice, not a case of fitting a parameter to a target and then 'predicting' that target. There are no load-bearing self-citations or imported uniqueness theorems: the thresholding function in Eq. (6) is introduced as the authors' own differentiable relaxation, and no cited prior work is used to force the method's choice. The reviewer's concern that the GPT-2/TinyLlama static-SVD baselines may not have been fine-tuned is a legitimate experimental-fairness and reproducibility issue, but it is not circularity: it does not reduce an equation or result to its own input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard SVD fine-tuning assumptions plus a domain assumption that singular-value magnitude tracks task importance. A key ad hoc assumption is that the soft-threshold relaxation's gradients guide discrete rank selection; this is not proven and depends on an unreported sharpness parameter. No new physical or architectural entities are introduced.

free parameters (4)
  • alpha_i = not reported (shown as rank distributions in Appendix B)
    Each layer's learnable threshold is adapted during fine-tuning; this is the central mechanism of the method.
  • sharpness s = not reported
    Controls how closely the soft threshold approximates a hard threshold in Eq. (6); never specified in the experiments.
  • gamma (compression loss weight) = 0.001, 0.01, 0.1 (ranges only)
    Balancing factor in the total loss; tuned per task but per-task values are not reported.
  • threshold learning rate = 1e-2, 1e-3
    Learning rate for alpha updates; chosen from a range and not reported per task.
assumptions (4)
  • domain assumption Singular value magnitude correlates with task importance.
    Motivates thresholding small singular values; stated in Section 4.2.
  • domain assumption SVD of the pretrained weight matrix is a good initialization for fine-tuning compressed layers.
    The method initializes U, Sigma, V from SVD of pretrained W and fine-tunes them; no convergence analysis is given.
  • ad hoc to paper The soft-threshold relaxation provides informative gradients for discrete rank selection.
    Assumed in Sections 4.3 and 4.4; not proven, and it depends on the unreported sharpness parameter s.
  • domain assumption Standard fine-tuning of U, V, and Sigma can recover task accuracy after compression.
    The entire method relies on fine-tuning the decomposed matrices; no theoretical or empirical robustness analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism." pith.science (2026). https://pith.science/paper/QDJQVOWO

@misc{pith2026241110543,
  author       = {Pith},
  title        = {Pith review of: SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QDJQVOWO}},
  note         = {Machine review of arXiv:2411.10543}
}
read the original abstract

Extensive efforts have been made to boost the performance in the domain of language models by introducing various attention-based transformers. However, the inclusion of linear layers with large dimensions contributes to significant computational and memory overheads. The escalating computational demands of these models necessitate the development of various compression techniques to ensure their deployment on devices, particularly in resource-constrained environments. In this paper, we propose a novel compression methodology that dynamically determines the rank of each layer using a soft thresholding mechanism, which clips the singular values with a small magnitude in a differentiable form. This approach automates the decision-making process to identify the optimal degree of compression for each layer. We have successfully applied the proposed technique to attention-based architectures, including BERT for discriminative tasks and GPT2 and TinyLlama for generative tasks. Additionally, we have validated our method on Mamba, a recently proposed state-space model. Our experiments demonstrate that the proposed technique achieves a speed-up of 1.33X to 1.72X in the encoder/ decoder with a 50% reduction in total parameters.

Figures

Figures reproduced from arXiv: 2411.10543 by the authors.

Figure 1
Figure 1. Accuracy trend of low-rank approximation for BERT-Base with GLUE MRPC dataset. (a) accuracy drop [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SoftBERT encoder with N blocks, where WQ, WK, WV , Wproj in Multi-Head Attention and Wf c1 and Wf c2 in the Feed Forward Network are substituted with U, S and V. The module S employs the Soft Threshold function to clamp the singular values in Σ, enabling dynamic rank for each block adaptively in fine-tuning. 94%) of the parameters in each block whereas the number of parameters inside SSM is significantly less. Thus,… view at source ↗
Figure 3
Figure 3. Learnable threshold (α) for selecting top sin￾gular values in Σ; values below (α) are set to zero. This threshold traverses diagonally across the Σ ma￾trix during the training so that values in Σ below the threshold are replaced by zero. A higher threshold corresponds to a greater level of compression. Con￾sequently, the corresponding rows and columns of the orthogonal matrices U and V , respectively, are also negle… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: The magnitude of compression loss over the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Threshold functions. (a) conventional non [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Rank of each block after fine-tuning Soft [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Inference speed-up achieved by SoftLMs. Baseline(BERT) Baseline(Mamba) 91.3% 90.5% 89.5% 87.0% 89.3% 88.8% 88.1% 86.7% Compression Ratio (CR) [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Performance variation with respect to CR in [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: MPRC [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: CoLA [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 13
Figure 13. Figure 13: QNLI [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    https://pythonhosted.org/nvidia-ml-py/

    pyNVML: Python bindings to the NVIDIA Management Library . https://pythonhosted.org/nvidia-ml-py/. Accessed: 2024-5-3

    work page 2024

  4. [4]

    https://blog.salesforceairesearch.com

    The WikiText Long Term Dependency Language Modeling Dataset . https://blog.salesforceairesearch.com. Accessed: 2024-5-3

    work page 2024

  5. [5]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://arxiv.org/abs/2305.14314 Qlora: Efficient finetuning of quantized llms . Preprint, arXiv:2305.14314

    arXiv 2023

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  7. [7]

    Gene H Golub and Christian Reinsch. 1971. Singular value decomposition and least squares solutions. In Handbook for Automatic Computation: Volume II: Linear Algebra, pages 134--151. Springer

    1971

  8. [8]

    Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307

    arXiv 2020

Show all 29 references
  1. [9]

    Albert Gu and Tri Dao. 2024. https://arxiv.org/abs/2312.00752 Mamba: Linear-time sequence modeling with selective state spaces . Preprint, arXiv:2312.00752

    2024 arXiv

  2. [10]

    Song Han, Huizi Mao, and William J. Dally. 2016. https://arxiv.org/abs/1510.00149 Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding . Preprint, arXiv:1510.00149

    2016 arXiv

  3. [11]

    Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. https://arxiv.org/abs/1506.02626 Learning both weights and connections for efficient neural networks . Preprint, arXiv:1506.02626

    2015 arXiv

  4. [12]

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. https://arxiv.org/abs/2207.00112 Language model compression with weighted low-rank factorization . Preprint, arXiv:2207.00112

    2022 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  6. [14]

    Ting Hua, Xiao Li, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.621 Dynamic low-rank estimation for transformer-based language models . In Findings of the Association for Computational Linguistics: EMNLP 2023,...

    2023 doi

  7. [15]

    Matan Ben Noach and Yoav Goldberg. 2020. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural La...

    2020

  8. [16]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  9. [17]

    Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran

    Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. 2013. https://doi.org/10.1109/ICASSP.2013.6638949 Low-rank matrix factorization for deep neural network training with high-dimensional output targets . In 2013 IEEE International Conferenc...

    2013

  10. [18]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . Preprint, arXiv:1910.01108

    2020 arXiv

  11. [19]

    Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. 2022. https://arxiv.org/abs/2205.13571 Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations . Preprint, arXiv:2205.13571

    2022 arXiv

  12. [20]

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. https://arxiv.org/abs/1908.09355 Patient knowledge distillation for bert model compression . Preprint, arXiv:1908.09355

    2019 arXiv

  13. [21]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. https://arxiv.org/abs/1706.03762 Attention is all you need . Preprint, arXiv:1706.03762

    2023 arXiv

  14. [22]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 a . https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

    2019 arXiv

  15. [23]

    Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. 2019 b . https://doi.org/10.1109/CVPR.2019.00881 Haq: Hardware-aware automated quantization with mixed precision . In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8604--8612

    2019

  16. [24]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. https://arxiv.org/abs/2002.10957 Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers . Preprint, arXiv:2002.10957

    2020 arXiv

  17. [25]

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2024. https://arxiv.org/abs/2403.07378 Svd-llm: Truncation-aware singular value decomposition for large language model compression . Preprint, arXiv:2403.07378

    2024 arXiv

  18. [26]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2024. https://arxiv.org/abs/2312.05821 Asvd: Activation-aware singular value decomposition for compressing large language models . Preprint, arXiv:2312.05821

    2024 arXiv

  19. [27]

    Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36--39. IEEE

    2019

  20. [28]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . Preprint, arXiv:2401.02385

    2024 arXiv

  21. [29]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. https://arxiv.org/abs/2303.10512 Adalora: Adaptive budget allocation for parameter-efficient fine-tuning . Preprint, arXiv:2303.10512

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.