Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

All-in-One Tuning and Structural Pruning for Domain-Specific LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One stage beats two for pruning domain-specific LLMs, claims ATP

desk verdict One-stage pruning with LoRA tuning is a real idea with promising results, but the paper's core guarantee—that pruned LoRA groups vanish so direct removal is safe—is asserted, not shown. read the letter →

arxiv 2412.14426 v2 pith:CKEPFR2H submitted 2024-12-19 cs.CL cs.AI

Lei Lu , Zhepeng Wang , Runxue Bao , Mengbing Wang , Fangyi Li , Yawen Wu , Weiwen Jiang , Jie Xu
show 2 more authors
Yanzhi Wang Shangqian Gao
This is my paper · ORCID
classification cs.CLcs.AI
keywords structuralpruningLoRAdomain-specificLLMsone-stagetuningpruning-decisiongeneratorgrouplassoLLMcompressionparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard two-stage recipe for compressing domain-specific LLMs—first prune the pretrained model, then fine-tune it—is suboptimal because pruning decisions are frozen while the weights change. ATP instead runs pruning and LoRA-based fine-tuning together, with a trainable generator that revises pruning decisions as the LoRA weights evolve, and a sparsity regularizer that pushes the weights slated for removal toward zero. The result is a compact, domain-tuned model that needs no second-stage fine-tuning after pruning. On legal and healthcare benchmarks, ATP recovers up to 88% (LLaMA2-7B) and 91% (LLaMA3-8B) of dense-model performance at 40% parameter removal, beating the two-stage baselines LLM-Pruner and SliceGPT. If correct, this changes how domain-specific LLM compression should be done: the search for what to cut and the tuning of what remains belong in the same optimization loop.

What carries the argument

The load-bearing object is the pruning-decision generator G: a small Transformer-encoder network whose outputs are converted to binary decisions by Gumbel-Sigmoid sampling with a straight-through estimator. Alongside it, the LoRA-aware group-lasso regularizer (Eq. 5) forces the rows and columns of the LoRA matrices that correspond to pruned dimensions toward zero, so that direct removal of the pruned groups (Algorithm 1, line 13) is valid. The two forward passes (Eq. 3 for generator training, Eq. 4 for LoRA tuning) let the same decisions serve both the search for structure and the adaptation of weights.

What would settle it

Measure the ℓ2 norm of the LoRA weight groups (I−D_prev)W_a and W_b(I−D) at the positions the final pruning decision marks for removal, right after training. If these norms are not orders of magnitude smaller than the norms of retained groups, then the direct-removal step is not actually justified, and ATP's performance would be expected to drop if one instead re-ran fine-tuning with those groups kept.

Watch

Extended reading notes

Core claim

ATP establishes, on its own terms, that pruning decisions and LoRA tuning should be co-optimized rather than applied sequentially. The central mechanism is a trainable pruning-decision generator that outputs binary masks via Gumbel-Sigmoid sampling with a straight-through estimator, coupled with two LoRA-aware forward passes: one that simulates the pruned model for training the generator, and one that masks only the pretrained weights so the LoRA parameters stay fully trainable. A group-lasso term on the LoRA weights at the pruned positions drives those groups toward zero, so at the end the pruned structures can be removed directly without further fine-tuning. Experimentally, ATP achieves the best relative performance among structural-pruning methods on HealthCare and Legal tasks, with only minor exceptions on label-prediction tasks such as MedNLI and PubMedQA.

Load-bearing premise

The claim that after ATP ends, the LoRA weights in pruned positions are small enough that removing them directly causes no meaningful performance loss; the paper does not quantitatively verify the magnitude of those removed weights.

Editorial extensions

If this is right

  • Domain-specific LLM compression can skip the separate recovery fine-tuning stage, saving compute and simplifying the deployment pipeline.
  • Pruning decisions should be treated as part of the optimization, not a precomputed artifact, whenever the weights are going to be updated afterward.
  • The 40–50% sparsity range becomes a practical operating point for legal and healthcare models, with relative performance of 75–91% of the dense model.
  • The framework transfers to any transformer whose linear projections can be grouped, since the decisions only select rows and columns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same joint search-and-tune idea could extend to other parameter-efficient adapters (e.g., DoRA, prefix tuning) or to full fine-tuning, as the paper itself sketches; the key requirement is that the adapter has identifiable groups to regularize.
  • Because the generator's decisions shift by 20–55% when tuning is added (Fig. 3), this suggests that frozen-decision pruning generally leaves performance on the table for any fine-tuned deployment, not only domain-specific ones.
  • A testable extension: run ATP with the calibration set replaced by random samples from the general domain to see whether the decision evolution is driven by domain alignment or just by weight movement during tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ATP, a one-stage method that jointly performs structured pruning and LoRA-based fine-tuning of domain-specific LLMs. A trainable pruning-decision generator G produces per-layer binary masks for attention and MLP dimensions, and the model is trained with a LoRA-aware masked forward pass plus group-lasso regularization on LoRA weights, so that pruned groups can be removed at the end without further tuning. Experiments on LLaMA2-7B and LLaMA3-8B in healthcare and legal domains compare ATP against LLM-Pruner and SliceGPT, reporting perplexity, NLI accuracy, QA macro-F1, summarization ROUGE scores, and a relative performance metric. The central claim is that dynamic one-stage pruning beats the two-stage pipeline, recovering 88% and 91% of dense-model performance at 40% sparsity on LLaMA2-7B and LLaMA3-8B, respectively.

Significance. ATP targets a genuine limitation of two-stage pruning pipelines: pruning decisions are made on pretrained weights and then frozen during fine-tuning, even though weight updates can change which substructures are important. The one-stage formulation with a trainable decision generator and LoRA-aware sparsity regularization is well motivated and potentially useful for deploying domain-adapted LLMs under limited data. The paper provides a complete algorithm, uses held-out evaluation corpora (HarrisonTextBook and MultiLegalPile) that are explicitly excluded from training and calibration, and includes sparsity sensitivity experiments and case studies. If the direct-removal mechanism is quantitatively validated, ATP would be a strong baseline for domain-specific LLM compression. At present, however, the key mechanism that makes the one-stage approach work is asserted rather than demonstrated, and the experimental comparison lacks uncertainty quantification.

major comments (4)
  1. [§3.4–3.5, Eq. (5), Algorithm 1 line 13] The guarantee that pruned LoRA groups can be directly removed is asserted but not quantitatively verified. Eq. (5) penalizes rows of (I−Dprev)Wa and columns of Wb(I−D); Section 4.3 and Figure 5(b) show loss curves, but the paper never reports the post-training L2 norms of these groups, the fraction of groups that actually converge to zero, or the final achieved structural sparsity. Also absent is an ablation comparing the masked model used during training (forward pass fL, Eq. (4)) with the physically pruned model produced at line 13 of Algorithm 1. Without such evidence, the reader cannot tell whether the evaluated compressed model is the same function as the trained model, which is the defining premise of the method.
  2. [§3.5, Eqs. (6)–(7)] The forward-pass cross-references are inconsistent. Eq. (6) states that fG(sc; dall) applies Eq. (4) to all projections, and Eq. (7) states that fL(st; dall) applies Eq. (3), but Eq. (3) defines fG and Eq. (4) defines fL. If implemented literally, the generator would be trained with the LoRA-tuning forward pass and the LoRA weights would be updated with the generator forward pass, swapping the roles described in Section 3.4 and in Algorithm 2. The authors should correct these references and state explicitly, for each loss, which forward pass is used.
  3. [Appendix A.2, Table 1] All sampled metrics are averaged over three evaluation runs, but Table 1 reports only point estimates with no standard deviations or confidence intervals. On several rows the differences between ATP and the better baseline are small or reversed on individual tasks (e.g., Table 1, LLaMA2-7B p=0.4, MedNLI accuracy 71.52 vs 72.29 for SliceGPT; LLaMA3-8B p=0.5, MedNLI accuracy 68.57 vs 75.04 for SliceGPT), so the headline claim that ATP greatly outperforms both baselines is not yet supported by uncertainty-aware comparisons. Reporting variances or significance tests is needed to support the comparative claim.
  4. [Appendix A.4] The reported LLM-Pruner numbers diverge substantially from the values reported by D-Pruner (e.g., perplexity 13.67 vs 44.56 for HealthCare at p=0.5 on LLaMA2-7B). The authors attribute this to different loss modeling and hyperparameters, but no sensitivity analysis is provided to quantify how the baselines' scores change under the altered loss modeling or to justify that the modified baselines still represent the original methods fairly. Because the main comparative claim rests on these numbers, the divergence needs to be quantified rather than explained by assertion.
minor comments (6)
  1. [Algorithm 1 / §3.5] Algorithm 1 uses Dc for the calibration dataset while Section 3.5 refers to a small calibration dataset Ds; please use one symbol consistently.
  2. [§3.5, Eq. (6)] In Eq. (6), fG(dc; dall) uses a semicolon while fG is defined as a function of X in Eq. (3); the argument ordering should be aligned for clarity.
  3. [Appendix A.1, Eq. (8)] Eq. (8) is not the standard Gumbel-Sigmoid parameterization; the role of the offset base b inside the sigmoid should be explained, and the chosen temperature and offset values (T=0.4, b=3) should be justified in the hyperparameter discussion.
  4. [§4.1] The phrase 'state-of-the-art two-stage pruning methods' is stronger than the evidence: only LLM-Pruner and SliceGPT are compared, while several methods cited in Related Work (e.g., LLM-Surgeon, DisP-LLM) are not evaluated.
  5. [Table 1] Model sizes should be reported with a note on what is counted (e.g., embeddings included or excluded), since the ATP sizes at p=0.5 are not exactly 50% of the dense sizes and this is not explained.
  6. [Appendix A.5] The periodic pattern in LLM_G is attributed to itertools.cycle over the calibration dataloader; this means calibration batches are always seen in the same order, which may bias G updates, and the authors should briefly discuss whether this affects the final pruning decisions.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ATP is evaluated on held-out data; the only self-citation (D-Pruner setup) is not load-bearing.

full rationale

ATP's central claims are empirically evaluated against benchmarks that are not used in either training or calibration. The pruning-decision generator G is optimized on Dc (Eq. 6, Algorithm 2) and the LoRA weights on Dt (Eq. 7, Algorithm 1); Appendix A.2 explicitly states: 'We DO NOT include any samples from HarrisionTextBook ... and MultiLegalPile ... in either Dt or Dc.' The task metrics (MedNLI, PubMedQA, HQS, BillSum) are reported on test splits distinct from the 15k training samples and 1.3k calibration samples. The relative performance metric (Eq. 9) is a ratio of independently measured pruned-model scores to dense-model scores, not a fitted parameter renamed as a prediction. The LoRA-aware group lasso (Eq. 5) is a regularization penalty, and the claim that pruned LoRA groups approach zero is supported only by loss curves (Fig. 5b); the absence of quantitative norms is a completeness limitation, not circularity, and the paper itself discloses training-dynamics artifacts (Appendix A.5). The one self-citation of note is Zhang et al. (2024b) with overlapping author Runxue Bao, used only for experimental setup ('We primarily adopt the experimental setup from Zhang et al. (2024b)'); it is not load-bearing for the method's validity, and ATP is compared against external baselines LLM-Pruner and SliceGPT. No equation in the paper reduces to its inputs by construction, and no 'prediction' is equivalent to its fit.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the pruning-decision generator is a neural module, not a postulated phenomenon with independent falsifiable handles.

free parameters (5)
  • sparsity constraint coefficient alpha = 5
    Hand-chosen hyperparameter in Eq. 6 that balances the calibration loss and the sparsity constraint. No sensitivity analysis except Fig. 5(b).
  • group lasso coefficient beta = 0.3 (increased to 30 after Tend)
    Hand-chosen coefficient in Eq. 7 and Eq. 5. The increase to 100x after Tend is critical for the direct removal step, but its necessity is only shown via loss curves.
  • Gumbel-Sigmoid temperature T and offset base b = T=0.4, b=3
    Fixed hyperparameters for discrete decision generation (Appendix A.1, Eq. 8). Not fitted to data, but they control the hardness of decisions.
  • LoRA rank r = 8
    Standard LoRA rank used for both ATP and baselines. Chosen by the authors, not justified by a sweep.
  • End of decision update step Tend = T/2
    The generator updates stop at half the total steps; after that only the LLM is tuned with stronger regularization. This schedule is set by hand.
assumptions (4)
  • domain assumption Gumbel-Sigmoid with straight-through estimator produces binary pruning decisions that are differentiable enough to train the generator.
    Invoked in Section 3.3 and Appendix A.1. This is a standard approximation, but there is no guarantee that the rounded binary decisions are the ones that minimize the calibration loss.
  • domain assumption The calibration dataset Dc (1,300 samples) is representative of the domain distribution for searching pruning decisions.
    Used in Algorithm 2 and Section 3.5. The paper notes in Appendix A.5 that decisions can vary with calibration samples due to the small set, so this representativeness is load-bearing but not proven.
  • ad hoc to paper Group lasso on LoRA weights drives pruned groups to zero so they can be directly removed.
    Assumed in Section 3.4 (Eq. 5) and Algorithm 1 line 13. The paper does not quantify the actual L2 norms of pruned groups after training.
  • domain assumption The forward pass fL(X) = X(WD + WaWb) with masked pretrained weights and unmasked LoRA weights captures the effect of the current pruning decision for LoRA tuning.
    Defined in Eq. 4 and used throughout training. This is a modeling choice that assumes LoRA updates can compensate for the masked weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of All-in-One Tuning and Structural Pruning for Domain-Specific LLMs." pith.science (2026). https://pith.science/paper/CKEPFR2H

@misc{pith2026241214426,
  author       = {Pith},
  title        = {Pith review of: All-in-One Tuning and Structural Pruning for Domain-Specific LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKEPFR2H}},
  note         = {Machine review of arXiv:2412.14426}
}
read the original abstract

Existing pruning techniques for large language models (LLMs) targeting domain-specific applications typically follow a two-stage process: pruning the pretrained general-purpose LLMs and then fine-tuning the pruned LLMs on specific domains. However, the pruning decisions, derived from the pretrained weights, remain unchanged during fine-tuning, even if the weights have been updated. Therefore, such a combination of the pruning decisions and the finetuned weights may be suboptimal, leading to non-negligible performance degradation. To address these limitations, we propose ATP: All-in-One Tuning and Structural Pruning, a unified one-stage structural pruning and fine-tuning approach that dynamically identifies the current optimal substructure throughout the fine-tuning phase via a trainable pruning decision generator. Moreover, given the limited available data for domain-specific applications, Low-Rank Adaptation (LoRA) becomes a common technique to fine-tune the LLMs. In ATP, we introduce LoRA-aware forward and sparsity regularization to ensure that the substructures corresponding to the learned pruning decisions can be directly removed after the ATP process. ATP outperforms the state-of-the-art two-stage pruning methods on tasks in the legal and healthcare domains. More specifically, ATP recovers up to 88% and 91% performance of the dense model when pruning 40% parameters of LLaMA2-7B and LLaMA3-8B models, respectively.

Figures

Figures reproduced from arXiv: 2412.14426 by the authors.

Figure 1
Figure 1. Comparison between regular pruning methods [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of one training step of ATP. (MLP). Let dh, dint and H denote the hidden di￾mension, intermediate dimension, and the num￾ber of attention heads, respectively, pruning these blocks can be mathematically expressed as follows: fMLP(X) = (A(XWGDGU) ⊙ (XWUDGU))WD, fAttn(X) = Concat(head1, . . . , headh)WO, headi = softmax  QiK⊤ i  Vi . (1) where A denotes the activation function, ⊙ indi￾cates element-wise mult… view at source ↗
Figure 3
Figure 3. Layer-wise difference ratio between the original pruning decisions generated on the pretrained model and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Layer-wise pruning ratio according to DQK, DV of attention and DGU of MLP for LLaMA2-7B under 50% sparsity level in HealthCare. Deeper color indicates a higher pruning ratio. pretrained dense models. Detailed dataset construction, sample template, evaluation metrics, a…
Figure 5
Figure 5. Figure 5: (a) Task performance when changing the sparsity levels. (b) Training dynamics given different [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Safe Screening Rules for Group SLOPE

    cs.LG 2025-06 reject novelty 5.0 of 10

    A doubly dynamic safe screening rule for Group SLOPE that discards inactive feature groups during training, reducing computation without changing the final solution.

Reference graph

Works this paper leans on

70 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Asma Ben Abacha and Dina Demner-Fushman. 2019. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2228--2234

    2019

  2. [2]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865--10873

    2024

  3. [3]

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations

    2024

  4. [4]

    Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. https://api.semanticscholar.org/CorpusID:8236317 Layer normalization . ArXiv, abs/1607.06450

    arXiv 2016

  5. [5]

    Runxue Bao, Bin Gu, and Heng Huang. 2020. Fast oscar and owl regression via safe screening rules. In International conference on machine learning, pages 653--663. PMLR

    work page 2020

  6. [6]

    Runxue Bao, Bin Gu, and Heng Huang. 2022 a . An accelerated doubly stochastic gradient method with faster explicit model identification. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 57--66

    work page 2022

  7. [7]

    Runxue Bao, Yiming Sun, Yuhe Gao, Jindong Wang, Qiang Yang, Zhi-Hong Mao, and Ye Ye. 2024. https://arxiv.org/abs/2310.08459 A recent survey of heterogeneous transfer learning . Preprint, arXiv:2310.08459

    arXiv 2024

  8. [8]

    Runxue Bao, Xidong Wu, Wenhan Xian, and Heng Huang. 2022 b . Doubly sparse asynchronous learning. In The 31st International Joint Conference on Artificial Intelligence (IJCAI 2022)

    work page 2022

Show all 70 references
  1. [9]

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432

    2013 arXiv

  2. [10]

    Hudson, Ehsan Adeli, and Russ Altman

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, and Russ Altman. 2021. https://crfm.stanford.edu/assets/report.pdf On the opportunities and risks of foundation models . ArXiv

    2021

  3. [11]

    Cl \'e ment Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, et al. 2024. Med42--evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-effi...

    2024 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR

    2023

  6. [14]

    Shangqian Gao, Chi-Heng Lin, Ting Hua, Tang Zheng, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024. Disp-llm: Dimension-independent structural pruning for large language models. arXiv preprint arXiv:2410.11988

    2024 arXiv

  7. [15]

    Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. 2023. Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015

    2023 arXiv

  8. [16]

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366

    2021 arXiv

  9. [17]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790--2799. PMLR

    2019

  10. [18]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  11. [19]

    Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144

    2016 arXiv

  12. [20]

    Cheonsu Jeong. 2024. Fine-tuning and utilization methods of domain-specific llms. arXiv preprint arXiv:2401.02981

    2024 arXiv

  13. [21]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on...

    2019

  14. [22]

    Anastassia Kornilova and Vlad Eidelman. 2019. Billsum: A corpus for automatic summarization of us legislation. arXiv preprint arXiv:1910.00523

    2019 arXiv

  15. [23]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105

    2012

  16. [24]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691

    2021 arXiv

  17. [25]

    Xiang Lisa Li and Percy Liang. 2021. https://arxiv.org/abs/2101.00190 Prefix-tuning: Optimizing continuous prompts for generation . Preprint, arXiv:2101.00190

    2021 arXiv

  18. [26]

    Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024 a . Modegpt: Modular decomposition for large language model compression. arXiv preprint arXiv:2408.09632

    2024 arXiv

  19. [27]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  20. [28]

    Xinyu Lin, Wenjie Wang, Yongqi Li, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. 2024 b . Data-efficient fine-tuning for llm-based recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 365--374

    2024

  21. [29]

    Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanq...

    2024 arXiv

  22. [30]

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  23. [31]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602

    2021 arXiv

  24. [32]

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. 2023. Full parameter fine-tuning for large language models with limited resources. arXiv preprint arXiv:2306.09782

    2023 arXiv

  25. [33]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  26. [34]

    Joel Niklaus, Veton Matoshi, Matthias St \"u rmer, Ilias Chalkidis, and Daniel E Ho. 2023. Multilegalpile: A 689gb multilingual legal corpus. arXiv preprint arXiv:2306.02069

    2023 arXiv

  27. [35]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1--67

    2020

  28. [36]

    https://arxiv.org/abs/1808.06752 Lessons from natural language inference in the clinical domain

    Alexey Romanov and Chaitanya Shivade. https://arxiv.org/abs/1808.06752 Lessons from natural language inference in the clinical domain

    arXiv

  29. [37]

    Zhengyan Shi, Adam X Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, and Aldo Lipani. 2024. Instruction tuning with loss over instructions. arXiv preprint arXiv:2405.14394

    2024 arXiv

  30. [38]

    Jovan Stojkovic, Esha Choukse, Chaojie Zhang, Inigo Goiri, and Josep Torrellas. 2024. Towards greener llms: Bringing energy-efficiency to the forefront of llm inference. arXiv preprint arXiv:2403.20306

    2024 arXiv

  31. [39]

    Zico Kolter

    Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. https://arxiv.org/abs/2306.11695 A simple and effective pruning approach for large language models . Preprint, arXiv:2306.11695

    2024 arXiv

  32. [40]

    Teo Susnjak, Peter Hwang, Napoleon H Reyes, Andre LC Barczak, Timothy R McIntosh, and Surangika Ranathunga. 2024. Automating research synthesis with domain-specific large language model fine-tuning. arXiv preprint arXiv:2404.08680

    2024 arXiv

  33. [41]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  34. [42]

    Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. 2024. The llm surgeon. In The Twelfth International Conference on Learning Representations

    2024

  35. [43]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  36. [44]

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. 2023. https://arxiv.org/abs/2312.03863 Efficient large language models: A survey . Preprint, arXiv:2312.03863

    2023 arXiv

  37. [45]

    Fali Wang, Runxue Bao, Suhang Wang, Wenchao Yu, Yanchi Liu, Wei Cheng, and Haifeng Chen. 2024 a . Infuserki: Enhancing large language models with knowledge graphs via infuser-guided knowledge integration. arXiv preprint arXiv:2402.11441

    2024 arXiv

  38. [46]

    Zhepeng Wang, Runxue Bao, Yawen Wu, Guodong Liu, Lei Yang, Liang Zhan, Feng Zheng, Weiwen Jiang, and Yanfu Zhang. 2024 b . Self-guided knowledge-injected graph neural network for alzheimer’s diseases. In International Conference on Medical Image Computing and Computer-Assisted...

    2024

  39. [47]

    Zhepeng Wang, Runxue Bao, Yawen Wu, Jackson Taylor, Cao Xiao, Feng Zheng, Weiwen Jiang, Shangqian Gao, and Yanfu Zhang. 2024 c . Unlocking memorization in large language models with dynamic soft prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural L...

    2024

  40. [48]

    Zhepeng Wang, Isaacshubhanand Putla, Weiwen Jiang, and Youzuo Lin. 2023. Edge-inversionnet: Enabling efficient inference of inversionnet on edge devices. In Third International Meeting for Applied Geoscience & Energy, pages 1059--1063. Society of Exploration Geophysicists and ...

    2023

  41. [49]

    Zhepeng Wang, Yawen Wu, Zhenge Jia, Yiyu Shi, and Jingtong Hu. 2021. Lightweight run-time working memory compression for deployment of deep neural networks on resource-constrained mcus. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pages 607--614

    2021

  42. [50]

    Xidong Wu, Shangqian Gao, Zeyu Zhang, Zhenzhen Li, Runxue Bao, Yanfu Zhang, Xiaoqian Wang, and Heng Huang. 2024. Auto-train-once: Controller network guided automatic network pruning from scratch. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  43. [51]

    Yawen Wu, Zhepeng Wang, Zhenge Jia, Yiyu Shi, and Jingtong Hu. 2020 a . Intermittent inference with nonuniformly compressed multi-exit neural network for energy harvesting powered devices. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1--6. IEEE

    2020

  44. [52]

    Yawen Wu, Zhepeng Wang, Yiyu Shi, and Jingtong Hu. 2020 b . Enabling on-device cnn training by self-supervised instance filtering and error map pruning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):3445--3457

    2020

  45. [53]

    Yawen Wu, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, and Jingtong Hu. 2021 a . Decentralized unsupervised learning of visual representations. arXiv preprint arXiv:2111.10763

    2021 arXiv

  46. [54]

    Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, and Jingtong Hu. 2021 b . Enabling on-device self-supervised contrastive learning with selective data contrast. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 655--660. IEEE

    2021

  47. [55]

    Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, and Jingtong Hu. 2023. Synthetic data can also teach: Synthesizing effective data for unsupervised visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2866--2874

    2023

  48. [56]

    Yawen Wu, Dewen Zeng, Zhepeng Wang, Yi Sheng, Lei Yang, Alaina J James, Yiyu Shi, and Jingtong Hu. 2021 c . Federated contrastive learning for dermatological disease diagnosis via on-device learning. In 2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD), p...

    2021

  49. [57]

    Yawen Wu, Dewen Zeng, Zhepeng Wang, Yiyu Shi, and Jingtong Hu. 2021 d . Federated contrastive learning for volumetric medical image segmentation. In Medical Image Computing and Computer Assisted Intervention--MICCAI 2021: 24th International Conference, Strasbourg, France, Sept...

    2021

  50. [58]

    Yawen Wu, Dewen Zeng, Zhepeng Wang, Yiyu Shi, and Jingtong Hu. 2022. Distributed contrastive learning for medical image segmentation. Medical Image Analysis, 81:102564

    2022

  51. [59]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations

    2024

  52. [60]

    Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, et al. 2024. Me llama: Foundation large language models for medical applications. arXiv preprint arXiv:2402.12749

    2024 arXiv

  53. [61]

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. 2024 a . When scaling meets llm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193

    2024 arXiv

  54. [62]

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2023. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403

    2023 arXiv

  55. [63]

    Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, and Haifeng Chen. 2024 b . Pruning as a domain-specific llm extractor. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1417--1428

    2024

  56. [64]

    Yanfu Zhang, Runxue Bao, Jian Pei, and Heng Huang. 2022. Toward unified data and algorithm fairness via adversarial data augmentation and adaptive model fine-tuning. In 2022 IEEE International Conference on Data Mining (ICDM), pages 1317--1322. IEEE

    2022

  57. [65]

    Yanfu Zhang, Guodong Liu, Runxue Bao, Liang Zhan, Paul Thompson, and Heng Huang. 2024 c . Neurodegenerative disease prediction via transferable deep networks. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1--5. IEEE

    2024

  58. [66]

    Jiawei Zheng, Hanghai Hong, Xiaoli Wang, Jingsong Su, Yonggui Liang, and Shikai Wu. 2024. Fine-tuning large language models for domain-specific machine translation. arXiv preprint arXiv:2402.15061

    2024 arXiv

  59. [67]

    Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificia...

    2021

  60. [68]

    Ou Zheng, Mohamed Abdel-Aty, Dongdong Wang, Chenzhu Wang, and Shengxuan Ding. 2023. Trafficsafetygpt: Tuning a pre-trained large language model to a domain-specific expert in transportation safety. arXiv preprint arXiv:2307.15311

    2023 arXiv

  61. [69]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.