Pith. sign in

REVIEW 4 major objections 5 minor 51 references

GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GradOT: training-free compression that preserves adapter gradients

desk verdict A useful training-free compression method for offsite-tuning with strong experiments, but the theoretical derivation leans on an unvalidated small-perturbation assumption. read the letter →

arxiv 2507.04455 v1 pith:RUF6BGSY submitted 2025-07-06 cs.CL

Kai Yao , Zhaorui Tan , Penglei Gao , Lichun Li , Kaixin Wu , Yinggui Wang , Yuan Zhao , Yixin Ji
show 2 more authors
Wei Wang Jianke Zhu
This is my paper
classification cs.CL
keywords offsite-tuninggradient-preservingcompressionLLMprivacy-preservingfine-tuningrankdecompositionchannelpruningFisherinformationparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offsite-tuning lets a model owner hand a compressed 'emulator' of a large language model to a data owner, who fine-tunes adapters on private data and sends them back; the adapters then plug into the full model. This paper claims that the right compression choice is the one that keeps the emulator's adapter gradients close to the full model's gradients while also making the emulator's own loss worse. It formalizes that trade-off in an objective and derives a Gradient-preserving Compression Score (GCS) from a first-order Taylor expansion, then uses GCS to decide which singular values to drop in attention layers and which channels to prune in MLP layers. Experiments on OPT-1.3B/6.7B and LLaMA-7B/13B report that the resulting training-free method matches or beats existing offsite-tuning methods while requiring only minutes of compression time instead of hours. The payoff, if the claim holds, is a faster, privacy-preserving way to adapt large models without ever sharing the full model weights.

What carries the argument

The central object is the Gradient-preserving Compression Score (GCS), defined in Eq. (8) as GCS(δ_i) = ||∂²ℓ/∂w_i² δ_i||₁ − λ (∂ℓ/∂w_i ⊙ δ_i), where δ_i is the weight perturbation caused by removing a singular value or a channel. The first term estimates how much the adapter gradients would be disturbed by the compression (via a Hessian-vector product), and the second term estimates how much the emulator's loss would rise (via a gradient inner product); λ sets the privacy–utility trade-off. The score is computed without training: first-order gradients come from a forward/backward pass on a support dataset, and the Hessian is replaced by a Kronecker-factored Fisher approximation (KFAC) that treats each linear layer independently. GCS then ranks candidate rank indices and channels, and GradOT keeps the components with the smallest scores until the target compression ratio is reached. This score is the machinery that turns the Taylor-expanded objective into a concrete, training-free compression policy.

What would settle it

Run a controlled experiment on OPT-1.3B where, for one linear layer, you compute the true Hessian-vector products and the true loss increments for each rank removal, and compare them to the KFAC-approximated GCS values; a large rank-correlation drop would show that the score does not preserve gradient fidelity as claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is that the offsite-tuning objective—minimize the loss gap between the plug-in model and full fine-tuning, and maximize the gap between the emulator and the plug-in—can be approximated entirely in terms of per-layer weight perturbations. For a perturbation δ_i, the change in adapter gradients is approximated by the Hessian-vector product ∂²ℓ/∂w_i² δ_i and the change in loss by the inner product (∂ℓ/∂w_i) ⊙ δ_i. The Gradient-preserving Compression Score combines these as ||∂²ℓ/∂w_i² δ_i||₁ − λ (∂ℓ/∂w_i ⊙ δ_i), and the paper argues that minimizing this score over allowed compressions achieves both objectives simultaneously. GradOT realizes this by applying Dynamic Rank Decomposition to attention weight matrices and Selective Channel Pruning to MLP weight matrices, choosing the components that keep the score smallest. The paper reports that GradOT's plug-in models are competitive with full fine-tuning and that its emulators are the weakest among the compared methods, indicating stronger privacy.

Load-bearing premise

The paper assumes that the first-order Taylor expansions in Eqs. (5) and (6) stay accurate for the actual perturbations δ_i caused by removing whole singular values or channels; if those perturbations are too large for the expansion, the Gradient-preserving Compression Score no longer estimates the true gradient or loss gaps.

Editorial extensions

If this is right

  • GradOT cuts the compression stage from hours (OT and ScaleOT take 60 and 5 hours on OPT-1.3B) to about 10 minutes on OPT-1.3B and 96 minutes on LLaMA-7B, all without any training-based step.
  • Plug-in models trained with GradOT-emulators reach within a few points of full fine-tuning on several benchmarks, while the emulator itself (before adapters are moved back) scores lowest among compared methods, a sign that privacy is preserved.
  • The same GCS indicator can guide compression choices for other offsite-tuning-oriented methods, because it scores any weight perturbation δ_i, not just rank or channel removals.
  • GradOT's success suggests that preserving adapter gradient fidelity, not just layer importance or reconstruction error, is the right criterion for offsite-tuning compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not test is using GCS to set per-layer compression ratios adaptively instead of one global ratio; the score already provides the per-component ranking needed for such a schedule.
  • A testable follow-up is measuring how sensitive GCS's ranking is to support-vs-target distribution shift; the paper's Fig. 5 only gives a coarse two-point comparison.
  • The signed loss term in GCS suggests privacy could be tuned as a knob by amplifying loss in directions that leave gradient information intact, which the paper does not investigate.
  • Applying GCS to non-transformer architectures would test whether the score is truly architecture-agnostic; the paper only reports experiments on transformers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes GradOT, a training-free offsite-tuning method that compresses the middle layers of a transformer LLM using Dynamic Rank Decomposition (DRD) for attention weights and Selective Channel Pruning (SCP) for MLP weights, guided by a Gradient-preserving Compression Score (GCS). The GCS is derived from first-order Taylor expansions of the loss and gradient around the original weights, with the aim of preserving adapter gradients while increasing the emulator's loss for privacy. Experiments on OPT-1.3B/6.7B and LLaMA-7B/13B across eight question-answering benchmarks show competitive plug-in performance and generally lower emulator zero-shot accuracy compared with existing OT methods, along with substantially lower compression cost.

Significance. If the theoretical derivation were sound, the paper would make a useful step by giving offsite-tuning a principled, gradient-based objective and a computationally cheap compression rule. The empirical results are interesting: GradOT matches or exceeds CRaSh and ScaleOT on several benchmarks with much lower compression time, and the ablation study supports the broad idea that gradient-aware compression helps. However, the central theoretical claim—that GCS accurately estimates gradient preservation and loss gap under the actual compression operations—is not yet supported by the evidence. The paper ships a reproducible compression pipeline and reports runtimes, which are valuable, but the theoretical foundation needs substantial strengthening or reframing.

major comments (4)
  1. [3.2, Eq. (5)] The derivation of the GCS rests on first-order Taylor expansions of the gradient and the loss. The only validation provided (Fig. 7) adds random noise with Frobenius norm up to 1 and shows that Eq. (15) tracks the loss change. DRD (Eq. (10)) removes all singular values below a retention threshold, keeping only 40–50% of ranks in the reported settings, and SCP (Eq. (12)) removes entire channels from the MLP; these structured deletions produce perturbations that are not small relative to the weights. The higher-order terms neglected in Eqs. (5) and (6) can therefore be substantial, and the paper provides no bound or direct correlation between GCS and the actual gradient/loss changes for the compression-induced perturbations. This is load-bearing because the score is defined as an approximation of the true objectives in Eq. (7).
  2. [3.2, Eq. (5)] Equation (5) approximates the perturbed gradient by keeping only the diagonal block ∂²ℓ/∂wi² δi and dropping all cross-layer Hessian terms ∂²ℓ/∂wi∂wj δj for j≠i. Since compression is applied simultaneously to every middle layer (Eq. (3)), the cross-layer terms are not automatically small. The paper does not justify this omission either theoretically or empirically; a sensitivity analysis varying the number of compressed layers would be needed to assess its impact. Without this, Score Term (1) in Eq. (8) may not correspond to the actual gradient change of the adapter.
  3. [3.2, Eqs. (4)–(7)] The paper claims that minimizing per-layer gradient changes of middle-layer weights preserves the gradients of the adapters, invoking the chain rule. This step is not derived: the adapters A1 and A2 are only the first and last two layers, and the objective in Eq. (4) is written for wi of each middle layer, not for the adapter parameters. The connection between the middle-layer gradient discrepancies and the adapter gradient discrepancies is non-trivial because the forward and backward passes propagate through many compressed layers. The authors should provide the chain-rule expansion or an empirical check, such as measuring adapter gradient cosine similarity before and after compression.
  4. [3.3, Appendix A.3, Table 8] The method is advertised as training-free, but the GCS depends on hyperparameters λmha and λmlp, the compression ratios rmha and rmlp, and a 5% rank-retention rule, all of which are tuned on downstream evaluation (Table 8; Appendix A.3). Additionally, the support dataset (BoolQ, TriviaQA, CoPA) is constructed from tasks of the same type as the downstream benchmarks, so the score is not evaluated in a fully task-agnostic setting. The authors should clarify what 'training-free' means (no gradient updates of the emulator) and provide evidence that the score transfers across task families without per-task hyperparameter tuning, or temper the claim.
minor comments (5)
  1. [Section 1] The abbreviation 'GSC' is used in the introduction for the score that is called 'GCS' elsewhere; please standardize the terminology.
  2. [Eq. (7)] Equation (7) contains a typesetting artifact ('nX' instead of a summation over layers) and the displayed approximation is difficult to parse; please reformat it for clarity.
  3. [Appendix A.2] The text in Appendix A.2 says 'select the components with the highest K scores' but the set definition s = {k | Score(w(k)) < Score(w(K))} selects components whose scores are below the K-th score; please clarify the selection rule.
  4. [Appendix B.4, Fig. 7] Figure 7 uses a log-scaled y-axis, but the text reports the error as 'within ±0.02'; please specify the exact error metric and noise distribution so that the validation can be reproduced.
  5. [Table 1] The average emulator fine-tuned accuracy for GradOT (45.0) is higher than for CRaSh (43.6), so on this privacy metric CRaSh is stronger; the abstract's claim that GradOT surpasses existing methods in both privacy and performance simultaneously is not uniformly supported by the table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GCS is an explicitly derived surrogate objective, and the paper's claims are tested on external downstream metrics rather than on the score's own construction.

full rationale

The derivation chain is not circular. GradOT's gradient-preserving compression score (Eq. 8) is an explicit first-order surrogate for the offsite-tuning objective in Eq. (7): Score Term (1) approximates the gradient mismatch via the Taylor expansion in Eq. (5), and Score Term (2) approximates the loss gap via the total differential in Eq. (6). The score is then used to select compression masks, while the paper's headline claims are evaluated on downstream benchmarks (plug-in accuracy, emulator fine-tuning accuracy, loss landscape), which are external to the score computation. The self-citations to ScaleOT (Yao et al., 2025) supply task setup and the general idea of block-wise compression, but they do not carry the derivation; there is no imported uniqueness theorem or fitted parameter that forces the result by definition. There are legitimate correctness concerns, notably that Eqs. (5) and (6) assume small perturbations while DRD and SCP delete large structured components, and that Fig. 7 validates the loss approximation only for random noise with Frobenius norm up to 1 rather than for the actual structured deletions or the gradient approximation. The trade-off parameters lambda and the compression ratios are also tuned in part on target tasks. These are validity and robustness issues, not circularity: the paper's conclusions do not reduce by construction to its inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about the Taylor expansion's validity for large weight perturbations, the representativeness of the support dataset, and the approximation of Hessians via Fisher/KFAC. It also depends on four hyperparameters (two lambdas and two compression ratios) that are tuned. No new physical entities are introduced.

free parameters (6)
  • lambda_mha = 1e4 (default for OPT-1.3B)
    Controls balance between gradient preservation and loss gap for MHA; tuned via grid search over {1e1, 1e2, 1e3, 1e4, 1e5} in Appendix B.2.
  • lambda_mlp = 1e2 (default for OPT-1.3B)
    Same role for MLP compression; tuned in sensitivity study in Appendix B.2.
  • compression ratio rmha = 0.4 for 1.3B, 0.5 for 7B
    Manually chosen to achieve target overall parameter ratio; no automated selection procedure.
  • compression ratio rmlp = 0.7 for 1.3B, 0.8 for 7B
    Manually chosen along with rmha; no sensitivity analysis across these values.
  • support dataset size = 1500 samples
    Chosen following prior work (Zhang et al., 2023a); no sensitivity analysis reported.
  • top 5% rank retention = 5%
    Heuristic to keep delta small for scoring accuracy, stated in Eq. (10) context.
assumptions (4)
  • ad hoc to paper First-order Taylor expansion of the loss and gradient is accurate for the actual compression-induced weight perturbations.
    Eqs. (5) and (6) truncate higher-order terms; actual delta from removing singular values or channels can be large, and validity is only tested for small random noise in Fig. 7.
  • domain assumption The Fisher information approximates the Hessian of the loss.
    KFAC approximation of the partial Hessian using gradient outer products is adopted from prior work (Kunstner et al., 2019; Daxberger et al., 2021) and is central to the GCS computation.
  • domain assumption Gradients and Hessians computed on the support dataset (BoolQ, TriviaQA, CoPA) are representative for downstream QA tasks.
    The GCS is computed from support data; Fig. 5 shows performance degrades when using general WikiText, so the choice of support data is highly influential.
  • ad hoc to paper Minimizing per-layer middle-weight gradient changes preserves the gradients of the adapters.
    The paper states this connection 'ultimately presses the gradients of the adapters' but does not derive the chain-rule relationship between middle-layer Hessian norms and top/bottom adapter gradients.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models." pith.science (2026). https://pith.science/paper/RUF6BGSY

@misc{pith2026250704455,
  author       = {Pith},
  title        = {Pith review of: GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUF6BGSY}},
  note         = {Machine review of arXiv:2507.04455}
}
read the original abstract

The rapid growth of large language models (LLMs) with traditional centralized fine-tuning emerges as a key technique for adapting these models to domain-specific challenges, yielding privacy risks for both model and data owners. One promising solution, called offsite-tuning (OT), is proposed to address these challenges, where a weaker emulator is compressed from the original model and further fine-tuned with adapter to enhance privacy. However, the existing OT-based methods require high computational costs and lack theoretical analysis. This paper introduces a novel OT approach based on gradient-preserving compression, named GradOT. By analyzing the OT problem through the lens of optimization, we propose a method that selectively applies compression techniques such as rank compression and channel pruning, preserving the gradients of fine-tuned adapters while ensuring privacy. Extensive experiments demonstrate that our approach surpasses existing OT methods, both in terms of privacy protection and model performance. Our method provides a theoretical foundation for OT and offers a practical, training-free solution for offsite-tuning of large-scale LLMs.

Figures

Figures reproduced from arXiv: 2507.04455 by the authors.

Figure 1
Figure 1. Illustration of our proposed Gradient￾preserving Offsite-tuning (GradOT). Li et al., 2020; Zou et al., 2023; Ye et al., 2024; Liu et al., 2024b), posing significant barriers to LLMs’ applications on sensitive downstream fine-tuning. Offsite-tuning (OT) (Xiao et al., 2023) has emerged as a promising solution to safeguard the privacy of both data and model owners [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Diagram of gradient preserving of adapters [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our Gradient-preserving Compression Score (GCS) guided compression strategy. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of the loss landscape of the ini [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of different data types for statisti [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance of applying GradOT to OPT￾1.3B with increased r on OBQA dataset [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Error between the actual loss and estimated [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 38 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pages 7432--7439

    work page 2020

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877--1901

    work page 2020

  5. [5]

    Terence Jie Chua, Wenhan Yu, Jun Zhao, and Kwok-Yan Lam. 2023. Fedpeat: Convergence of federated learning, parameter-efficient fine tuning, and emulator assisted tuning for artificial intelligence foundation models with mobile edge computing. arXiv preprint arXiv:2310.17491

    work page Pith review arXiv 2023

  6. [6]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. B ool Q : Exploring the surprising difficulty of natural yes/no questions. In Association for Computational Linguistics, pages 2924--2936

    work page 2019

  7. [7]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  8. [8]

    Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. 2021. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089--20103

    work page 2021

Show all 51 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hum...

    2019

  2. [10]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, and Zhifang Sui. 2024. A survey on in-context learning. In Conference on Empirical Methods in Natural Language Processing, pages 1107--1128

    2024

  3. [11]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM: general language model pretraining with autoregressive blank infilling. In Association for Computational Linguistics, pages 320--335

    2022

  4. [12]

    Carl Eckart and Gale Young. 1936. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211--218

    1936

  5. [13]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  6. [14]

    Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang

    Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang. 2024. Dp-opt: Make large language model your privacy-preserving prompt engineer. In International Conference on Learning Representations

    2024

  7. [15]

    Yixin Ji, Yang Xiang, Juntao Li, Wei Chen, Zhongyi Liu, Kehai Chen, and Min Zhang. 2024. Feature-based low-rank compression of large language models via bayesian optimization. In Findings of the Annual Meeting of the Association for Computational Linguistics

    2024

  8. [16]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics

    2017

  9. [17]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. Squeezellm: Dense-and-sparse quantization. In International Conference on Machine Learning

    2024

  10. [18]

    Frederik Kunstner, Philipp Hennig, and Lukas Balles. 2019. Limitations of the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32

    2019

  11. [19]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Conference on Empirical Methods in Natural Language Processing, pages 785--794

    2017

  12. [20]

    Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: adversarial attack against BERT using BERT . In Conference on Empirical Methods in Natural Language Processing, pages 6193--6202

    2020

  13. [21]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 a . Visual instruction tuning. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, and Xia Hu. 2024 b . Lora-as-an-attack! piercing llm safety under the share-and-play scenario. arXiv preprint arXiv:2403.00108

    2024 arXiv

  15. [23]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing

    2018

  16. [24]

    Dinh C Nguyen, Ming Ding, Pubudu N Pathirana, Aruna Seneviratne, Jun Li, and H Vincent Poor. 2021. Federated learning for internet of things: A comprehensive survey. IEEE Communications Surveys & Tutorials, 23(3):1622--1658

    2021

  17. [25]

    Kazuki Osawa, Shigang Li, and Torsten Hoefler. 2023. Pipefisher: Efficient training of large language models using pipelining and fisher information matrices. Proceedings of Machine Learning and Systems, 5:708--727

    2023

  18. [26]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...

    2022

  19. [27]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  20. [28]

    Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. 2024. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science, 383(6690):1461--1467

    2024

  21. [29]

    Siyu Ren and Kenny Q Zhu. 2023. Low-rank prune-and-factorize for language model compression. arXiv preprint arXiv:2306.14152

    2023 arXiv

  22. [30]

    Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. A scalable laplace approximation for neural networks. In 6th international conference on learning representations, ICLR 2018-conference track proceedings, volume 6. International Conference on Representation Learning

    2018

  23. [31]

    Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429

    2023

  24. [32]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

    2019 arXiv

  25. [33]

    Madisetti, and Arshdeep Bahga

    Tanmay Singh, Harshvardhan Aditya, Vijay K. Madisetti, and Arshdeep Bahga. 2024. Whispered tuning: Data privacy preservation in fine-tuning llms through differential privacy. Journal of Software Engineering and Applications, 17(1):1--22

    2024

  26. [34]

    Zhaorui Tan, Xi Yang, Qiufeng Wang, Anh Nguyen, and Kaizhu Huang. 2024. Interpret your decision: Logical reasoning regularization for generalization in visual classification. In Advances in Neural Information Processing Systems

    2024

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30

    2017

  29. [37]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32

    2019

  30. [38]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations

    2022

  31. [39]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In EMNLP Workshop, pages 94--106. Association for Computational Linguistics

    2017

  32. [40]

    Noam Wies, Yoav Levine, and Amnon Shashua. 2024. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36

    2024

  33. [41]

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, and Hongseok Namkoong. 2022. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2022

  34. [42]

    Guangxuan Xiao, Ji Lin, and Song Han. 2023. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiv:2302.04870

    2023 arXiv

  35. [43]

    Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. 2024. Bayesian low-rank adaptation for large language models. In international conference on learning representations

    2024

  36. [44]

    Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, and Jianke Zhu. 2024. Layer-wise importance matters: Less memory for better performance in parameter-efficient fine-tuning of large language models. In Findings of the Association for Computational Linguistic...

    2024

  37. [45]

    Kai Yao, Zhaorui Tan, Tiandi Ye, Lichun Li, Yuan Zhao, Wenyan Liu, Wei Wang, and Jianke Zhu. 2025. Scaleot: Privacy-utility-scalable offsite-tuning with dynamic layerreplace and selective rank compression. In AAAI

    2025

  38. [46]

    Tiandi Ye, Cen Chen, Yinggui Wang, Xiang Li, and Ming Gao. 2024. Bapfl: You can backdoor personalized federated learning. Transactions on Knowledge Discovery from Data, 18(7):166

    2024

  39. [47]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Association for Computational Linguistics, pages 4791--4800

    2019

  40. [48]

    Kaiyan Zhang, Ning Ding, Biqing Qi, Xuekai Zhu, Xinwei Long, and Bowen Zhou. 2023 a . CR a S h: Clustering, removing, and sharing enhance fine-tuning without full large language model. In Conference on Empirical Methods in Natural Language Processing, pages 9612--9637

    2023

  41. [49]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2023 b . Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2023 arXiv

  42. [50]

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816--16825

    2022

  43. [51]

    Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.