Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs

Aleksey Kudelya; Alexander Shirnin

REVIEW 4 major objections 5 minor 17 references

Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-phase unlearning keeps 0.469 MMLU while hitting 0.283 regurgitation.

desk verdict The paper's core update rule appears to have the wrong sign for unlearning, and the abstract numbers don't match Table 2; as written, it can't be trusted. read the letter →

arxiv 2506.04044 v1 pith:UQJRKO5O submitted 2025-06-04 cs.CL cs.AI

Aleksey Kudelya , Alexander Shirnin This is my paper

classification cs.CLcs.AI

keywords machineunlearninglargelanguagemodelsinfluencefunctionsFisherinformationmatrixsecond-orderoptimizationLoRASemEval-2025Task4MMLU

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LIBU is a two-phase recipe for removing specific knowledge from a large language model without retraining it from scratch. The paper claims that combining an influence-function-style update, weighted by a Fisher information diagonal estimated from retained data, with a second-order optimizer called Sophia erases forget-set influence while keeping overall capability intact. On the SemEval-2025 Task 4 unlearning benchmark the authors report a regurgitation rate of 0.283 while maintaining an MMLU accuracy of 0.469, above the competition's 0.371 threshold. If this holds, unlearning becomes practical on a single GPU for 1B and 7B models.

What carries the argument

The load-bearing object is the diagonal Fisher weighting $w_\theta = 1/(\mathbb{E}[g_{\text{retain}}^2] + \lambda)$, computed from squared retain-set gradients and used to scale forget-set gradients before each LoRA parameter update. The second piece is Sophia's update $\Delta \theta_t = -\eta \cdot g_t / \max(\gamma \cdot h_t, \epsilon)$, where $h_t$ is a stochastically sampled Hessian-diagonal estimate from squared gradients; it stabilizes Phase 2. LoRA (low-rank adaptation) confines both phases to small adapter matrices, and gradient accumulation simulates larger batches under memory limits.

What would settle it

Run one subtask with the Fisher weighting replaced by a constant, $w_\theta = 1$, keeping everything else identical. If regurgitation and MMLU are essentially unchanged, the influence-based update is inert and the two-phase claim collapses; if performance degrades, the weighting is doing real work. The same comparison with Sophia replaced by Adam would isolate the second phase.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the unlearning update $\theta_{t+1} \leftarrow \theta_t - \eta \cdot w_\theta \cdot g_{\text{forget}}$, with $w_\theta = 1/(\mathbb{E}[g_{\text{retain}}^2] + \lambda)$, provides a workable way to erase data: parameters that are important for retained knowledge receive small updates, while less critical parameters are moved more aggressively along the forget-set gradient. Phase 2 then applies the Sophia optimizer, which uses squared-gradient Hessian estimates to dampen and clip updates in high-curvature directions, stabilizing utility. The authors test this on OLMo-7B and OLMo-1B across three subtasks (synthetic documents, PII biographies, and real pretraining documents) and report that the balanced configuration reaches 0.254 aggregate task score and 0.483 MMLU in local evaluation, while the submitted system is stated to reach a regurgitation rate of 0.283 and MMLU 0.469, beating the 0.371 threshold. No component-level ablation is reported.

Load-bearing premise

The method stands on the untested idea that multiplying forget-set gradients by the inverse of average squared retain-set gradients erases target knowledge without damaging retained knowledge; nothing in the paper isolates or proves that weighting assumption.

Editorial extensions

If this is right

If LIBU works as claimed, sensitive content can be removed from 1B and 7B models on a single A100 GPU in under the competition's one-hour limit, with no full retraining.
The Fisher weighting gives a parameter-level mechanism for protecting retained knowledge, so unlearning becomes a targeted update rather than a global degradation of the model.
The two-phase pattern—influence-based erasure followed by second-order stabilization—can be lifted into other unlearning pipelines that currently rely on plain gradient ascent.
Because the method only trains LoRA adapters, it suggests a cheap route to compliance workflows where each data-removal request requires a new model state.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never ablates the Fisher weighting or the Sophia phase, so the central claim that the two-phase combination is responsible for the result is still a hypothesis; a reader should treat the influence-function framing as promising rather than proven.
If the weighting $w_\theta$ is nearly constant across parameters—for example if $\lambda$ dominates—then LIBU reduces to LoRA-based gradient ascent with second-order stabilization, and the 'influence-based' label would be decorative; a single ablation with $w_\theta = 1$ would settle this.
The reported MIA score of 0.0 in all in-paper setups suggests the forget set becomes fully indistinguishable from nonmembers, which may mean the method over-erases; testing on a more granular membership metric would show whether utility loss is the price of that strong signal.
The 0.283 regurgitation / 0.469 MMLU headline appears to come from the official competition evaluation, while Table 2 reports local setups; reconciling these numbers would clarify which configuration is the actual submission.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about this one. First, the core update rule in Section 3.1 looks like it has the wrong sign for unlearning. The paper states the objective is to maximize loss on the forget set, but the update θ_{t+1} ← θ_t − η·w·g_forget, with w positive and g_forget computed by standard backpropagation, is gradient descent on the forget loss. That reduces the forget loss, which would improve recall, not erase it. The correct influence-function update for removing a point is positive H^{-1}∇L_forget; the paper uses the negative. Unless the code implements the opposite of what is printed, the method cannot do what it claims. If the code uses the opposite sign, the paper misdescribes the algorithm. There is no derivation or code excerpt in the text to resolve this.

Second, the abstract's headline numbers (regurgitation 0.283, MMLU 0.469) do not appear in Table 2, and the third LIBU row is garbled. This makes the empirical claims hard to verify.

What is new: the specific pipeline — Fisher-diagonal weighting from Foster et al., Sophia optimizer from Liu et al., LoRA, and gradient accumulation on OLMo — is a new system configuration for the SemEval task, and the code is publicly released. That is legitimate engineering credit.

The soft spots beyond the sign issue: no variance estimates, hyperparameters chosen post hoc on the test set, and the authors explicitly defer ablations of the two phases to future work. The limitations section is honest about these gaps.

If the sign is a typo, a one-line correction and a cleaned-up table would turn this into a passable system report. As written, the central equation contradicts the stated objective, so I would not trust the method description. I agree with the reader that the results are conditional, but the sign problem is more fundamental than the missing ablations. I would not cite this paper in its current state and would not send it to a serious referee until the authors fix the equation and reconcile the abstract with the results.

Referee Report

4 major / 5 minor

Summary. The paper presents LIBU, a two-phase unlearning method for LLMs submitted to SemEval-2025 Task 4. Phase 1 applies a Fisher-information-weighted gradient update to the forget set using LoRA adapters, and Phase 2 runs Sophia second-order optimization to stabilize utility. The authors report experiments on OLMo-7B and claim a regurgitation rate of 0.283 with MMLU accuracy 0.469, exceeding the competition threshold, plus three hyperparameter configurations that trade off unlearning aggressiveness against retained-task performance. The paper includes a public code repository and a short limitations section.

Significance. If the reported numbers are reliable, LIBU would be a useful low-cost baseline for LLM unlearning: it uses only LoRA adapters and a small number of hyperparameters, and it is evaluated on an external benchmark (LUME). The public code release and the use of standard baselines are strengths. However, the core update rule as written has the wrong sign for unlearning, the result table is internally inconsistent and partially unreadable, and the absence of ablations and variance estimates means the central claim that the two-phase combination works is not currently established. The paper is a credible competition system description, but not yet a supported empirical study.

major comments (4)

[§3.1, update rule] The printed update θ_{t+1} ← θ_t − η·w_θ·g_forget is gradient descent on the forget-set loss. Since the task objective in Section 2 is to maximize the loss on D_forget, and w_θ is positive, this update reduces the forget loss and therefore improves recall of forget data rather than erasing it. Please correct the sign, or equivalently state that the update maximizes the negated loss, and confirm the implemented update against the released code. If the code uses θ_t + η·w_θ·g_forget, the paper currently misdescribes the core algorithm.
[Table 2 and abstract] The headline numbers do not match the result table. The abstract reports regurgitation 0.283 and MMLU accuracy 0.469; Table 2 contains three LIBU rows, none of which displays 0.283/0.469, and the third row is printed as '0.2540.280.00.483', which cannot be parsed into the four listed metrics. The MIA entries are 0.0 for all LIBU rows, but the paper never states whether lower or higher MIA is better. Without a readable and internally consistent table, the central empirical claim that LIBU exceeds the competition threshold cannot be verified.
[§4 and §5] Section 4 explicitly leaves 'fine-grained ablations of individual components (Sophia, Influence Functions)' to future work, and each configuration is run once. Section 5 then compares three setups after observing their task scores and selects Setup 3 as the reported result. This post hoc selection, without a validation split or variance estimates, does not support the claim that the two-phase influence-plus-Sophia design is responsible for the performance, nor does it rule out plain gradient ascent or gradient difference as equally effective alternatives listed in Table 2.
[§3.1, Fisher weighting] The update is described as 'influence-based,' but no derivation connects w_θ = 1/(E[g_retain^2]+λ) to influence functions (Koh and Liang, 2017). The weighting is a heuristic, and the paper does not ablate it against an unweighted gradient-ascent update with the same LoRA rank and learning rate. As written, the specific contribution of the Fisher weighting is not isolated, so the paper's claim that influence-based weighting preserves retain knowledge is not supported.

minor comments (5)

[§5] The experiments section says two model versions (7B and 1B) were evaluated, but only OLMo-7B results appear; please add the 1B results or remove the claim.
[Table 1 and §3.1] Section 3.1 states λ=10^-3, but Table 1 lists DAMPING_FACTOR values 5e-5, 8e-4, and 1e-3; please reconcile the reported default with the table.
[Figure 1] The pipeline diagram uses informal iconography and is hard to read; a clean vector diagram with labeled tensors and update rules would substantially improve reproducibility.
[§2, evaluation metrics] The MIA metric is described only as 'equivalent to the PrivLeak metric,' without stating the direction of improvement; please specify whether lower or higher values are better and why the LIBU MIA scores are exactly 0.0.
[References] There are formatting errors in the references, including 'V olkan' for Volkan and 'San Diega' for San Diego; please correct them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical results on an external benchmark, with no self-citation chain or construction-level reduction of the prediction to its inputs.

full rationale

The paper's central claim is that LIBU achieves a regurgitation rate of 0.283 and MMLU accuracy of 0.469 on SemEval-2025 Task 4, exceeding the competition threshold. This is an empirical result evaluated by the external LUME benchmark and OLMo-Eval framework, not a quantity derived by construction from the method's own definitions. The update rule in Section 3.1, theta_{t+1} <- theta_t - eta * w_theta * g_forget with w_theta = 1/(E[g_retain^2] + lambda), is a heuristic weighting derived from retain-set gradients and forget-set gradients; it is not defined in terms of the reported regurgitation or MMLU scores, so the outcome is not forced by definition. The paper explicitly leaves ablations of individual components (Sophia, influence functions) to future work, which weakens the evidence for the two-phase design but does not make the derivation circular. References to influence functions (Koh and Liang), Sophia (Liu et al.), LoRA (Hu et al.), and LUME (Ramakrishna et al.) are external prior work; there is no load-bearing self-citation, uniqueness theorem, or ansatz smuggled in via citation by the same authors. The potential sign issue in the update rule—gradient descent rather than ascent on the forget loss—is a correctness or soundness concern, not a circularity reduction, and cannot be classified as circularity without inspecting the released code.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim rests on heuristic update rules with many hand-tuned hyperparameters; no ablation isolates the contribution of either phase.

free parameters (9)

Learning rate (eta) = 2e-5 (Setup 3)
Hand-tuned per setup; no sensitivity analysis; choice affects both phases.
Damping factor lambda = 1e-3 (Setup 3)
Added to stabilize Fisher inversion; value selected by experiments.
Sophia gamma = 1.2 (Setup 3)
Controls update conservatism; tuned per setup.
Sophia rho = 0.06 (Setup 3)
Probability of sampling Hessian diagonal estimates; tuned.
LoRA rank = 16 (Setup 3)
Low-rank adapter dimension; chosen by hand.
Gradient accumulation steps = 8 (Setup 3)
New hyperparameter introduced by authors; chosen for memory and stability.
Batch size = 4 (Setup 3)
Chosen per setup.
Num epochs = 4 (Setup 3)
Chosen per setup.
Max length = 1024
Tokenization length cap.

assumptions (5)

domain assumption Diagonal Fisher information E[g_retain^2] estimates parameter importance for retained knowledge
Section 3.1 states squared retain gradients quantify importance for retained tasks; this is a heuristic from selective synaptic dampening, not derived in the paper.
domain assumption Gradient on the forget set indicates directions that correlate with recalling forget data
Section 3.1 says 'These gradients indicate directions in parameter space that correlate with the model's ability to recall the forget data.' No proof is given.
ad hoc to paper Sophia second-order updates stabilize utility after influence update
Section 3.2 asserts Phase 2 stabilizes retained knowledge; no ablation supports this; authors state preliminary results indicated components work better together but show no data.
domain assumption The competition metrics (regurgitation, MIA, MMLU) faithfully measure unlearning and utility
Evaluation relies on SemEval-2025 Task 4 metrics (Sections 2 and 4); the paper does not critique metric validity.
domain assumption LoRA parameter updates are sufficient to erase knowledge
Only LoRA adapter weights are updated; paper does not test full fine-tuning comparisons.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs." pith.science (2026). https://pith.science/paper/UQJRKO5O

@misc{pith2026250604044,
  author       = {Pith},
  title        = {Pith review of: Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQJRKO5O}},
  note         = {Machine review of arXiv:2506.04044}
}

read the original abstract

This paper describes LIBU (LoRA enhanced influence-based unlearning), an algorithm to solve the task of unlearning - removing specific knowledge from a large language model without retraining from scratch and compromising its overall utility (SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models). The algorithm combines classical \textit{influence functions} to remove the influence of the data from the model and \textit{second-order optimization} to stabilize the overall utility. Our experiments show that this lightweight approach is well applicable for unlearning LLMs in different kinds of task.

Figures

Figures reproduced from arXiv: 2506.04044 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 5 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Lorenzo Bertetto, Francesca Bettinelli, Alessio Buda, Marco Da Mommio, Simone Di Bari, Claudio Savelli, Elena Baralis, Anna Bernasconi, Luca Cagliero, Stefano Ceri, and Francesco Pierri. 2024. Towards an explorable conceptual map of large language models. In Intelligent Information Systems, pages 82--90, Cham. Springer Nature Switzerland

work page 2024
[4]

Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot

Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. https://doi.org/10.1109/SP40001.2021.00019 Machine unlearning . In Proceedings - 2021 IEEE Symposium on Security and Privacy, SP 2021, Proceedings - IEEE Symposium on Security and Privacy, pages 141--159, U...

arXiv 2021
[5]

Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. https://doi.org/10.1609/aaai.v38i11.29092 Fast machine unlearning without retraining through selective synaptic dampening . Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12043--12051

doi:10.1609/aaai.v38i11.29092 2024
[6]

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Cry...

2024
[7]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

2021
[8]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

2022

Show all 17 references

[9]

Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.245 SOUL : Unlocking the power of second-order optimization for LLM unlearning . In Proceedings of the 2...

2024 doi
[10]

Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA

2015
[11]

Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1885–1894. JMLR.org

2017
[12]

Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. 2024. https://openreview.net/forum?id=3xHDeA8Noi Sophia: A scalable stochastic second-order optimizer for language model pre-training . In The Twelfth International Conference on Learning Representations

2024
[13]

Alessandro Mantelero. 2013. https://doi.org/10.1016/j.clsr.2013.03.010 The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’ . Computer Law & Security Review, 29(3):229--235

2013 doi
[14]

Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, and Rahul Gupta. 2025 a . Lume: Llm unlearning with multitask evaluations. arXiv preprint arXiv:2502.15097

2025 arXiv
[15]

Anil Ramakrishna, Yixin Wan, Xiaomeng Jin, Kai-Wei Chang, Zhiqi Bu, Bhanukiran Vinzamuri, Volkan Cevher, Mingyi Hong, and Rahul Gupta. 2025 b . Semeval-2025 task 4: Unlearning sensitive content from large language models. arXiv preprint

2025
[16]

Smith, and Chiyuan Zhang

Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. 2025. https://openreview.net/forum?id=TArmA033BU MUSE : Machine unlearning six-way evaluation for language models . In The Thirte...

2025
[17]

Tarun, Vikram S

Ayush K. Tarun, Vikram S. Chundawat, Murari Mandal, and Mohan Kankanhalli. 2024. https://doi.org/10.1109/tnnls.2023.3266233 Fast yet effective machine unlearning . IEEE Transactions on Neural Networks and Learning Systems, 35(9):13046–13055

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Lorenzo Bertetto, Francesca Bettinelli, Alessio Buda, Marco Da Mommio, Simone Di Bari, Claudio Savelli, Elena Baralis, Anna Bernasconi, Luca Cagliero, Stefano Ceri, and Francesco Pierri. 2024. Towards an explorable conceptual map of large language models. In Intelligent Information Systems, pages 82--90, Cham. Springer Nature Switzerland

work page 2024

[4] [4]

Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot

Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. https://doi.org/10.1109/SP40001.2021.00019 Machine unlearning . In Proceedings - 2021 IEEE Symposium on Security and Privacy, SP 2021, Proceedings - IEEE Symposium on Security and Privacy, pages 141--159, U...

arXiv 2021

[5] [5]

Jack Foster, Stefan Schoepf, and Alexandra Brintrup. 2024. https://doi.org/10.1609/aaai.v38i11.29092 Fast machine unlearning without retraining through selective synaptic dampening . Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12043--12051

doi:10.1609/aaai.v38i11.29092 2024

[6] [6]

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Cry...

2024

[7] [7]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

2021

[8] [8]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

2022