PRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs

Maluna Menke; Thilo Hagendorff

REVIEW 3 major objections 5 minor 12 references

PRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One epoch of parameter-efficient LoRA fine-tuning on queer news text cuts measured anti-LGBTQIA+ bias in open 7B–8B models by up to 50 points, while soft-prompt tuning does not.

desk verdict The paper's empirical direction is credible and the cheap recipe is worth testing, but the undefined WinoQueer score keeps the headline numbers from being verifiable — send it to review anyway. read the letter →

arxiv 2507.13743 v1 pith:3LJDUQJE submitted 2025-07-18 cs.CL cs.CY

Maluna Menke , Thilo Hagendorff This is my paper · ORCID

classification cs.CLcs.CY

keywords LGBTQIA+biasparameter-efficientfine-tuningLoRAsoft-prompttuningWinoQueeropen-weightLLMsdebiasingnewscorpus

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that anti-LGBTQIA+ bias in open-weight language models can be reduced cheaply: one epoch of LoRA fine-tuning, training under 0.1% of the weights, on a corpus of queer news articles. On an extended WinoQueer benchmark, baseline bias values reach as high as 98 on a 0–100 scale where 50 is neutral, and the LoRA pass reduces them by up to 50 points, lifting neutral outputs from near zero to as much as 36%. The paper contrasts this with soft-prompt tuning, which moves the numbers only a few points and is not statistically significant. If true, this gives practitioners a low-compute route to continuously refresh fairness in deployed models.

What carries the argument

The load-bearing object is the WinoQueer counterfactual-pair measurement, extended by the authors to 111,685 sentence pairs that include colloquial identity terms such as 'enby', 'trans*', 'inter*', 'bi', 'ace', 'pan', and 'demi'. For each pair, the model assigns a conditional log-likelihood to a negative or stereotypical suffix given the identity token, and the difference between marginalized and dominant terms is converted to a 0–100 scale, with 50 as neutrality. The training mechanism is LoRA with rank 8, alpha 16, and dropout 0.1, which injects low-rank update matrices into the frozen model; the paper's PRIDE workflow combines this LoRA pass with the QueerNews corpus, one epoch, Adafactor, and gradient accumulation, compared against ten-token soft-prompt tuning. The machinery works because the counterfactual likelihood gap is what the fine-tuning directly compresses.

What would settle it

Run the extended WinoQueer pairs through a matched set of non-identity homographs (e.g., 'pan' as cookware, 'ace' as a card, 'bi' as a prefix) and compare the likelihood gaps; if the gaps for queer identity terms and the homographs move together after LoRA, the reported debiasing is partly a lexical-frequency effect. Alternatively, have blind raters judge whether the LoRA-tuned model's high-neutrality continuations are actually non-derogatory toward the identity in question.

Watch

Extended reading notes

Core claim

The central discovery is that parameter-efficient adaptation changes not just the model's surface behavior but the measured preference asymmetry between queer and dominant identity terms. For three 7B–8B open models (Llama 3 8B, Mistral 7B 0.3, Gemma 7B), the authors report baseline WinoQueer means of roughly 65–86 depending on identity category, with individual terms above 95; after one LoRA epoch on the QueerNews corpus the same terms move toward 50, often below it, and neutral outputs appear where they were virtually absent. Soft-prompt tuning with ten virtual tokens fails to move the needle, which the authors attribute to prompt tuning's dependence on much larger models. The paper's claim is that this asymmetry reduction is a genuine debiasing effect, not just perplexity change, and that it can be achieved with <0.1% trainable parameters in about 25 hours on four A100 GPUs.

Load-bearing premise

The load-bearing assumption is that the extended WinoQueer numbers, especially for short ambiguous terms like 'pan', 'bi', 'ace', and 'inter*', actually measure anti-LGBTQIA+ bias rather than lexical ambiguity or template artifacts.

Editorial extensions

If this is right

If the reported bias reductions are accurate, a one-epoch LoRA pass on QueerNews is enough to bring several identity terms from extreme bias values of 95–99 to near or below the 50 neutrality line across three 7B–8B open models.
Neutral output rates rise from under 0.1% to roughly 33–36% in Mistral 7B after LoRA, showing that the reduction is not just a small shift in a continuous measurement.
Ten-token soft-prompt tuning at this model scale does not produce a statistically reliable bias reduction, so LoRA is the practical parameter-efficient choice for 7B–8B models.
The compute footprint, under 0.1% trainable parameters and about 25 hours on four A100 GPUs, makes periodic fairness fine-tuning feasible for modestly resourced teams.
The same PRIDE workflow can be applied to other underrepresented identity groups, provided a counterfactual-pair evaluation exists for them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The near-neutral baseline values for abbreviated terms like 'pan' and 'bi' hint that lexical ambiguity contaminates the metric; a human-rating or homograph-control version of WinoQueer would tell whether LoRA is reducing prejudice or merely flattening token surprisal.
Because QueerNews is outsider-authored journalism, a queer-authored corpus of similar size should, under the paper's mechanism, produce larger or more durable reductions; that is a direct testable extension.
The observed over-correction in Gemma 7B, with several terms far below 50, suggests that one-epoch LoRA without a stopping rule can overshoot into reverse bias; adding a neutrality-anchored early stop would be a natural follow-up.
The paper itself flags in its limitations that polysemous tokens like 'pan' can blur identity signals, so the reported reductions for those terms should be read with that caveat.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful empirical datapoint on PEFT for queer-identity bias, and the direction of the effect is probably right, but the headline numbers are not verifiable as written because the score is not defined. Don't desk-reject it; send it back for a revision that pins down the metric.

What's new: the paper actually runs the comparison — LoRA vs soft prompts on three 7-8B open models with WinoQueer, and extends the benchmark with colloquial identity terms (pan, bi, ace, inter*). That is a concrete artifact others can build on, even if the technique itself is off-the-shelf. The pattern across tables is consistent, and the qualitative examples in Tables 3 and 4 give real evidence that LoRA changes continuations from derogatory to merely odd. The authors also list limitations honestly, including outsider-authored corpus bias and lexical ambiguity.

The soft spots: the central issue is that Section 2.2 says the score comes from 'converting' a conditional log-likelihood difference into a 0-100 scale, but the formula is never given. Later, Section 3.3 talks about 'log softmax difference,' which is not obviously the same thing. Without the normalization and without the neutrality threshold, a 50-point drop could partly reflect a global shift in identity-token marginal probabilities from fine-tuning on QueerNews, rather than a genuine fairness change. The paper does not release code or data, so that ambiguity cannot be resolved from the paper alone. Add to that single runs without error bars, a paired t-test that treats per-identity scores as independent, and deliberately polysemous tokens like 'pan' and 'bi' that the authors themselves flag as ambiguous. These are not fatal — the qualitative evidence makes me think LoRA genuinely shifts outputs — but they should be fixed before anyone reuses the numbers.

Who this is for: anyone working on bias evaluation or parameter-efficient alignment. It deserves serious referee time, but with the expectation of major revision: report the exact score function, define 'neutral' in advance, release code/data, and ideally report multiple seeds or confidence intervals.

I would not cite the numeric claims in their current form, but I'd cite the experimental setup once the metric is pinned down.

Referee Report

3 major / 5 minor

Summary. The paper proposes PRIDE, a parameter-efficient fine-tuning workflow that applies LoRA or soft-prompt tuning to QueerNews data to reduce anti-LGBTQIA+ bias in three 7-8B open-source LLMs (Llama 3 8B, Mistral 7B, Gemma 7B). Bias is measured with an extended WinoQueer benchmark, split into gender-identity and sexual-orientation subsets. The authors report that one LoRA epoch (<0.1% trainable parameters) reduces mean WinoQueer scores by up to roughly 40-50 points and raises the neutrality rate for Mistral to 32-36%, while 10-token soft prompts produce no reliable improvement. A paired t-test over per-identity scores is used to support the LoRA result, and qualitative inspection of top-gap sentences is reported.

Significance. If the quantitative results are taken at face value, the paper makes a useful practical contribution: it suggests that a one-epoch LoRA fine-tune on a modest-size community-curated corpus can move three widely used open models from strongly anti-queer WinoQueer scores toward neutrality, at a parameter cost that is far below full fine-tuning. The tables are internally consistent, the direction of the effect is uniform across models and identity categories, and the authors are candid about limitations. The main obstacle to accepting the headline numbers is that the paper never specifies the score formula or the neutrality decision rule; without these, the reported 50-point reductions and 36% neutrality rates cannot be verified or given a precise interpretation. The qualitative analysis and the paired t-test provide supporting but secondary evidence; the central claim needs the metric definition before I can recommend acceptance. No code or data is released, so reproducibility is currently conditional.

major comments (3)

[§2.2, §3.3, Tables 1-2] The WinoQueer score is never defined. Section 2.2 says that log p(suffix | identity) is 'converted into a bias score on a 0-100 scale' with 50 as neutrality, but it gives no formula, no normalization over templates, and no mapping from the likelihood difference to the 0-100 scale. Section 3.3 and Tables 3-4 instead describe 'log softmax output probabilities,' which is not the same quantity as the conditional log-likelihood in Section 2.2. The paper also reports 'neutrality rates' (Tables 1-2) without stating the decision rule that classifies an output as neutral; a score of 50 as a central tendency does not determine a per-sentence rate. These definitions are load-bearing for the abstract's claims of 'up to 50 points' reduction and '36%' neutrality, so they must be supplied, ideally with code or the exact evaluation script.
[§2.3, §3.2, §4(4)] The extended benchmark adds polysemous identity tokens ('pan', 'bi', 'ace', 'inter*') and the authors acknowledge in Limitation (4) that lexical ambiguity can blur identity signals. Without the score formula, the large post-LoRA drops for some terms—for example Gemma's 'demisexual' from 99.59 to 1.86 and 'pansexual' from 94.42 to 4.93 in Table 2—could be an artifact of the model raising the marginal probability of the identity token after training on QueerNews, rather than evidence that stereotypical associations were removed. Please report the exact normalization used in the score (e.g., conditional on the template, or a difference of joint log-likelihoods), and provide a sensitivity analysis that excludes or flags ambiguous tokens. This is necessary to establish that the improvements reflect fairness rather than a prior shift.
[§3.4, §2.5] The paired t-test in Section 3.4 treats the per-identity WinoQueer scores as paired observations, but Tables 1-2 contain single runs with no variance estimates, and Section 2.5 describes one training run per model and method. The identity terms within a model are not independent observations because they share the same WinoQueer templates, which can inflate the effective sample size. Please state the number of paired observations used in the test, report results across multiple random seeds or at least confidence intervals, and either justify the independence assumption or use a model-level analysis. This does not overturn the visible direction of the effect, but it affects the strength of the 'significant reduction' claim.

minor comments (5)

[§2.2] Section 2.2 cites 'WinoQueer (Zhao et al. 2018)', but the WinoQueer benchmark is introduced by Felkner et al. (2024) in the reference list; Zhao et al. is WinoBias. Please correct the citation.
[§3.3] Section 3.3 refers to 'section 4.1 and 4.2' for the quantitative results, but these are in Sections 3.1 and 3.2.
[§2.3] Section 2.3 says the original corpus has '45,540 counterfactual pairs' and the enlarged corpus contains '111,685 counterfactual sentences'; the move from pairs to sentences makes the 2.5x claim ambiguous. Please state the units consistently.
[Tables 1-2] Tables 1-2: the header row does not specify which columns are 0-100 bias scores and which are neutrality rates; add explicit column labels and units.
[§2.5] Section 2.5 reports LoRA hyperparameters (rank, alpha, dropout) but no learning rate or optimization schedule for either LoRA or soft-prompt tuning; please add these for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline bias reductions come from an external benchmark (WinoQueer) after LoRA training on a disjoint corpus (QueerNews), with hyperparameters taken from prior work rather than fitted to the evaluation.

full rationale

The paper's central claim is a before/after comparison on an external evaluation benchmark. Bias scores are computed with WinoQueer (Felkner et al. 2024), extended by the authors from existing counterfactual templates; LoRA and soft-prompt models are trained with standard language-modeling loss on the QueerNews corpus, which is disjoint from the WinoQueer evaluation templates. No WinoQueer score or neutrality percentage is used as a training objective, and no hyperparameter is selected on WinoQueer results: the LoRA rank, alpha, dropout, and the 10-token prompt are explicitly taken from prior work (Hu et al. 2021; Lester et al. 2021) without refinement. The paper's own equations therefore do not reduce to its inputs, and no fitted parameter is renamed as a prediction. The cited works are external, and the only author self-citations (Hagendorff 2020, 2024) are background context in related work, not load-bearing for the empirical result. The underspecified score-to-0-100 mapping, the undefined neutrality threshold, and the ambiguity of polysemous identity terms are validity and reproducibility concerns, not circularity: even if those problems were fixed, the derivation would still be an external benchmark evaluation of a training intervention, not a self-referential loop. Overall, the main result is self-contained with respect to circularity, so the score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No invented entities. The main result does not fit parameters to the evaluation benchmark, but it does depend on hand-chosen PEFT hyperparameters and on the WinoQueer operationalization of bias being valid.

free parameters (5)

LoRA rank = 8
Chosen from Hu et al. (2021) defaults; no HPO (Section 2.5).
LoRA alpha = 16
Scaling factor from prior LoRA usage; not tuned (Section 2.5).
LoRA dropout = 0.1
Chosen from prior work; no HPO (Section 2.5).
Soft prompt length = 10 virtual tokens
Set based on Lester et al. (2021); no HPO (Section 2.5).
Gradient accumulation factor = 32 (effective batch size ~128)
Chosen for GPU memory; affects optimization trajectory, not tuned.

assumptions (3)

domain assumption Conditional log-likelihood differences between counterfactual identity pairs validly operationalize anti-queer bias.
Borrowed from WinoQueer; the paper does not validate this operationalization and omits the exact formula (Section 2.2).
domain assumption The scraped QueerNews corpus is representative enough to debias without destroying model utility.
Only WinoQueer scores are reported; no downstream capability evaluation is given (Sections 2.4, 5).
ad hoc to paper Per-identity WinoQueer scores are independent, approximately normal observations suitable for a paired t-test.
The roughly 23 identity terms are correlated within the same model run and come from a single run; Shapiro-Wilk on this small sample is weak evidence of normality (Section 3.4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of PRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs." pith.science (2026). https://pith.science/paper/3LJDUQJE

@misc{pith2026250713743,
  author       = {Pith},
  title        = {Pith review of: PRIDE -- Parameter-Efficient Reduction of Identity Discrimination for Equality in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LJDUQJE}},
  note         = {Machine review of arXiv:2507.13743}
}

read the original abstract

Large Language Models (LLMs) frequently reproduce the gender- and sexual-identity prejudices embedded in their training corpora, leading to outputs that marginalize LGBTQIA+ users. Hence, reducing such biases is of great importance. To achieve this, we evaluate two parameter-efficient fine-tuning (PEFT) techniques - Low-Rank Adaptation (LoRA) and soft-prompt tuning - as lightweight alternatives to full-model fine-tuning for mitigating such biases. Using the WinoQueer benchmark, we quantify bias in three open-source LLMs and observe baseline bias scores reaching up to 98 (out of 100) across a range of queer identities defined by gender and/or sexual orientation, where 50 would indicate neutrality. Fine-tuning with LoRA (< 0.1% additional parameters) on a curated QueerNews corpus reduces those scores by up to 50 points and raises neutrality from virtually 0% to as much as 36%. Soft-prompt tuning (10 virtual tokens) delivers only marginal improvements. These findings show that LoRA can deliver meaningful fairness gains with minimal computation. We advocate broader adoption of community-informed PEFT, the creation of larger queer-authored corpora, and richer evaluation suites beyond WinoQueer, coupled with ongoing audits to keep LLMs inclusive.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

[3]

Nadeem, Moin; Bethke, Anna; Reddy, Siva (2020): StereoSet: Measuring stereotypical bias in pretrained language models

Available online at https://ai.meta.com/blog/meta-llama-3/, checked on 7/25/2024. Nadeem, Moin; Bethke, Anna; Reddy, Siva (2020): StereoSet: Measuring stereotypical bias in pretrained language models. In ArXiv:2004.09456, pp. 1–15. Parrish, Alicia; Chen, Angelica; Nangia, Nikita; Padmakumar, Vishakh; Phang, Jason; Thompson, Jana (2021): BBQ: A Hand-Built ...

arXiv 2020
[5]

2021)—offer a low‑cost alternative

and soft‑prompt tuning (Lester et al. 2021)—offer a low‑cost alternative. Prior works applied LoRA for domain adaptation or RLHF alignment (Sidahmed et al. 2024; Santacroce et al

work page 2021
[6]

its use for identity‑bias mitigation remains sparse. 2 Method 2.1 Models To ensure both computational fe asibility and societal relevance, we limit our analysis to open -source LLMs with fewer than 10 billion parameters—models that can be fine-tuned using modest GPU resources and are widely accessible to researchers and practitioners. Moreover, we priorit...

work page 2024
[7]

No one likes … people because …

and Google’s Gemma 7B (Google 2024), which is part of the Gemini Family. 2.2 Bias Metric – WinoQueer To quantify model biases, we use WinoQueer (Zhao et al. 2018). It evaluates bias through counterfactual sentence pairs. Each pair consists of two otherwise‑identical sentences that differ only in a single identity token (e.g., gay vs straight) and present ...

work page 2024
[12]

doesn’t count

Zhao, Jieyu; Wang, Tianlu; Yatskar, Mark; Ordonez, Vicente; Chang, Kai -Wei (2018): Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In ArXiv:1804.06876, pp. 1–6. 8 Appendix A Model neutrality Overall non-binary enby gender-fluid transgender trans* intersex inter* Llama 3 8B pretr. 00.08% 74.83 83.95 74.88 83.66 80.52 86.71 75.19 4...

arXiv 2018
[40]

non‑binary

seen for Gemma 7B. (4) Polysemous tokens or lexical ambiguity (e.g., “pan”) can blur identity signals and complicate debiasing. (5) QueerNews contains copyrighted, outsider‑authored journalism; redistribution is restricted, and framing may skew toward non‑community perspectives. (6) WinoQueer omits intersectional attributes, conversational safety, and cod...

arXiv 2024
[100]

across a range of queer identities defined by gender and/or sexual orientation, where 50 w ould indicate neutrality. Fine‑tuning with LoRA (< 0.1 % additional parameters) on a curated QueerNews corpus reduces those scores by up to 50 points and raises neutrality from virtually 0 % to as much as 36 %. Soft‑prompt tuning (10 virtual tokens) delivers only ma...

work page 2024
[128]

For soft-prompt tuning, we train 10 virtual tokens with random initialization, which corresponds to only about 0.0005% of the model's parameters

The full training process takes approximately 25 hours on four A100 GPUs for one epoch. For soft-prompt tuning, we train 10 virtual tokens with random initialization, which corresponds to only about 0.0005% of the model's parameters . We again use the Adafactor optimizer and a single epoch. Training follows the same schedule as LoRA but is early -stopped ...

work page 2021

Show all 12 references

[2018]

2020), and BBQ (Parrish et al

, StereoSet (Nadeem et al. 2020), and BBQ (Parrish et al

2020
[2021]

LGBTQIA+ bias has received limited scholarly attention: Tomasev et al

extend evaluation to multiple protected attributes. LGBTQIA+ bias has received limited scholarly attention: Tomasev et al. (2021) first flagged its near‑absence in fairness research, and Felkner (2024) later introduced WinoQueer, revealing persistent hetero‑normative bias acro...

2021
[2023]

Numerous studies reveal that LLMs inherit and sometimes magnify biases contained in their massive training corpora (Mehrabi et al. 2022). While racial and (binary) gender biases have received attention, bias against LGBTQIA+ identities remains under‑studied. Recent evidence sh...

2022
[2024]

To capture a richer spectrum of identities, we expand the vocabulary by adding colloquial and short forms that are common in community discourse

, which contains 45,540 sentence pairs built around four dominant terms ( straight, heterosexual, cisgender, cis ) and eight marginalized counterparts (LGBTQ, queer, transgender, bisexual, pansexual, lesbian, asexual, gay, non‑binary). To capture a richer spectrum of identitie...

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [3]

Nadeem, Moin; Bethke, Anna; Reddy, Siva (2020): StereoSet: Measuring stereotypical bias in pretrained language models

Available online at https://ai.meta.com/blog/meta-llama-3/, checked on 7/25/2024. Nadeem, Moin; Bethke, Anna; Reddy, Siva (2020): StereoSet: Measuring stereotypical bias in pretrained language models. In ArXiv:2004.09456, pp. 1–15. Parrish, Alicia; Chen, Angelica; Nangia, Nikita; Padmakumar, Vishakh; Phang, Jason; Thompson, Jana (2021): BBQ: A Hand-Built ...

arXiv 2020

[2] [5]

2021)—offer a low‑cost alternative

and soft‑prompt tuning (Lester et al. 2021)—offer a low‑cost alternative. Prior works applied LoRA for domain adaptation or RLHF alignment (Sidahmed et al. 2024; Santacroce et al

work page 2021

[3] [6]

its use for identity‑bias mitigation remains sparse. 2 Method 2.1 Models To ensure both computational fe asibility and societal relevance, we limit our analysis to open -source LLMs with fewer than 10 billion parameters—models that can be fine-tuned using modest GPU resources and are widely accessible to researchers and practitioners. Moreover, we priorit...

work page 2024

[4] [7]

No one likes … people because …

and Google’s Gemma 7B (Google 2024), which is part of the Gemini Family. 2.2 Bias Metric – WinoQueer To quantify model biases, we use WinoQueer (Zhao et al. 2018). It evaluates bias through counterfactual sentence pairs. Each pair consists of two otherwise‑identical sentences that differ only in a single identity token (e.g., gay vs straight) and present ...

work page 2024

[5] [12]

doesn’t count

Zhao, Jieyu; Wang, Tianlu; Yatskar, Mark; Ordonez, Vicente; Chang, Kai -Wei (2018): Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In ArXiv:1804.06876, pp. 1–6. 8 Appendix A Model neutrality Overall non-binary enby gender-fluid transgender trans* intersex inter* Llama 3 8B pretr. 00.08% 74.83 83.95 74.88 83.66 80.52 86.71 75.19 4...

arXiv 2018

[6] [40]

non‑binary

seen for Gemma 7B. (4) Polysemous tokens or lexical ambiguity (e.g., “pan”) can blur identity signals and complicate debiasing. (5) QueerNews contains copyrighted, outsider‑authored journalism; redistribution is restricted, and framing may skew toward non‑community perspectives. (6) WinoQueer omits intersectional attributes, conversational safety, and cod...

arXiv 2024

[7] [100]

across a range of queer identities defined by gender and/or sexual orientation, where 50 w ould indicate neutrality. Fine‑tuning with LoRA (< 0.1 % additional parameters) on a curated QueerNews corpus reduces those scores by up to 50 points and raises neutrality from virtually 0 % to as much as 36 %. Soft‑prompt tuning (10 virtual tokens) delivers only ma...

work page 2024

[8] [128]

For soft-prompt tuning, we train 10 virtual tokens with random initialization, which corresponds to only about 0.0005% of the model's parameters

The full training process takes approximately 25 hours on four A100 GPUs for one epoch. For soft-prompt tuning, we train 10 virtual tokens with random initialization, which corresponds to only about 0.0005% of the model's parameters . We again use the Adafactor optimizer and a single epoch. Training follows the same schedule as LoRA but is early -stopped ...

work page 2021