REVIEW 4 major objections 5 minor 16 references
On Adversarial Robustness of Language Models in Transfer Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Transfer learning often raises attack success even when accuracy improves, and scale appears to soften the effect.
desk verdict A useful new empirical grid on sequential fine-tuning and adversarial robustness, but the abstract's scale-resilience claim is contradicted by the paper's own GPT-2 family numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is a controlled comparison protocol: for each model, train (a) direct fine-tuning on the target dataset and (b) transfer learning from one large bias dataset to the target small dataset, then measure the relative change in Attack Success Rate (ΔASR) under two attacks. TextFooler generates adversarial examples by replacing important words with semantically similar substitutes; A2T uses gradient-based word importance and counter-fitted embeddings. For billion-parameter models the authors use LoRA (low-rank adaptation), which freezes base weights and trains small random adapters. The load-bearing quantity is the relative ΔASR between the two training regimes, and the paper's size-resilience claim is an observed inverse relationship between model parameter count and this ΔASR.
What would settle it
Re-run the GPT-2 family (base, medium, large, XL) with at least 10 random seeds per condition, identical learning rates and epochs, and measure ΔASR between direct fine-tuning and transfer. The size-resilience claim is falsified if the median ΔASR of GPT-2-large is not smaller than that of GPT-2-medium, or if the sign of the effect reverses across seeds.
Extended reading notes
Core claim
The central discovery is an empirical regularity: the average Attack Success Rate (ASR) of a model fine-tuned directly on a target dataset is usually lower than the ASR of the same model after it has first been trained on a different but related dataset and then transferred to the target. This holds across most of the 30 model-dataset combinations reported, with an average relative ASR increase often in the 8–30% range for smaller models, while original accuracy typically rises by 1–8%. The paper further finds that the size of the ASR increase shrinks as model size grows within the BERT, RoBERTa, and GPT-2 families, and that models adapted with LoRA show mixed, sometimes beneficial robustness shifts. The authors also report that adding 10% adversarial examples during transfer learning reduces ASR and raises accuracy-under-attack, at the cost of clean accuracy.
Load-bearing premise
The paper assumes the difference in attack success rate between transfer learning and direct fine-tuning is caused by the transfer process itself, not by single-run training noise, learning-rate tuning, or by pairing small models with full fine-tuning and large models with LoRA.
Editorial extensions
If this is right
- If the pattern holds, safety evaluation for fine-tuned LLMs should measure attack success rate or accuracy-under-attack, not just accuracy, because transfer learning can raise vulnerability while benchmarks improve.
- Smaller models deployed after transfer learning would be the highest-risk targets, since their ASR increases are largest.
- LoRA-adapted large models cannot be assumed to follow the same trend; their robustness after transfer varies by sequence and can even improve, so they need individual robustness checks.
- Including a fraction of adversarial examples during transfer learning is a viable mitigation: it lowers ASR and boosts accuracy-under-attack at a moderate cost to clean accuracy.
- The size-resilience trend implies that scaling model size may be one route to safer transfer, independent of architecture family.
Reading between the lines
- Because the large models were adapted with LoRA and the small ones with full fine-tuning, model size is entangled with adaptation method; the size-resilience claim may partly be an effect of LoRA's parameter isolation rather than scale itself.
- The datasets are all bias-detection tasks that share a similar label scheme; the transfer here is between closely related domains, so the 'transfer learning' effect may be smaller than in more distant domain adaptation, meaning the robustness loss could be even larger in real multi-stage pipelines.
- A practical metric suggested by the data: compute ΔASR per point of ΔOAcc ('robustness elasticity of transfer'); models with high positive elasticity would be flagged as unsafe to transfer even when accuracy improves.
- The variability in LoRA results (e.g., improved ASR on political bias for Phi-2 and Gemma) hints that adapter rank, initialization, or training order, rather than model scale, may be the controlling factor; a controlled study varying adapter rank at fixed size would test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether transfer learning (pre-training on a related bias dataset followed by fine-tuning on a target bias dataset) affects the adversarial robustness of text classifiers. Using three MBIB bias datasets, two attacks (TextFooler and A2T), and models ranging from BERT/GPT-2 to Gemma/Phi, the authors report that transfer learning often increases Attack Success Rate (ASR) while sometimes improving original accuracy, and they claim that larger models show greater resilience to this effect. The paper also includes an adversarial-training experiment (RQ4). The appendix provides per-sequence raw results for all transfer sequences.
Significance. If the scale-resilience finding were sound, it would be practically relevant: practitioners could expect larger models to be safer under transfer. The paper's strength is that it reports raw per-sequence results in the appendix and covers several architectures and bias domains. However, the headline scale claim is not supported by the paper's own Table 1: the GPT-2 family is non-monotonic in size, and model size is confounded with LoRA versus full fine-tuning. Because the paper's main novelty rests on this unsupported claim, the current contribution is limited to a descriptive observation that transfer learning often increases ASR.
major comments (4)
- [Section 3.2.1 and Table 1] The claim that 'larger models within each family (GPT-2, BERT, RoBERTa) exhibited smaller increases in ASR' is directly contradicted by the GPT-2 family: the average delta-ASR values are 11.6% for GPT-2, 12.7% for GPT-2-medium, 1.3% for GPT-2-large, and 5.3% for GPT-2-XL. The non-monotonic pattern, with medium worse than base, breaks the size-resilience premise stated in the abstract. This undermines the paper's central novelty.
- [Section 2.6 and Section 3.1.1] Model size is confounded with adaptation method: BERT, RoBERTa, and GPT-2 (up to 762M parameters) use full fine-tuning, while Gemma-2B, Phi-2, and GPT-2-XL use LoRA. Consequently, cross-family comparisons of 'larger models' cannot be attributed to scale alone. Section 3.1.1 itself concedes 'mixed results' for LoRA models, yet the abstract states the scale claim unconditionally.
- [Section 2.3 and Table 1] All results are reported from single runs with no error bars, standard deviations, or significance tests. The learning rate is selected from a wide range (5e-6 to 4e-4) and the best model is chosen on validation, so the observed delta-ASR differences (e.g., 1.9% vs. 20.4%) could reflect training noise or hyperparameter selection rather than a true effect of transfer learning. This lack of statistical grounding weakens every quantitative comparison in the paper.
- [Section 3.1 and Table 1] The example 'on the Hate Speech dataset, GPT-2 experiences a mean 20.4% increase in ASR' is factually incorrect. Table 1 shows GPT-2's Hate Speech delta-ASR is 1.94%; the 20.4% value corresponds to the Gender Bias dataset. This misattribution undermines the credibility of the narrative around the performance-robustness trade-off.
minor comments (5)
- [Table 8] The header of Table 8 lists 'OAcc ASR ASR' with a duplicated 'ASR' column; the third column should presumably be 'AUA'.
- [Section 3.1.1] The explanations involving 'false memories' and 'shortcut learning' are speculative and not tested by the experiments; they should be framed as hypotheses rather than conclusions.
- [Section 2.3] The learning rate range is very broad; the paper should report the exact learning rate used for each model, since this is essential for reproducibility.
- [Table 2] The percentages in Table 2 are not accompanied by the number of sequences considered per model; without this denominator, the percentages are difficult to interpret.
- [Figures 2 and 3] Figures 2 and 3 are referenced but do not appear in the text provided; they should include axis labels and, if possible, error bars or per-point values consistent with Table 1.
Circularity Check
No circularity found: the paper's claims are direct measurements, not derived quantities.
full rationale
This paper presents an empirical study of adversarial robustness under transfer learning. The headline quantities (OAcc, ASR, AUA, and their relative changes) are directly measured on trained models and reported in Tables 1 and 4 through 13. No parameter is fitted to a subset of data and then renamed as a prediction; no equation defines one reported quantity in terms of another in a way that would make the conclusion true by construction. The interpretive claims about model size, architecture, LoRA, and transfer learning are generalizations from these measurements, and the lack of multiple seeds or controlled comparisons is a correctness and evidence-quality concern, not a circularity concern. The paper does not rely on any self-citation to justify a load-bearing premise; citations are used for standard methods and prior work, not to force the paper's conclusions. Thus, under the stated circularity criteria, the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption ASR and AUA computed on the test set are valid and sufficient measures of adversarial robustness for these models.
- domain assumption The three MBIB bias datasets form a meaningful transfer-learning chain (related but distinct tasks).
- domain assumption TextFooler and A2T attacks, with default or unspecified parameters, are representative of realistic adversarial threats.
- domain assumption Training for 1 epoch on the source dataset and up to 6 epochs on the target, with early stopping on validation accuracy, is a fair operationalization of transfer learning versus direct fine-tuning.
Cite this review
Pith. "Pith review of On Adversarial Robustness of Language Models in Transfer Learning." pith.science (2026). https://pith.science/paper/ALDATNUC
@misc{pith2026250100066,
author = {Pith},
title = {Pith review of: On Adversarial Robustness of Language Models in Transfer Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/ALDATNUC}},
note = {Machine review of arXiv:2501.00066}
}
read the original abstract
We investigate the adversarial robustness of LLMs in transfer learning scenarios. Through comprehensive experiments on multiple datasets (MBIB Hate Speech, MBIB Political Bias, MBIB Gender Bias) and various model architectures (BERT, RoBERTa, GPT-2, Gemma, Phi), we reveal that transfer learning, while improving standard performance metrics, often leads to increased vulnerability to adversarial attacks. Our findings demonstrate that larger models exhibit greater resilience to this phenomenon, suggesting a complex interplay between model size, architecture, and adaptation methods. Our work highlights the crucial need for considering adversarial robustness in transfer learning scenarios and provides insights into maintaining model security without compromising performance. These findings have significant implications for the development and deployment of LLMs in real-world applications where both performance and robustness are paramount.
Figures
Reference graph
Works this paper leans on
-
[1]
Towards assurance of llm adversarial robustness using ontology-driven argumentation
Tomas Bueno Momˇcilovi´c, Beat Buesser, Giulio Zizzo, Mark Purcell, and Dian Balta. Towards assurance of llm adversarial robustness using ontology-driven argumentation. Valletta, Malta, July 2024. xAI 2024: World Conference on eXplainable Artificial Intelligence
work page 2024
-
[2]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018
arXiv 2018
-
[3]
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding.Communications of the ACM, 67(1): 110–120, 2023
work page 2023
-
[4]
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations?arXiv preprint arXiv:2405.05904, 2024
arXiv 2024
-
[5]
Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks.arXiv preprint arXiv:2004.10964, 2020
arXiv 2004
-
[6]
An overview of catastrophic ai risks
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001, 2023
arXiv 2023
-
[7]
Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021
arXiv 2021
-
[8]
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. InProceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018–8025, 2020
work page 2020
Show all 16 references
-
[9]
Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019
1907 arXiv
-
[10]
An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, 2023
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, 2023
2023 arXiv
-
[11]
Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019
2019
-
[12]
Adversarial attacks and defenses in large language models: Old and new threats
Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. Adversarial attacks and defenses in large language models: Old and new threats. InProceedings on, pp. 103–117. PMLR, 2023
2023
-
[13]
Introducing mbib-the first media bias identification benchmark task and dataset collection
Martin Wessel, Tomas Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, and Timo Spinde. Introducing mbib-the first media bias identification benchmark task and dataset collection. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informati...
2023
-
[14]
It is all about data: A survey on the effects of data on adversarial robustness.ACM Computing Surveys, 56(7):1–41, 2024
Peiyu Xiong, Michael Tegegn, Jaskeerat Singh Sarin, Shubhraneel Pal, and Julia Rubin. It is all about data: A survey on the effects of data on adversarial robustness.ACM Computing Surveys, 56(7):1–41, 2024
2024
-
[15]
Assessing adversarial robustness of large language models: An empirical study.arXiv preprint arXiv:2405.02764, 2024
Zeyu Yang, Zhao Meng, Xiaochen Zheng, and Roger Wattenhofer. Assessing adversarial robustness of large language models: An empirical study.arXiv preprint arXiv:2405.02764, 2024
2024 arXiv
-
[16]
Towards improving adversarial training of nlp models.arXiv preprint arXiv:2109.00544, 2021
Jin Yong Yoo and Yanjun Qi. Towards improving adversarial training of nlp models.arXiv preprint arXiv:2109.00544, 2021. 7 A Social Impact Statement Our research rigorously examines the balance between performance enhancements and security vulnerabilities in large language mode...
2021 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.