pith. sign in

arxiv: 2606.10610 · v1 · pith:ZZC6IAHPnew · submitted 2026-06-09 · 💻 cs.CL

Small Data, Big Noise: Adversarial Training for Robust Parameter-Efficient Fine-Tuning

Pith reviewed 2026-06-27 13:18 UTC · model grok-4.3

classification 💻 cs.CL
keywords adversarial trainingparameter-efficient fine-tuningrobustnesslow-resource NLPnoise robustnessdiscrete uncertainty setscharacter-level edits
0
0 comments X

The pith

A framework called SDBN integrates adversarial training into parameter-efficient fine-tuning to improve robustness and generalization on limited noisy data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SDBN as a way to combine adversarial training with PEFT methods, which have not been studied much together. It targets the common problem that PEFT models lose performance when training data is scarce or inputs contain noise like typos or word changes. Two variants create adversarial examples: one selects worst-case character edits using gradients, the other uses language model outputs for generative cases. Experiments across benchmarks show gains especially in low-resource conditions and under both word and character corruptions. The approach adds no parameters and only modest computation.

Core claim

SDBN brings adversarial training to PEFT by optimizing over discrete uncertainty sets of input variants, where SDBN-h selects gradient-chosen character-level edits and SDBN-p uses LLM-generated variants, yielding better robustness and generalization than standard PEFT without extra parameters.

What carries the argument

The SDBN framework, which performs robust optimization over discrete uncertainty sets of adversarial input variants during PEFT training.

If this is right

  • Performance gains are largest in low-resource settings and under word-level or character-level corruptions.
  • The method works for both classification and generative tasks via the two variants.
  • No additional model parameters are introduced.
  • Computational overhead remains modest compared to standard PEFT.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same discrete-set approach could be tested on modalities beyond text if similar uncertainty sets can be defined.
  • Combining SDBN with existing data-augmentation techniques might compound the robustness effect.
  • Deployment in production systems with unpredictable user input could be more reliable if the uncertainty sets match actual error patterns.

Load-bearing premise

The chosen character edits and LLM-generated variants represent enough real-world noise to deliver actual robustness gains.

What would settle it

Testing SDBN-trained models on noise distributions outside the character-edit and LLM-variant sets shows the reported robustness improvements disappear or reverse.

Figures

Figures reproduced from arXiv: 2606.10610 by Eitan Cohen, Idan Simai, Uri Shaham.

Figure 1
Figure 1. Figure 1: Performance degradation of PEFT meth￾ods on limited training data. Accuracy of three com￾mon PEFT methods: Adapter, BitFit, and LoRA–on a test set with word-swapping noise as the size of the clean Banking77 training data corpus is reduced (x-axis, number of examples). All methods exhibit a marked drop in noisy-test accuracy as the available training data dwindles, in some cases losing more than 50% below 1… view at source ↗
Figure 2
Figure 2. Figure 2: Perturbation regimes addressed by Adversarial Training. (a) PCA visualization of embedding space for TREC sentences, showing how character/word perturbations (triangles/squares) - fall within uncertainty regions (ellipses) around clean sentences (circles). This illustrates how our adversarial training approach effectively can improve the model’s performance on diverse linguistic variations that occur withi… view at source ↗
Figure 3
Figure 3. Figure 3: Adversarial Training Effect on Limited Data Scenarios. Relative performance improvement of adversarial training over baseline PEFT methods (e.g., 10% means SDBN-LoRA/LoRA = 1.10) across training sizes on BANKING77 and TREC. Adversarial training not only maintains but often improves clean-data perfor￾mance, with gains increasing as the training set shrinks. data size decreases. This validates our hypothesis… view at source ↗
Figure 4
Figure 4. Figure 4: Adversarial training for variable-intensity noise. Performance comparison of LoRA PEFT implemen￾tations under variable-intensity noise with DeBERTa-v3 on 1,000 BANKING77 samples. SDBN shows superior robustness in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Histogram of embedding differences between noisy and clean sentences across multiple noise types. The [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Heatmap of coordinate-wise embedding differences for multiple noise types. Each row is a noise type; [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Adversarial training for variable-intensity noise, BitFit PEFT. Performance comparison of BitFit PEFT implementations under variable-intensity noise conditions in different amplitudes with DeBERTa-v3 on 1,000 samples from Banking77 dataset. These results demonstrate the superior robustness of Adversarial train￾ing gradient-based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Adversarial training for variable-intensity noise, QLoRA PEFT. Performance comparison of QLoRA PEFT implementations under variable-intensity noise conditions in different amplitudes with DeBERTa￾v3 on 1,000 samples from Banking77 dataset. These results demonstrate the superior robustness of Adver￾sarial training gradient-based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 8
Figure 8. Figure 8: Adversarial training for variable-intensity noise, Adapter PEFT. Performance comparison of Adapter PEFT implementations under variable-intensity noise conditions in different amplitudes with DeBERTa￾v3 on 1,000 samples from Banking77 dataset. These results demonstrate the superior robustness of Adver￾sarial training gradient-based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Adversarial training for constant-intensity noise, LoRA PEFT. Performance comparison of LoRA PEFT implementations under constant-intensity noise conditions with DeBERTa-v3 on 1,000 samples from Banking77 dataset. These results demonstrate the supe￾rior robustness of Adversarial training gradient-based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Adversarial training for constant-intensity noise, QLoRA PEFT. Performance comparison of QLoRA PEFT implementations under constant-intensity noise conditions with DeBERTa-v3 on 1,000 samples from Banking77 dataset. These results demonstrate the superior robustness of Adversarial training gradient￾based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Adversarial training for constant-intensity noise, Adapter PEFT. Performance comparison of Adapter PEFT implementations under constant-intensity noise conditions with DeBERTa-v3 on 1,000 samples from Banking77 dataset. These results demonstrate the superior robustness of Adversarial training gradient￾based to noise in low-resource settings [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Domain Shifts. Comparison of LoRA PEFT methods on Arabic dialect sentiment classification us￾ing multilingual BERT-base trained on 270 samples of Egyptian Arabic without exposure to the other dialects during training. SDBN consistently outperforms alter￾natives across all dialects, with notable advantages on both the source and target domains [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) has become essential for adapting foundation models to downstream NLP tasks. However, current PEFT methods often struggle with robustness to noise and performance degradation on limited training data. We propose SDBN (Small Data Big Noise), a unified framework that brings adversarial training to PEFT - a combination that remains less studied in the PEFT setting despite its complementary strengths - to enhance model robustness and generalization, outperforming alternative approaches. We also introduce two variants of the method that use discrete uncertainty sets: SDBN-h, which enumerates character-level edits and selects worst-case variants using gradients, and SDBN-p, which uses LLM-generated variants for robust optimization in generative tasks. Experiments across multiple benchmarks reveal substantial improvements, particularly in low-resource settings and under both word-level and character-level corruptions. This framework addresses the less explored intersection of adversarial training and parameter-efficient adaptation, without introducing additional parameters or only modest computational overhead, making PEFT deployments more reliable in real-world scenarios where data scarcity and linguistic variability often coexist

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes SDBN, a unified framework that applies adversarial training to parameter-efficient fine-tuning (PEFT) methods. It introduces two variants using discrete uncertainty sets: SDBN-h (gradient-selected character-level edits) and SDBN-p (LLM-generated variants) for robust optimization, claiming substantial improvements in robustness and generalization on NLP benchmarks under word- and character-level noise, especially in low-resource settings, with no or modest additional parameters or overhead.

Significance. If the empirical claims hold after proper validation, the work would address a practically important gap at the intersection of adversarial training and PEFT, potentially improving reliability of adapted models in noisy, data-scarce deployments. The emphasis on parameter efficiency and modest overhead is a positive aspect for real-world applicability.

major comments (2)
  1. [Abstract] Abstract: the central claim of 'substantial improvements' and 'outperforming alternative approaches' is asserted without any quantitative results, baseline comparisons, effect sizes, or experimental details, preventing assessment of whether the data support the robustness claims.
  2. [SDBN-h and SDBN-p] Description of SDBN-h and SDBN-p: the robustness gains rest on the assumption that the discrete uncertainty sets (gradient-selected character edits in SDBN-h; LLM-generated variants in SDBN-p) represent real-world noise distributions. No validation against held-out noise distributions or ablation of post-hoc choices in variant generation is reported, which is load-bearing for the outperformance claims under corruptions.
minor comments (2)
  1. [Abstract] The title and abstract introduce SDBN without immediately clarifying that it stands for 'Small Data Big Noise'.
  2. Notation for the two variants (SDBN-h, SDBN-p) is introduced without an explicit table or diagram contrasting their uncertainty-set construction and optimization procedures.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful comments. We provide point-by-point responses to the major comments below.

read point-by-point responses
  1. Referee: [Abstract] Abstract: the central claim of 'substantial improvements' and 'outperforming alternative approaches' is asserted without any quantitative results, baseline comparisons, effect sizes, or experimental details, preventing assessment of whether the data support the robustness claims.

    Authors: The abstract serves as a high-level overview, while the manuscript's experimental section details the quantitative results, baseline comparisons, and effect sizes across NLP benchmarks under noise conditions. To better support the claims in the abstract, we will revise it to include specific quantitative highlights from our experiments. revision: yes

  2. Referee: [SDBN-h and SDBN-p] Description of SDBN-h and SDBN-p: the robustness gains rest on the assumption that the discrete uncertainty sets (gradient-selected character edits in SDBN-h; LLM-generated variants in SDBN-p) represent real-world noise distributions. No validation against held-out noise distributions or ablation of post-hoc choices in variant generation is reported, which is load-bearing for the outperformance claims under corruptions.

    Authors: Our uncertainty sets are designed based on prevalent noise types in real-world NLP data, with SDBN-h targeting character-level edits common in typos and SDBN-p capturing generative variations. The reported experiments evaluate robustness on held-out test sets with injected word- and character-level corruptions, providing evidence for the claims. We recognize the value of additional validation using independent real-world noise datasets and ablations on variant generation choices. We will incorporate these in the revised version to strengthen the manuscript. revision: yes

Circularity Check

0 steps flagged

No circularity detected in derivation or claims

full rationale

The paper presents SDBN as an empirical framework proposal combining adversarial training with PEFT, supported by benchmark experiments on robustness under noise. No equations, derivations, or parameter-fitting steps are described that reduce by construction to the inputs (e.g., no fitted uncertainty sets renamed as predictions, no self-definitional relations, and no load-bearing self-citations invoked as uniqueness theorems). The method variants (SDBN-h/p) are defined explicitly via gradient selection and LLM generation, with performance claims resting on external validation rather than internal tautology. This is a standard non-circular empirical proposal.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no equations, parameters, or explicit assumptions; ledger entries cannot be populated beyond noting the introduction of the SDBN framework and its two variants.

pith-pipeline@v0.9.1-grok · 5719 in / 1009 out tokens · 23027 ms · 2026-06-27T13:18:44.273182+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 15 canonical work pages · 8 internal anchors

  1. [1]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , title =. arXiv preprint arXiv:2106.09685 , year =

  2. [2]

    MEL o RA : M ini- E nsemble L ow- R ank A dapters for P arameter- E fficient F ine- T uning

    Ren, Pengjie and Shi, Chengshun and Wu, Shiguang and Zhang, Mengqi and Ren, Zhaochun and de Rijke, Maarten and Chen, Zhumin and Pei, Jiahuan. MEL o RA : M ini- E nsemble L ow- R ank A dapters for P arameter- E fficient F ine- T uning. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi...

  3. [3]

    International Conference on Learning Representations (ICLR) , year =

    Qingru Zhang and Minshuo Chen and Alexander Bukharin and Nikos Karampatziakis and Pengcheng He and Yu Cheng and Weizhu Chen and Tuo Zhao , title =. International Conference on Learning Representations (ICLR) , year =

  4. [4]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Tim Dettmers and Artidoro Pagnoni and Ari Holtzman and Luke Zettlemoyer , title =. arXiv preprint arXiv:2305.14314 , year =

  5. [5]

    arXiv preprint arXiv:2404.13425 , year =

    Yuheng Ji and Yue Liu and Zhicheng Zhang and Zhao Zhang and Yuting Zhao and Gang Zhou and Xingwei Zhang and Xinwang Liu and Xiaolong Zheng , title =. arXiv preprint arXiv:2404.13425 , year =

  6. [6]

    2024 , pages=

    Fu, Jiadong and Fang, Jiang and Sun, Jiyan and Zhuang, Shangyuan and Geng, Liru and Liu, Yinlong , booktitle=. 2024 , pages=

  7. [7]

    International Conference on Learning Representations (ICLR) , year =

    Chen Zhu and Yu Cheng and Zhe Gan and Siqi Sun and Tom Goldstein and Jingjing Liu , title =. International Conference on Learning Representations (ICLR) , year =

  8. [8]

    Neurocomputing , volume =

    Understanding adversarial training:. Neurocomputing , volume =. 2018 , issn =. doi:https://doi.org/10.1016/j.neucom.2018.04.027 , url =

  9. [9]

    2021 , organization=

    Ben-Zaken, Elad and Ravfogel, Shauli and Goldberg, Yoav , booktitle=. 2021 , organization=

  10. [10]

    2019 , organization=

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanisław and Morrone, Bruna and de Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , booktitle=. 2019 , organization=

  11. [11]

    Kim, Yeachan and Kim, Junho and Lee, SangKeun , booktitle=. Towards. 2024 , organization=

  12. [12]

    Towards Deep Learning Models Resistant to Adversarial Attacks

    Towards deep learning models resistant to adversarial attacks , author=. arXiv preprint arXiv:1706.06083 , year=

  13. [13]

    International Conference on Learning Representations (ICLR) , year =

    Explaining and Harnessing Adversarial Examples , author =. International Conference on Learning Representations (ICLR) , year =

  14. [14]

    Robust Optimization , author =

  15. [15]

    arXiv preprint arXiv:2003.04807 , year =

    Iñigo Casanueva and Tadas Temčinas and Daniela Gerz and Matthew Henderson and Ivan Vulić , title =. arXiv preprint arXiv:2003.04807 , year =

  16. [16]

    Proceedings of the Eighth Text REtrieval Conference (TREC-8) , year =

    Amit Singhal and Steve Abney and Michiel Bacchiani and Michael Collins and Donald Hindle and Fernando Pereira , title =. Proceedings of the Eighth Text REtrieval Conference (TREC-8) , year =

  17. [17]

    1995 , booktitle =

    Ken Lang , title =. 1995 , booktitle =

  18. [18]

    Convex optimization , author=

  19. [19]

    Preprint , year=

    NEFTune: Noisy Embeddings Improve Instruction Finetuning , author=. Preprint , year=

  20. [20]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Towards Robust and Generalized Parameter-Efficient Fine-Tuning for Noisy Label Learning , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2024 , organization=

  21. [21]

    arXiv preprint arXiv:2110.07602 , year =

    P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks , author =. arXiv preprint arXiv:2110.07602 , year =

  22. [22]

    Prefix-Tuning: Optimizing Continuous Prompts for Generation

    Prefix-Tuning: Optimizing Continuous Prompts for Generation , author =. arXiv preprint arXiv:2101.00190 , year =

  23. [23]

    2019 , url=

    Jiang, Haoming and He, Pengcheng and Chen, Weizhu and Liu, Xiaodong and Gao, Jianfeng and Zhao, Tuo , journal=. 2019 , url=

  24. [24]

    2021 , url=

    Miyato, Takeru and Dai, Andrew M and Goodfellow, Ian , journal=. 2021 , url=

  25. [25]

    Proceedings of the 25th International Conference on Machine Learning (ICML 2008) , pages =

    Extracting and Composing Robust Features with Denoising Autoencoders , author =. Proceedings of the 25th International Conference on Machine Learning (ICML 2008) , pages =. 2008 , url =

  26. [26]

    Proceedings of the Workshop on Understanding Foundation Models at ICLR , year =

    Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs , author =. Proceedings of the Workshop on Understanding Foundation Models at ICLR , year =

  27. [27]

    Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic

    Abu Farha, Ibrahim and Zaghouani, Wajdi and Magdy, Walid. Overview of the WANLP 2021 Shared Task on Sarcasm and Sentiment Detection in Arabic. Proceedings of the Sixth Arabic Natural Language Processing Workshop. 2021

  28. [28]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author =. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages =

  29. [29]

    He, Pengcheng and Gao, Jianfeng and Chen, Weizhu , booktitle =

  30. [30]

    SQuAD: 100,000+ Questions for Machine Comprehension of Text

    Pranav Rajpurkar and Jian Zhang and Konstantin Lopyrev and Percy Liang , title =. arXiv preprint arXiv:1606.05250 , year =

  31. [31]

    Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, EMNLP 2011 , pages =

    Marco Baroni and Alessandro Lenci , title =. Proceedings of the GEMS 2011 Workshop on Geometrical Models of Natural Language Semantics, EMNLP 2011 , pages =. 2011 , address =

  32. [32]

    Neural Information Processing (ICONIP 2019) , series=

    Models in the Wild: On Corruption Robustness of Neural NLP Systems , author=. Neural Information Processing (ICONIP 2019) , series=. 2019 , url=

  33. [33]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Combating Adversarial Misspellings with Robust Word Recognition , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  34. [34]

    2018 , url=

    Ebrahimi, Javid and Rao, Anyi and Lowd, Daniel and Dou, Dejing , booktitle=. 2018 , url=

  35. [35]

    Findings of the Association for Computational Linguistics: ACL 2022 , year=

    Improving Zero-Shot Cross-lingual Transfer Between Closely Related Languages by Injecting Character-Level Noise , author=. Findings of the Association for Computational Linguistics: ACL 2022 , year=

  36. [36]

    arXiv preprint arXiv:1901.11196 , year =

    Jason Wei and Kai Zou , title =. arXiv preprint arXiv:1901.11196 , year =

  37. [37]

    2025 , month = dec, howpublished =

    Update to. 2025 , month = dec, howpublished =

  38. [38]

    The Llama 3 Herd of Models

    The. arXiv preprint arXiv:2407.21783 , year =

  39. [39]

    arXiv preprint arXiv:1911.12237 , year =

    Gliwa, Bogdan and Mochol, Iwona and Biesek, Maciej and Wawer, Aleksander , title =. arXiv preprint arXiv:1911.12237 , year =

  40. [40]

    Computational Linguistics , volume =

    Jiehang Zeng and Jianhan Xu and Xiaoqing Zheng and Xuanjing Huang , title =. Computational Linguistics , volume =. 2023 , url =

  41. [41]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (

    Po-Sen Huang and Robert Stanforth and Johannes Welbl and Chris Dyer and Dani Yogatama and Sven Gowal and Krishnamurthy Dvijotham and Pushmeet Kohli , title =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (. 2019 , url =

  42. [42]

    Certified Robustness to Adversarial Word Substitutions , booktitle =

    Robin Jia and Aditi Raghunathan and Kerem Gokhan G. Certified Robustness to Adversarial Word Substitutions , booktitle =. 2019 , url =

  43. [43]

    Yi Zhou and Xiaoqing Zheng and Cho-Jui Hsieh and Kai-Wei Chang and Xuanjing Huang , title =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages =. 2021 , url =

  44. [44]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =

    Maor Ivgi and Jonathan Berant , title =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages =. 2021 , url =

  45. [45]

    Ajinkya More , title =

  46. [46]

    2019 , address =

    Xiong, Wenhan and Wu, Jiawei and Wang, Hong and Kulkarni, Vivek and Yu, Mo and Chang, Shiyu and Guo, Xiaoxiao and Wang, William Yang , booktitle =. 2019 , address =

  47. [47]

    Qwen2.5 Technical Report

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =. 2412.15115 , archivePrefix =

  48. [48]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author =. arXiv preprint arXiv:2307.09288 , year =. 2307.09288 , archivePrefix =