Limited-Resource Adapters Are Regularizers, Not Linguists

E. Djeride Jean-Baptiste; Ernests Lavrinovics; Heather Lent; Johannes Bjerva; Marcell Fekete; Nathaniel R. Robinson; Raj Dabre

REVIEW 3 major objections 5 minor 20 references

Limited-Resource Adapters Are Regularizers, Not Linguists

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that in low-resource machine translation, adapter gains come from parameter regularization, not from linguistic information stored in trained adapter weights, because untrained adapters perform as well as trained ones.

desk verdict The random-adapter baseline result is real and useful, but the 'regularizer, not linguist' mechanism claim is not supported by the shared-initialization control. read the letter →

arxiv 2505.24525 v1 pith:I4L3RZIV submitted 2025-05-30 cs.CL

Marcell Fekete , Nathaniel R. Robinson , Ernests Lavrinovics , E. Djeride Jean-Baptiste , Raj Dabre , Johannes Bjerva , Heather Lent This is my paper

classification cs.CL

keywords adaptersoupingcross-attentionfine-tuninglow-resourcemachinetranslationparameter-efficientCreolelanguagesregularizationcross-lingualtransferrandominitialization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that, in low-resource machine translation with cross-attention fine-tuning, the gains credited to adapter souping are due to parameter regularization rather than to linguistic knowledge carried by trained adapter weights. Working with three Creole languages, Haitian, Papiamento, and Sango, and with transfer languages chosen by phylogeny, typology, model embeddings, and subword statistics, the authors find that performance is essentially the same whether the adapters were trained on related languages, unrelated languages, or not trained at all. They conclude that adapter benefit in this setting lies in regularization of fine-tuning on small parallel data, and they support this with lower dev loss, lower gradient norms, and reduced parameter variance under souping. If the claim is right, it reframes a common PEFT assumption: the choice of transfer language can matter far less than the regularizing effect of weight averaging.

What carries the argument

The machinery is adapter souping: weight-space averaging of bottleneck adapters, combined with cross-attention fine-tuning of a frozen distilled NLLB-200 model. Denoising adapters are trained per language on monolingual data, source and target adapters are inserted in encoder and decoder, and only decoder cross-attention is unfrozen and fine-tuned on small parallel segments. The decisive instrument is the control: an untrained adapter with the same random initialization as every trained adapter, used either alone or souped with the Creole adapter, which isolates regularization from information transfer.

What would settle it

Soup a trained adapter with an untrained adapter initialized from a different random seed under the same cross-attention fine-tuning setup. If the gain remains, untrained adapters act as a generic noise regularizer; if the gain vanishes, the paper's control conflates regularization with shrinkage toward the shared initialization.

Watch

Extended reading notes

Core claim

The central claim is that bottleneck adapters combined with cross-attention fine-tuning improve low-resource MT through parameter regularization, not meaningful information transfer. The evidence is that BLEU gains over the base model are comparable for linguistically motivated transfer, for unrelated control language families, and for an untrained adapter that shares the same random initialization as all trained adapters; untrained souping even gives the highest Haitian-to-English score. Supporting analyses show that souping reduces parameter variance, lowers dev loss and gradient norms relative to unregularized adapters, and a native-speaker evaluation finds that untrained souping avoids grammatical errors at least as well as transfer from Indo-European relatives.

Load-bearing premise

The conclusion depends on treating the randomly initialized adapter as carrying no useful information; since all trained and untrained adapters share the same starting point, averaging with the untrained one pulls the trained weights back toward that start, and the experiment may be measuring that pull rather than generic noise-based regularization.

Editorial extensions

If this is right

If the central claim is right, choosing a transfer language by linguistic criteria such as phylogeny, typology, embedding similarity, or Subword Evenness yields no reliable BLEU advantage over unrelated or untrained adapters in this setting.
The method still gives a practical gain: for Creole-to-English, souping with untrained or unrelated adapters improves over the cross-attention-fine-tuned base model by roughly 2 to 8 BLEU depending on the language.
The same qualitative pattern appears with only 800 parallel fine-tuning segments, so the regularization effect appears robust to very small data conditions.
Untrained souping shows lower dev loss and gradient norms than the unregularized adapter setup, which is direct evidence for the regularization reading.
An untrained adapter can replace the trained target adapter and still beat the base model, so even the standalone adapter gain may be regularizing rather than transferring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive extension would soup a trained adapter with an untrained adapter initialized at a different random seed; if the gain disappears, the effect is shrinkage toward the shared initialization rather than generic noise-based regularization.
The regularization framing is testable outside bottleneck adapters: rerun the same comparison with LoRA or other low-rank adaptation methods to see whether untrained adapters regularize there as well.
For practitioners, this suggests a cheap baseline: cross-attention fine-tuning plus an untrained-adapter soup may capture most of the benefit of linguistically motivated transfer-language selection in low-resource MT.
If the pattern generalizes, PEFT gains in extremely low-resource settings should not be read as evidence of linguistic knowledge transfer unless an untrained-initialization control is included.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear X,

The headline: the empirical observation holds. When you soup adapters with a randomly initialized adapter and do cross-attention fine-tuning on a distilled NLLB model, you get the same BLEU as souping with 'related' languages. That is a useful negative result, and it should make random-adapter baselines standard in PEFT transfer work.

The paper does several things well. It compares five principled transfer-selection criteria against untrained and unrelated-language controls across three Creoles, includes a Catalan sanity check where even close relatives don't help, and shows parameter-variance reduction when souping. The manual Haitian evaluation is a nice touch. The empirical claim is likely robust.

The soft spot is the title and abstract. 'Regularizers, not linguists' is a mechanism claim, and the evidence doesn't get there. The untrained adapter control is not a no-information control: all adapters share the same random initialization, so souping a trained adapter with the untrained one is interpolation toward the common start point. With the 1:3 ratio, the trained weights are diluted to a quarter. Equal BLEU across conditions could be produced by shrinkage toward init, which is a form of regularization, but not the 'random noise' story the paper tells. So the title overstates what the control shows. The paper itself walks this back in the Limitations section, saying rigorous verification is outside scope, but the abstract doesn't carry that caution.

Other soft spots: no error bars or significance tests, one seed for main results, and the main direction was chosen after preliminary runs with 800 segments. The into-Creole negative result only appears in the appendix. No code is released, so the numbers aren't independently checkable. These are minor compared to the interpretational issue.

The citation pattern is fine. Related work on adapter souping, regularization, and Creole transfer is covered.

Bottom line: the empirical result deserves serious referee time. It reframes how people should read PEFT transfer gains. But the mechanism claim needs either a cleaner control—different init seeds for the 'untrained' adapter—or a softer framing. I'd send it to review with a request for major revision. For my own work, I'd cite the empirical finding, not the title.

Referee Report

3 major / 5 minor

Summary. The paper studies adapter-based cross-lingual transfer for three low-resource Creole languages (Haitian, Papiamento, Sango) using a 600M-parameter NLLB-200 model. The method combines monolingual denoising adapters, weight-space averaging ('souping') of target and helper-language adapters, and cross-attention fine-tuning on 10k parallel segments. The authors compare six criteria for selecting transfer languages (phylogeny, Niger-Congo relatives, typological distance, NLLB embeddings, Subword Evenness, and Creole-to-Creole transfer) against unrelated-language controls (Uralic, CJK, Dravidian) and an untrained-adapter control. Across conditions, BLEU gains over the base CA-FT baseline are similar, and the untrained-adapter soup matches or exceeds the trained-adapter conditions. The paper interprets this as evidence that adapters act as regularizers rather than as carriers of linguistic information, and supports this with loss/variance analyses, a Catalan control, and a small native-speaker evaluation.

Significance. The empirical observation that linguistically motivated transfer-language selection provides no measurable benefit over unrelated languages or an untrained adapter, in this setup, is a valuable cautionary result for PEFT-based cross-lingual transfer. The paper's strengths include a systematic comparison of several transfer criteria, explicit unrelated-language and untrained controls, a supplementary low-resource condition (800 segments), preliminary analyses of loss and parameter variance, and an honest limitations section that concedes the difficulty of verifying the mechanism. If the findings are robust, they support the broad claim that much of the adapter benefit in limited-resource settings is regularization-like rather than linguistic transfer. However, the title-level mechanism claim is currently underdetermined by the shared-initialization confound in the control and by the absence of error bars or significance tests, so the paper requires revision before the strong 'regularizers, not linguists' conclusion is fully supported.

major comments (3)

[§2.2, §2.3, Table 2] The 'untrained adapter' control does not cleanly separate noise-based regularization from shrinkage. All adapters, trained and untrained, are initialized from the same seed (§2.2), and the 1:3 soup is exactly 0.25·θ_trained + 0.75·θ0, i.e., linear interpolation of the trained adapter toward the common initialization. This is a structured intervention, not an independent random perturbation. Equal BLEU across the IE/NC/typology soups and the untrained soup could therefore arise from a shrinkage mechanism that damps update magnitudes, rather than from the 'random noise' regularization the paper invokes. Please add a control that does not share initialization (e.g., an untrained adapter with a different seed, or an additive Gaussian perturbation of the trained adapter), or rephrase the conclusion to say that the benefits are consistent with shrinkage/regularization rather than with linguistic content and that 'no information transfer' is not demonstrated by this control.
[§3, Table 2 and Table 6] All main results appear to come from a single run per condition, with no error bars, confidence intervals, or significance tests. For a paper whose central empirical claim is that conditions are 'virtually indistinguishable'—a null result—reporting per-condition variance is essential. A 0.5-BLEU spread across conditions can be within run-to-run noise, while some apparent differences (e.g., 37.42 vs. 36.41 for hat→eng) could be meaningful if measured with more runs. Please report multiple seeds with standard deviations or bootstrap intervals, and ideally paired tests across conditions, to support the claim that transfer criteria and the untrained control are equivalent.
[Limitations section, 'Difficulty in Verifying Neural Phenomena'] The paper's final limitations paragraph states that rigorous regularization analysis is 'outside the scope of this current work' and that verifying the mechanism is difficult. Yet the title and abstract assert more categorically that adapters 'are regularizers, not linguists' and that the benefit 'lies in parameter regularization.' The analytical support provided—lower validation loss with the untrained soup (Table 7) and reduced parameter variance (Figure 2), both from single runs—is suggestive but not decisive. Please align the title/abstract claims with the admitted evidentiary status, e.g., formulate the mechanism as a hypothesis consistent with the observations, or provide the rigorous analysis that the limitations section defers to future work.

minor comments (5)

[Abstract and §3] The phrase 'randomly initialized adapters' is ambiguous: these are untrained adapters that share the initialization with all trained adapters, not random noise added at inference. Consider using 'untrained adapters' consistently and clarifying the shared-initialization design early in the paper.
[Appendix E, Table 7] The table is labeled 'Validation loss' but the appendix text says the loss is evaluated on a subset of the CA-FT training data. This is train loss, not validation loss; please correct the terminology and clarify what the metric measures.
[Figure 2] The figure would benefit from labeled axes and a description of which boxplot corresponds to which condition; currently it is difficult to interpret the claim that parameter variance 'decreases substantially' without a legend or numerical values.
[Appendix D, Table 6] The rows 'Random (same init) (1 : 1)' and 'Random (same init) (1 : 3)' are presented alongside the main results, but the main text (Table 2) only reports the 1:3 ratio. Clarify which ratio is used in the main results and why the 1:1 condition is not reported there, to avoid confusion.
[§2.3] The sentence describing the soup ratio with the init adapter ('1 : 3, to closer approximate the other setups') is grammatically unclear. Rewriting it as 'the Creole adapter is averaged with the init adapter in a 1:3 ratio, matching the three-way averaging used elsewhere' would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the untrained-adapter control is an external baseline, not a fitted prediction.

full rationale

The paper's central claim is empirical: linguistically motivated transfer-language selection performs no better than unrelated-language or untrained-adapter controls in a low-resource MT setup. This is a direct experimental comparison, not a derivation that reduces to its inputs. No parameter is fitted to the reported BLEU values and then relabeled as a prediction; the untrained adapter is an external control constructed by sharing a random initialization (Sections 2.2 and 2.3), and its construction does not presuppose the regularization conclusion. The interpretive step attributing gains to regularization is post hoc, and the authors acknowledge this uncertainty in the Limitations section (“it is difficult to verify this with certainty”). The skeptic's point that souping with the untrained adapter is shrinkage toward the shared initialization (0.25·θ_trained + 0.75·θ_0) is a plausible alternative mechanism, but an alternative explanation of a control is a confound or interpretive weakness, not circularity: the empirical equality of scores would support the broad claim whether the mechanism is noise-based regularization or weight-space shrinkage. Citations to prior work (Chronopoulou et al.; Üstün et al.) supply experimental procedures rather than the paper's conclusion, and self-citations (Lent et al.) provide background on Creole transfer without carrying the load-bearing argument. No equation identifies a derived quantity with an input by construction, and no fitted constant is presented as a prediction. The derivation chain is therefore self-contained, and any concerns about the strength of the evidence belong to correctness/interpretation rather than circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No parameters are fitted to a target result; the main hand-chosen value is the 1:3 soup ratio for the untrained adapter. The claim relies on imported assumptions about language relatedness measures, the validity of the shared-initialization control, and the adequacy of BLEU/chrF on a single model, all stated in the paper.

free parameters (1)

Souping ratio for untrained adapter = 1:3 (Creole:init)
Chosen by hand in Section 2.3 to approximate the equal-weight soup of one Creole adapter with three helper adapters; affects the Untrained Souping condition.

assumptions (4)

standard math Weight averaging of adapters in parameter space preserves or improves task performance (Eq. 1).
Imported from Wortsman et al. 2022 and Chronopoulou et al. 2023a; the paper relies on this for souping.
domain assumption lang2vec distances, NLLB language embeddings, and SuE scores are valid proxies for cross-lingual transfer usefulness.
Used in Section 2.1 and Appendix C to select transfer languages; if these measures do not capture relevant similarity, the comparison of linguistic relatedness is weakened.
domain assumption The distilled NLLB-600M model and BLEU/chrF metrics adequately represent MT behavior for these languages.
All experiments use one 600M-parameter NLLB-200 distillation and Flores-200 BLEU/chrF; the conclusion may not generalize to other model families or metrics.
domain assumption The identified Indo-European and Niger-Congo relatives for Haitian, Papiamento, and Sango are correct.
Creole phylogeny is disputed; the paper relies on APICS and selected references in Section C.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Limited-Resource Adapters Are Regularizers, Not Linguists." pith.science (2026). https://pith.science/paper/I4L3RZIV

@misc{pith2026250524525,
  author       = {Pith},
  title        = {Pith review of: Limited-Resource Adapters Are Regularizers, Not Linguists},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4L3RZIV}},
  note         = {Machine review of arXiv:2505.24525}
}

read the original abstract

Cross-lingual transfer from related high-resource languages is a well-established strategy to enhance low-resource language technologies. Prior work has shown that adapters show promise for, e.g., improving low-resource machine translation (MT). In this work, we investigate an adapter souping method combined with cross-attention fine-tuning of a pre-trained MT model to leverage language transfer for three low-resource Creole languages, which exhibit relatedness to different language groups across distinct linguistic dimensions. Our approach improves performance substantially over baselines. However, we find that linguistic relatedness -- or even a lack thereof -- does not covary meaningfully with adapter performance. Surprisingly, our cross-attention fine-tuning approach appears equally effective with randomly initialized adapters, implying that the benefit of adapters in this setting lies in parameter regularization, and not in meaningful information transfer. We provide analysis supporting this regularization hypothesis. Our findings underscore the reality that neural language processing involves many success factors, and that not all neural methods leverage linguistic knowledge in intuitive ways.

Figures

Figures reproduced from arXiv: 2505.24525 by the authors.

**Figure 1.** Overview of the MT transfer experiments between English and Creoles. The ↑ path shows the use of source language adapter in the encoder and the ‘souped’ (weight-averaged) target and transfer language adapters in the decoder layers. The ↑ path shows the opposite direction. (Hu et al., 2021), and (IA)3 (Liu et al., 2022a) are widely used to adapt language models to lowresource languages (Parovic et al. ´ , 2022; Paro… view at source ↗

**Figure 2.** Parameter variance between the pretrained [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Training loss curves for a subset of adapter training experiments. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

[1]

Sentence ends with a terminal punctuation mark (’.’; ’!’, ’ " ’, ’?’)

work page
[2]

Ratio of numerical characters with respect to all other characters is less than 0.25

work page
[3]

Sentence starting with a bullet-point

work page
[4]

In Proceedings of the 29th International Conference on Computational Linguistics, pages 4397–4406, Gyeongju, Republic of Korea

How to parse a creole: When martini- can creole meets French. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4397–4406, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Reza Moradi, Reza Berangi, and Behrouz Minaei. 2020. A survey of regularization strategies for deep models. Artif...

work page 2020
[5]

Ratio of stop-words with respect to all other tokens is less than 0.6 using whitespace tok- enization

work page
[6]

PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to English

No language left behind: Scaling human- centered machine translation. Kelechi Ogueji and Orevaoghene Ahia. 2019. Pidg- inunmt: Unsupervised neural machine translation from west african pidgin to english. arXiv preprint arXiv:1912.03444. Wuraola Fisayo Oyewusi, Olubayo Adekanmbi, and Olalekan Akinsande. 2020. Semantic enrichment of nigerian pidgin english ...

work page Pith review arXiv 2019
[7]

Sentence length is less than 14 and more than 300 characters

work page
[8]

Kreyòl-MT: Building MT for Latin Amer- ican, Caribbean and colonial African creole lan- guages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume1: Long Papers), pages 3083– 3110, Mexico City, Mexico. Association for Compu- tational Linguistics. Nathan...

work page 2024

Show all 20 references

[9]

In 4th Workshop on African Natural Language Processing

African substrates rather than european lex- ifiers to augment african-diaspora creole transla- tion. In 4th Workshop on African Natural Language Processing. Hedvig Skirgård, Hannah J. Haynie, Damián E. Blasi, Harald Hammarström, Jeremy Collins, Jay J. Latarche, Jakob Lesage, ...

2024 arXiv
[10]

For this we employ FastText model 5

Language identification does not match the ex- pected language. For this we employ FastText model 5. This preprocessing removed approximately 20- 23% of sentences, depending on the language. The same preprocessing steps were also applied to NLLB parallel data used for cross-at...

2022
[13]

Sentence containing a lorem ipsum substring
[15]

4https://github.com/mediacloud/sentence-splitter

Ratio of token count that does not contain alpha characters with respect to total token count is less than 0.25, using whitespace tok- enization. 4https://github.com/mediacloud/sentence-splitter
[17]

Ratio of capitalized characters is less than 0.2
[18]

Ratio of alpha characters with respect to all other characters is more than 0.6
[2014]

CoRR, abs/1406.1831

Analyzing noise in autoencoders and deep networks. CoRR, abs/1406.1831. Vipul Rathore, Rajdeep Dhingra, Parag Singla, and Mausam. 2023. ZGUL: Zero-shot generalization to unseen languages using multi-source ensem- bling of language adapters. In Proceedings of the 2023 Conferenc...

2023 arXiv
[2017]

many beautiful buildings to look at

assembles typological feature vectors from different sources, enabling measurement of holis- tic typological distance between languages. The drawbacks of lang2vec include the lack of equal coverage of languages in terms of the number of features, and the coarseness of the ling...

2022
[2020]

Revista Economia & Gestão, 20(55)

Latin american, african and asian immigrants working in brazilian organizations: facing the lan- guage barrier. Revista Economia & Gestão, 20(55). NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Jani...
[2022]

In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 7676–7685, Dublin, Ireland

Make the best of cross-lingual transfer: Ev- idence from POS tagging with over 100 languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 7676–7685, Dublin, Ireland. Association for Computational Li...

2022
[2023]

Online database

Glottolog. Online database. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard...

2022 arXiv
[2024]

Transactions of the Association for Computational Linguistics, 12:950–978

CreoleVal: Multilingual multitask bench- marks for creoles. Transactions of the Association for Computational Linguistics, 12:950–978. Patrick Littell, David R. Mortensen, Ke Lin, Kather- ine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing langu...

2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Sentence ends with a terminal punctuation mark (’.’; ’!’, ’ " ’, ’?’)

work page

[2] [2]

Ratio of numerical characters with respect to all other characters is less than 0.25

work page

[3] [3]

Sentence starting with a bullet-point

work page

[4] [4]

In Proceedings of the 29th International Conference on Computational Linguistics, pages 4397–4406, Gyeongju, Republic of Korea

How to parse a creole: When martini- can creole meets French. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4397–4406, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Reza Moradi, Reza Berangi, and Behrouz Minaei. 2020. A survey of regularization strategies for deep models. Artif...

work page 2020

[5] [5]

Ratio of stop-words with respect to all other tokens is less than 0.6 using whitespace tok- enization

work page

[6] [6]

PidginUNMT: Unsupervised Neural Machine Translation from West African Pidgin to English

No language left behind: Scaling human- centered machine translation. Kelechi Ogueji and Orevaoghene Ahia. 2019. Pidg- inunmt: Unsupervised neural machine translation from west african pidgin to english. arXiv preprint arXiv:1912.03444. Wuraola Fisayo Oyewusi, Olubayo Adekanmbi, and Olalekan Akinsande. 2020. Semantic enrichment of nigerian pidgin english ...

work page Pith review arXiv 2019

[7] [7]

Sentence length is less than 14 and more than 300 characters

work page

[8] [8]

Kreyòl-MT: Building MT for Latin Amer- ican, Caribbean and colonial African creole lan- guages. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume1: Long Papers), pages 3083– 3110, Mexico City, Mexico. Association for Compu- tational Linguistics. Nathan...

work page 2024