Pith. sign in

REVIEW 5 major objections 6 minor 74 references

Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Federated averaging of small prompt encoders raises low-resource-language accuracy by 6.9 points over local tuning, while cutting communication cost by more than 99%.

desk verdict Straightforward FedAvg+prompt-tuning combination with a promising XNLI signal that is undercut by single-run reporting and a contradictory MasakhaNEWS result. read the letter →

arxiv 2507.03003 v1 pith:6LU4SFUH submitted 2025-07-02 cs.CL

Wanru Zhao , Yihong Chen , Royson Lee , Xinchi Qiu , Yan Gao , Hongxiang Fan , Nicholas D. Lane This is my paper · ORCID
classification cs.CL
keywords federatedlearningprompttuningmultilinguallanguagemodelslow-resourcelanguagescross-lingualtransferparameter-efficientfine-tuningdistancedataprivacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multilingual models struggle to serve low-resource languages, languages with little digitized text and few labeled examples, when each region's data cannot be pooled and each region lacks compute for full fine-tuning. This paper proposes federated prompt tuning: each client trains only a small prompt encoder on its own language data, and a server averages the encoders across clients. The claim is that this simple averaging beats local monolingual tuning, on XNLI average accuracy rising from 32.94% to 39.83%, while cutting communication cost by more than 99% and staying effective when one language has as few as 30 training examples. If correct, it offers a practical route to cross-lingual transfer under privacy rules and compute constraints.

What carries the argument

The load-bearing object is the virtual prompt encoder, a compact network that turns a task's virtual tokens into soft prompt embeddings inserted into the frozen model's input sequence. Federated Prompt Averaging updates these encoders locally on each client and aggregates them on the server weighted by each client's dataset size. All other model parameters stay frozen, which is what makes the communication and compute savings possible.

What would settle it

Re-run the XNLI comparison over many seeds and check whether federated prompt tuning's average stays more than one standard deviation above monolingual prompt tuning; if the gap vanishes, the headline claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that aggregating lightweight prompt encoders, rather than full model weights or raw data, transfers cross-lingual knowledge without breaking the frozen pretrained model. Across news classification, XNLI, and MasakhaNEWS, federated prompt tuning outperforms per-language local prompt tuning on nearly every language, with the largest gains among low-resource and typologically distant languages. The authors attribute the effect to federated averaging acting as implicit regularization, weight averaging finding flatter optima, and to the prompt encoder's small parameter count preventing the model from overwriting its linguistic foundations.

Load-bearing premise

The central 6.9-point improvement rests on treating a single run with a fixed, unreported seed as representative, so the gap is signal rather than run-to-run noise.

Editorial extensions

If this is right

  • Under GDPR-style data-sharing restrictions, multilingual models can still be adapted to many languages by exchanging only small prompt encoders, not raw text.
  • Languages with tiny datasets and large linguistic distance from the pretrained corpus gain the most from federated prompt averaging.
  • The method keeps most accuracy when training data for a language drops to tens of examples, pointing to use in genuinely low-resource deployment.
  • Communication cost falls from over 100 GB to under 500 MB for the reported setup, making federated multilingual tuning feasible on constrained networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to weight the server aggregation by the language-distance metric the paper introduces, so clients close to the pretrained language contribute less and distant ones more.
  • The same encoder-averaging recipe should work with other parameter-efficient methods, and the paper's own federated LoRA results already hint at comparable benefits.
  • Because the authors explicitly leave out differential privacy and secure aggregation, prompt averaging is best seen as reducing, not eliminating, privacy risk until those defenses are added.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Multilingual Federated Prompt Tuning, in which each client tunes a small prompt encoder on its local language-specific data while the XLM-R backbone stays frozen, and the server aggregates these encoders via Federated Prompt Averaging. The method is evaluated on News Classification (NC), XNLI, and MasakhaNEWS. The paper reports that federated prompt tuning outperforms local monolingual prompt tuning by 6.9 points average accuracy on XNLI, retains performance when one client's data volume is reduced, and reduces trainable parameters and communication cost by more than 99% relative to full-parameter federated fine-tuning. It also introduces a language-distance measure based on typological feature vectors and pretraining token counts, arguing that federated prompt tuning degrades more gracefully for languages far from the pretraining distribution.

Significance. If the XNLI result is robust, the method is a valuable baseline for privacy-preserving cross-lingual transfer: prompt encoders are extremely communication-light and avoid moving raw data across borders. The paper's strengths include a simple and well-defined algorithm; a concrete comparison of trainable parameters and communication bytes; and a language-distance analysis that relies on external typological vectors rather than fitted constants, so the circularity burden is low. However, the headline claims of consistent improvement, greater stability and generalization, and a 99% reduction in computational cost are not currently supported by the evidence as presented.

major comments (5)
  1. [§5.1, Tables 1–3, Appendix B] The central quantitative claim is based on what appear to be single unreplicated runs. Appendix B reports learning rate, batch size, sequence length, early stopping, and virtual-token count, but never a random seed, number of trials, standard deviation, or confidence interval. The 6.89-point average gap between PE_Monolingual (32.94) and PE_FL (39.83) on XNLI, and the per-language gaps in Table 2, are therefore not distinguishable from run-to-run noise. The sentence in §5.1 stating that federated prompt tuning 'consistently outperformed' the monolingual method is too strong without variability estimates. Please report mean and standard deviation over repeated seeds, or otherwise justify single-run reporting.
  2. [Table 3 / Abstract] On MasakhaNEWS the pattern reverses: PE_FL Prompt Tuning has average accuracy 76.4, below PE_Centralized (79.3) and below PE_FL LoRA (81.0). The abstract's claim that 'our approach achieves 6.9% higher accuracy' and demonstrates 'greater stability and generalization' as a general property is contradicted by this task. The 6.9% figure is a single-task comparison (XNLI PE_FL versus PE_Monolingual), not a property of the approach across tasks. Please rewrite the claims to state the specific comparison and to discuss the MasakhaNEWS results explicitly.
  3. [§5.2, Figure 4] The data-efficiency ablation is confounded. The experiment reduces data for one language (German) but appears to compare local monolingual fine-tuning on that reduced German set with federated prompt tuning in which other clients presumably contribute their own language data. The federated method then has access to far more total training data, so the observed robustness under data reduction is not an apples-to-apples comparison. The paper should specify exactly what data each client holds in this ablation and, if the goal is per-client sample efficiency, control for total data volume or compare against centralized training on the same total data.
  4. [§1, §5.4, Table 4] The claim that the method 'reduces computational and communication cost by more than 99%' conflates communication and trainable parameters with compute. Table 4 reports trainable parameter counts and communication bytes; prompt tuning still runs full forward and backward passes through the frozen 270M-parameter XLM-R, so compute per training step is not reduced by 99%. The statement in §5.1 that 'the fine-tuning time is considerably reduced' is not backed by any timing table. Please either restrict the cost claim to communication and trainable parameters, or provide measured FLOPs or wall-clock training time.
  5. [§5.3, Figure 5] The generalization claim that 'the decline is notably less steep' for languages far from the pretrained language is based on visual inspection of Figure 5 with the same unreplicated runs as the rest of the paper. No slope, confidence interval, or statistical test is reported, and the figure panels do not show error bars. This is load-bearing because the abstract's 'greater stability and generalization' claim rests partly on this figure. Please quantify the interaction between method and language distance and report variability.
minor comments (6)
  1. [§2] The SeamlessM4T reference is formatted as 'Loic Barrault, 2023' and is not listed with the full author list; it should be cited as Barrault et al. (2023).
  2. [§3.2 / Algorithm 1] The aggregation formula ht+1_g = sum |D_k| / sum |D_k| * ht_k is missing parentheses; as written it is ambiguous. Write the weighting factor explicitly as |D_k| / (sum_j |D_j|).
  3. [§4.2, Table 1] The notation is inconsistent: Table 1 uses 'FL', 'PE_FL (IID)', and 'PE_FL (Prompt Tuning) (Non-IID)', while §5.1 refers to 'PE_FL' without specifying the setting. Define all abbreviations and use them uniformly throughout the paper.
  4. [Appendix C] The appendix heading 'Datasets for Generative Tasks' includes the UN Corpus, a machine-translation dataset that is not used in the reported experiments; either remove it or explain its role.
  5. [§5.2 / Figure 4] The x-axis label '#Data (data reduction)' is unclear about units and direction; clarify whether the axis shows the absolute number of training examples or the percentage of the original 8,000 examples.
  6. [§5.1] Typo: 'overfiting' should be 'overfitting'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's accuracy, data-efficiency, and language-distance claims are empirical comparisons or external-data correlations, not derivations from fitted inputs.

full rationale

The paper makes no first-principles prediction that reduces to its own inputs. The headline 6.9% accuracy gain (Abstract; Section 5.1, Tables 1-2) is a direct empirical comparison between PE_FL and PE_Monolingual runs on XNLI; no parameter is fitted to the outcome and then renamed as a prediction. The language-distance measure in Section 5.3.1 is defined from external typological vectors (Littell et al. 2017; Malaviya et al. 2017) and pretraining token counts, then used post-hoc to stratify observed accuracy trends; it is not used to generate the accuracy numbers, so it is not self-definitional. The federated prompt-averaging update (Section 3.2, Algorithm 1) is a stated method, not a derived result that assumes the conclusion. Self-citations (Flower framework, Zhao et al. 2022; efficiency work, Qiu et al.; regularization analogy, Rehman et al. 2022) are supporting tooling or explanatory hypotheses, not load-bearing uniqueness theorems or ansatz smuggling. The data-efficiency ablation (Figure 4) and communication-cost comparison (Table 4) are arithmetic or straightforward measurements. Concerns about single-seed evaluation and the MasakhaNEWS ordering (Table 3) are empirical-validity issues, not circularity: they do not show that any claim is equivalent by construction to its inputs. No quoted reduction of a prediction to a fit or to a self-citation chain exists in the manuscript.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted constants or new entities. The language-distance measure relies on external typological vectors and pretraining token counts, which are treated as given inputs rather than fitted to the reported results. The central assumptions are that prompt tuning can extract task knowledge from frozen XLM-R, that FedAvg aggregation transfers across languages, and that the external language-distance data is a valid proxy for transfer difficulty.

assumptions (3)
  • domain assumption XLM-RoBERTa-base's frozen representations are sufficient for task adaptation via prompt embeddings.
    Section 3.1 freezes the PLM and relies entirely on prompt tuning to encode task knowledge. If prompt tuning cannot capture the needed task-specific features, the method fails.
  • domain assumption Federated averaging of prompt encoder weights yields a global prompt that transfers across languages.
    Algorithm 1 in Appendix D applies weighted averaging, but the paper provides no convergence or transfer analysis. The claimed benefit of federation is an empirical assumption.
  • domain assumption The typological vectors and token-count weighting from Littell et al. (2017) and Malaviya et al. (2017) accurately represent language similarity for cross-lingual transfer.
    Section 5.3.1 defines language distance using these external resources. If the distance metric is uninformative, the analysis in Figure 5 loses meaning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages." pith.science (2026). https://pith.science/paper/6LU4SFUH

@misc{pith2026250703003,
  author       = {Pith},
  title        = {Pith review of: Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LU4SFUH}},
  note         = {Machine review of arXiv:2507.03003}
}
read the original abstract

Pre-trained large language models (LLMs) have become a cornerstone of modern natural language processing, with their capabilities extending across a wide range of applications and languages. However, the fine-tuning of multilingual LLMs, especially for low-resource languages, faces significant challenges arising from data-sharing restrictions (the physical border) and inherent linguistic differences (the linguistic border). These barriers hinder users of various languages, particularly those in low-resource regions, from fully benefiting from the advantages of LLMs. To address these challenges, we propose the Federated Prompt Tuning Paradigm for multilingual scenarios, which utilizes parameter-efficient fine-tuning while adhering to data sharing restrictions. We design a comprehensive set of experiments and analyze them using a novel notion of language distance to highlight the strengths of our paradigm: Even under computational constraints, our method not only improves data efficiency but also facilitates mutual enhancements across languages, particularly benefiting low-resource ones. Compared to traditional local cross-lingual transfer tuning methods, our approach achieves 6.9\% higher accuracy with improved data efficiency, and demonstrates greater stability and generalization. These findings underscore the potential of our approach to promote social equality and champion linguistic diversity, ensuring that no language is left behind.

Figures

Figures reproduced from arXiv: 2507.03003 by the authors.

Figure 1
Figure 1. Linguistic coverage of different large language models. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of three different learning paradigms for multilingual tasks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of prompt tuning. Instead of selecting discrete text prompts in a manual or automated fashion, in our Multilin￾gual Federated Prompt Tuning paradigm, we utilize virtual prompt embeddings that can be op￾timized via gradient descent. Specifically, each prompt encoder, whether global or local, takes a series of virtual tokens, which are updated during tuning to better aid the model [PITH_FULL_IMAGE:figure… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Performance comparison between tradi￾tional local fine-tuning and our federated prompt tuning method. From our results in section 5.1, we observe that some languages demonstrate superior accuracy with the FL method compared to the centralized approach. This enhanced pe…
Figure 5
Figure 5. Figure 5: Comparative performance for both XNLI and NC tasks. (a)(c) reports the fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Communication Cost Compar￾ison between Federated Prompt Tuning and Federated LoRA. the pre-trained language decreases, the model’s accuracy tends to drop. However, when we apply our Federated Prompt method, this decline is notably less steep. This means that even when …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 34 canonical work pages

  1. [1]

    Towards afrocentric NLP for A frican languages: Where we are and where we can go

    Ife Adebara and Muhammad Abdul-Mageed. Towards afrocentric NLP for A frican languages: Where we are and where we can go. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3814--3841, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-long.265

    doi:10.18653/v1/2022.acl-long.265 2022

  2. [2]

    Masakhaner: Named entity recognition for african languages

    David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9: 0 1116--1131, 2021

    work page 2021

  3. [3]

    David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-Azzawi, Blessing K. Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Oluwaseyi Ajayi, Tatiana Moteu Ngoli, Br...

    work page 2023

  4. [4]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  5. [5]

    Composable sparse fine-tuning for cross-lingual transfer

    Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vuli \'c . Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1778--1796, 2022

    2022

  6. [6]

    On the cross-lingual transferability of monolingual representations

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 4623--4637, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.421

    doi:10.18653/v1/2020.acl-main.421 2020

  7. [7]

    Federated learning review: Fundamentals, enabling technologies, and future applications

    Syreen Banabilah, Moayad Aloqaily, Eitaa Alsayed, Nida Malik, and Yaser Jararweh. Federated learning review: Fundamentals, enabling technologies, and future applications. Information Processing & Management, 59 0 (6): 0 103061, 2022. ISSN 0306-4573. doi:https://doi.org/10.1016/j.ipm.2022.103061

    arXiv 2022

  8. [8]

    B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.\ 1--9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.1865...

    doi:10.18653/v1/2022.acl-short.1 2022

Show all 74 references
  1. [9]

    Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, and Nicholas D

    Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, and Nicholas D. Lane. Flower: A friendly federated learning research framework, 2020

    2020

  2. [10]

    K. A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. In NIPS Workshop on Private Multi-Party Machine Learning, 2016

    2016

  3. [11]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  4. [12]

    Chau, Lucy H

    Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. Parsing with multilingual BERT , a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.\ 1324--1334, Online, November 2020. Association for Computational Linguistics. do...

    2020 doi

  5. [13]

    Improving language plasticity via pretraining with active forgetting

    Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Ifeoluwa Adelani, Pontus Stenetor, Sebastian Riedel, and Mikel Artetx. Improving language plasticity via pretraining with active forgetting. NeurIPS 2023, 2023

    2023

  6. [14]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

    2022

  7. [15]

    Improving multilingual models with language-clustered vocabularies

    Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 4536--4546, Online, November 2020. Associ...

    2020 doi

  8. [16]

    Cross-Lingual Language Model Pretraining

    Alexis Conneau and Guillaume Lample. Cross-Lingual Language Model Pretraining. Curran Associates Inc., Red Hook, NY, USA, 2019

    2019

  9. [17]

    Bowman, Holger Schwenk, and Veselin Stoyanov

    Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In EMNLP, 2018

    2018

  10. [18]

    Unsupervised cross-lingual representation learning at scale

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting ...

    2020 doi

  11. [19]

    Sumanth Doddapaneni, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M. Khapra. A primer on pretrained multilingual language models. CoRR, abs/2107.00676, 2021

    2021 arXiv

  12. [20]

    A mericas NLI : Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages

    Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Gim \'e nez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina...

    2022

  13. [21]

    Indo-European language and culture: An introduction

    Benjamin W Fortson IV. Indo-European language and culture: An introduction. John Wiley & Sons, 2011

    2011

  14. [22]

    Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33: 0 16937--16947, 2020

    Jonas Geiping, Hartmut Bauermeister, Hannah Dr \"o ge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33: 0 16937--16947, 2020

    2020

  15. [23]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP . In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Intern...

    2019

  16. [24]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp.\ 2790--2799. PMLR, 2019 b

    2019

  17. [25]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021

    2021 arXiv

  18. [26]

    Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization

    Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080, 2020

    2003 arXiv

  19. [27]

    Evaluating gradient inversion attacks and defenses in federated learning

    Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34: 0 7232--7241, 2021

    2021

  20. [28]

    Garipov, Dmitry P

    Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, 2018

    2018

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015

    2015

  22. [30]

    Federated learning: Strategies for improving communication efficiency

    Jakub Kone c n \`y , H Brendan McMahan, Felix X Yu, Peter Richt \'a rik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016

    2016 arXiv

  23. [31]

    From zero to hero: O n the limitations of zero-shot language transfer with multilingual T ransformers

    Anne Lauscher, Vinit Ravishankar, Ivan Vuli \'c , and Goran Glava s . From zero to hero: O n the limitations of zero-shot language transfer with multilingual T ransformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ ...

    2020 doi

  24. [32]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al - Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. CoRR, abs/2104.08691, 2021

    2021 arXiv

  25. [33]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap...

    2021 doi

  26. [34]

    Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation

    Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, ...

    2004 arXiv

  27. [35]

    Federated learning in mobile edge networks: A comprehensive survey

    Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22 0 (3): 0 2031--2063, 2020

    2020

  28. [36]

    F ed NLP : Benchmarking federated learning methods for natural language processing tasks

    Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. F ed NLP : Benchmarking federated learning methods for natural language processing tasks. In Findings of the Association for C...

    2022 doi

  29. [37]

    Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors

    Patrick Littell, David R Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. Uriel and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association fo...

    2017

  30. [38]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586, 2021 a

    2021 arXiv

  31. [39]

    Gpt understands, too

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv:2103.10385, 2021 b

    2021 arXiv

  32. [40]

    Seamlessm4t—massively multilingual & multimodal machine translation

    Mariano Cora Meglioli Loic Barrault, Yu-An Chung. Seamlessm4t—massively multilingual & multimodal machine translation. ArXiv, 2023

    2023

  33. [41]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

    2019

  34. [42]

    Findings of the americasnlp 2021 shared task on open machine translation for indigenous languages of the americas

    Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios Gonzales, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Gim \'e nez-Lugo, Ricardo Ramos, et al. Findings of the americasnlp 2021 shared task on open machine translation for indigenous langu...

    2021

  35. [43]

    Learning language representations for typology prediction

    Chaitanya Malaviya, Graham Neubig, and Patrick Littell. Learning language representations for typology prediction. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark, September 2017

    2017

  36. [44]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022

    2022

  37. [45]

    Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training

    Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training. ACL 2023, Findings of the Association for Computational Linguistics, 2022

    2023

  38. [46]

    On-device federated learning with flower

    Akhil Mathur, Daniel J Beutel, Pedro Porto Buarque de Gusmao, Javier Fernandez-Marques, Taner Topal, Xinchi Qiu, Titouan Parcollet, Yan Gao, and Nicholas D Lane. On-device federated learning with flower. arXiv preprint arXiv:2104.03042, 2021

    2021 arXiv

  39. [47]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 2017

    2017

  40. [48]

    Naijasenti: A nigerian twitter sentiment corpus for multilingual sentiment analysis

    Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Said Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahudeen Abdullahi, Anuoluwapo Aremu, et al. Naijasenti: A nigerian twitter sentiment corpus for mu...

    2022 arXiv

  41. [49]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  42. [50]

    MAD-X : A n A dapter- B ased F ramework for M ulti- T ask C ross- L ingual T ransfer

    Jonas Pfeiffer, Ivan Vuli \'c , Iryna Gurevych, and Sebastian Ruder. MAD-X : A n A dapter- B ased F ramework for M ulti- T ask C ross- L ingual T ransfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 7654--7673, Online...

    2020 doi

  43. [52]

    How multilingual is multilingual BERT ? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4996--5001, Florence, Italy, July 2019 b

    Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT ? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4996--5001, Florence, Italy, July 2019 b . Association for Computational Linguistics. doi:10.1...

    2019 doi

  44. [53]

    XCOPA : A multilingual dataset for causal commonsense reasoning

    Edoardo Maria Ponti, Goran Glava s , Olga Majewska, Qianchu Liu, Ivan Vuli \'c , and Anna Korhonen. XCOPA : A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\ 2362--23...

    2020 doi

  45. [54]

    Zerofl: Efficient on-device training for federated learning with local sparsity

    Xinchi Qiu, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Titouan Parcollet, and Nicholas Donald Lane. Zerofl: Efficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507, 2022

    2022 arXiv

  46. [55]

    A first look into the carbon footprint of federated learning

    Xinchi Qiu, Titouan Parcollet, Javier Fernandez-Marques, Pedro PB Gusmao, Yan Gao, Daniel J Beutel, Taner Topal, Akhil Mathur, and Nicholas D Lane. A first look into the carbon footprint of federated learning. Journal of Machine Learning Research, 24 0 (129): 0 1--23, 2023

    2023

  47. [56]

    Federated self-supervised learning for video understanding

    Yasar Abbas Ur Rehman, Yan Gao, Jiajun Shen, Pedro Porto Buarque de Gusm \ a o, and Nicholas Lane. Federated self-supervised learning for video understanding. In Shai Avidan, Gabriel Brostow, Moustapha Ciss \'e , Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Visio...

    2022

  48. [57]

    True few-shot learning with P rompts --- A real-world perspective

    Timo Schick and Hinrich Sch \"u tze. True few-shot learning with P rompts --- A real-world perspective. Transactions of the Association for Computational Linguistics, 10: 0 716--731, 2022. doi:10.1162/tacl_a_00485

    2022 doi

  49. [58]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt : Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020

    2020

  50. [59]

    The sino-tibetan languages

    Graham Thurgood and Randy J LaPolla. The sino-tibetan languages. Routledge, 2016

    2016

  51. [60]

    Memorization without overfitting: Analyzing the training dynamics of large language models

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35: 0 38274--38290, 2022

    2022

  52. [61]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  53. [62]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  54. [63]

    The eu general data protection regulation (gdpr)

    Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A practical guide, 1st ed., Cham: Springer International Publishing, 10 0 (3152676): 0 10--5555, 2017

    2017

  55. [64]

    Fedkc: Federated knowledge composition for multilingual natural language understanding

    Haoyu Wang, Handong Zhao, Yaqing Wang, Tong Yu, Jiuxiang Gu, and Jing Gao. Fedkc: Federated knowledge composition for multilingual natural language understanding. In Proceedings of the ACM Web Conference 2022, WWW '22, pp.\ 1839–1850, New York, NY, USA, 2022. Association for C...

    2022

  56. [65]

    Federated learning with differential privacy: Algorithms and performance analysis

    Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security, 15: 0 3454--3469, 2020

    2020

  57. [66]

    Pretrained models for multilingual federated learning

    Orion Weller, Marc Marone, Vladimir Braverman, Dawn J Lawrie, and Benjamin Van Durme. Pretrained models for multilingual federated learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec...

    2022

  58. [67]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  59. [68]

    Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT

    Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP...

    2019 doi

  60. [69]

    Are all languages created equal in multilingual BERT ? In Proceedings of the 5th Workshop on Representation Learning for NLP, pp.\ 120--130, Online, July 2020

    Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT ? In Proceedings of the 5th Workshop on Representation Learning for NLP, pp.\ 120--130, Online, July 2020. Association for Computational Linguistics. doi:10.18653/v1/2020.repl4nlp-1.16

    2020 doi

  61. [70]

    Protea: Client profiling within federated systems using flower

    Wanru Zhao, Xinchi Qiu, Javier Fernandez-Marques, Pedro PB de Gusm \ a o, and Nicholas D Lane. Protea: Client profiling within federated systems using flower. In Proceedings of the 1st ACM Workshop on Data Privacy and Federated Learning Technologies for Mobile Edge Network, pp...

    2022

  62. [71]

    The U nited N ations parallel corpus v1.0

    Micha Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The U nited N ations parallel corpus v1.0. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Pi...

    2016

  63. [72]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  64. [73]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  65. [74]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  66. [75]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.