Pith. sign in

REVIEW 5 major objections 5 minor 26 references

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LoRA-tuned Marathi Gemma models win human evaluations even as automated benchmarks decline, the paper claims, arguing that current metrics miss qualitative gains from language adaptation.

desk verdict Plausible observation about the automated-vs-manual gap for Marathi LoRA tuning, but the manual evidence is too under-reported to settle it. read the letter →

arxiv 2411.18571 v1 pith:ZMLAHFNF submitted 2024-11-27 cs.CL cs.LG

Omkar Khade , Shruti Jagdale , Abhishek Phaltankar , Gauri Takalikar , Raviraj Joshi This is my paper · ORCID
classification cs.CLcs.LG
keywords LoRAPEFTlow-resourcelanguagesMarathiGemmainstructiontuninghumanevaluationautomatedbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that parameter-efficient fine-tuning (LoRA PEFT) of multilingual Gemma models on a translated Marathi instruction dataset produces models that human raters prefer over the base models on open-ended questions, despite automated NLU and reasoning benchmarks mostly showing a decline. The authors argue that current logit-based automated metrics are poorly suited for evaluating instruction-tuned models in low-resource languages because they overlook improvements in cultural relevance, fluency, and instruction-following. If true, the result would mean that benchmark-driven evaluations understate the practical value of adapting large language models to low-resource languages, and that better evaluation methodologies and native datasets are needed.

What carries the argument

The central mechanism is Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning technique that updates only small low-rank matrices rather than all model weights, combined with an Alpaca-style instruction dataset machine-translated into Marathi. The argument is carried by the contrast between two evaluation instruments: five automated AI4Bharat benchmarks that score F1 on classification and reasoning tasks, and a 150-question manual evaluation in which humans compare base and fine-tuned model outputs. The paper's core evidence is the win-rate gap in the manual evaluation, which it presents as evidence that automated metrics miss qualitative language improvements.

What would settle it

Conduct a blind, rubric-based human evaluation of the same 150 questions with at least three independent native Marathi-speaking raters, reporting inter-annotator agreement, and separately score factual correctness versus style; if the fine-tuned models no longer show higher win rates under blinding, or if the win-rate advantage disappears when factual accuracy is isolated, the paper's central claim is overturned.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuning Gemma models for Marathi using LoRA PEFT on 52,000 translated Alpaca instruction-response pairs improves target-language generation as judged by humans, while simultaneously degrading performance on standard automated benchmarks such as IndicSentiment, ARC-easy, ARC Challenge, Indic COPA, and Indic XNLI. The paper reports that fine-tuned variants like gemma-2-2b-it (Mr) and gemma-2b-it (Mr) show higher win rates than their base counterparts in a manual evaluation of 150 open-ended questions covering knowledge, culture, mathematics, and problem-solving. The authors conclude that the observed divergence between manual and automated assessments reveals a fundamental limitation of current evaluation practices for low-resource languages, which rely on logit-based metrics that cannot capture culturally grounded response quality.

Load-bearing premise

The manual evaluation of 150 questions is an unbiased and accurate measure of response quality; the paper does not specify how questions were chosen, what rubric or rating scale was used, who the raters were, whether they were blind to model identity, or any inter-annotator agreement, so the central comparison rests entirely on this unverified assessment.

Editorial extensions

If this is right

  • Current logit-based benchmarks may systematically underreport the benefit of language adaptation for low-resource languages, so leaderboard rankings could mislead practitioners selecting models for real users.
  • Instruction-tuned multilingual models fine-tuned on translated data can gain fluency and cultural appropriateness in the target language, even when their performance on abstract reasoning tasks declines.
  • Evaluation suites for low-resource languages should incorporate human judgment or human-aligned metrics rather than relying solely on F1 scores from translated benchmarks.
  • Fine-tuning strategies for low-resource languages may need to balance target-language generation quality against preserving reasoning capabilities, possibly through mixed training data or selective adaptation.
  • The quality of the translation step in creating fine-tuning data is directly implicated in the reasoning degradation, since translated Alpaca pairs may introduce artifacts that erode skills like entailment and commonsense inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported pattern generalizes, task-specific automated benchmarks in low-resource languages may be measuring something orthogonal to what human users value, and win rates on open-ended questions could serve as a complementary evaluation axis.
  • The observed reasoning decline may stem from the machine-translated training data rather than from LoRA adaptation itself; testing the same method with naturally occurring Marathi instruction data would isolate the cause.
  • Human raters may be rewarding style, fluency, and politeness rather than factual correctness; a manual evaluation that separately scores factuality and style would reveal whether the fine-tuned models actually increase usable accuracy.
  • The win-rate gap might diminish if base models were given Marathi prompts that are better tuned or if few-shot examples were provided, suggesting that the manual evaluation conflates language capability with instruction-following behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies LoRA-based PEFT adaptation of four Gemma base models (gemma-2b, gemma-2b-it, gemma-2-2b, gemma-2-2b-it) to Marathi using a 52,000-pair translated Alpaca dataset. It reports automated F1 scores on five AI4Bharat benchmarks (IndicSentiment, ARC-easy, ARC Challenge, Indic COPA, Indic XNLI) and describes a manual evaluation of 150 questions in which fine-tuned models "frequently" outperform base models. The paper concludes that automated logit-based benchmarks understate the benefits of language adaptation for low-resource languages and calls for better evaluation methods and native datasets.

Significance. If the manual-evaluation result could be substantiated, the paper would provide a valuable, counterintuitive finding: standard benchmarks may not capture qualitative improvements from low-resource adaptation, and language consistency may be a major hidden factor in perceived quality. The study is also relevant to practitioners because it compares several Gemma sizes and checkpoints under a parameter-efficient method. Credit is due for using publicly available AI4Bharat benchmarks, for directly comparing base and adapted models, and for transparently listing limitations (translated data, compute constraints, scarcity of Marathi evaluation sets). However, as submitted, the headline claim is not supported by the evidence actually present in the manuscript: the referenced figure is missing and the manual evaluation protocol is unspecified. Therefore the current significance is conditional on a revision that supplies the missing evidence.

major comments (5)
  1. [3.3 / 4.1 / Figure 1] The central claim that fine-tuned models win more often in manual evaluation is not substantiated in the text. The manuscript refers to "Figure 1: Manual Evaluation Performance" and "Appendix Figure 2: Responses," but no actual figure or numerical win rates appear in the submission; the only quantitative statement in §4.1 is that fine-tuned versions "showed higher win rates," without counts, percentages, or a definition of a "win." Please include the figures and the pairwise win/loss/tie counts for each base-versus-fine-tuned comparison, and define what constituted a win.
  2. [3.3] The manual evaluation protocol is under-specified: there is no description of how the 150 questions were selected from the "curated sheet," what rating scale or rubric was used, who the raters were, how many raters scored each response, whether they were blind to model identity, or what the inter-annotator agreement was. Because the paper's main conclusion relies on this evaluation, these elements must be reported. Additionally, §4.1 states that base models "occasionally generated responses in English"; if raters were not instructed to disregard language, the observed preference could reflect language consistency rather than content quality. Please report the instructions given to raters and, ideally, breakdowns of win rates by language-consistency status of the response.
  3. [3.2 / 4.1] The model inventory is internally inconsistent. Section 3.2 defines fine-tuned models gemma-2b (Mr), gemma-2-2b (Mr), and gemma-2-2b-it (Mr), but §4.1 and Figure 1 also discuss "gemma-2b-it (Mr)," which is never defined. If gemma-2b-it was fine-tuned as well, add it to the model list and results; if the claim refers to another model, correct the label throughout.
  4. [3.2] No LoRA hyperparameters (rank, alpha, target modules, learning rate, batch size, number of epochs, or equivalent) are reported, so the fine-tuning setup cannot be reproduced or compared with other LoRA studies. Please include a hyperparameter table or state the exact values used for each model.
  5. [Tables 1–2 / 4.1] Automated F1 scores are presented as single numbers with no variance, confidence intervals, or significance tests. The claim that fine-tuning leads to a "degradation in NLU and reasoning benchmarks" is based on comparisons of these single numbers; without repeated runs or paired tests, some differences (e.g., gemma-2-2b-it versus gemma-2-2b-it (Mr) on ARC Challenge: 0.7210 versus 0.6374) could be noise. Please report standard deviations or at least explicitly state that each benchmark was run once and treat the differences accordingly.
minor comments (5)
  1. [References] The reference list contains a duplicated entry: Lankford et al. 2023a and 2023b are identical (same title, venue, volume, and page range). Please remove the duplicate and update citations accordingly.
  2. [Introduction / Related Work] The sentence in §1 that PEFT "avoids catastrophic forgetting due to usage of non-English data only" is unclear; catastrophic forgetting is normally about forgetting previous capabilities, not about the language of the training data. Please rephrase to describe what is actually being claimed.
  3. [3.1] "Google translate API" should be capitalized as "Google Translate API," and the manuscript would benefit from a brief note on whether any post-translation filtering or manual spot-checking of the 52,000 pairs was performed.
  4. [Appendix] The captions for Figure 1 and Figure 2 are present but the figures themselves are missing from the PDF; ensure the final version includes the images, or remove the cross-references.
  5. [4.1] The phrase "In the evaluation of the F1 score, represented in Table 1 for gemma-1 models and Table 2 for gemma-2 models" is grammatically awkward, and the model family names ("gemma-1" versus "Gemma1") are used inconsistently; please harmonize the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical comparisons are self-contained and not derived from their own outputs.

full rationale

The paper makes no formal derivation that could reduce to its inputs: there are no fitted parameters, no equations, and no theoretical claim that is defined in terms of its own conclusion. The central comparison is empirical: LoRA fine-tuned Gemma models are evaluated on external AI4Bharat benchmarks (IndicSentiment, ARC-easy, ARC Challenge, Indic COPA, Indic XNLI) and on an author-created 150-question manual evaluation. The manual evaluation is internal to the paper, but it is presented as direct observation of model outputs, not as a quantity that is mathematically forced by the training data or by a self-cited prior result. The only 'prediction' in the paper is a qualitative claim that fine-tuned models are better in manual assessment, and that claim is not shown to be equivalent to any input or fitted quantity. References to Gemma and AI4Bharat are standard external citations, not self-citations used to forbid alternatives or import an unproven uniqueness theorem. The paper's own limitations section candidly acknowledges the translated-dataset weakness, computational constraints, and scarcity of Marathi evaluation data; those are methodological limitations, not circularity. The manual evaluation would benefit from reporting the question set, rubric, rater blinding, and inter-annotator agreement, but that is a reproducibility concern, not a circular-derivation concern. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities appear. The empirical claim rests on three main domain assumptions: the translated dataset is adequate, the automated benchmarks are valid, and the author-designed manual evaluation is a reliable signal. The LoRA configuration is also unstated, which makes the results hard to interpret.

assumptions (4)
  • domain assumption The translated Alpaca Marathi data is a usable proxy for Marathi instruction tuning despite machine-translation artifacts.
    Section 3.1 uses Google Translate output as the training corpus; the authors later list translation bias as a limitation.
  • domain assumption AI4Bharat benchmarks (IndicSentiment, ARC, COPA, XNLI) are valid for evaluating Marathi instruction-tuned Gemma models.
    Section 3.3 uses these benchmarks without adaptation or validation for the fine-tuned models.
  • domain assumption A manual evaluation of 150 self-curated questions can detect quality differences that automated metrics miss.
    Section 3.3 and Section 4.1 rely on this manual evaluation, but no protocol, rubric, or rater details are given.
  • domain assumption The LoRA configuration used is appropriate for Gemma Marathi adaptation.
    Section 3.2 reports model names but not rank, alpha, learning rate, or number of epochs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning." pith.science (2026). https://pith.science/paper/ZMLAHFNF

@misc{pith2026241118571,
  author       = {Pith},
  title        = {Pith review of: Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMLAHFNF}},
  note         = {Machine review of arXiv:2411.18571}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.

Figures

Figures reproduced from arXiv: 2411.18571 by the authors.

Figure 1
Figure 1. Manual Evaluation Performance. reasoning benchmarks following language adapta￾tion. However, the adapted model performs better on the open-ended question answering dataset dur￾ing manual evaluation. This suggests the need for a more comprehensive evaluation strategy and more suitable datasets to fully assess the benefits of lan￾guage adaptation. While automated benchmarks indicate degradation, they may not be the id… view at source ↗
Figure 2
Figure 2. Responses [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [4]

    Firoj Alam, Shammur Absar Chowdhury, Sabri Boughorbel, and Maram Hasanain. 2024. Llms for low resource languages in multilingual, multimodal and dialectal settings. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 27--33

    work page 2024

  4. [13]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Tao Song, Yingce Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. In Conference on Empirical Methods in Natural Language Processing

    work page 2023

  5. [14]

    Lankford, H

    S. Lankford, H. Afli, and A. Way. 2023 a . adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated llm playgrounds. Information, 14(12):638

    work page 2023

  6. [15]

    Lankford, H

    S. Lankford, H. Afli, and A. Way. 2023 b . adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated llm playgrounds. Information, 14(12):638

    work page 2023

  7. [16]

    Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116--126

    work page 2021

  8. [23]

    adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds,

    Lankford, S., Afli, H., and Way, A., "adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds," Information, vol. 14, no. 12, pp. 638, 2023

    work page 2023

Show all 26 references
  1. [24]

    LoRA: Low-Rank Adaptation of Large Language Models,

    Hu, E. J., Shen, Y., Wallis, P., Li, Y., Wang, S., Wang, L., and Chen, W., "LoRA: Low-Rank Adaptation of Large Language Models," ArXiv, 2021. Available: https://arxiv.org/abs/2106.09685

    2021 arXiv

  2. [25]

    Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey,

    Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q., "Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey," ArXiv, 2024. Available: https://arxiv.org/abs/2403.14608

    2024 arXiv

  3. [26]

    Airavata: Introducing Hindi Instruction-tuned LLM,

    Gala, J., Jayakumar, T., Husain, J. A., M, A. K., Khan, M. S., Kanojia, D., Puduppully, R., Khapra, M. M., Dabre, R., Murthy, R., and Kunchukuttan, A., "Airavata: Introducing Hindi Instruction-tuned LLM," ArXiv, 2024. Available: https://arxiv.org/abs/2401.15006

    2024 arXiv

  4. [27]

    Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models,

    Barnett, S., Brannelly, Z., Kurniawan, S., and Wong, S., "Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models," ArXiv, 2024. Available: 10.48550/arXiv.2406.11201

    2024 doi arXiv

  5. [28]

    LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings,

    Alam, F., Chowdhury, S. A., Boughorbel, S., and Hasanain, M., "LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings," in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstract...

    2024

  6. [29]

    Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters,

    Gurgurov, D., Hartmann, M., and Ostermann, S., "Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters," ArXiv, 2024. Available: 10.48550/arXiv.2407.01406

    2024 doi arXiv

  7. [30]

    Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting,

    Huang, H., Tang, T., Zhang, D., Zhao, W. X., Song, T., Xia, Y., and Wei, F., "Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting," Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  8. [31]

    Gemma: Open Models Based on Gemini Research and Technology,

    Team, G., Mesnard, T., Hardin, C., et al., "Gemma: Open Models Based on Gemini Research and Technology," ArXiv, 2024. Available: https://arxiv.org/abs/2403.08295

    2024 arXiv

  9. [32]

    Gemma 2: Improving Open Language Models at a Practical Size,

    Team, G., Riviere, M., Pathak, S., et al., "Gemma 2: Improving Open Language Models at a Practical Size," ArXiv, 2024. Available: https://arxiv.org/abs/2408.00118

    2024 arXiv

  10. [33]

    How Multilingual Are Large Language Models Fine-Tuned for Translation?

    Richburg, A., and Carpuat, M., "How Multilingual Are Large Language Models Fine-Tuned for Translation?" ArXiv, 2024. Available: https://arxiv.org/abs/2405.20512

    2024 arXiv

  11. [34]

    Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?

    Zhu, D., Chen, P., Zhang, M., Haddow, B., Shen, X., and Klakow, D., "Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?" ArXiv, 2024. Available: https://arxiv.org/abs/2404.14122

    2024 arXiv

  12. [35]

    MultiFiT: Efficient Multi-lingual Language Model Fine-tuning,

    Eisenschlos, J. M., Ruder, S., Czapla, P., Kardas, M., Gugger, S., and Howard, J., "MultiFiT: Efficient Multi-lingual Language Model Fine-tuning," ArXiv, 2019. Available: https://arxiv.org/abs/1909.04761

    2019 arXiv

  13. [36]

    Multilingual Translation with Extensible Multilingual Pretraining and Finetuning,

    Tang, Y., Tran, C., Li, X., Chen, P., Goyal, N., Chaudhary, V., Gu, J., and Fan, A., "Multilingual Translation with Extensible Multilingual Pretraining and Finetuning," ArXiv, 2020. Available: https://arxiv.org/abs/2008.00401

    2020 arXiv

  14. [37]

    Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies,

    Weng, B., "Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies," ArXiv, 2024. Available: https://arxiv.org/abs/2404.09022

    2024 arXiv

  15. [38]

    Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages,

    Ogueji, K., Zhu, Y., and Lin, J., "Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages," in Proceedings of the 1st Workshop on Multilingual Representation Learning, Punta Cana, Dominican Republic, pp. 116–126, 2021

    2021

  16. [39]

    Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages,

    Dhamecha, T. I., Murthy, V. R., Bharadwaj, S., Sankaranarayanan, K., and Bhattacharyya, P., "Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages," ArXiv, 2021. Available: https://arxiv.org/abs/2109.10534

    2021 arXiv

  17. [40]

    MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models,

    Aggarwal, D., Sathe, A., Watts, I., and Sitaram, S., "MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Models," ArXiv, 2024. Available: https://arxiv.org/abs/2401.07598

    2024 arXiv

  18. [41]

    When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages,

    Chang, T. A., Arnett, C., Tu, Z., and Bergen, B. K., "When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages," ArXiv, 2023. Available: https://arxiv.org/abs/2311.09205

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.