Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

Abhishek Phaltankar; Gauri Takalikar; Omkar Khade; Raviraj Joshi; Shruti Jagdale

arxiv: 2411.18571 · v1 · pith:ZMLAHFNFnew · submitted 2024-11-27 · 💻 cs.CL · cs.LG

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

Omkar Khade , Shruti Jagdale , Abhishek Phaltankar , Gauri Takalikar , Raviraj Joshi This is my paper

classification 💻 cs.CL cs.LG

keywords languagemodelslow-resourcemultilingualadaptationadaptingcapabilitieschallenges

0 comments

read the original abstract

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Cross-Lingual Transfer and Parameter-Efficient Adaptation in the Turkic Language Family: A Theoretical Framework for Low-Resource Language Models
cs.CL 2026-03 unverdicted novelty 7.0

The paper introduces the Turkic Transfer Coefficient (TTC) as a theoretical measure of transfer potential and a scaling model linking adaptation performance to model capacity, data size, and adaptation module expressi...
Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety
cs.CL 2025-12 unverdicted novelty 6.0

Distilling safe refusal behavior from OpenAI o1-mini into Llama-3, Gemma-2, and Qwen3 models via response-based LoRA on multilingual jailbreak data increases jailbreak success rates on MultiJail by up to 16.6 points.
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
cs.CL 2026-05 accept novelty 5.0

QLoRA on 7B-scale models like Mistral achieves perplexity within 0.45 of full fine-tuning on GPT-2 medium for Bashkir while using over 40 times fewer trainable parameters, though best perplexity does not guarantee coh...
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
cs.CL 2026-05 unverdicted novelty 5.0

QLoRA on Mistral-7B and Phi-2 yields perplexity 3.79-3.81 on Bashkir, close to full fine-tuning's 3.34 but with over 40x fewer trainable parameters, though some base models degrade sharply and best-perplexity models o...