REVIEW 6 cited by
Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model's overall factual accuracy. We perform an ablation study to highlight the impact of Hindi pre-training, showing significant improvements in Hindi chat capabilities and factual accuracy, which cannot be achieved through Hindi alignment alone.
Forward citations
Cited by 6 Pith papers
-
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
For Hindi vocabulary extension of a 30B LLM, the best embedding initialization is uniform subword averaging with Hindi norm calibration on the input and character-length-weighted averaging on the output, cutting conti...
-
A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script
Homophone normalization in Amharic training data can harm cross-lingual machine translation to Tigrinya and Ge'ez, but applying the normalization only at scoring time recovers BLEU gains.
-
FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models
A new benchmark shows state-of-the-art LLMs perform poorly on three Taiwanese indigenous languages across MT, ASR, and summarization.
-
Teaching a Language Model to Speak the Language of Tools
LoRA fine-tuning of BgGPT models on a bilingual Bulgarian function-calling dataset yields large gains on a self-built 120-case benchmark while keeping knowledge benchmarks stable.
-
PARAM-1 BharatGen 2.9B Model
A technical report on a 2.9B English-Hindi model whose headline evaluation numbers are internally inconsistent and whose promoted tokenizer was not used to train the final model.
- Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
Discussion (0). Sign in to comment.