REVIEW 8 cited by
LLaMA Beyond English: An Empirical Study on Language Capability Transfer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their performance in other non-English languages. In this paper, we focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. To answer this question, we conduct an extensive empirical investigation based on LLaMA, accumulating over 1440 GPU hours. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. To accurately assess the model's level of knowledge, we employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a comprehensive evaluation of the model's response quality is conducted, considering aspects such as accuracy, fluency, informativeness, logical coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting instruction tasks from 17 diverse categories. Our evaluation results demonstrate that comparable performance to state-of-the-art transfer models can be achieved with less than 1% of the pretraining data, both in terms of knowledge alignment and response quality. Furthermore, the experimental outcomes across the thirteen low-resource languages also exhibit similar trends. We anticipate that the conclusions revealed by the experiments will aid the community in developing non-English LLMs.
Forward citations
Cited by 8 Pith papers
-
Multilingual Self-Taught Faithfulness Evaluators
STEMF trains multilingual faithfulness evaluators from synthetic data alone, and English-only training yields the best average results across languages.
-
Exploring the Potential of LLMs for Serendipity Evaluation in Recommender Systems
Basic and multi-model LLM prompts can evaluate recommendation serendipity as well as or better than standard proxy formulas, reaching 21.5% Pearson correlation with user-study ratings.
-
How and Where to Translate? The Impact of Translation Strategies in Cross-lingual LLM Prompting
For multilingual RAG intent classification, the best translation strategy depends on the model and language; translating instructions into the user's language helps some models, while making the model answer in low-re...
-
Teaching a Language Model to Speak the Language of Tools
LoRA fine-tuning of BgGPT models on a bilingual Bulgarian function-calling dataset yields large gains on a self-built 120-case benchmark while keeping knowledge benchmarks stable.
-
Text2Cypher Across Languages: Evaluating and Finetuning LLMs
A new multilingual Text2Cypher benchmark shows LLMs rank English highest, Spanish next, and Turkish lowest, and multilingual finetuning narrows the language gap more than English-only finetuning.
-
Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages
Filipino language models attribute gender and sexuality bias to concrete nouns such as people and objects, diverging from the action-heavy bias drivers observed in English models.
-
Beyond Text Compression: Evaluating Tokenizers Across Scales
Tokenizer choice matters mostly for multilingual tasks, and 350M-parameter models can predict 2.7B model ranking on translation but not on English benchmarks.
-
From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
A neuron-activation-based alignment score for LLMs correlates highly with downstream multilingual performance and transferability across nine open models.
Discussion (0). Sign in to comment.