Pith. sign in

REVIEW 8 cited by

Few-shot Learning with Multilingual Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2112.10668 v3 pith:CZSH7UJR submitted 2021-12-20 cs.CL cs.AI

Xi Victoria Lin , Todor Mihaylov , Mikel Artetxe , Tianlu Wang , Shuohui Chen , Daniel Simig , Myle Ott , Naman Goyal
show 13 more authors
Shruti Bhosale Jingfei Du Ramakanth Pasunuru Sam Shleifer Punit Singh Koura Vishrav Chaudhary Brian O'Horo Jeff Wang Luke Zettlemoyer Zornitsa Kozareva Mona Diab Veselin Stoyanov Xian Li
This is my paper
classification cs.CLcs.AI
keywords modelslanguagesfew-shotgpt-3languagelearningmultilingualshot
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large-scale generative language models such as GPT-3 are competitive few-shot learners. While these models are known to be able to jointly represent many different languages, their training data is dominated by English, potentially limiting their cross-lingual generalization. In this work, we train multilingual generative language models on a corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks. Our largest model with 7.5 billion parameters sets new state of the art in few-shot learning in more than 20 representative languages, outperforming GPT-3 of comparable size in multilingual commonsense reasoning (with +7.4% absolute accuracy improvement in 0-shot settings and +9.4% in 4-shot settings) and natural language inference (+5.4% in each of 0-shot and 4-shot settings). On the FLORES-101 machine translation benchmark, our model outperforms GPT-3 on 171 out of 182 directions with 32 training examples, while surpassing the official supervised baseline in 45 directions. We conduct an in-depth analysis of different multilingual prompting approaches, showing in particular that strong few-shot learning performance across languages can be achieved via cross-lingual transfer through both templates and demonstration examples. Finally, we evaluate our models in social value tasks such as hate speech detection in five languages and find it has limitations similar to comparable sized GPT-3 models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 76 citations worldwide. Full citation record

  1. ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models

    cs.CL 2025-10 conditional novelty 7.0 of 10

    ChiKhaPo is an 8-subtask benchmark that measures word-level comprehension and generation in 2,700+ languages and shows state-of-the-art models perform poorly on low-resource languages.

  2. Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Across eight languages, n-gram metrics such as ROUGE correlate less with human ratings in fusional languages than in isolating and agglutinative ones, while the neural metric COMET correlates better, especially in low...

  3. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language

    cs.CL 2025-06 conditional novelty 6.0 of 10

    An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...

  4. Evaluating Morphological Alignment of Tokenizers in 70 Languages

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Morphological alignment of tokenizers across 70 languages explains only about 0.5% to 6% of variance in language model task performance, with a small negative trend.

  5. mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

    cs.CL 2025-06 conditional novelty 5.0 of 10

    mSTEB is a new 200+ language speech and text benchmark showing that LLMs perform substantially worse on low-resource African and Americas/Oceania languages, especially in speech tasks.

  6. Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A survey of context-aware machine translation with large language models, categorizing prompting, fine-tuning, and agent-based approaches.

  7. Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks

    cs.CL 2025-07 reject novelty 3.0 of 10

    Across Arabic, English, and Kannada benchmarks, 4-bit and 8-bit quantization preserves most accuracy while aggressive pruning degrades larger multilingual models more than smaller ones.

  8. Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

    cs.CL 2025-06

Pith tools