Pith. sign in

REVIEW 3 cited by

Are All Languages Created Equal in Multilingual BERT?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2005.09093 v2 pith:Z7VUJ5FM submitted 2020-05-18 cs.CL

Shijie Wu , Mark Dredze This is my paper
classification cs.CL
keywords languagesmbertbertcross-lingualperformanceresourcebettereven
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross-lingual transfer with high-resource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging, and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    Biaffine LSTM outperforms transformer parsers like AfroXLMR and RemBERT in low-resource dependency parsing, with transformers gaining advantage as data increases and morphological complexity as a secondary predictor.

  2. How do datasets, developers, and models affect biases in a low-resourced language?: The Case of the Bengali Language

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Bengali sentiment analysis models exhibit persistent identity-based biases across datasets and developer backgrounds despite similar semantic content.

  3. The Role of Orthographic Consistency in Multilingual Embedding Models for Text Classification in Arabic-Script Languages

    cs.CL 2025-07 reject novelty 4.0 of 10

    Language-specific RoBERTa models for four Arabic-script languages beat multilingual baselines on news classification, though the claimed orthographic-consistency mechanism is not demonstrated.

Pith tools