Pith. sign in

REVIEW 3 cited by

Larger-Scale Transformers for Multilingual Masked Language Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.00572 v1 pith:AQQWSTSH submitted 2021-05-02 cs.CL

Naman Goyal , Jingfei Du , Myle Ott , Giri Anantharaman , Alexis Conneau This is my paper
classification cs.CL
keywords modelslanguagelanguagesmodelxlm-raveragecross-linguallarger
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed XLM-R XL and XLM-R XXL outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests pretrained models with larger capacity may obtain both strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OPT: Open Pre-trained Transformer Language Models

    cs.CL 2022-05 unverdicted novelty 7.0 of 10

    OPT releases open decoder-only transformers up to 175B parameters that match GPT-3 performance at one-seventh the carbon cost, along with code and training logs.

  2. EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries

    cs.DB 2026-06 unverdicted novelty 6.0 of 10

    EcoTable is the first NL-based data integration framework that builds a join-likelihood graph, uses two-stage schema linking and Steiner tree search to find paths, then generates transformations with LLMs, reporting >...

  3. EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries

    cs.DB 2026-06 conditional novelty 6.0 of 10

    Query-driven table integration that uses Steiner-tree search to choose which joins LLMs must verify, reporting 30%+ accuracy gains at 5x lower LLM cost.

Pith tools