Pith. sign in

REVIEW 6 cited by

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.04662 v1 pith:ZITBFQGG submitted 2023-09-09 cs.CL cs.LG

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

Sneha Kudugunta , Isaac Caswell , Biao Zhang , Xavier Garcia , Christopher A. Choquette-Choo , Katherine Lee , Derrick Xin , Aditya Kusupati
show 3 more authors
Romi Stella Ankur Bapna Orhan Firat
This is my paper
classification cs.CL cs.LG
keywords datasetmadlad-400auditedavailableb-parameterdatalanguagesmodel
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Scalable Extraction of Training Data from (Production) Language Models

    cs.LG 2023-11 conditional novelty 7.0

    Adversaries can scalably extract gigabytes of training data from open, semi-open, and closed language models via querying attacks, including a divergence method that increases extraction rates 150x on aligned models l...

  2. Cross-Lingual Transfer for Machine Translation in Turkic Languages

    cs.CL 2026-07 conditional novelty 6.0

    Among five Turkic languages, mT5 transfer is strongest for Turkish–Azerbaijani and Kazakh–Kyrgyz, direction and translation target matter, and Latinization helps surface metrics mainly in script-mismatched pairs.

  3. How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP

    cs.CL 2024-11 unverdicted novelty 6.0

    The study filters non-English Wikipedia, reveals quality problems, proposes a 4-level ranking, and shows filtered data matches or beats raw data in language modeling with largest gains for lower-quality editions.

  4. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    cs.CL 2025-07 unverdicted novelty 4.0

    Gemini 2.5 Pro and Flash models are presented as achieving frontier performance in reasoning, coding, and long-context multimodal tasks while spanning a cost-capability Pareto curve.

  5. Gemma: Open Models Based on Gemini Research and Technology

    cs.CL 2024-03 accept novelty 4.0

    Gemma introduces open 2B and 7B LLMs derived from Gemini technology that beat comparable open models on 11 of 18 text tasks and come with safety assessments.

  6. Gemma 2: Improving Open Language Models at a Practical Size

    cs.CL 2024-07 conditional novelty 3.0

    Gemma 2 models achieve leading performance at their sizes by combining established Transformer modifications with knowledge distillation for the 2B and 9B variants.