REVIEW 69 cited by
mT5: A massively multilingual pre-trained text-to-text transformer
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
Forward citations
Showing 60 of 69 Pith papers that cite this
-
STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection
STATE ToxiCN provides the first span-level Chinese hate speech dataset with 9,533 target-argument-hateful-group quadruples and a 830-term annotated hateful slang lexicon, and baseline results show fine-tuned models ou...
-
On the Fitness Landscape in the $NK$ Model
For the NK fitness landscape with K/N tending to alpha, exact limits for free energy and maximum fitness are identified, together with the geometry of near-fittest peaks.
-
Meta CLIP 2: A Worldwide Scaling Recipe
A data curation and training recipe that scales CLIP from English-only data to 300+ languages from scratch, breaking the curse of multilinguality at ViT-H/14 scale.
-
Fine-Grained Chinese Hate Speech Understanding: Span-Level Resources, Coded Term Lexicon, and Enhanced Detection Frameworks
The paper creates a span-level Chinese hate speech dataset and a 830-term coded hate lexicon, but its two-stage training method's reported superiority is contradicted by the paper's own COLD results.
-
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Under a fixed model-FLOPs budget, the fitted loss for sparse MoE models decreases monotonically with sparsity, so the optimum sits at the boundary; an interior sparsity optimum appears only when hardware-deliverable F...
-
Effective vocabulary expansion of multilingual language models for extremely low-resource languages
Using bilingual-dictionary-aligned embeddings to initialize expanded vocabulary improves mBERT continued pre-training for seven low-resource languages, with average gains of 0.54% (POS) and 2.60% (NER).
-
Building Large-Scale English-Romanian Literary Translation Resources with Open Models
LoRA-fine-tuning a 12B open model on 15,000 GPT-o3 fable translations yields rubric scores close to GPT-o3 (4.83 vs 4.92) for English-to-Romanian literary translation at roughly one percent of the API cost.
-
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
mmBERT, a 3T-token encoder-only model pretrained on over 1,800 languages with inverse mask-rate and temperature schedules, substantially outperforms prior multilingual encoders like XLM-R and approaches ModernBERT on English.
-
Evaluating LLMs on Chinese Idiom Translation
Across 900 annotated translation pairs from nine MT systems, the best system still mistranslates Chinese idioms in 28% of cases, and standard metrics miss these errors (Pearson correlation below 0.48).
-
Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries
CEMA converts multi-task black-box text attacks into attacks on a binary classifier trained on cluster pseudo-labels, achieving high attack success with 100 queries.
-
Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan
Multi-TW is the first Traditional Chinese benchmark to evaluate multimodal models on both image-text and audio-text questions while also measuring inference latency.
-
CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
CONCAP combines retrieved captions with retrieved concepts to improve multilingual image captioning, reaching 34.2 average CIDEr on XM3600 against 31.8 for Pangea and 25.9 for mBLIP while training on 566K pairs.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages
In a large benchmark, prompting and translate-test outperform gradient-based adaptation for in-context learning in low-resource languages, with degradation traced to catastrophic forgetting.
-
Cost-Optimal Active AI Model Evaluation
The paper derives budget-optimal rules for mixing cheap weak raters and expensive strong raters so that the mean rating is estimated with minimum variance, and shows large cost savings when example difficulty varies.
-
OneSug: The Unified End-to-End Generative Framework for E-commerce Query Suggestion
A unified encoder-decoder model with prefix representation enhancement and reward-weighted DPO outperforms Kuaishou's online multi-stage query suggestion system in offline and live A/B evaluations.
-
Entity Image and Mixed-Modal Image Retrieval Datasets
A new benchmark, MMIR, built from Wikipedia and WIT by masking entity names and supplying canonical entity images, tests mixed-modal retrieval with single- and multi-entity queries.
-
RepText: Rendering Visual Text via Replicating
A FLUX-based control module renders multilingual text by replicating glyph shapes from canny and position inputs, using glyph-latent initialization, region masks, and an OCR loss, with qualitative parity to closed-sou...
-
Training Sparse Mixture Of Experts Text Embedding Models
Nomic Embed v2 applies sparse mixture-of-experts upcycling to a multilingual biencoder, reporting competitive BEIR and MIRACL scores with fewer active parameters than dense models of similar size.
-
Scaling Pre-training to One Hundred Billion Data for Vision Language Models
Scaling VLM pretraining from 10B to 100B image-text pairs yields saturation on standard benchmarks but large gains on cultural diversity, low-resource language retrieval, and subgroup disparity.
-
PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts
PiKE adaptively re-weights pretraining data sources by gradient magnitude and variance, exploiting low gradient conflicts to speed up convergence and improve downstream accuracy in LLM pretraining.
-
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile
A three-stage pipeline combining sparse 'tile' attention with multi-step consistency distillation makes Open-Sora-Plan video generation up to 7.8x faster while keeping the aggregate VBench final score within 1%.
-
Towards the Development of Balanced Synthetic Data for Correcting Grammatical Errors in Arabic: An Approach Based on Error Tagging Model and Synthetic Data Generating Model
A tag-conditioned Arabic synthetic data pipeline is claimed to set a new GEC state of the art, but the reported 79.36% F1 is the F0.5 score from the paper's own table.
-
Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning
Training-order effects in multi-domain learning are governed by the Lie bracket of the per-domain gradient flows, yielding a local optimality criterion for weight schedules.
-
Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction
A DQN-based candidate selection and a competitive correction block improve MT ensembling quality while reducing inference cost on English-Hindi and Hindi-English tasks.
-
V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.
-
Infusing Prompts with Syntax and Semantics
Appending syntactic and semantic analyses to prompts improves text-to-SQL accuracy in four low-resource languages and speeds fine-tuning.
-
Yankari: A Monolingual Yoruba Dataset
Yankari is a curated 30-million-token monolingual Yoruba dataset built from 13 web sources without religious or machine-translated content.
-
MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
A 20M-parameter adapter trained only on English image-text pairs adapts frozen diffusion models to generate images from prompts in over 110 languages, using a multilingual image-text encoder as its text encoder.
-
INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
INCLUDE is a multilingual benchmark of 197,243 exam questions from local sources that evaluates how well LLMs handle regional and cultural knowledge.
-
DiffSLT: Enhancing Diversity in Sign Language Translation via Diffusion Model
DiffSLT uses a latent diffusion model conditioned on fused multi-level visual features to produce diverse, accurate sign language translations, and DiffSLT-P conditions on pseudo-glosses to improve accuracy further.
-
Training Bilingual LMs with Data Constraints in the Targeted Language
Higher-quality auxiliary English pretraining data improves target-language performance for languages close to English (about 2% on translated QA tasks), but not for distant languages, when target-language data is limi...
-
Heuristic-Free Multi-Teacher Learning
By turning each teacher's labels into a separate confidence-prediction task conditioned on a teacher token, Teacher2Task claims to outperform the best individual teacher on open-vocabulary tagging without manual aggre...
-
MultimodalHugs: Enabling Sign Language Processing in Hugging Face
MultimodalHugs provides a standardized TSV-based dataset format, modular processors, and Hugging Face integration to enable reproducible sign language and multimodal translation experiments.
-
Two Spelling Normalization Approaches Based on Large Language Models
Fine-tuned mT5 and mBART perform competitively on Spanish and Slovene historical spelling normalization, but character-based SMT achieves the best overall scores, confirming its continued state-of-the-art status.
-
Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languages
Generating Python programs as intermediate guidance before SQL, then voting on Python execution results to select the fastest matching SQL, improves text-to-SQL execution accuracy and efficiency on BIRD and Archer.
-
Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
MSumBench is a new English/Chinese benchmark that grades summaries across six domains on faithfulness, completeness, and conciseness, using domain-specific key-facts and multi-agent-debate-assisted human annotations, ...
-
Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Model
A compact 1.5B Arabic-English model beats GPT-4o mini only on the authors' own Tarjama-25 benchmark, while trailing large models on standard WMT24++ and IWSLT2017 tests.
-
SELF: Self-Extend the Context Length With Logistic Growth Function
SELF replaces Self-Extend's fixed token grouping with a logistic-growth grouping schedule, giving mixed but sometimes large gains on long-context benchmarks.
-
LAGO: Few-shot Crosslingual Embedding Inversion Attacks via Language Similarity-Aware Graph Optimization
LAGO shows that constraining alignment matrices of linguistically similar languages to be close improves few-shot cross-lingual embedding inversion accuracy over independent per-language baselines.
-
Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks
A survey that categorizes multilingual prompting techniques by NLP task and language family, and designates potential state-of-the-art prompting methods for each dataset.
-
Investigating Task Arithmetic for Zero-Shot Information Retrieval
Adding domain or language task vectors to an MS-MARCO fine-tuned reranker improves zero-shot retrieval on several biomedical, scientific, and multilingual datasets, with the largest gains after tuning the injection strength.
-
Trillion 7B Technical Report
Trillion-7B pairs Korean documents with English documents during pretraining and lets them attend to each other, claiming competitive Korean performance with only about 10% multilingual tokens.
-
Kuwain 1.5B: An Arabic SLM via Language Injection
Kuwain 1.5B, built by freezing TinyLlama's original layers and training eight added layers plus a 26K Arabic tokenizer, improves Arabic benchmark average from 36.95 to 44.49 while keeping English near 53.28.
-
Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs
Selective pre-translation, translating only some prompt components into English, generally outperforms both full prompt translation and direct inference across tasks and languages, with the largest gains for low-resou...
-
Multilingual Open QA on the MIA Shared Task
Zero-shot question-generation reranking boosts Korean and Japanese retrieval but hurts Finnish and Bengali, and translated training data yields no consistent QA improvement.
-
Empowering Bengali Education with AI: Solving Bengali Math Word Problems through Transformer Models
Fine-tuning mT5, BanglaT5, mBART50 and a basic Transformer on a newly translated Bengali math word problem dataset yields up to 97.3% solution accuracy on elementary arithmetic problems.
-
DIRECT: Direct Decoding for Efficient and Aligned Sequence Labeling with Large Language Models
DPO after SFT plus constrained template-filling decoding improves LLM sequence-labeling accuracy and cuts inference time by reusing KV cache for non-label tokens.
-
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
The paper surveys Indic NLP evolution and proposes 'Culture Sensing' to integrate indigenous oral knowledge into foundation models for cultural preservation.
-
SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System
A bidirectional MSA-Syrian Arabic translation system built by fine-tuning AraT5v2 on the Nabra corpus; only the MSA-to-Shami direction is evaluated, with a GPT-4.1 score of 4.01/5.
-
Enhancing Entity Aware Machine Translation with Multi-task Learning
A multi-task mT5 model that predicts named entities, translates them, and translates the full sentence outperforms mT5 and mBART baselines in BLEU on three of four SemEval 2025 Task 2 language pairs.
-
Token-free Models for Sarcasm Detection
ByT5-small reaches 89.87% and CANINE reaches 72.88% on news-headline and Twitter sarcasm detection, each edging a T5 baseline by less than one accuracy point.
-
Analysis of Indic Language Capabilities in LLMs
A desk-research review finds that LLM performance is strongest for Hindi, Bengali, Marathi, Telugu, and Tamil, and recommends prioritizing these five languages for safety benchmarks.
-
A Method for Multi-Hop Question Answering on Persian Knowledge Graph
A decomposition-based Persian KGQA method and a new 5,600-question decomposition dataset improve F1 from 62.98% to 75.55% on PeCoQ.
-
The First Multilingual Model For The Detection of Suicide Texts
mT5 fine-tuned on Spanish suicide tweets translated into five languages outperforms mBERT and XLM-R across six languages, but the evaluation is under-specified and the F1 'above 85%' claim is contradicted for two languages.
-
Deploying Foundation Model Powered Agent Services: A Survey
This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...
-
The Generative AI Ethics Playbook
A structured playbook that collects existing guidance, checklists, and case studies to help generative AI practitioners identify and mitigate ethical harms across six lifecycle stages.
-
Enabling Low-Resource Language Retrieval: Establishing Baselines for Urdu MS MARCO
A machine-translated Urdu MS MARCO dataset and fine-tuned mT5 reranker achieve MRR@10 0.248 and Recall@10 0.438, outperforming zero-shot baselines and providing first baselines for Urdu IR.
-
7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement
The authors trained and openly released a 7B LLM, an instruction-tuned variant, a GRPO-based reasoning variant, and a VLM, claiming competitive or superior performance on zero-shot, few-shot, CoT, and VLM benchmarks.
-
BanglaDialecto: An End-to-End AI-Powered Regional Speech Standardization
An ASR + MT + TTS pipeline converts Noakhali dialect speech to standard Bangla, with Whisper-large V2 achieving 0.8% CER and BanglaT5 a 41.6 BLEU on the authors' NDD dataset.
Discussion (0). Continue with ORCID to comment.