REVIEW 3 major objections 6 minor 19 references
Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A vision-language model can be trained on 100 languages simultaneously, with as little as 25–50% of the data non-English, without degrading English performance.
desk verdict A useful, mostly solid empirical recipe for multilingual LVLM training, but the xGQA training/evaluation overlap makes the headline numbers provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a controlled data-mix experiment: identical high-quality English data (the LLaVA-Next instruction mix and ShareGPT4v dense captions) is machine-translated via NLLB into 99 languages, and models are trained under a fixed budget in the standard two-stage LLaVA protocol — caption pre-training, then instruction tuning — with LoRA on a multilingual LLM, changing only the language count and the English percentage. This isolates language composition as the single driver under study. The second mechanism is synthetic OCR: Synthdog-generated plot-style images with text sampled from each language's Wikipedia, probed by a new benchmark, SMPQA (Synthetic Multilingual Plot Question Answering), which splits reading text from an image and grounding prompt text to image text into separately scored tasks.
What would settle it
Re-run the key English-only versus L100 comparison on xGQA after removing GQA from the instruction-tuning mix, or on a never-released multilingual VQA test set; if the reported multilingual gains shrink toward zero, the central in-language training effect is largely an artifact of test-train overlap.
Extended reading notes
Core claim
The paper's central claim is that the multilingual behavior of a vision-language model is set almost entirely by which languages are in the training data, and hardly at all by how much data each language gets. In fixed-budget controlled runs, the authors machine-translate the identical English instruction and caption datasets into up to 99 other languages and vary only (a) the number of training languages, from 7 to 100, and (b) the English share of the budget, from 1% to 90%. Scaling to 100 languages sharply improves performance for the newly added languages — especially language fidelity, the ability to answer in the language asked — while costing previously covered languages and English only a negligible tax. The effective mixing ratio is English-heavy: 25–50% non-English data captures most of the multilingual gain, and larger non-English shares can even degrade some tasks. For text rendered inside images, translated prompts alone do nothing; only synthetic OCR data in the target script, added in pre-training and instruction tuning with the image encoder unfrozen, moves performance, and mostly for Latin scripts. Assembling these rules into a final recipe yields Centurio, a 100-language model that matches leading open vision-language models on English and high-resource languages and surpasses them on lower-resource ones across 14 tasks and 56 languages.
Load-bearing premise
The evaluation assumes the test benchmarks were not seen in training, and that assumption fails for xGQA, whose images and translated questions come from GQA, a dataset the model is trained on, so part of the reported multilingual gain may be memorization rather than genuine in-language transfer.
Editorial extensions
If this is right
- A 100-language vision-language model is affordable: keep roughly half the training data in English and machine-translate the rest, and low-resource languages improve sharply while English performance is retained.
- The curse of multilinguality does not bite at these budgets: growing the language set from 7 to 100 costs previously covered languages almost nothing, so coverage decisions can be made optimistically.
- More multilingual data is not automatically better: 25–50% non-English data saturates the gains in both pre-training and instruction tuning, and larger non-English shares can degrade some tasks.
- Multilingual text-in-image ability requires explicit synthetic OCR data in the target script plus an unfrozen image encoder; Latin-script languages respond quickly while non-Latin scripts remain far behind.
- Following the recipe end-to-end produces Centurio, which matches leading open multilingual vision-language models on English and high-resource languages and outperforms them on low-resource languages across 14 tasks and 56 languages.
Reading between the lines
- Editorial inference: because GQA appears in the training mix while xGQA reuses GQA's images and translated questions, part of the measured multilingual gain on xGQA may be test-train overlap; a contamination-free re-run could leave the paper's other benchmarks carrying more of the argument.
- Editorial inference: the persistent Latin-versus-non-Latin gap suggests the vision encoder, not language data volume, bounds text-in-image ability; a directly testable consequence is that an OCR-pretrained, script-aware encoder should close the gap faster than more synthetic data.
- Editorial inference: the 25–50% sweet spot was found at one architecture scale with 384px inputs, so the ratio may shift for much larger models, native-resolution encoders, or much bigger OCR budgets rather than transferring unchanged.
- Editorial inference: if the ratio transfers, the cheapest route to massively multilingual assistants is machine-translating a minority share of an existing English-only pipeline — a cost argument that follows from the paper's data but is not its headline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a systematic empirical study of multilingual training-data composition for large vision-language models (LVLMs). Using machine-translated versions of English instruction-tuning and pre-training data, the authors vary (1) the number of training languages, (2) the English-to-multilingual ratio in instruction tuning, (3) the same ratio in pre-training, and (4) the amount of synthetic multilingual OCR data. They evaluate across 13 tasks and 43 languages (plus CVQA for 56 languages total), report tier-wise averages, and conclude that training with up to 100 languages causes little English degradation, that 25-50% non-English data is a practical sweet spot, and that multilingual synthetic OCR data helps text-in-image understanding, especially for Latin scripts. They introduce a new benchmark, SMPQA, and train Centurio (two variants) using the derived recipe, obtaining strong multilingual results against open-weight LVLMs.
Significance. If the main claims hold, the paper would provide unusually actionable guidance for a practical question in multilingual LVLM training: how many languages and what data mix should be used, and at what cost. The study is broad (two LLM backbones, 100 languages, 13-14 tasks), the training pipeline is controlled via translation, and the authors are transparent about limitations. The introduction of SMPQA as a targeted OCR diagnostic is a useful community asset. However, the quantitative headline is weakened by a training-evaluation overlap for xGQA, by the lack of repeated runs or uncertainty estimates for the central 'optimal mix' comparisons, and by selecting the final recipe on the same evaluation suite used to demonstrate Centurio's state-of-the-art results. The qualitative direction of the findings is likely robust, but the specific 25-50% recommendation and the claimed 'greatly improved multilingual performance' need stronger support.
major comments (3)
- [§2.2-2.3, Table 8, Appendix C.4] xGQA is not demonstrably held out: the instruction-tuning mix in Table 8 includes GQA (72k images) and VG (86k Visual Genome images), while Appendix C.4 states that xGQA consists of 9,666 manually translated questions over 300 unique Visual Genome images taken from the GQA balanced test-dev set. Even if the GQA train/test split is image-disjoint, the VG portion draws from the same Visual Genome image pool as xGQA, so the exact evaluation images are plausibly seen during training. Because xGQA contributes to the RQ1/RQ2 tier averages (Tables 1-2) and to the final Centurio comparison (Table 5), a nontrivial overlap would inflate the apparent multilingual gains from in-language instruction tuning. Please quantify the image-level overlap, exclude overlapping images, or report whether the conclusions are unchanged when xGQA is removed from the averages.
- [§2.3-2.4, Tables 2-3; §3.1-3.2] Each configuration is trained once, without multiple seeds or error bars, and the differences among the ratios used to select the 'optimal' 50% English setting are small (e.g., Table 2: T3 scores are 29.9, 30.7, 30.7 for 25%, 50%, 75%; T4 scores are 27.9, 28.9, 28.4; T5 scores are 33.0, 34.4, 34.6). The claim that 25-50% is a robust optimum is therefore not statistically supported. In addition, the same evaluation suite used in §2 to choose E_IT=50 and E_PT=50 is then reused in §3 to benchmark Centurio, creating a selection-on-test effect. Please provide repeated runs or confidence intervals, and validate the recipe on a benchmark not used in the analysis (CVQA is a useful step, but it is only one additional task).
- [Abstract and §2.3-2.4] The abstract's recommendation of 'as little as 25-50% non-English data' is not directly supported by the reported tables. In Table 2 the lower-resource tiers T1 and T2 perform best at 25% English, while T5 and English perform best at 75-90% English, and Table 3 shows E_PT=1% and E_PT=50% give nearly identical results for most tiers. The 25-50% sweet spot is thus a judgment call across tiers and training stages rather than a quantitative finding. Please either revise the claim to state the tier-dependent pattern precisely or provide an explicit averaging criterion that justifies the selected operating point.
minor comments (6)
- [§2.3, Table 2 caption] The text says performance peaks 'between 25% and 75%', while the abstract says '25-50%'; these should be reconciled, ideally with the tier-dependent numbers in the main text.
- [Appendix D.3] There is a typo: 'Qwen achievs' should be 'Qwen achieves'.
- [Table 9] The row for Design2Code lists '0k' as the dataset size, which appears to be a typo; please clarify the actual number of samples.
- [Table 5 caption and several appendix tables] The label 'A VG.' appears in several table captions and column headers; this should be formatted as 'Avg.' for readability.
- [§5, conclusion] The phrase '56 language tasks' should read '56 languages' (or 'language-task combinations') for accuracy.
- [§2.3, last sentence] There is a missing space in 'E = 50as a robust value'; it should read 'E = 50 as a robust value'.
Circularity Check
No significant circularity: the paper’s central claims are empirical, grounded in controlled experiments and external benchmarks, and the self-citations are not load-bearing.
full rationale
I examined the derivation chain from the controlled training-mix experiments (RQ1–RQ3), the OCR study (RQ4), and the final Centurio model. The central quantitative claims are supported by comparisons across training-language sets, English/multilingual ratios, and pre-training/instruction-tuning conditions; these are empirical search results, not quantities defined in terms of the outcomes they are said to predict. The choice E=50% English is a hyperparameter selected from the analysis experiments and then applied to Centurio; this is model selection, not a fitted parameter renamed as a prediction, and the final comparison includes external baselines. SMPQA is introduced as a new evaluation benchmark and the training OCR data is generated separately with Synthdog, so RQ4 does not train on its own test set. The self-citations (mBLIP, Babel-ImageNet) are contextual or constitute one evaluation signal among many; Babel-ImageNet is anchored in external ImageNet labels and is not used to define the training objective. The strongest validity concern, noted by the skeptic, is that xGQA may share Visual Genome images with the VG/GQA training portions (Appendix C.4 vs. Table 8); that is a potential contamination/held-out threat that could inflate multilingual gains, but it is not a circular reduction by construction. The paper’s Section 6 limitations candidly discuss MT data quality, translationese, and synthetic OCR limitations, reinforcing that the contribution is empirical rather than definitional. No quoted equation or parameter is shown to be equivalent to its own input, so no circular step meets the required evidentiary bar.
Assumptions & free parameters
free parameters (4)
- English share in instruction tuning (E_IT) =
50%
- English share in pre-training (EPT) =
50%
- Synthetic OCR data budget per non-English language =
5k for Latin script, 10k for other scripts, 500k total
- Number of training languages =
100
assumptions (4)
- domain assumption Machine-translated English data is a sufficient proxy for native multilingual training data across all 99 non-English languages.
- domain assumption The LLaVA-style architecture with SigLIP encoder, Phi-3.5 LLM, and LoRA tuning is representative of LVLM training.
- domain assumption The selected 13-task evaluation suite, later 14 with CVQA, measures multilingual ability.
- domain assumption Language resource tiers from Joshi et al. (2020) are a valid grouping for studying language difficulty.
invented entities (1)
-
SMPQA (Synthetic Multilingual Plot QA) benchmark
Cite this review
Pith. "Pith review of Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model." pith.science (2026). https://pith.science/paper/PVM4J7ML
@misc{pith2026250105122,
author = {Pith},
title = {Pith review of: Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/PVM4J7ML}},
note = {Machine review of arXiv:2501.05122}
}
read the original abstract
Most Large Vision-Language Models (LVLMs) to date are trained predominantly on English data, which makes them struggle to understand non-English input and fail to generate output in the desired target language. Existing efforts mitigate these issues by adding multilingual training data, but do so in a largely ad-hoc manner, lacking insight into how different training mixes tip the scale for different groups of languages. In this work, we present a comprehensive investigation into the training strategies for massively multilingual LVLMs. First, we conduct a series of multi-stage experiments spanning 13 downstream vision-language tasks and 43 languages, systematically examining: (1) the number of training languages that can be included without degrading English performance and (2) optimal language distributions of pre-training as well as (3) instruction-tuning data. Further, we (4) investigate how to improve multilingual text-in-image understanding, and introduce a new benchmark for the task. Surprisingly, our analysis reveals that one can (i) include as many as 100 training languages simultaneously (ii) with as little as 25-50\% of non-English data, to greatly improve multilingual performance while retaining strong English performance. We further find that (iii) including non-English OCR data in pre-training and instruction-tuning is paramount for improving multilingual text-in-image understanding. Finally, we put all our findings together and train Centurio, a 100-language LVLM, offering state-of-the-art performance in an evaluation covering 14 tasks and 56 languages.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
For each plot type, we define 50 configurations, so we have 100 plots/images in total per language
We define a list of diverse pie charts and bar plots by randomly sampling the number of bars/s- lices, the size of each, their colors, the plot size and aspect ratio, and vertical/horizontal orientation for bar plots, and exploding some slices in pie charts. For each plot type, we define 50 configurations, so we have 100 plots/images in total per language
-
[2]
Using word lists of common words in the lan- guages, we sample words for use as labels for the bars and pie slices to fill and ultimately render the pre-defined plots. This means the plots are iden- tical between languages except for the labels and some size adjustments caused by different word lengths
-
[3]
For each plot, we use templates to generate 5 questions for reading and 8 questions for grounding (with balanced ‘yes’ and ‘no’ as answers). The questions are always the same for a plot, so each language has the same questions, just with different labels. Language Selection: We selected the languages as follows: For Latin-script languages, we chose Englis...
-
[6]
Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?
Turning English-centric LLMs Into Poly- glots: How Much Multilinguality Is Needed? CoRR, abs/2312.12683. ArXiv: 2312.12683. Tannon Kew, Florian Schottmann, and Rico Sennrich
-
[7]
Association for Computational Linguistics
Turning English-centric LLMs into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13097–13124, Miami, Florida, USA. Association for Computational Linguistics. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- seok Hwang, Sangdoo Yun, Dongy...
arXiv 2024
-
[8]
SPHINX: A Mixer of Weights, Visual Em- beddings and Image Scales for Multi-modal Large 13 Language Models. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXII, volume 15120 of Lecture Notes in Computer Science, pages 36–55. Springer. Adam Dahlgren Lindström and Savitha Sam Abraham
work page 2024
-
[10]
DeepSeek-VL: Towards Real-World Vision- Language Understanding. CoRR, abs/2403.05525. ArXiv: 2403.05525. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021a. Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning. In Proceedings of the 59th Annual Meeting of the Asso- cia...
arXiv 2021
-
[13]
LAION-5B: An open large-scale dataset for training next generation image-text models. CoRR, abs/2210.08402. ArXiv: 2210.08402. Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: A Benchmark for Visual Question An- swering Using World Knowledge. In Computer Vi- sion - ECCV 2022 - 17th European Confere...
arXiv 2022
Show all 19 references
-
[15]
{HYPOTHESIS}
introduces the task of Cross-lingual Vi- sual Natural Language Inference where a model needs to predict whether a textual hypothesis en- tails, contradicts, or is neutral concerning a visual premise. XVNLI comprises five languages cov- ering three scripts and 357 unique images...
2015
-
[16]
Is the bar with label ’reward’ the biggest?
is a large multilingual image captioning dataset comprising 36 languages with 261375 cap- tions covering 13 different scripts for 100 unique images per language. The images are selected to match the language’s cultural background, ensuring cultural and linguistic diversity. Th...
2024
-
[2016]
A Diagram is Worth a Dozen Images. In Com- puter Vision - ECCV 2016 - 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 235–251. Springer. Aniruddha Kembhavi, Min Joon Seo,...
2016
-
[2019]
TallyQA: Answering Complex Counting Ques- tions. In The Thirty-Third AAAI Conference on Artifi- cial Intelligence, AAAI 2019, The Thirty-First Innova- tive Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educa- tional Advances in Arti...
2019 arXiv
-
[2020]
arXiv:2005.14165 [cs]
Language Models are Few-Shot Learners. arXiv:2005.14165 [cs]. ArXiv: 2005.14165. Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulic. 2022. IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Lan...
2005 arXiv
-
[2022]
CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning. In Proceedings of the 16th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 2nd International Joint Conference on Learn- ing & Reasoning (IJCLR 2022), Cumberland...
2022 arXiv
-
[2023]
CoRR, abs/2305.06500
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. CoRR, abs/2305.06500. ArXiv: 2305.06500. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terre...
2024 arXiv
-
[2024]
CoRR, abs/2407.07726
PaliGemma: A versatile 3B VLM for transfer. CoRR, abs/2407.07726. ArXiv: 2407.07726. Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluís Gómez i Bigorda, Marçal Rusiñol, C. V . Jawahar, Ernest Valveny, and Dimosthenis Karatzas. 2019. Scene Text Visual Question Answering. In 2019...
2019 arXiv
-
[2591]
Minesh Mathew, Dimosthenis Karatzas, and C
IEEE. Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawa- har. 2021. DocVQA: A Dataset for VQA on Docu- ment Images. In IEEE Winter Conference on Applica- tions of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 2199–2208. IEEE. Anand Mishra, Shashank...
2021 arXiv
-
[6786]
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan
Association for Computational Linguistics. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Sci- ence Question Answering. In Advanc...
2022 arXiv
-
[7298]
Association for Computational Linguistics. Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yu- liang Liu, Hao Liu, Xiang Bai, and Can Huang. 2024. MTVQA: Benchmarking Multilingual...
2020 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.