REVIEW 3 major objections 4 minor 12 references
Multilingual Definition Modeling
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Per-language performance on definition modeling correlates strongly with multilingual LLM benchmark scores, making dictionary definitions a potential cheap and stable language-proficiency probe.
desk verdict Useful multilingual definition modeling study with new dictionary datasets, but the headline BERTScore correlation claim is misreported in Table 12 and does not hold for Llama-3. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the definition modeling task itself, reduced to a text-to-text generation problem: prompt a model to "Define the {language} word X" and score the generated definition against a dictionary gold standard with BERTScore, a learned semantic-similarity metric between generated and reference text. The argument then compares those per-language BERTScore values against per-language accuracy on three multilingual benchmarks (HellaSwag, ARC-Challenge, MMLU) and reports the Pearson correlation across the five languages. The supporting data are new monolingual dictionary datasets (DRAE, DICIO, Larousse, Duden) restricted to monosemic entries, which are used to fine-tune mT5 and to build zero- and few-shot prompts for the chat LLMs.
What would settle it
Compute the same per-language BERTScore correlations after adding another ten or more languages, or after replacing BERTScore with human quality ratings; the proxy claim would be refuted if the average correlation with HellaSwag, ARC, and MMLU falls below roughly 0.8 or if the language ranking changes materially.
Extended reading notes
Core claim
The paper's central claim is that the ability to define simple, monosemic words in a language, measured automatically by BERTScore, is a reliable stand-in for how well an LLM performs on standard multilingual benchmarks in that same language, at a fraction of the compute. On per-language results for English, Spanish, Portuguese, French, and German, Pearson correlations with HellaSwag, ARC-Challenge, and MMLU average 0.94 for Llama 2, 0.95 for Mistral, and 0.87 for Llama 3. The authors interpret this as evidence that definition modeling offers a compute-constrained, stable, and natural alternative to multilingual benchmarks, whose multilingual versions are often translations of English tests. They also report that fine-tuning mT5 on multiple Romance languages does not improve per-language performance over monolingual fine-tuning, and that in-context learning does not reliably improve LLM-generated definitions.
Load-bearing premise
The headline correlation rests on only five languages per model, computed without confidence intervals or significance tests, and assumes BERTScore measures definition quality equally well across those languages.
Editorial extensions
If this is right
- If the correlation holds, researchers can probe multilingual proficiency by generating dictionary definitions instead of running large multiple-choice benchmarks, cutting token cost to roughly 0.5% of MMLU's budget.
- Fine-tuning a multilingual encoder-decoder on several related languages neither helps nor hurts per-language definition quality, so cross-lingual synergy should not be assumed from data alone.
- Chat LLMs outperform fine-tuned multilingual models on definition quality, but their outputs must be filtered for language compliance because they sometimes answer in English.
- In-context learning and dictionary fine-tuning do not reliably improve over zero-shot prompting, suggesting the bottleneck is recall of lexical knowledge from pretraining rather than task recognition.
- Portuguese and German definitions score lowest in both automatic and human evaluation, consistent with the comparatively lower quality of their dictionary gold standards.
Reading between the lines
- If the correlation is causal rather than incidental, definition modeling could be extended to lower-resourced languages whose dictionaries exist but whose benchmark translations do not, giving a cheap first-pass language-proficiency estimate.
- The reported Portuguese BLEU scores are inflated by a lemmatization-based data leak described in Appendix C.1, so the Portuguese correlation point may carry less weight than the other four languages.
- The BERTScore backbone is unspecified in the paper; choosing a different embedding model could shift scores and change the correlation, so the proxy's stability should be tested across metric backbones.
- Since the paper's own pilot shows LLMs can generate usable example sentences for polysemic terms, a natural next step is to extend the benchmark from monosemic words to sense-level definitions with LLM-generated contexts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents the first multilingual definition-modeling study, introducing four new dictionary-derived datasets (Spanish DRAE, Portuguese DICIO, French LAROUSSE, German DUDEN) alongside the existing English OXFORD data, and filtering to monosemic terms. It fine-tunes mT5/mBART models and evaluates Llama-2, Llama-3.1, and Mistral in zero-shot and few-shot settings using BLEU, BERTScore, COMET, language-compliance checks, and human evaluation. The main empirical findings are that multilingual LMs match but do not improve on English-only performance, that LLMs generally outperform finetuned multilingual LMs, and that per-language BERTScore on the definition-modeling task correlates strongly with HellaSwag, MMLU, and ARC-Challenge, leading the authors to propose the task as a compute-constrained alternative benchmark.
Significance. If the correlation claim were supported, the proposed task would be a valuable low-cost, language-specific evaluation tool; the new datasets and the first multilingual experiments on definition modeling are useful contributions to the field. The paper is transparent about data provenance, carefully documents quantization effects, reports human judgments with agreement measures, and releases reconstruction recipes and code, which aids reproducibility. However, the headline claim in Section 5.3 currently rests on internally inconsistent numbers and a very small language sample, so the paper's central practical proposal is not yet established.
major comments (3)
- [§5.3 / Table 12] Section 5.3 states that per-language performance measured via BERTScore 'strongly correlates' with HellaSwag, MMLU, and ARC-Challenge, with averages of 0.94, 0.95, and 0.87 for Llama 2, Mistral, and Llama 3, respectively. In Table 12, the BS column (BERTScore, following the notation used throughout the paper) gives averages of 0.78, 0.73, and -0.18 for the same three models; the stated averages instead coincide with the BL (BLEU) column. The Llama-3 BERTScore correlations are negative (-0.18, -0.10, -0.25), so the claim that BERTScore strongly correlates is contradicted by the table as printed. Because the 'viable compute-constrained alternative' conclusion in the abstract and Section 6 depends directly on this correlation, the authors must either correct the table or the text, state which metric the analysis actually uses, and re-check Figure 1, whose 'Corr. w/ours' labels also match the BL column rather than the BS column.
- [§5.3 / Figure 1] The correlation analysis uses only five language points (en, es, pt, fr, de), with no confidence intervals, significance tests, or leave-one-out diagnostics. With n=5, even an r of 0.78 has a two-sided p-value around 0.12 and is not significant at the 0.05 level, and the reported 0.95 average would have a very wide interval. The acknowledged Portuguese leakage in Appendix C.1 is a further threat because Portuguese is one of the five points and its BLEU scores are explicitly described as unusually high due to 'limitations of our lemmatization-based data cleaning technique'. The authors should report intervals or per-language sensitivity analyses (for example, omitting each language in turn) before claiming that the task is a 'stable' alternative.
- [§4 / Evaluation] Section 4 does not specify the BERTScore backbone model. BERTScore is computed relative to a contextual encoder, and the per-language comparability of the scores, which is the foundation of the Section 5.3 correlation, depends on using a genuinely multilingual and language-fair encoder. Please report the exact checkpoint used (e.g., a specific mBERT or XLM-R variant), state whether the same backbone was applied to all languages, and verify that the Section 5.3 conclusions are stable under an alternative backbone.
minor comments (4)
- [§1 / §5.3 / §6] The proposed alternative is described inconsistently as 'static' in the introduction, 'stable' in Section 5.3, and 'natural, stable' in the conclusion; the wording should be unified.
- [Appendix C / §4] The heading 'Finetuninig LLMs' and the word 'perfomance' in §4 are typos; 'on-pair' in the abstract and Section 5 should read 'on par'.
- [Table 1] The percentage '73,17%' for German uses a comma as the decimal separator, which is inconsistent with the rest of the table.
- [Tables 3, 4, 7–9, 11–13] The abbreviations BL, BS, and CMT are defined only in the caption of Table 3 but are reused in many later tables without repetition; given the confusion in Table 12, a global notation table or a repeated definition in each caption would improve readability.
Circularity Check
No significant circularity: the benchmark-correlation claim is an empirical measurement, not a fitted or self-referential construction.
full rationale
The paper's central quantitative claim is that per-language definition-modeling performance, measured by BERTScore, correlates with per-language scores on HellaSwag, MMLU, and ARC-Challenge. This is presented in Section 5.3 and Table 12 as a Pearson correlation computed from observed model outputs and external benchmark runs over five languages. No parameter is fitted to force the relationship, and the benchmark values are not derived from definition-modeling scores by construction. The conclusion that definition modeling 'can offer a viable compute-constrained, stable and natural alternative' is an interpretive extrapolation from that measured correlation, not a result whose definition is equivalent to its inputs. Self-citations to Reid et al. (2020) and Jhirad et al. (2023) are used as prior-work comparisons and as a source for the COMET metric choice; they are not load-bearing premises that define the headline finding, and no uniqueness theorem or modeling ansatz is imported from the authors' own prior work. The limitations section appropriately notes that generalization to other languages is untested, which is a scope caveat rather than a circular step. A separate concern, not a circularity issue, is that the printed Table 12 appears inconsistent with the text: the claimed BERTScore averages of 0.94 and 0.95 match the BLEU column instead of the BERTScore column, and the Llama-3 BERTScore correlations are negative; this is a reporting or consistency problem that affects evidential strength but does not make the derivation circular.
Assumptions & free parameters
assumptions (6)
- domain assumption Monosemic word subsets are representative of definition modeling ability.
- domain assumption BERTScore provides a language-fair similarity measure.
- domain assumption The multilingual versions of HellaSwag, MMLU, and ARC are valid measures of language proficiency.
- domain assumption The dictionary data from DRAE, DICIO, Larousse, and Duden are high-quality ground truth.
- domain assumption Definition pairs in the test sets were not seen during pretraining of the LLMs.
- ad hoc to paper The lemmatization pipeline does not create train/test leakage.
Cite this review
Pith. "Pith review of Multilingual Definition Modeling." pith.science (2026). https://pith.science/paper/YJ674W4N
@misc{pith2026250601489,
author = {Pith},
title = {Pith review of: Multilingual Definition Modeling},
year = {2026},
howpublished = {\url{https://pith.science/paper/YJ674W4N}},
note = {Machine review of arXiv:2506.01489}
}
read the original abstract
In this paper, we propose the first multilingual study on definition modeling. We use monolingual dictionary data for four new languages (Spanish, French, Portuguese, and German) and perform an in-depth empirical study to test the performance of pre-trained multilingual language models on definition modeling of monosemic words when finetuned on this data. Furthermore, we use a zero-shot approach to test the multilingual capabilities of two popular chat-based Large Language Models (LLMs) in the task. Results show that multilingual language models can perform on-pair with English but cannot leverage potential cross-lingual synergies, with LLMs generally offering better performance overall. A comprehensive human evaluation of the LLM-generated definition highlights the zero and few-shot capabilities of these models in this new task, also showing their shortcomings. Finally, we show that performance on our task via BERTScore strongly correlates to the performance on multilingual LLM benchmarks, suggesting that our task offers a viable compute-constrained, stable and natural alternative to these.
Figures
Reference graph
Works this paper leans on
-
[10]
In Forty-first International Conference on Machine Learning
tinybenchmarks: evaluating LLMs with fewer examples. In Forty-first International Conference on Machine Learning. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computa- tional Linguistics. Colin Raffel, Noam Sh...
work page 2018
-
[11]
NLP evaluation in trouble: On the need to mea- sure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore. Association for Computational Linguistics. Franck Sajous, Basilio Calderone, and Nabil Hathout
work page 2023
-
[12]
You are a helpful assistant. Always answer as helpfully and concisely as possible
ENGLAWI: From Human- to Machine- Readable Wiktionary. In 12th International Confer- ence on Language Resources and Evaluation (LREC 2020), page 3016. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual Translation with Exten- sible Multilingual Pretraining and Finetuning. Hugo T...
arXiv 2012
-
[161]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa
Springer. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large Lan- guage Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems. Cunliang Kong, Liner Yang, Tianzuo Zhang, Qinan Fan, Zhenghao Liu, Yun Chen, and Erhong Yang
work page 2022
-
[1901]
Basilio Calderone, Matteo Pascoli, Franck Sajous, and Nabil Hathout
Curran Associates, Inc. Basilio Calderone, Matteo Pascoli, Franck Sajous, and Nabil Hathout. 2017. Hybrid Method for Stress Pre- diction Applied to GLAFF-IT, a Large-Scale Italian Lexicon. In Language, Data, and Knowledge, Lec- ture Notes in Computer Science, pages 26–41, Cham. Springer International Publishing. Pinzhen Chen and Zheng Zhao. 2022. A Unifie...
arXiv 2017
-
[2014]
GLÀFF, a Large Versatile French Lexicon. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1007–1012, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA). Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt
-
[2018]
Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266–271, Melbourne, Australia. Association for Computational Linguistics. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding...
arXiv 2023
-
[2020]
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen
Toward Cross-Lingual Definition Generation for Language Learners. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen
Show all 12 references
-
[2021]
In International Conference on Learning Representations
Measuring massive multitask language under- standing. In International Conference on Learning Representations. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Com...
2016 arXiv
-
[2022]
In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8298–8317, Dublin, Ireland
Generating Scientific Definitions with Control- lable Complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 8298–8317, Dublin, Ireland. Association for Computational Lin- guistics. Timothy Baldwi...
2010
-
[2023]
In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations , pages 318–327, Singapore
Okapi: Instruction-tuned large language mod- els in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations , pages 318–327, Singapore. Association fo...
2023
-
[2024]
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA
Automatically Generated Definitions and their utility for Modeling Word Meaning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational Linguistics. Felipe Maia Polo, Luca...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.