{"id":"1370dff0-bbd5-4d56-b405-0c35533d06e2","arxiv_id":"2509.06401","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs produce more commonsensical sentences in English than in Spanish, Dutch, or Valencian, across automatic, LLM-judge, and human evaluations on the new MULTICOM benchmark.","lead":"Researchers created a four-language commonsense test and found LLMs generate much more sensible sentences in English than in Spanish, Dutch, or Valencian. The result matters because it quantifies how language bias persists even in modern open-source LLMs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Machine-translated references may not be equivalent gold standards; without validation, the English advantage in MULTICOM could be a translation-quality artifact.","rationale":"I agree with the reader's verdict and weakest assumption. The most load-bearing concern is the validity of machine-translated references in MULTICOM. The abstract's claim of 'significantly lower performance in less-resourced languages' is supported primarily by automatic metrics computed against these references. The limitations section acknowledges encoder and prompt bias but does not address the possibility that the references themselves are not equivalent gold standards across languages. This is a structural issue: the benchmark measures similarity to a potentially flawed reference, so it may not measure the same construct across languages. The proposed check directly tests the impact of reference quality on the central metric. If the gap persists with human references, the claim is robust; if it shrinks, the paper's main conclusion is an artifact. This does not change the reader's CONDITIONAL verdict, as the concern is already the stated condition, but it pinpoints the exact evidence needed.","tokens_in":15076,"tokens_out":4479,"duration_ms":51493,"concrete_test":"Select a random sample of 100 instances per language from MULTICOM's test set. Replace the existing non-English reference sentences with professionally human-translated references (or human-written originals) for Spanish, Dutch, and Valencian, keeping the same input triples and contexts. Re-run the automatic metric evaluation (BERTScore, USE, dependency-based Levenshtein/cosine) for LLaMA-3.2-3B-Instruct outputs, comparing against the human references instead of the MT references. If the mean reference-vs-counterfactual gap for any non-English language increases by >0.05 absolute (or the English-minus-non-English gap shrinks by >0.05), the MT references were suppressing scores and the central claim is confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on MULTICOM reference sentences being equally natural and valid in all four languages. Section 3.1 describes a translation pipeline: Spanish COCOTEROS was translated to English via OPUS-MT-ES-EN, English to Dutch via OPUS-MT-EN-NL, and Spanish to Valencian via the Salt tool. The only post-hoc verification was a Grok-based check that each reference contains the intended keywords; no human or automatic assessment of grammaticality, idiomaticity, or commonsense plausibility of the translated references is reported. The main quantitative evidence—the reference-vs-counterfactual gap under BERTScore, USE, and dependency-based metrics—compares each generated sentence with these references. If the non-English references are stiff, unnatural, or contain translationese, then even a perfectly fluent commonsensical output in Spanish/Dutch/Valencian will be scored closer to the counterfactual and farther from the reference, mechanically shrinking the gap and inflating the apparent English superiority. The human evaluation (Section 4.3) is limited to 20 instances from one model and cannot independently validate the broad cross-lingual claim. Thus the equivalence of reference quality is the load-bearing assumption; the paper provides no evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Martínez-Murillo et al. present MULTICOM, a four-language extension (English, Spanish, Dutch, Valencian) of the Spanish COCOTEROS corpus, and use it to evaluate open-weight LLMs from five families (LLaMA, Qwen, Gemma, EuroLLM, Salamandra) on constrained commonsense generation from a keyword triple, with and without a supporting context. The evaluation triangulates automatic metrics (BERTScore, Universal Sentence Encoder cosine similarity, dependency/Levenshtein and dependency/vector cosine similarity), two LLM-as-a-judge models (Prometheus-2, JudgeLM), and a 20-instance human annotation study of one model. The paper reports that the reference-vs-counterfactual gap is consistently largest in English, that performance is lower in Spanish, Dutch, and Valencian, that context has mixed but generally positive effects for lower-resourced languages, and that human and LLM judgments broadly agree. The dataset is released publicly.","tokens_in":1922,"tokens_out":1827,"duration_ms":72168,"significance":"If the empirical conclusion holds, the paper provides a useful multilingual benchmark and a broad comparison of open-source LLMs; the public release of MULTICOM and the multi-method evaluation are strengths. The claim is not circular: the reference/counterfactual gap is an external, fixed measurement. However, the central comparison rests on the assumption that the translated reference sentences are equally natural, idiomatic, and commonsense-valid gold standards in all four languages, and that the automatic metrics are language-invariant. The manuscript does not currently validate that assumption, and several design choices—English-only prompts and rubrics, English-centric encoders, no statistical testing—are acknowledged or self-evident confounds. Thus the paper is a potentially valuable contribution, but it needs substantial supporting evidence before the cross-lingual claim is established.","major_comments":[{"comment":"The load-bearing assumption is that the non-English references are equally valid gold standards. The translation pipeline uses OPUS-MT and Salt, with only a Grok-based keyword-presence check. Appendix A's alignment prompt explicitly allows the model to replace a missing keyword with 'another word from the sentence', so some non-English instances may no longer instantiate the original concept triple; this changes the task across languages. No evaluation of grammaticality, idiomaticity, or commonsense plausibility of the translated references is reported. Because the gap score is computed relative to these references, translationese or unnatural references would mechanically reduce the gap in non-English languages, independent of model quality. The paper needs either manual validation of a sample of translations or a robustness analysis restricted to high-confidence translations.","section":"§3.1 and Appendix A"},{"comment":"The automatic metrics are not shown to be language-fair. The USE model is not specified; the SpaCy models used for dependency/vector similarity have different coverage and training distributions; the Levenshtein match threshold and cosine-similarity threshold are free parameters but are not reported; and BERTScore with bert-base-multilingual-cased may not be equally reliable for Valencian. The Limitations section acknowledges the general encoder/parser issue, but no sensitivity analysis or per-language calibration is provided. Given that the automatic metrics are the primary quantitative evidence, the paper should report model versions, thresholds, and results of a language-appropriateness check (e.g., comparing metric decisions against human judgments on the same items).","section":"§3.3 and §4.1"},{"comment":"The abstract states 'significantly lower performance' in less-resourced languages, but no statistical tests, confidence intervals, or effect-size measures are reported. Table 1 gives only point estimates for one model, and the appendix figures are descriptive. Given that outputs for the same model across languages are correlated and that the metrics may have different ranges or floor/ceiling effects, the 'significant' language differences are unsupported. The authors should report per-language score distributions, paired comparisons across models, or at least bootstrap intervals for the reported gaps.","section":"Abstract and §4.1"},{"comment":"The human evaluation is limited to 20 instances, one model, no-context condition, and three annotators per language; Valencian was judged by Catalan speakers. The reported majority agreement (0.75–0.95) is a weak reliability measure and does not address the reference-equivalence problem: human raters scored generated sentences without being asked whether the reference translations are equally natural. This small sample cannot validate the broad cross-lingual claim, nor can it disambiguate model quality from reference quality. A stratified human evaluation that also rates reference naturalness across languages would be needed.","section":"§4.3"},{"comment":"Prometheus and JudgeLM are given English prompts and rubrics even for non-English outputs, as shown in Appendices C and D. The manuscript acknowledges this as a limitation, but because the cross-lingual comparison is the central claim, it is load-bearing. A judge with weaker target-language competence may under-score non-English outputs. The paper should include a per-language calibration or prompt-language invariance check, e.g., judge scores against human scores on a stratified sample per language, to rule out an LLM-judge bias as the source of the English advantage.","section":"§3.3 and §4.2"}],"minor_comments":[{"comment":"Typographical issues: 'several several' and 'commonense' appear in the Related Work section; 'Gemmma' appears in §3.2. These should be corrected.","section":"§2"},{"comment":"The column header order 'ES EN CA-VA NL' is inconsistent with the enumeration in the text ('English (EN), Spanish (ES), Valencian (CA-VA), Dutch (NL)'). Align the header order with the text or vice versa for readability.","section":"Table 1"},{"comment":"The alignment prompt references 'keywords_es' even when the source language for Dutch was English. Clarify how the post-processing prompt was adapted for each translation direction, and how the final keywords were validated after replacement.","section":"Appendix A"},{"comment":"The Grok citation is incomplete; the Miller et al. reference contains a stray 'Damerau?Levenshtein'; the Prolific reference lacks a date/access detail. These should be cleaned up.","section":"References"},{"comment":"The main text presents only LLaMA results; comprehensive results are in appendices as figures, which are difficult to read in PDF form. Consider providing numeric tables or higher-resolution figures so that the reported cross-model patterns can be checked.","section":"§4.1"},{"comment":"The caption says 'the tested models' but only one model (LLaMA-3.2-3B-Instruct) is evaluated. Correct the caption to reflect the single-model evaluation.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about its limitations, but those limitations are not peripheral: the fairness of the translated references is the core of the comparison. The dataset is a useful contribution, and the multi-method design is a strength. I would be willing to accept after the authors validate the translations, add statistical support for the 'significant' differences, and address the metric/language-invariance concerns. There is no circularity problem: the reference/counterfactual gap is a reasonable external measurement, and the reuse of the authors' earlier COCOTEROS corpus is transparent. The main risk is that the English advantage is inflated by translationese and English-centric evaluators, not by an actual commonsense deficit in non-English outputs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: MULTICOM is a real new resource—COCOTEROS extended with counterfactual and unrelated contrastive sentences across English, Spanish, Dutch, and Valencian, plus a systematic sweep of five open-weight model families. The dataset and the contrastive setup are worth having. The empirical claim—English wins, low-resource languages lose—is plausible and consistent with prior work, but the paper states it more confidently than the evidence supports. The soft spot is not the central idea; it is that the non-English references are machine translations with no validation of naturalness or idiomaticity. That matters because the main automatic metrics score generated sentences by similarity to those references. If the Spanish/Dutch/Valencian references are stiff or contain translationese, even strong model outputs will look closer to the counterfactual and farther from the reference, mechanically widening the English gap. The paper's own Limitations section partially acknowledges encoder and judge bias, but it does not confront the reference-equivalence problem. The stress-test note has it right: this is load-bearing, and the Grok keyword check does not address it.\n\nWhat is genuinely new and good: the contrastive sentence types, the four-language alignment, the context-support condition, and the breadth of models tested. The three-way evaluation (automatic, LLM judge, human) is a sound triangulation strategy, and the human sample is small but honestly reported. I also credit the authors for flagging the Valencian-related caveats and for releasing the data.\n\nWhere it frays: no significance tests anywhere—the 'significantly lower' in the abstract is not backed by inference. The automatic metrics use multilingual encoders and SpaCy/Stanza models, but these are still English-centric in training. The LLM judges use English rubrics even for non-English outputs; the paper acknowledges this but the abstract does not. The human evaluation covers 20 instances from one model, which is fine for a spot-check but cannot carry the cross-lingual generalization.\n\nMy take: the resource deserves peer review and the paper should be revised, not shelved. The English-superiority result is probably true in the broad sense, but as presented it could be partly a translation-quality artifact. The fix is doable: human ratings of reference naturalness per language, or per-language human judgments on a larger sample, and significance testing. I would cite the benchmark.\n\nRecommendation: send to peer review with a request for the reference-validation experiments. It is a serious contribution with a fixable flaw.","headline":"A genuinely useful multilingual commonsense benchmark, but the headline English-superiority result rests on machine-translated references and English-centric metrics that may inflate the gap.","tokens_in":15824,"tokens_out":1621,"would_cite":true,"duration_ms":18601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that open-source LLMs generate commonsense-consistent sentences more reliably in English than in Spanish, Dutch, or Valencian, and presents MULTICOM, a four-language benchmark that makes the gap measurable.","keywords":["multilingual commonsense generation","constrained text generation","LLM evaluation","English bias","low-resource languages","multilingual benchmark","commonsense reasoning","LLM-as-a-judge"],"falsifier":"Native-speaker rewrites of the MULTICOM reference sentences and LLM judges prompted in each target language, with humans rating the same outputs: if English no longer ranks first under both changes, the claim of an inherent English advantage fails.","tokens_in":14992,"feed_emoji":"🌍","tokens_out":9996,"duration_ms":83947,"temperature":0.7,"pith_summary":"This paper asks whether large language models show the same commonsense reasoning ability when generating text in different languages. To test this, the authors build MULTICOM, a benchmark in English, Spanish, Dutch, and Valencian, in which a model must produce a plausible everyday sentence containing three given words. Evaluating five open-source model families with automatic metrics, LLM judges, and human raters, they find a consistent pattern: outputs are judged more commonsensical in English, with notably lower scores in the less-resourced languages. The paper's central claim is that this English advantage is real across evaluation methods, while adding a supporting context has mixed but often helpful effects for Dutch and Valencian.","feed_headline":"LLMs show a clear English edge in commonsense generation","feed_subtitle":"A four-language benchmark shows open models lag outside English; context helps Dutch and Valencian.","key_machinery":"The engine of the argument is the contrast between reference and counterfactual sentences in MULTICOM. For each triplet of keywords, the dataset provides a commonsense-consistent target sentence and a fluent but nonsensical counterfactual; a model's commonsense quality is measured by the gap between its similarity to the target and its similarity to the counterfactual. A wider gap means the model can tell everyday logic from nonsense. Building on that, the benchmark combines semantic metrics (BERTScore, Universal Sentence Encoder cosine similarity), syntactic-dependency comparisons (via Stanza and Levenshtein distance or vector cosine), plus two LLM-as-judge scorers and human ratings, so the","core_discovery":"The central claim is that open-source LLMs' commonsense generation is not language-uniform: the same models that reliably produce plausible sentences in English produce weaker, less commonsensical sentences in Spanish, Dutch, and Valencian. The evidence comes from MULTICOM, which extends the Spanish COCOTEROS corpus with counterfactual and unrelated references and translates it into four languages. Models are scored by how much closer their outputs sit to a commonsense reference than to a deliberately nonsensical counterfactual, and by LLM judges and human annotators. All three evaluation strands agree on the ranking, with English first and Valencian last; the authors interpret this as a con","pith_inferences":["Because the LLM judges were prompted in English with English rubrics, part of the measured gap could be judge bias rather than purely generation quality; re-running the judges in each target language would isolate this.","The counterfactual-contrast design could be reused as a lightweight probe: since only the contrast sentences need translating, the benchmark could be extended to many more low-resource languages cheaply.","If the English edge comes mainly from pretraining-data volume, fine-tuning on a modest amount of target-language commonsense text might shrink the gap faster than scaling model size, which the paper found to have only modest effects."],"forward_implications":["If the claim holds, English-only commonsense evaluations overstate the general commonsense competence of current open LLMs.","Adding a supporting context sentence is a cheap intervention that tends to improve generation in Dutch and Valencian, even though it helps little or hurts in English and Spanish.","The target-versus-counterfactual gap is a usable, reference-contrast measure of commonsense that can be ported to other languages.","Model families whose pretraining emphasized language diversity show a flatter language profile, suggesting data curation can narrow, but not remove, the English gap."],"supporting_citations":[{"why":"Supplies the Spanish COCOTEROS corpus that MULTICOM extends with counterfactual and unrelated references.","marker":"Maestre et al., 2024"},{"why":"Defines the CommonGen constrained generation task—generate a plausible sentence from a keyword set—that MULTICOM adapts.","marker":"Lin et al., 2020"},{"why":"Provides the OPUS-MT translation models used to produce MULTICOM's English and Dutch versions.","marker":"Tiedemann and Thottingal, 2020"},{"why":"Introduces the LLaMA 3.2 family used to generate counterfactual sentences and as the main model for human evaluation.","marker":"Grattafiori et al., 2024"},{"why":"Supplies BERTScore, one of the automatic semantic metrics used to compare outputs against references.","marker":"Zhang et al., 2020"},{"why":"Provides the Stanza dependency-parsing pipeline used by the two syntactic-structure metrics.","marker":"Qi et al., 2020"},{"why":"Introduces Prometheus-2, one of the two LLM-as-judge evaluators used to score commonsense quality.","marker":"Kim et al., 2024"},{"why":"Introduces JudgeLM, the second LLM-as-judge evaluator used in the study.","marker":"Zhu et al., 2025"}],"fun_headline_variants":["LLMs show uneven commonsense across four languages","English dominates LLM commonsense, Dutch and Valencian lag","Context helps LLMs' commonsense in low-resource languages","Multilingual test: LLMs' commonsense is English-centric","LLMs' commonsense skills drop outside English"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The translated non-English reference sentences are natural, idiomatic gold standards; if machine translation produced awkward or non-typical sentences, models score worse against them regardless of their true commonsense quality, inflating the English advantage.","fun_headline_variants_meta":{"raw":{"variants":["LLMs show uneven commonsense across four languages","English dominates LLM commonsense, Dutch and Valencian lag","Context helps LLMs' commonsense in low-resource languages","Multilingual test: LLMs' commonsense is English-centric","LLMs' commonsense skills drop outside English"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1076,"prompt_tokens":692,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":317}},"tokens_in":436,"tokens_out":384,"duration_ms":4721,"temperature":1.0,"reasoning_tokens":317,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:37:30.362111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Native-speaker rewrites of the MULTICOM reference sentences and LLM judges prompted in each target language, with humans rating the same outputs: if English no longer ranks first under both changes, the claim of an inherent English advantage fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Spanish COCOTEROS corpus that MULTICOM extends with counterfactual and unrelated references."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OPUS-MT translation models used to produce MULTICOM's English and Dutch versions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces JudgeLM, the second LLM-as-judge evaluator used in the study."}],"review_version":1}