{"id":"fe542f2b-9d99-43fd-9d91-4d2416a4f5ea","arxiv_id":"2507.08342","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across eight languages, n-gram metrics such as ROUGE correlate less with human ratings in fusional languages than in isolating and agglutinative ones, while the neural metric COMET correlates better, especially in low-resource languages.","lead":"Researchers compared automatic summarization metrics against about 20,000 human ratings across eight languages and found that word-overlap metrics like ROUGE match human judgment worse for fusional languages such as Hebrew and Arabic, while neural metrics, especially COMET, hold up better. The study also shows that adapting tokenization or lemmatization can reverse some negative correlations and argues the field should move toward neural evaluation metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 5 pools Arabic and Hebrew into one family-level correlation, so the negative high-fusional result may be an artifact of between-language mean differences; Table 11 shows Arabic positive and Hebrew negative, so the typology claim is not identified.","rationale":"The reader's weakest assumption identifies the same problem I regard as most load-bearing: family-level correlations are computed by pooling two languages, and per-language Table 11 shows the high-fusional effect is not homogeneous. I sharpen the point by noting that pooled Pearson correlations across languages with different mean human scores can produce a negative association even if every within-language correlation is positive, so Table 5's negative high-fusional values cannot be interpreted as evidence about morphology without a language-level analysis. The same logic applies to the COMET low-resource claim in Table 6, where the pooled advantage is not consistently present in Yoruba or Turkish. The resource itself is valuable and the paper's conditional verdict is appropriate; the condition should be that the headline claims survive a language-as-random-effect analysis. No change to the reader's verdict is needed, because the conditional verdict already requires strengthened statistical evidence.","tokens_in":18608,"tokens_out":9507,"duration_ms":101258,"concrete_test":"Recompute the family-level results as a mixed-effects meta-analysis: Fisher z-transform each per-language Pearson r in Table 11, fit a model with typological family and criterion as fixed effects and language as a random intercept, and test the high-fusional contrast. Then run leave-one-language-out on the high-fusional cluster, dropping Hebrew and then dropping Arabic. If the high-fusional deficit is not significant under the random-effects model, or changes sign when Hebrew is dropped, the central typology claim fails. Apply the same procedure to Table 6 with resource level as the fixed effect and language as the random effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the aggregation in Table 5. Section 6 says family correlations were 'measured across all the languages within the respective linguistic family,' i.e., one pooled Pearson r over the combined Arabic and Hebrew samples. A pooled r across groups with different mean human scores (Table 4: Hebrew coherence mean 3.8 vs Arabic 2.6) conflates within-language signal with between-language mean shifts, so it can be negative even when both within-language correlations are positive. Table 11 shows the high-fusional row is not a reproducible family property: for coherence, ROUGE1 is 0.10* in Arabic but -0.31** in Hebrew; BLEU is 0.06* vs -0.16*; ROUGE-L is 0.06* vs -0.26**. The negative family-level correlations are therefore an artifact of pooling, or of Hebrew alone, not evidence about fusional morphology generally. The same aggregation problem affects the low-resource COMET claim in Table 6: per-language low-resource COMET correlations for coherence are 0.01 (Yoruba), 0.21 (Turkish), 0.38 (Hebrew), and 0.11 (Ukrainian), so the pooled COMET advantage is not consistent across low-resource languages. No test compares correlations or models language as a random effect, so the claimed family and resource effects are not statistically identified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies automatic evaluation metrics for abstractive summarization in eight non-English languages spanning four typological families (isolating, agglutinative, low-fusional, and high-fusional) at both high- and low-resource levels. The authors collect roughly 20,000 human annotations of coherence and completeness for GPT-3.5 and Gemini generated summaries, and compute Pearson correlations with n-gram metrics (ROUGE, BLEU, CHRF), adapted variants (mBERT tokenization, language-specific tokenizers, lemmatization), and neural metrics (BERTScore, MoverScore, COMET, Gemini-as-a-judge). The paper claims that n-gram metrics correlate less with human judgments in fusional languages, that tokenization and lemmatization can mitigate or reverse these negative correlations, and that COMET consistently outperforms other metrics in low-resource settings.","tokens_in":18842,"tokens_out":6455,"duration_ms":68526,"significance":"If the headline claims are valid, the paper would provide actionable guidance for choosing evaluation metrics in multilingual summarization and would strengthen the case for neural, task-specific metrics over n-gram metrics. The paper's concrete strengths are the substantial annotation resource (~20,000 human judgments, eight languages, four families), the released code and data, the inclusion of both n-gram and neural metric families, and the explicit reporting of significance values. These assets are real regardless of the analysis issues detailed below. However, the central typological and resource-level claims are not statistically identified by the current pooled analyses, so the paper's conclusions, as stated, require substantial revision.","major_comments":[{"comment":"The family-level correlations are computed by pooling all languages within a family, but Table 11 shows opposite correlation signs within the high-fusional family: for coherence, Arabic ROUGE1 is 0.10 while Hebrew ROUGE1 is -0.31; Arabic BLEU is 0.06 while Hebrew BLEU is -0.16; Arabic ROUGE-L is 0.06 while Hebrew ROUGE-L is -0.26. Table 4 reports large between-language differences in mean human scores (Hebrew coherence mean 3.8 vs. Arabic 2.6), so the pooled negative correlations in Table 5 can be driven entirely by between-language mean shifts rather than by any within-language association. The claim that n-gram metrics show lower correlation with human judgments in fusional languages is therefore not identified. Please report per-language correlations as primary results, or model language as a random effect and test whether family membership explains correlation magnitude.","section":"Section 6, Table 5"},{"comment":"The statement that COMET 'consistently outperform[s]' other neural metrics in low-resource languages is not supported by the per-language correlations in Table 11. For coherence, the low-resource COMET correlations are Yoruba 0.01, Turkish 0.21, Hebrew 0.38, and Ukrainian 0.11, while the pooled low-resource value in Table 6 is 0.18. This heterogeneity is substantial, and the paper provides no test of whether COMET's correlation is significantly higher than other metrics within the low-resource group. Please present per-language results for each resource level and add a formal interaction or bootstrap test to substantiate the resource-level claim.","section":"Section 6, Table 6"},{"comment":"The claimed reversal of negative correlations by applying BLEU to lemmatized text is confounded with the corruption protocol. Section A.5 states that coherence corruption is implemented by replacing verbs with their lemma forms, producing ungrammatical summaries. When BLEU is computed on lemmatized text, those corrupted summaries are normalized by the same transformation that was used to corrupt them, which can preferentially increase their metric scores and artificially inflate the correlation with human coherence judgments. The paper therefore does not establish a general benefit of lemmatization for fusional languages. Please re-evaluate the mitigation on uncorrupted data or with a corruption method that does not overlap with the tested transformation.","section":"Section 6, Table 5, line 16"},{"comment":"The paper compares many metric-by-language-by-criterion correlations but reports no confidence intervals, no tests of differences between correlations (e.g., Fisher's z), and no multiple-comparison correction. Verbal claims such as 'consistently showing a better correlation' and 'the advantage of agglutinative languages' rest on point estimates that may not be statistically distinguishable. Please add formal comparisons, such as bootstrap confidence intervals or a mixed-effects model with metric and language as factors, to support the qualitative conclusions.","section":"Section 5 and Section 6"}],"minor_comments":[{"comment":"In the country-of-residence list, 'mibya' appears to be a typo for 'Libya'; please correct it and also clarify entries such as 'West Africa' that are regions rather than countries.","section":"Table 3"},{"comment":"The term 'Abstactness' should be 'Abstractness' in the description of the generated-summaries analysis.","section":"Section 5.1"},{"comment":"The Limitations section says the criteria are 'coherence and consistency,' but the body of the paper consistently uses 'coherence and completeness'; please align the terminology.","section":"Limitations"},{"comment":"Arabic is classified as high-resource despite being below the 0.1% token threshold stated in Section 3; the footnote justifies this by worker availability, but this exception should be discussed as a sensitivity analysis because it affects the high- versus low-resource comparisons in Table 6.","section":"Section 3, Table 8"},{"comment":"The number of annotators varies considerably across languages, with Japanese having only one worker; the paper should explicitly discuss how this imbalance affects the reliability of the aggregated human scores used as the gold standard.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The annotation resource and the breadth of the evaluation setup are genuine contributions, but the central empirical claims are not identified by the current pooling strategy. The per-language tables already contain the information needed to re-analyze the data, so the issues are fixable within the scope of a revision. I would also encourage the authors to revisit the corruption-lemmatization confound before resubmitting, as the headline mitigation result may disappear under a cleaner design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper is a useful, honest resource with a central claim that doesn't hold up under inspection. The dataset—about 20,000 human annotations across eight languages, with quality checks, inter-annotator agreement, and significance testing per correlation—is a real contribution. The coverage of typological families and resource levels is better than prior meta-evaluations, and the appendix tables are transparent enough to show the per-language numbers that undermine the family-level story. That transparency deserves credit.\n\nThe main problem is the pooling. Table 5 reports family-level correlations by pooling the two languages in each typological group. For high-fusional, that pools Arabic and Hebrew. Table 11 shows the two go in opposite directions: ROUGE1 coherence is +0.10 for Arabic and -0.31 for Hebrew; BLEU is +0.06 vs -0.16. Pooling two groups with different mean human scores (Hebrew coherence mean 3.8, Arabic 2.6) can produce a negative r even when both within-language correlations are positive. So the headline 'n-gram metrics underperform in fusional languages' is actually 'n-gram metrics underperform in Hebrew.' The abstract overgeneralizes.\n\nThe COMET low-resource claim has the same issue. Table 6 pools low-resource languages, but per-language coherence correlations for COMET are 0.01 (Yoruba), 0.21 (Turkish), 0.38 (Hebrew), 0.11 (Ukrainian). That is not consistent.\n\nA second soft spot: the lemmatization mitigation. The coherence corruption procedure includes replacing verbs with lemmas, and the result that BLEU on lemmatized text reverses negative correlations (Table 5, line 16) is measured on summaries whose main corruption is exactly lemmatization. That is partly circular. It would be far more convincing on naturally occurring model errors.\n\nThere is also a small inconsistency: Arabic is labeled high-resource in Table 1, but its GPT-3 token share in Appendix A.1 is 0.031%, below the paper's own 0.1% threshold. That should be corrected.\n\nWho is this for? People working on multilingual summarization evaluation will want the annotation resource and the metric comparison. The statistical lesson—always check pooled correlations against per-language results—is worth a reading group discussion. It deserves peer review: the resource is valuable and the research question is important, but the analysis needs to be reframed around per-language effects, and the lemmatization claim needs a non-circular test.","headline":"Valuable 20k-annotation resource, but the headline typology claim is an artifact of pooling Arabic and Hebrew; per-language data show only Hebrew drives the negative correlation.","tokens_in":19422,"tokens_out":3863,"would_cite":true,"duration_ms":43599,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper finds that n-gram metrics correlate poorly with human judgments of summaries in fusional languages, while neural evaluation-trained metrics like COMET correlate better, and tokenization can rescue n-gram metrics.","keywords":["multilingual summarization","automatic evaluation metrics","ROUGE","BLEU","COMET","typological families","fusional morphology","low-resource languages"],"falsifier":"Run the same annotation-correlation protocol on additional fusional languages with rich morphology (e.g., Russian, Polish, or Czech); if space-delimited ROUGE and BLEU show positive correlations with human judgments in several such languages, the claim that n-gram metrics systematically under-correlate in fusional languages would be contradicted.","tokens_in":18374,"feed_emoji":"🌐","tokens_out":10342,"duration_ms":92955,"temperature":0.7,"pith_summary":"Summarization quality for English is usually measured by n-gram overlap metrics such as ROUGE, but it is unclear whether those numbers mean the same thing in other languages. This paper tests ten evaluation metrics against about 20,000 human ratings of generated summaries in eight languages spanning four typological families (isolating, agglutinative, low-fusional, high-fusional), with both high- and low-resource members in each family. It tries to establish that n-gram metrics under-correlate with human judgment in fusional languages, that adapting tokenization or lemmatizing can substantially improve (sometimes reverse) those correlations, and that the neural metric COMET consistently outperforms other metrics in low-resource settings. If true, the paper would imply that multilingual leaderboards built on ROUGE/BLEU are unreliable for fusional languages and that evaluation practice should shift toward neural metrics trained for the evaluation task.","feed_headline":"N-gram metrics misjudge fusional-language summaries","feed_subtitle":"Across 8 languages and 20,000 ratings, COMET tracks human judgment better, especially at low resource levels.","key_machinery":"The load-bearing mechanism is the evaluation-correlation suite: for each language, human annotators rate model-generated summaries on coherence and completeness, and each candidate metric's score is then Pearson-correlated with the averaged human ratings, with p-values reported. The typological axis is the paper's central experimental design: eight languages grouped into four families, each containing a high- and a low-resource member, so that metric behavior can be attributed to morphological type rather than mere language identity. Within this design, the decisive technical objects are the metric adaptations: ROUGE with mBERT BPE tokenization, ROUGE with language-specific tokenizers, and BLEU over lemmatized text, which change what counts as a 'word' and thereby change the overlap signal; and COMET, a neural regression metric trained on human quality judgments, whose weights are reused for summarization by omitting the source side.","core_discovery":"The paper reports that the reliability of automatic summarization metrics depends on the typological family of the language being evaluated. For isolating and agglutinative languages, ROUGE-family metrics show modest positive correlations with human coherence and completeness judgments; for fusional languages these correlations weaken, and for high-fusional languages (Arabic, Hebrew) they become negative for standard space-delimited ROUGE, BLEU, and CHRF. Replacing the default tokenizer with an mBERT or language-specific tokenizer, or applying BLEU to lemmatized text, improves the correlations and can turn negative values positive in the high-fusional group. Among neural metrics, COMET — a regression model trained for machine translation evaluation and adapted here by omitting the source — shows the strongest and most consistent correlations with human judgments, and in low-resource languages it beats both n-gram and other neural metrics. The authors conclude that n-gram metrics are ill-suited to fusional languages and advocate for investment in neural metrics trained specifically for evaluation.","pith_inferences":["The per-language results in the paper's appendix suggest the family-level negative correlation for high-fusional languages is driven mainly by Hebrew; Arabic's correlations are positive or near zero, so the typological conclusion may be about Hebrew's specific morphology or data rather than high-fusional morphology as a whole.","A direct test would rerun the protocol on additional fusional languages (e.g., Russian, Polish, Czech); if negative n-gram correlations do not replicate, the family-level claim needs refinement.","Because COMET was trained only on machine-translation quality judgments yet transfers to summarization, evaluation-specific training may transfer across generation tasks, possibly to dialogue or long-form QA in low-resource languages.","The tokenization results imply that standardizing subword tokenization across languages could make n-gram metrics more comparable, but the paper's own data show it can hurt isolating languages like Chinese, so any standardization must be typology-aware."],"forward_implications":["ROUGE-, BLEU-, and CHRF-based leaderboards for abstractive summarization in fusional languages (e.g., Arabic, Hebrew) will misalign with human judgments, as these metrics show negative or near-zero correlations in the paper's high-fusional group.","Using subword tokenization (mBERT or language-specific tokenizers) before computing ROUGE, or applying BLEU to lemmatized text, can raise correlations substantially and even reverse negative trends in high-fusional languages.","COMET should be preferred over other neural metrics and n-gram metrics for evaluating summaries in low-resource languages, since it correlates best with human judgments there.","Evaluation studies should report per-language correlations with p-values and use at least about 400 samples per language, the sample size the paper's power analysis finds necessary for detecting effects at p ≤ 0.05.","Multilingual model development should not treat English-derived n-gram metrics as a universal yardstick; the choice of metric needs to be language-aware."],"supporting_citations":[{"why":"Supplies ROUGE, the n-gram metric whose correlations with human judgments are the paper's main object of study.","marker":"(Lin, 2004)"},{"why":"Defines BLEU, another n-gram metric that the paper evaluates and finds poorly correlated in fusional languages.","marker":"(Papineni et al., 2002)"},{"why":"Introduces COMET, the neural evaluation metric that the paper adapts to summarization and reports as the strongest correlate with human judgments.","marker":"(Rei et al., 2020)"},{"why":"Provides the XL-Sum dataset whose articles and human summaries are used to generate the model outputs annotated in this study.","marker":"(Hasan et al., 2021)"},{"why":"Supplies the HeSum dataset for Hebrew and earlier evidence of negative ROUGE correlations in Hebrew that the current study extends.","marker":"(Paz-Argaman et al., 2024)"},{"why":"Defines BERTScore, one of the neural metrics the paper compares against COMET and n-gram metrics.","marker":"(Zhang et al., 2019)"},{"why":"Prior multilingual summarization evaluation that excluded high-fusional languages; the paper's design builds on and extends this work.","marker":"(Koto et al., 2021)"},{"why":"Prior meta-evaluation with small per-language samples; the paper's ~400-sample power analysis responds to this limitation.","marker":"(Han et al., 2024)"}],"fun_headline_variants":["Fusional languages break ROUGE and BLEU","COMET beats n-gram metrics for multilingual summaries","N-gram metrics fail for fusional languages","For summary metrics, language type matters more than size","Tokenization fixes ROUGE's blind spot for fusional tongues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that two languages per typological family are enough to represent how that entire family behaves in evaluation; if Hebrew and Arabic are not typical of high-fusional languages, the family-level negative correlations may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Fusional languages break ROUGE and BLEU","COMET beats n-gram metrics for multilingual summaries","N-gram metrics fail for fusional languages","For summary metrics, language type matters more than size","Tokenization fixes ROUGE's blind spot for fusional tongues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1431,"prompt_tokens":993,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":609,"tokens_out":438,"duration_ms":4320,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:21:44.102908+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same annotation-correlation protocol on additional fusional languages with rich morphology (e.g., Russian, Polish, or Czech); if space-delimited ROUGE and BLEU show positive correlations with human judgments in several such languages, the claim that n-gram metrics systematically under-correlate in fusional languages would be contradicted.","supporting_citations":[{"cited_title":"XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages","cited_arxiv_id":"2106.13822","evidence_quote":"Provides the XL-Sum dataset whose articles and human summaries are used to generate the model outputs annotated in this study."},{"cited_title":"HeSum: a Novel Dataset for Abstractive Text Summarization in Hebrew","cited_arxiv_id":"2406.03897","evidence_quote":"Supplies the HeSum dataset for Hebrew and earlier evidence of negative ROUGE correlations in Hebrew that the current study extends."},{"cited_title":"Evaluating the Efficacy of Summarization Evaluation across Languages","cited_arxiv_id":"2106.01478","evidence_quote":"Prior multilingual summarization evaluation that excluded high-fusional languages; the paper's design builds on and extends this work."}],"review_version":1}