{"id":"052ca6fb-6717-4c89-b8b8-2bac68fd7e33","arxiv_id":"2508.21389","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On SummEval, LLM-based summary evaluators are expensive and unstable, and several published correlations did not reproduce when using open-weight models.","lead":"This paper measures six summary-evaluation metrics on the SummEval benchmark and reports that several correlations with human judgment differ from values published in earlier papers, with LLM-based metrics costing hours to run and varying between model runs. It also introduces an open-source framework, AllSummedUp, for running such comparisons in a shared pipeline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'significant discrepancies' claim is confounded: G-Eval/SEval-Ex were run with different LLMs than the originals, and no repeated-run variance is reported, so observed instability may be model substitution rather than inherent metric instability.","rationale":"The reader's weakest assumption correctly focuses on the open-model substitution as a confound for the reproducibility claim. My analysis agrees and sharpens it: the paper also fails to report the variance of multiple runs that it claims to have performed. This is a load-bearing gap because the entire 'instability' and 'significant discrepancies' narrative depends on separating model choice from stochasticity. The reader's verdict of CONDITIONAL is appropriate; the claims need revision or additional evidence before being accepted at face value. I find no additional independent issue that would change the verdict.","tokens_in":10331,"tokens_out":1990,"duration_ms":22665,"concrete_test":"Re-run G-Eval and SEval-Ex on the same SummEval subset multiple times (e.g., 10 seeds) with the same open model (Gemma-3-27b) and with the original GPT-4 (or at least a comparable closed model) using the exact prompts from the original papers. Compare the within-model variance of Spearman correlations to the between-model differences. If the between-model gap (e.g., G-Eval fluency vs. literature) far exceeds the within-model standard deviation, then the discrepancy is due to model substitution, not instability, undermining the central 'instability' claim. If within-model variance is comparable to the gap, the instability claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (§4.3, abstract) is that high-correlation LLM metrics are also computationally costly and unstable, with 'significant discrepancies' from literature (abstract). This rests on comparing correlations obtained with Gemma-3-27b / Qwen-2.5-72b to literature values that used GPT-4 (G-Eval) or other closed models (SEval-Ex). The authors admit in §5: 'nous avons privilégié l'utilisation d'alternatives open-source, quitte à perdre en fidélité par rapport aux conditions d'évaluation originales.' If the reported gaps (e.g., G-Eval fluency: −0.45 for Gemma, −0.51 for Qwen) come primarily from using different, weaker LLMs rather than from run-to-run stochasticity, then the 'instability' and 'reproducibility' conclusions are overstated. Critically, the paper states (§4.2.1) that 'Toutes les métriques ont été testées sur plusieurs exécutions distinctes' but only reports execution times (Table 3), not the variance of Spearman correlations across those runs. Without confidence intervals or standard deviations, a single-point comparison with literature cannot distinguish between model substitution effects and genuine instability, yet the 'structural trade-off' hinges on this distinction. The load-bearing assumption is that replacing the model is an acceptable way to test reproducibility; the paper itself flags this as a fidelity loss, so the assumption is insecure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AllSummedUp, a modular open-source framework for comparing automatic summarization evaluation metrics, applied to the SummEval dataset. Six metrics (ROUGE, BERTScore, BARTScore, QuestEval, UniEval, G-Eval, and SEval-Ex) are run in a unified environment; Spearman correlations with human judgments on four dimensions are reported and compared with reference values from the literature, alongside total execution times. The central claims are that there are significant discrepancies between reproduced and reported correlations, that the best-aligned metrics are the most computationally expensive, and that LLM-based metrics such as G-Eval are unstable across conditions, motivating calls for standardized evaluation protocols.","tokens_in":10724,"tokens_out":3996,"duration_ms":42463,"significance":"The topic is timely and the framework's design—standardized metric interface, evaluator orchestration, and report generation—is a useful contribution to reproducible evaluation research. If the empirical claims were fully supported, the paper would provide valuable evidence that high-correlation LLM-based metrics carry hidden costs and instability. The authors also make a useful methodological point by highlighting the heterogeneity of correlation measures in the literature and the need for reporting experimental configurations. However, the central empirical claims are currently weakened by a confounded comparison (different LLMs than the original publications), missing within-run variance information, and an unreleased code repository, so the significance of the conclusions is not yet established.","major_comments":[{"comment":"The abstract's 'significant discrepancies' claim is confounded by LLM substitution. G-Eval was originally run with GPT-4, and SEval-Ex's original conditions are not reproduced; here Gemma-3-27b and Qwen-2.5-72b are used with default temperature=1. The observed gap in G-Eval fluency (−0.45 for Gemma, −0.51 for Qwen) is exactly the kind of difference expected when changing the underlying model, not necessarily evidence of run-to-run instability. The authors themselves concede in §5: 'nous avons privilégié l'utilisation d'alternatives open-source, quitte à perdre en fidélité par rapport aux conditions d'évaluation originales.' To support the reproducibility claim, the paper should either add control experiments with the original model where feasible or explicitly reframe the findings as measuring cross-model sensitivity, with the models named in the abstract and conclusion.","section":"§4.2.1, Table 1; §5"},{"comment":"The paper states 'Toutes les métriques ont été testées sur plusieurs exécutions distinctes' but only reports execution times in Table 3. No standard deviations, confidence intervals, or per-run Spearman correlations are given. Consequently, the central assertion that high-performing metrics like G-Eval are 'less stable across runs' (abstract and §4.3) is not empirically supported by the reported data. Without run-level variance, the observed discrepancies could be fully explained by the different choice of LLM or by other implementation differences. The authors should report the distribution of correlations across runs, at least for G-Eval and SEval-Ex, and use it to distinguish model-substitution effects from stochastic instability.","section":"§4.2.2, Table 3"},{"comment":"The framework is described as open-source in the abstract and §3.1, but footnote 1 states 'Le github sera partagé après acceptation de l'article'. This means the central reproducibility artifact is currently unavailable to readers or reviewers. The paper's own reproducibility claims cannot be verified, and the proposed framework cannot be used or extended. The code should be released under a public repository at submission time, or the manuscript should clearly state its current availability status and avoid presenting the framework as an existing open-source contribution.","section":"§3.1, footnote 1"},{"comment":"The provenance of the reference correlations is underspecified. The text says that when Spearman correlations were not provided in the original papers, 'nous avons pris comme références des articles ayant reproduit les expériences', but no citations are provided for these secondary sources. Moreover, BERTScore appears in Table 1 but is not listed among the implemented metrics in §3.2. Without a precise mapping from each metric to its reference value and implementation version, the 'discordance' calculation is not reproducible and the significance of the discrepancies cannot be assessed. Please provide a supplementary table listing reference values, the paper they come from, and the exact implementation/version used.","section":"Table 1; §4.2.1"}],"minor_comments":[{"comment":"The labels 'Rouge - nous' and 'Rouge - réel' are unclear. Clarify what 'réel' refers to (the BARTScore paper's reported values?) and describe the reference-handling difference that explains the mismatch.","section":"Table 2"},{"comment":"The sentence 'Seul GEval et SEval-Ex sont impactées' is ambiguous; QuestEval also has execution times in the table and appears LLM-based. Clarify whether the statement refers to the impact of model substitution on execution time.","section":"§4.2.2, Table 3"},{"comment":"There are several formatting errors in the reference list, e.g., 'Yuanet al.' missing a space, 'Kry´sci´nski' with improperly encoded diacritics, and inconsistent use of capitalization.","section":"References"},{"comment":"The methodology section does not list BERTScore among implemented metrics, although it is evaluated and reported in Table 1. Add it to the list or remove it from the experiments.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The inclusion of SEval-Ex, a metric by the same authors, in the comparison is not itself a reason for concern, but it should be clearly flagged as author-proposed and not presented as an independent external baseline. The main concern is the gap between the strength of the abstract claims and the evidence actually reported: the comparison is confounded by model substitution, no run-to-run variance is shown, and the code is not yet public. These are fixable with additional experiments and careful reframing, so I recommend major revision rather than rejection. The paper would be substantially stronger if the authors explicitly positioned their results as measuring sensitivity to open-weight LLM substitutes and provided the missing variance data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth a look for its wall-clock cost numbers and its argument for standardized evaluation protocols, but the headline claim about 'significant discrepancies' is weaker than it looks. The authors swapped the LLMs used in the original papers (e.g., GPT-4 for G-Eval) for open-weight models running locally, and they report no run-to-run variance, so they cannot separate model substitution from genuine instability.\n\nWhat's actually new: a head-to-head run of seven metrics on SummEval with Gemma-3-27b and Qwen-2.5-72b, including execution times. Those specific numbers are a new data point. The modular framework, if released, could make future metric comparisons easier and more consistent.\n\nThe paper does several things well. It is transparent about its choices: it admits in Section 5 that it traded fidelity for open-source access, it flags that SEval-Ex is the authors' own metric, and its calls for publishing prompts, weights, temperatures, and seeds are sensible. The cost table alone is a useful contribution.\n\nThe soft spots are real but not fatal. The central comparison in Table 1 is confounded: G-Eval's literature reference uses GPT-4, while this study runs Gemma/Qwen. The large negative gaps on fluency are almost certainly model behavior, not run-to-run randomness. The abstract still frames these as reproducibility failures. Section 4.2.1 says all metrics were tested on several runs, but only execution times are reported—no standard deviations or confidence intervals for the Spearman correlations. Without those, the 'instability' conclusion is unsubstantiated. There is also a citation mistake: UniEval is attributed to \"Zhang, 2024\" in the text, while the reference list correctly has Zhong et al. 2022. And the framework is not yet available on GitHub, so the engineering contribution cannot be verified.\n\nProportionate assessment: these issues weaken the paper but do not sink it. If the authors either measure variance across repeated runs or re-frame the differences as 'results under open-weight substitution' rather than non-reproducibility, the core message about cost/quality tradeoffs stands. The cost-vs-correlation pattern is plausible and the specific numbers are new.\n\nThis paper is for people working on summarization evaluation who need practical guidance on metric selection and a starting point for open-source evaluation. It deserves peer review, but the revision should require explicit variance reporting and a toned-down framing of the discrepancies.\n\nRecommendation: send it out. It is an honest, potentially useful empirical contribution that needs revision, not rejection.","headline":"Useful cost data and a reasonable plea for standardization, but the 'significant discrepancies' finding is confounded by model substitution and missing variance.","tokens_in":11149,"tokens_out":3570,"would_cite":true,"duration_ms":38404,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Re-running the top summary-evaluation metrics under one protocol shows that the best-aligned are also the costliest and least reproducible.","keywords":["evaluation metrics","automatic summarization","reproducibility","large language models","SummEval","ROUGE","G-Eval","SEval-Ex"],"falsifier":"A decisive check would be to run G-Eval and SEval-Ex on the same SummEval summaries under (a) the original closed-model configuration, (b) the open-weight configuration used here, and (c) repeated open-weight runs with different temperatures and seeds, then compare the spread of Spearman correlations with the paper's reported gaps. If the original configuration matches published reference values and only the open-weight runs drift, the discrepancy is an artifact of model substitution; if within-configuration spread is as large as the between-configuration spread, the instability is intrinsic.","tokens_in":10285,"feed_emoji":"📊","tokens_out":11645,"duration_ms":106038,"temperature":0.7,"pith_summary":"The paper tries to establish that published performance scores for automatic summary-evaluation metrics do not reliably reproduce when the metrics are re-run under one controlled protocol. It compares metrics spanning ROUGE, BARTScore, BERTScore, QuestEval, and UniEval, up to the LLM-based G-Eval and SEval-Ex, on the SummEval benchmark, measuring Spearman correlations against human judgments and contrasting them with values reported in the original papers. Some gaps are large, including sign reversals among LLM-based metrics, and the best-correlated metrics take hours to run while cheap metrics take seconds. The conclusion is a three-way structural tradeoff: human alignment, computational frugality, and reproducibility do not come together. If the paper is right, a single published run of an LLM-based metric is not a reliable basis for choosing an evaluation metric.","feed_headline":"LLM summary metrics fail to reproduce in re-run tests","feed_subtitle":"Re-tests on SummEval show published correlations shift, sometimes flip sign; best-aligned metrics cost hours to run.","key_machinery":"The argument is carried by a comparative reimplementation protocol: every metric runs through the same TextMetric interface, is executed on the same SummEval instances in the same local environment (same GPU, same model versions from HuggingFace and Ollama), and is scored by Spearman correlation against human judgments with total runtime recorded. The decisive operation is a reported-versus-measured correlation table, because converting all metrics to the same output format and correlation convention removes the methodological heterogeneity that normally prevents direct comparison.","core_discovery":"The central empirical finding is that under one common execution environment, the correlation scores that made each metric attractive do not all survive contact with a different but reasonable setting. For example, G-Eval's fluency correlation falls from a positive reference value to about -0.45 with Gemma-3-27b and -0.51 with Qwen-2.5-72b, while SEval-Ex's relevance correlation rises by +0.12 in one open-weight configuration; BERTScore and BARTScore also show weak and inconsistent correlations across dimensions. The paper interprets these gaps as evidence of a structural tradeoff: metrics that align best with human annotators, such as G-Eval, SEval-Ex, and QuestEval, pay for that alignment","pith_inferences":["A natural extension the paper leaves implicit is that any single-run correlation for an LLM evaluator should be treated as one draw: averaging over repeated runs and reporting the spread would likely change the practical ranking of metrics, especially for cheap metrics whose spread is negligible.","One testable follow-up is to feed prompt-format perturbations (added spaces, line breaks, JSON vs plain text) through the same framework and measure whether G-Eval-style metrics are as sensitive to prompt changes as they are to backbone changes, linking the reproducibility results to the prompt-sensitivity literature the paper cites.","Because the study uses only SummEval, a direct way to test whether the tradeoff is structural or dataset-specific is to apply the framework's pipeline to another summarization benchmark and see whether high-correlation metrics are still the least frugal and least stable there."],"forward_implications":["Published correlation coefficients for LLM-based summary metrics should not be read as stable properties of the metric; reports should specify backbone model, prompt, decoding parameters, library versions, and seed.","Large-scale or repeated evaluations should favor frugal metrics, while hour-long LLM metrics are better reserved for small, high-stakes evaluations.","Metric comparisons need a common environment and a common correlation convention, with variability across runs reported rather than a single point estimate.","Using open-weight models as substitutes for closed proprietary models is a workable but fidelity-losing path to reproducibility, and both the substitute and the loss should be documented.","Metrics that look superior in one published run may change rank when re-run, so benchmark comparisons should include spread or interval estimates."],"supporting_citations":[{"why":"Shows that repeated runs with supposedly deterministic LLM settings can yield different outputs, grounding the paper's variability premise.","marker":"Atil et al., 2025"},{"why":"Supplies the repeatability/reproducibility distinction and the documentation-obstacle framing behind the paper's protocol.","marker":"Belz et al., 2021"},{"why":"Documents how model version, decoding choices, library dependencies, and closed-model deprecation such as GPT-3 break reproducible evaluation.","marker":"Biderman et al., 2024"},{"why":"Provides SummEval, the benchmark with human judgments on four dimensions used in every metric comparison.","marker":"Fabbri et al., 2021"},{"why":"Defines ROUGE, the classical lexical-overlap metric that anchors the cheap-but-limited side of the tradeoff.","marker":"Lin, 2004"},{"why":"Defines G-Eval and reports the high GPT-4-based correlations that the paper re-runs and fails to reproduce.","marker":"Liu et al., 2023"},{"why":"Defines BARTScore and provides first SummEval correlations used as reference values for several metrics.","marker":"Yuan et al., 2021"},{"why":"Defines QuestEval, a reference-free question-generation metric whose reported Pearson correlations motivate the paper's choice of Spearman.","marker":"Scialom et al., 2021"},{"why":"Defines SEval-Ex, the explainable statement-level metric re-run with two open-weight models.","marker":"Herserant & Guigue, 2025"},{"why":"Defines UniEval, the multi-dimensional evaluator that emerges as the stable mid-cost comparison point.","marker":"Zhong et al., 2022"}],"fun_headline_variants":["Summary metric correlations flip sign under re-test","High-alignment summary metrics prove unstable","Open-source tool reveals metric reproducibility gaps","G-Eval correlation swings to negative on re-run","Trade-off: best metrics for humans are least stable"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that replacing the original closed models (e.g., GPT-4 in G-Eval) with open-weight ones (Gemma-3-27b, Qwen-2.5-72b) is a fair reproducibility test; the paper explicitly concedes it traded fidelity for open-source access, so a large part of the observed gap could come from that substitution rather than from inherent metric instability.","fun_headline_variants_meta":{"raw":{"variants":["Summary metric correlations flip sign under re-test","High-alignment summary metrics prove unstable","Open-source tool reveals metric reproducibility gaps","G-Eval correlation swings to negative on re-run","Trade-off: best metrics for humans are least stable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1408,"prompt_tokens":692,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":648}},"tokens_in":436,"tokens_out":716,"duration_ms":7570,"temperature":1.0,"reasoning_tokens":648,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:19:00.754185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to run G-Eval and SEval-Ex on the same SummEval summaries under (a) the original closed-model configuration, (b) the open-weight configuration used here, and (c) repeated open-weight runs with different temperatures and seeds, then compare the spread of Spearman correlations with the paper's reported gaps. If the original configuration matches published reference values and only the open-weight runs drift, the discrepancy is an artifact of model substitution; if within-configuration spread is as large as the between-configuration spread, the instability is intrinsic.","supporting_citations":[{"cited_title":"R., Kry \\'s ci \\'n ski W., McCann B., Xiong C., Socher R","cited_arxiv_id":null,"evidence_quote":"Provides SummEval, the benchmark with human judgments on four dimensions used in every metric comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ROUGE, the classical lexical-overlap metric that anchors the cheap-but-limited side of the tradeoff."},{"cited_title":"& Guigue V","cited_arxiv_id":null,"evidence_quote":"Defines SEval-Ex, the explainable statement-level metric re-run with two open-weight models."}],"review_version":1}