REVIEW 4 major objections 4 minor 21 references
AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Re-running the top summary-evaluation metrics under one protocol shows that the best-aligned are also the costliest and least reproducible.
desk verdict Useful cost data and a reasonable plea for standardization, but the 'significant discrepancies' finding is confounded by model substitution and missing variance. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a comparative reimplementation protocol: every metric runs through the same TextMetric interface, is executed on the same SummEval instances in the same local environment (same GPU, same model versions from HuggingFace and Ollama), and is scored by Spearman correlation against human judgments with total runtime recorded. The decisive operation is a reported-versus-measured correlation table, because converting all metrics to the same output format and correlation convention removes the methodological heterogeneity that normally prevents direct comparison.
What would settle it
A decisive check would be to run G-Eval and SEval-Ex on the same SummEval summaries under (a) the original closed-model configuration, (b) the open-weight configuration used here, and (c) repeated open-weight runs with different temperatures and seeds, then compare the spread of Spearman correlations with the paper's reported gaps. If the original configuration matches published reference values and only the open-weight runs drift, the discrepancy is an artifact of model substitution; if within-configuration spread is as large as the between-configuration spread, the instability is intrinsic.
Extended reading notes
Core claim
The central empirical finding is that under one common execution environment, the correlation scores that made each metric attractive do not all survive contact with a different but reasonable setting. For example, G-Eval's fluency correlation falls from a positive reference value to about -0.45 with Gemma-3-27b and -0.51 with Qwen-2.5-72b, while SEval-Ex's relevance correlation rises by +0.12 in one open-weight configuration; BERTScore and BARTScore also show weak and inconsistent correlations across dimensions. The paper interprets these gaps as evidence of a structural tradeoff: metrics that align best with human annotators, such as G-Eval, SEval-Ex, and QuestEval, pay for that alignment
Load-bearing premise
The load-bearing premise is that replacing the original closed models (e.g., GPT-4 in G-Eval) with open-weight ones (Gemma-3-27b, Qwen-2.5-72b) is a fair reproducibility test; the paper explicitly concedes it traded fidelity for open-source access, so a large part of the observed gap could come from that substitution rather than from inherent metric instability.
Editorial extensions
If this is right
- Published correlation coefficients for LLM-based summary metrics should not be read as stable properties of the metric; reports should specify backbone model, prompt, decoding parameters, library versions, and seed.
- Large-scale or repeated evaluations should favor frugal metrics, while hour-long LLM metrics are better reserved for small, high-stakes evaluations.
- Metric comparisons need a common environment and a common correlation convention, with variability across runs reported rather than a single point estimate.
- Using open-weight models as substitutes for closed proprietary models is a workable but fidelity-losing path to reproducibility, and both the substitute and the loss should be documented.
- Metrics that look superior in one published run may change rank when re-run, so benchmark comparisons should include spread or interval estimates.
Reading between the lines
- A natural extension the paper leaves implicit is that any single-run correlation for an LLM evaluator should be treated as one draw: averaging over repeated runs and reporting the spread would likely change the practical ranking of metrics, especially for cheap metrics whose spread is negligible.
- One testable follow-up is to feed prompt-format perturbations (added spaces, line breaks, JSON vs plain text) through the same framework and measure whether G-Eval-style metrics are as sensitive to prompt changes as they are to backbone changes, linking the reproducibility results to the prompt-sensitivity literature the paper cites.
- Because the study uses only SummEval, a direct way to test whether the tradeoff is structural or dataset-specific is to apply the framework's pipeline to another summarization benchmark and see whether high-correlation metrics are still the least frugal and least stable there.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces AllSummedUp, a modular open-source framework for comparing automatic summarization evaluation metrics, applied to the SummEval dataset. Six metrics (ROUGE, BERTScore, BARTScore, QuestEval, UniEval, G-Eval, and SEval-Ex) are run in a unified environment; Spearman correlations with human judgments on four dimensions are reported and compared with reference values from the literature, alongside total execution times. The central claims are that there are significant discrepancies between reproduced and reported correlations, that the best-aligned metrics are the most computationally expensive, and that LLM-based metrics such as G-Eval are unstable across conditions, motivating calls for standardized evaluation protocols.
Significance. The topic is timely and the framework's design—standardized metric interface, evaluator orchestration, and report generation—is a useful contribution to reproducible evaluation research. If the empirical claims were fully supported, the paper would provide valuable evidence that high-correlation LLM-based metrics carry hidden costs and instability. The authors also make a useful methodological point by highlighting the heterogeneity of correlation measures in the literature and the need for reporting experimental configurations. However, the central empirical claims are currently weakened by a confounded comparison (different LLMs than the original publications), missing within-run variance information, and an unreleased code repository, so the significance of the conclusions is not yet established.
major comments (4)
- [§4.2.1, Table 1; §5] The abstract's 'significant discrepancies' claim is confounded by LLM substitution. G-Eval was originally run with GPT-4, and SEval-Ex's original conditions are not reproduced; here Gemma-3-27b and Qwen-2.5-72b are used with default temperature=1. The observed gap in G-Eval fluency (−0.45 for Gemma, −0.51 for Qwen) is exactly the kind of difference expected when changing the underlying model, not necessarily evidence of run-to-run instability. The authors themselves concede in §5: 'nous avons privilégié l'utilisation d'alternatives open-source, quitte à perdre en fidélité par rapport aux conditions d'évaluation originales.' To support the reproducibility claim, the paper should either add control experiments with the original model where feasible or explicitly reframe the findings as measuring cross-model sensitivity, with the models named in the abstract and conclusion.
- [§4.2.2, Table 3] The paper states 'Toutes les métriques ont été testées sur plusieurs exécutions distinctes' but only reports execution times in Table 3. No standard deviations, confidence intervals, or per-run Spearman correlations are given. Consequently, the central assertion that high-performing metrics like G-Eval are 'less stable across runs' (abstract and §4.3) is not empirically supported by the reported data. Without run-level variance, the observed discrepancies could be fully explained by the different choice of LLM or by other implementation differences. The authors should report the distribution of correlations across runs, at least for G-Eval and SEval-Ex, and use it to distinguish model-substitution effects from stochastic instability.
- [§3.1, footnote 1] The framework is described as open-source in the abstract and §3.1, but footnote 1 states 'Le github sera partagé après acceptation de l'article'. This means the central reproducibility artifact is currently unavailable to readers or reviewers. The paper's own reproducibility claims cannot be verified, and the proposed framework cannot be used or extended. The code should be released under a public repository at submission time, or the manuscript should clearly state its current availability status and avoid presenting the framework as an existing open-source contribution.
- [Table 1; §4.2.1] The provenance of the reference correlations is underspecified. The text says that when Spearman correlations were not provided in the original papers, 'nous avons pris comme références des articles ayant reproduit les expériences', but no citations are provided for these secondary sources. Moreover, BERTScore appears in Table 1 but is not listed among the implemented metrics in §3.2. Without a precise mapping from each metric to its reference value and implementation version, the 'discordance' calculation is not reproducible and the significance of the discrepancies cannot be assessed. Please provide a supplementary table listing reference values, the paper they come from, and the exact implementation/version used.
minor comments (4)
- [Table 2] The labels 'Rouge - nous' and 'Rouge - réel' are unclear. Clarify what 'réel' refers to (the BARTScore paper's reported values?) and describe the reference-handling difference that explains the mismatch.
- [§4.2.2, Table 3] The sentence 'Seul GEval et SEval-Ex sont impactées' is ambiguous; QuestEval also has execution times in the table and appears LLM-based. Clarify whether the statement refers to the impact of model substitution on execution time.
- [References] There are several formatting errors in the reference list, e.g., 'Yuanet al.' missing a space, 'Kry´sci´nski' with improperly encoded diacritics, and inconsistent use of capitalization.
- [§3.2] The methodology section does not list BERTScore among implemented metrics, although it is evaluated and reported in Table 1. Add it to the list or remove it from the experiments.
Circularity Check
No circular derivation; empirical benchmark with a minor, non-load-bearing self-citation.
full rationale
The paper's central claims are empirical measurements, not derived results. It reports Spearman correlations (Table 1) and runtimes (Table 3) for six metrics on SummEval and compares them with literature values. No parameter is fitted to the target claim, no equation is defined in terms of the conclusion, and no uniqueness theorem is invoked. The only self-citation is SEval-Ex (Herserant & Guigue, 2025), a metric authored by the same researchers and included as one of the six comparators. That citation is descriptive, not load-bearing: the 'structural trade-off' between correlation and cost is supported by the full set of metrics (ROUGE, BARTScore, QuestEval, UniEval, G-Eval), and removing SEval-Ex would not change the qualitative conclusion. The admitted limitation in Section 5 — 'nous avons privilégié l'utilisation d'alternatives open-source, quitte à perdre en fidélité par rapport aux conditions d'évaluation originales' — is a validity threat to the reproducibility comparison, but it is not a circularity: the observed gaps are compared against external literature values, not against the paper's own inputs. Accordingly, no circular step can be exhibited, and the score is minimal.
Assumptions & free parameters
assumptions (4)
- domain assumption SummEval human annotations are the ground truth for correlation
- domain assumption Spearman correlation is the appropriate alignment measure
- domain assumption Open-source implementations of the metrics reproduce the published metrics
- domain assumption Literature-reported correlations are accurate references
Cite this review
Pith. "Pith review of AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume." pith.science (2026). https://pith.science/paper/JFBNVYS3
@misc{pith2026250821389,
author = {Pith},
title = {Pith review of: AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume},
year = {2026},
howpublished = {\url{https://pith.science/paper/JFBNVYS3}},
note = {Machine review of arXiv:2508.21389}
}
read the original abstract
This paper investigates reproducibility challenges in automatic text summarization evaluation. Based on experiments conducted across six representative metrics ranging from classical approaches like ROUGE to recent LLM-based methods (G-Eval, SEval-Ex), we highlight significant discrepancies between reported performances in the literature and those observed in our experimental setting. We introduce a unified, open-source framework, applied to the SummEval dataset and designed to support fair and transparent comparison of evaluation metrics. Our results reveal a structural trade-off: metrics with the highest alignment with human judgments tend to be computationally intensive and less stable across runs. Beyond comparative analysis, this study highlights key concerns about relying on LLMs for evaluation, stressing their randomness, technical dependencies, and limited reproducibility. We advocate for more robust evaluation protocols including exhaustive documentation and methodological standardization to ensure greater reliability in automatic summarization assessment.
Reference graph
Works this paper leans on
-
[1]
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Atil B., Aykent S., Chittams A., Fu L., Passonneau R. J., Radcliffe E., Rajagopal G. R., Sloan A., Tudrej T., Ture F., Wu Z., Xu L. & Baldwin B. (2025). Non- Determinism of " Deterministic " LLM Settings . arXiv:2408.04667 [cs], doi:10.48550/arXiv.2408.04667
-
[3]
Belz A., Agarwal S., Shimorina A. & Reiter E. (2021). A Systematic Review of Reproducibility Research in Natural Language Processing . In P. Merlo, J. Tiedemann & R. Tsarfaty , \'Eds., Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics : Main Volume , p.\ 381--393, Online: Association for Computatio...
-
[4]
Bhandari M., Gour P., Ashfaq A., Liu P. & Neubig G. (2020). Re-evaluating evaluation in text summarization. arXiv preprint arXiv:2010.07100
arXiv 2020
-
[5]
Biderman S., Schoelkopf H., Sutawika L., Gao L., Tow J., Abbasi B., Aji A. F., Ammanamanchi P. S., Black S., Clive J., DiPofi A., Etxaniz J., Fattori B., Forde J. Z., Foster C., Hsu J., Jaiswal M., Lee W. Y., Li H., Lovering C., Muennighoff N., Pavlick E., Phang J., Skowron A., Tan S., Tang X., Wang K. A., Winata G. I., Yvon F. & Zou A. (2024). Lessons fr...
-
[6]
Branco A., Calzolari N., Vossen P., Van Noord G., Van Uytvanck D., Silva J., Gomes L., Moreira A. & Elbers W. (2020). A shared task of a new, collaborative type to foster reproducibility: A first exercise in the area of language science and technology with reprolang2020. In Proceedings of The 12th Language Resources and Evaluation Conference , p.\ 5539--5...
work page 2020
-
[7]
R., Kry \'s ci \'n ski W., McCann B., Xiong C., Socher R
Fabbri A. R., Kry \'s ci \'n ski W., McCann B., Xiong C., Socher R. & Radev D. (2021). Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics , 9 , 391--409
work page 2021
-
[8]
He J., Rungta M., Koleczek D., Sekhon A., Wang F. X. & Hasan S. (2024). Does Prompt Formatting Have Any Impact on LLM Performance ? arXiv:2411.10541 [cs], doi:10.48550/arXiv.2411.10541
Show all 21 references
-
[9]
& Guigue V
Herserant T. & Guigue V. (2025). Seval-ex: A statement-level framework for explainable summarization evaluation. Accepté pour publication, prépublication en cours sur arXiv
2025
-
[10]
& Socher R
Kry \'s ci \'n ski W., McCann B., Xiong C. & Socher R. (2019). Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840
2019 arXiv
-
[11]
Lin C.-Y. (2004). ROUGE : A Package for Automatic Evaluation of Summaries . In Text Summarization Branches Out , p.\ 74--81, Barcelona, Spain: Association for Computational Linguistics
2004
- [12]
- [13]
-
[14]
& Wang A
Scialom T., Dray P.-A., Gallinari P., Lamprier S., Piwowarski B., Staiano J. & Wang A. (2021). Questeval: Summarization asks for fact-based evaluation. arXiv preprint arXiv:2103.12693
2021 arXiv
-
[15]
Whitaker K. (2017). The mt reproducibility checklist. https://www.cs.mcgill.ca/ jpineau/ReproducibilityChecklist.pdf. Accessed: 2025-04-29
2017
-
[16]
Xue Y., Cao X., Yang X., Wang Y., Wang R. & Li J. (2023). We Need to Talk About Reproducibility in NLP Model Comparison . In H. Bouamor, J. Pino & K. Bali , \'Eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , p.\ 9424--9434, Singapo...
2023 doi
- [17]
- [18]
- [19]
- [20]
- [21]
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.