{"id":"32a88766-8e2c-48a5-a692-6406b213acf3","arxiv_id":"2606.05494","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-model selection framework that picks the best of three summaries by averaged lexical/semantic scores; the headline performance claims are contradicted by the paper's own tables.","lead":"Three summarization models each write a candidate summary, and the paper picks the one with the best averaged ROUGE-L/BLEU/BERTScore. The evaluation is compromised: scores are computed against the source article rather than human references, one table contradicts the paper's 'highest BERTScore' claim, and another table's numbers are mathematically inconsistent with the selection rule.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported tables contradict the central claim: Table III base-line MASF average (39.59%) is below T5-small (39.75%), and Table II lists DAMB BERTScore 88.70% above the claimed 'highest' 88.63%.","rationale":"The reader's weakest assumption is that the article-relative selection score of Eq. (6) is a valid proxy for reference-relative quality. That is a legitimate concern, but it is not the most decisive one: even granting the proxy, the paper's own printed results contradict the central claim. Table III arithmetic shows base-line MASF average 39.59% < T5-small 39.75%, so the conclusion that MASF 'consistently outperforms' individual base-line models is false on the paper's own evaluation metric. Additionally, Table II explicitly contains DAMB with BERTScore 88.70%, above the claimed highest 88.63%. These are internal mismatches between the manuscript's claims and its data, not matters of interpretation. A concrete rerun of the evaluation would settle them. I therefore treat the central concern as internal inconsistency rather than the proxy assumption, hence partial agreement with the reader. The reader's REJECT verdict remains appropriate; no adjustment is needed.","tokens_in":8651,"tokens_out":6699,"duration_ms":61653,"concrete_test":"Release the generated candidate summaries and per-article scores on the CNN/DailyMail test split and rerun the Table III/IV evaluation against human reference summaries. Then check whether (1) the unweighted average of BERTScore, ROUGE-L, and BLEU for base-line MASF is 39.59% while T5-small is 39.75%, and (2) any row in Table II (DAMB) has BERTScore above 88.63%. If the numbers reproduce as printed, the claims 'outperforms individual base-line ... models' and 'highest BERTScore among all compared methods' are false. This rerun also tests whether the source-relative selection score of Eq. (6) actually selects summaries with higher reference-based scores than the best individual model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MASF 'outperforms the individual base-line and fine-tuned summarization models in terms of overall summarization quality' (Abstract, Section V, Conclusion). This is contradicted by the paper's own Table III. Using the paper's average of ROUGE-L, BLEU, and BERTScore (Eq. (6) with reference-based scores), MASF scores (87.07+24.30+7.40)/3 = 39.59%, while T5-small scores (86.31+25.19+7.75)/3 = 39.75%. So in the base-line setting, MASF is not the best individual model on the paper's own aggregate quality metric. Section V acknowledges T5-small has higher ROUGE-L and BLEU, then calls MASF 'highly competitive' and 'close'; the Conclusion nonetheless asserts MASF 'consistently outperforms' individual base-line models. This is an internal inconsistency, not a question of metric proxy. Independently, the Abstract and Section V state MASF 'achieves the highest BERTScore among all compared methods with a score of 88.63%.' Table II includes DAMB with BERTScore 88.70%, which is higher. The claimed 'highest' is therefore false under the paper's own comparison table. These two discrepancies are load-bearing because they directly falsify the strongest_claim. If the printed numbers are correct, the framework does not deliver the claimed improvement; if they are not correct, the evaluation is unreliable. Either way, the central claim fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Multi-Model Adaptive Summarization Framework (MASF) that generates three candidate summaries from fine-tuned T5-small, PEGASUS-xsum, and LED-base models, scores each candidate with an average of ROUGE-L, BLEU, and BERTScore, and selects the highest-scoring summary as the final output. The models are fine-tuned on CNN/DailyMail and compared against the individual models and several prior/LLM baselines. The central claims are that MASF achieves the highest BERTScore among all compared methods (88.63%) and consistently outperforms the individual baseline and fine-tuned summarization models in overall summarization quality.","tokens_in":9068,"tokens_out":2801,"duration_ms":29109,"significance":"The idea of combining multiple summarization models with a lightweight metric-based selector is practically appealing and could be a useful robustness technique if it works. The paper uses a standard public dataset, standard fine-tuned models, and a simple transparent selection rule, which are strengths in terms of reproducibility. However, the reported results do not support the stated claims: the paper's own tables contradict the claimed BERTScore superiority and the claimed consistent outperformance over baselines. Moreover, because the selection metric is computed against the source article rather than human reference summaries, and the same metric is then used to report system performance, the reported gains are partly an artifact of the selection operator. These issues are load-bearing and make the current manuscript unsuitable for publication in its present form.","major_comments":[{"comment":"The central claim that MASF 'consistently outperforms the individual baseline and fine-tuned summarization models' is contradicted by the paper's own Table III in the baseline setting. Using the paper's average of ROUGE-L, BLEU, and BERTScore as in Eq. (6), T5-small receives (86.31 + 25.19 + 7.75)/3 = 39.75%, while MASF receives (87.07 + 24.30 + 7.40)/3 = 39.59%. Section V explicitly acknowledges that T5-small has higher ROUGE-L and BLEU, yet the Conclusion asserts consistent outperformance. This internal inconsistency directly undermines the main contribution.","section":"Table III / Section V"},{"comment":"The abstract and Section V state that MASF 'achieves the highest BERTScore among all compared methods with a score of 88.63%.' Table II lists DAMB with a BERTScore of 88.70%, which is higher than 88.63%. Under the paper's own comparison table, the claimed 'highest' BERTScore is false. This is not a cosmetic wording issue: it pertains to the primary reported result.","section":"Abstract / Section V / Table II"},{"comment":"The selection rule in Eq. (7) chooses the candidate with the highest average of ROUGE-L, BLEU, and BERTScore, where Eq. (6) states these scores are computed between the generated summary and the article, not against reference summaries. The same article-relative scores are then reported in Tables III–IV as MASF's performance. This creates a circularity: the selected summary's score is, by construction, at least as high as each candidate's score at the instance level, so reporting these scores as evidence of improved summary quality is partly a mathematical consequence of the argmax. Furthermore, in the baseline table, MASF's average (39.59%) is below T5-small's average (39.75%); if the reported averages were article-relative and selection used Eq. (7), the selected average would have to be at least as large as every candidate's average. The observed violation suggests either the table sc","section":"Eq. (6), Eq. (7), Tables III–IV"},{"comment":"Even setting aside the internal contradictions, the framework's validity depends on whether article-relative ROUGE-L/BLEU/BERTScore is a proxy for reference-relative summary quality. The paper provides no evidence, such as correlation with reference-based metrics or human judgments, that selecting the candidate most similar to the source article yields summaries of higher quality. Without such evidence, the method may simply favor extractive or source-copying candidates, and the reported 'gains' over fine-tuned models are not established.","section":"Section IV-D / Selection metric validity"}],"minor_comments":[{"comment":"Use consistent terminology: the title, abstract, and body alternate among 'Metric-based Selection', 'Adaptive Selection', and 'Multi-Model Adaptive Summarization Framework'. This makes the contribution harder to identify.","section":"Throughout"},{"comment":"The table columns do not specify whether scores are averages over the test set with standard deviations, and no significance tests are reported. Some values are missing (e.g., DAMB's BLEU in Table II), and the 'AverageScore' column is not defined as the mean of the three preceding columns.","section":"Tables II–IV"},{"comment":"The phrase 'higher than all reported models in the comparison' is immediately followed by the observation that text-davinci-003 has a higher average score (53.75%), which is confusing. Clarify whether BERTScore or average score is the claimed ranking criterion.","section":"Section V"},{"comment":"There are typos in the article sample text, e.g., 'Firoentina' and 'PSG and Spanish clubs' with inconsistent capitalization. These do not affect the technical content but should be cleaned.","section":"Table I"},{"comment":"Several references are incompletely formatted (e.g., [19], [26] lack page numbers or venue details in places), and the reference list numbering should be checked against in-text citations.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript has a clear, reproducible experimental design, but the central claims are directly contradicted by the paper's own tables. The discrepancy in Table III (MASF baseline average below T5-small) and the BERTScore claim in Table II (DAMB 88.70% vs. 88.63%) are not presentation-level issues; they falsify the main conclusions. The additional circularity concern about scoring against the source rather than references reinforces the reject recommendation. I would not consider this a borderline case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Sam,\n\nQuick take on arXiv:2606.05494. The paper's own tables contradict its central claim. In the base-line comparison, MASF's average — computed the paper's own way, as a mean of ROUGE-L/BLEU/BERTScore — is 39.59%, lower than T5-small's 39.75%. So the 'adaptive selection' did not select the highest-scoring candidate, or the reported numbers are wrong. And the abstract's claim of 'the highest BERTScore among all compared methods' fails against DAMB's 88.70% in Table II, versus MASF's 88.63%. These are not subtle issues.\n\nWhat's the paper actually doing? It combines three off-the-shelf summarizers, fine-tunes them on CNN/DailyMail, and picks the candidate with the highest average of ROUGE-L, BLEU, and BERTScore. The writing is clear, and the framework is easy to follow. Honest credit: the methodology section is readable, and the authors cite the directly relevant prior work — DAMB, plus model-agnostic reranking papers — which makes the novelty problem self-inflicted.\n\nThe soft spots are substantial. First, Eq. (6) defines the metrics between the summary and the source article, not the reference summary. That makes the selection score a proxy for extractiveness/copy-overlap, and the paper offers no evidence that it tracks human-judged quality. If the tables report these same article-relative scores, then the reported 'performance' is partly a mathematical artifact of the argmax. If the tables instead use reference-based scores (as is standard), the paper never says so, and then there is no explanation of why selecting on article-relative scores improves reference-relative scores. Second, the base-line numbers violate the selection rule's own guarantee, as noted. Third, the comparison with prior work mixes numbers from other papers with different evaluation setups and omits DAMB's BLEU score, making the table hard to parse. No code or data is provided.\n\nWhat's good: the idea of multi-model selection is legitimate, and for news summarization a carefully executed version could be a modest engineering contribution. But this paper's evaluation doesn't support its claims, and the internal contradictions are load-bearing. A serious referee would need the evaluation redone, not just patched. I would desk-reject this one in its current form. If the authors fix the evaluation and clearly position against DAMB, it might become a short empirical paper.\n\nAlex","headline":"The paper's own tables contradict its central claim: the base-line average is below T5-small, and the 'highest BERTScore' claim ignores DAMB in the same table.","tokens_in":9501,"tokens_out":4059,"would_cite":false,"duration_ms":39997,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting the best of three candidate summaries by the average of ROUGE-L, BLEU, and BERTScore measured against the source article outperforms each constituent summarization model on CNN/DailyMail.","keywords":["abstractive summarization","multi-model ensemble","metric-based selection","ROUGE-L","BLEU","BERTScore","CNN/DailyMail","transformer fine-tuning"],"falsifier":"For a held-out set of CNN/DailyMail test articles, compute the three candidate summaries, apply the MASF selection rule, and evaluate the selected summary with reference-based metrics (e.g., ROUGE-1/2/L and BERTScore against the human highlights). If the selected summaries do not beat the best single model's summaries on those reference-based metrics — or if a random/majority baseline does equally well — the central claim fails. Also check Table II's DAMB BERTScore of 88.70%; if it holds, the paper's 'highest BERTScore' claim is false.","tokens_in":8550,"feed_emoji":"🎯","tokens_out":9780,"duration_ms":85155,"temperature":0.7,"pith_summary":"The paper tries to establish that a simple post-hoc selection rule can make multi-model summarization more robust than any single model: generate one candidate summary from each of three fine-tuned transformers (T5-small, PEGASUS-xsum, LED-base), score each candidate against the original article with the average of ROUGE-L, BLEU, and BERTScore, and output the highest-scoring candidate. On CNN/DailyMail, this Multi-Model Adaptive Summarization Framework (MASF) reports a fine-tuned BERTScore of 88.63% and higher ROUGE-L and BLEU than its individual models, and it is competitive with much larger LLMs. The authors argue that the framework leverages complementary strengths of different architectures and improves consistency across diverse articles. If true, the result matters because it suggests quality gains can come from selection rather than further model scale or training.","feed_headline":"Best-of-three summary selection outranks its models","feed_subtitle":"A metric-based picker scores 88.63% BERTScore on CNN/DailyMail and beats all three constituent models.","key_machinery":"The load-bearing mechanism is the selection rule defined by Eqs. (6)–(7): for each candidate summary S_i, compute Score(S_i) = (ROUGE-L(S_i) + BLEU(S_i) + BERTScore-F1(S_i)) / 3, where all three metrics are computed between the candidate and the source article, then choose S* = argmax Score(S_i). This composite score is meant to capture both lexical overlap and semantic similarity, and the argmax turns three candidate summaries into one final output. The framework also uses LoRA-based parameter-efficient fine-tuning for the LED-base model, but the selection rule is the element that carries the paper's argument.","core_discovery":"The paper's central claim is that candidate summaries generated independently by three fine-tuned transformer models can be reliably ranked by a composite score — the simple average of ROUGE-L, BLEU, and BERTScore computed between each candidate and the source article — and that the argmax selection (Eq. 7) identifies the most informative summary. The authors report that in the fine-tuned setting MASF achieves 88.63% BERTScore, 32.75 ROUGE-L, and 16.00 BLEU, improving the average score over the strongest individual model, T5-small, by 3.01 percentage points, and state that it achieves the highest BERTScore among all compared methods, outperforming several LLMs such as GPT3-D2, Falcon-7b, and","pith_inferences":["The selection metric is article-relative, not reference-relative; the paper gives no evidence that scoring high against the source correlates with scoring high against human highlights, so the framework may in practice select the most extractive or copy-heavy candidate. A direct test would compare selected summaries to human references across the test set.","The paper's 'highest BERTScore' statement is, on its face, inconsistent with its own Table II, which lists DAMB at 88.70% versus MASF's 88.63%; if that table is accurate, the claim should be read as 'highest among the models, excluding DAMB' or the comparison is in error.","A natural extension, mentioned in the paper's future work, is to swap the article-relative metrics for reference-free, human-aligned or factuality metrics; this would test whether the selection mechanism generalises beyond lexical overlap to true summary quality.","The largest fine-tuned gain appears in BLEU (+3.5 points over T5-small), a lexical-precision metric measured against the article; this is consistent with the extractive-bias concern and suggests the selector may disproportionately favour candidates that reuse source phrases."],"forward_implications":["Combining a handful of small fine-tuned models plus a metric-based selector can match or beat much larger LLM summarizers on CNN/DailyMail, suggesting scale is not the only path to quality.","The selection rule works without human reference summaries at inference time, since it scores candidates against the source article, so it can be dropped into existing single-model pipelines at negligible cost.","Because the method is model-agnostic, it applies to any set of candidate summaries — from different prompts, decodings, or architectures — as a post-hoc robustness layer.","The consistent BERTScore gains imply the selector tends to prefer summaries semantically aligned with the source, which may translate to fewer hallucinations (though the paper does not test this directly).","The framework's competitive average score (45.80%) against BRIO (46.00%) with smaller models suggests lightweight ensembles could be a practical alternative to training larger summarizers."],"fun_headline_variants":["Best-of-three summary selection beats its own models","Three candidate summaries, one metric pick, top score","Multi-model selection outperforms each single model","Metric-based selection raises summary BERTScore by 3 pts","Composite score picker wins vs individual summarizers"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework's selection rule assumes that a candidate summary's average ROUGE-L, BLEU, and BERTScore against the source article is a reliable proxy for its quality relative to human-written reference summaries — an assumption the paper does not test.","fun_headline_variants_meta":{"raw":{"variants":["Best-of-three summary selection beats its own models","Three candidate summaries, one metric pick, top score","Multi-model selection outperforms each single model","Metric-based selection raises summary BERTScore by 3 pts","Composite score picker wins vs individual summarizers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1531,"prompt_tokens":780,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":677}},"tokens_in":524,"tokens_out":751,"duration_ms":8013,"temperature":1.0,"reasoning_tokens":677,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:19:54.672238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a held-out set of CNN/DailyMail test articles, compute the three candidate summaries, apply the MASF selection rule, and evaluate the selected summary with reference-based metrics (e.g., ROUGE-1/2/L and BERTScore against the human highlights). If the selected summaries do not beat the best single model's summaries on those reference-based metrics — or if a random/majority baseline does equally well — the central claim fails. Also check Table II's DAMB BERTScore of 88.70%; if it holds, the paper's 'highest BERTScore' claim is false.","supporting_citations":[],"review_version":4}