{"id":"14fe466b-fc73-4ba4-afee-55d7c4eda64e","arxiv_id":"2506.10346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A meta-analysis of keyphrase generation shows benchmark datasets are highly correlated, evaluation protocols inflate scores, and a released BART-large baseline provides a stronger reference point.","lead":"This paper analyzes 52 keyphrase generation papers and finds the five most common benchmark datasets are nearly redundant, and that common evaluation tweaks inflate reported scores. It also releases a fine-tuned BART-large baseline and evaluation code to standardize future comparisons.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset-redundancy claim rests on Pearson correlations among best-reported scores from heterogeneous evaluation protocols; the paper's own §3.3 shows protocol shifts of several F1 points, so the >0.9 correlations may reflect protocol artifacts or model-strength confounds rather than intrinsic…","rationale":"The reader's weakest_assumption correctly identifies the central risk: the redundancy conclusion is inferred from correlations among best reported scores rather than from a controlled evaluation, leaving room for confounds from evaluation-protocol heterogeneity and general model strength. My read agrees with this concern and sharpens it by pointing to the paper's own §3.3 replication results, which quantify how much protocol choices can shift scores—enough to materially affect cross-dataset correlations. The paper also has genuine strengths: it releases artifacts, includes direct domain-overlap evidence, and its second finding about metric overestimation is supported by direct replications. Therefore the appropriate outcome is to keep the reader's CONDITIONAL verdict: the core contribution is valuable but the strongest quantitative claim needs a controlled check before the field should act on it. My concern does not move the verdict to reject or unverified, because the issue is addressable and the paper's released code and model make the decisive test straightforward. I mark agreement as partial rather than full because the reader's formulation emphasizes small sample sizes and model-strength effects, while I see the protocol-confounding mechanism as the more specific and actionable risk, directly tied to the paper's own evidence in §3.3.","tokens_in":22096,"tokens_out":3146,"duration_ms":40221,"concrete_test":"Use the released evaluation framework and available model outputs (at minimum the released BART-large checkpoint, plus catSeqTG-2RF1, ExHiRD-h, and SetTrans, for which outputs are available) to evaluate every model on all five datasets under one identical protocol: same normalization choice, same F1@M/F1@5 definition with or without dummy padding, same present/absent matching, and same decoding. Recompute the Figure 2 Pearson correlation matrix on this controlled set. If the mean pairwise ρ remains above 0.9, the redundancy claim is supported; if it drops materially (e.g., below 0.8) or pairwise rankings change, the original correlations were inflated by protocol heterogeneity. Also report the number of models contributing to each pairwise correlation in both the original and controlled analyses, to assess small-sample bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The first headline finding—that the five common benchmark datasets are so similar that reporting results on more than one adds no value—depends on the correlation matrix in Figure 2, computed from the best reported scores extracted from 52 papers. Those scores were produced under materially different evaluation protocols: at least 30 of 50 papers apply some form of normalization, two incompatible F1@k variants coexist (with or without dummy phrase padding), present/absent matching methods differ (the paper's own §A.3 notes its subsequence matching is stricter than regex matching), and model selection rules favor the best overall or best-on-KP20k variant. §3.3 demonstrates that normalization alone changes F1@M and F1@5 by +2.2 to +3.5 absolute points. If a paper's protocol systematically inflates scores across all datasets, the cross-dataset Pearson correlations will be inflated by this shared protocol factor, independent of dataset content. Additionally, a single latent 'model quality' factor—stronger models tend to score higher on every benchmark—can produce ρ>0.9 even for datasets with no document overlap. The paper does not report how many models have scores on all five datasets; with sparse overlapping entries, the correlations may be computed on a small, non-representative subset, and the reported p-values do not correct for the 28 pairwise comparisons in Figure 2. The direct evidence of overlap (all scientific abstracts, three sharing ACM DL sources) is suggestive but weaker than the quantitative redundancy claim, which requires showing that rankings and absolute scores are stable under a single controlled evaluation protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a meta-analysis of 52 keyphrase-generation papers published after Meng et al. (2017). The authors manually extracted 826 best-score triples across 50 models, analyzed contribution types, benchmark-dataset usage, evaluation metrics, and model architectures, and released a fine-tuned BART-large baseline along with an evaluation framework. The three headline findings are that the five commonly used benchmark datasets are so correlated that reporting results on more than one adds no value, that evaluation-protocol inconsistencies such as keyphrase normalization overestimate model performance, and that despite a shift to PLM-based models, overall progress has been modest. The paper also includes replication experiments measuring the effect of the Meng et al. (2017) normalization procedure on three published models.","tokens_in":22416,"tokens_out":6788,"duration_ms":78312,"significance":"If the findings were fully established, they would give the field a clear, actionable message: consolidate redundant benchmarks, standardize metric computation, and treat many published performance gaps with caution. The paper provides a useful service by making the collected metadata, replication scripts, and model weights publicly available, and the replication experiments are a concrete step beyond a standard survey. The main significance currently rests on two empirical claims that need stronger support: the dataset-redundancy claim in Section 3.2 and the overestimation claim in Section 3.3. I do not see a circularity problem: the analysis compares externally reported results and evaluates its released baseline under a standard benchmark, rather than deriving its conclusions from assumptions secretly built into the data.","major_comments":[{"comment":"The dataset-redundancy claim is not established by the reported correlation analysis. The figure is computed from \"best scores\" on each dataset, but the paper does not specify which of the 42 extracted metrics was used to obtain one score per model and dataset, how many models have scores on each dataset pair, or whether the compared scores were produced under the same evaluation protocol. This matters because, under the paper's own Section 3.3, normalization changes F1 by 2.2 to 3.5 absolute points and two incompatible F1@k padding rules coexist. A paper that uses one protocol for all five datasets will have its scores shifted on all five datasets, inflating cross-dataset Pearson correlations even if the datasets share no documents; a shared latent \"model quality\" factor will do the same. The statement that \"reporting results on more than one adds no value\" therefore needs a matched-protocol analysis, such as restricting correlations to models evaluated with identical normalization and F1@k conventions, reporting pairwise sample sizes, and correcting for the 28 pairwise comparisons shown in Figure 2.","section":"Section 3.2, Figure 2"},{"comment":"The replication evidence as printed is internally inconsistent and should be repaired before it can support the overestimation claim. The figure appears to show that \"Ours w/o norm\" exceeds \"Ours\" for present F1@M on ExHiRD and SetTrans, and exceeds \"Ours\" for all three models on absent F1@5 (e.g., catSeqTG absent F1@5: 5.8 vs 1.6 in the printed bars). The text says normalization \"significantly increases the scores for the majority of the evaluation metrics\" and gives +2.2 (F1@M) and +3.5 (F1@5) for present keyphrases, but these numbers are not derived from the figure in an obvious way. The authors should report the per-model, per-metric values in a table, define which pairwise comparison yields the claimed averages, and explain the sign of the effect for absent keyphrases; otherwise the central claim that normalization overestimates performance is not quantitatively supported.","section":"Section 3.3, Figure 4"},{"comment":"The state-of-the-art-over-time plot and the statement that only 3.1% present F1@M separates Chan et al. (2019) from Thomas and Vajjala (2024a) inherit the protocol-mixing problem identified in Sections 3.2 and 3.3. The plotted scores are the best scores extracted from 50 papers under different normalization rules, different F1@k padding conventions, and different present/absent matching methods; Section 3.3 shows these choices alone move scores by several points. Restricting the SOTA lines to a single protocol, or clearly labeling them as illustrative rather than directly comparable, is necessary for the claim about limited overall progress.","section":"Section 3.5, Figure 6"}],"minor_comments":[{"comment":"The selection section says the sample includes papers \"published at major NLP venues in the last seven years,\" but the same paragraph states that AAAI, SIGIR, and CIKM papers are included; the sentence should be reworded to acknowledge that non-ACL venues are also represented.","section":"Section 2"},{"comment":"The caption for Figure 2 should state the exact metric used for the correlation, the number of models per pair, and the method used to aggregate multiple reported metrics per dataset.","section":"Section 3.2, Figure 2 caption"},{"comment":"Footnote 3 notes that normalization information is often located only in source code; the sentence in the main text says normalization was observed \"in at least 30 out of 50 papers\" but does not say how much uncertainty remains or how the count was determined; a short note would make the statistic reproducible.","section":"Section 3.3"},{"comment":"There is a typo \"ONE2M ANY\" in the baseline description; elsewhere the paper writes \"ONE2MANY\" and \"ONE2SET\"; these should be made consistent.","section":"Section 4"},{"comment":"The appendix says keyphrases are lowercased, stemmed, and deduplicated before score calculation, but the main analysis does not state whether the 826 extracted triples were all normalized in this way; adding this detail would help readers interpret the correlation matrix.","section":"Appendix A.3"},{"comment":"The Jiang et al. (2023a) and Jiang et al. (2023b) references appear to point to the same paper; this duplicate should be resolved, and the in-text citations should match the reference list.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real gap and the released resources are valuable. My concern is that the two strongest empirical claims are currently overstated relative to the evidence: the correlation analysis does not control for protocol confounds and the replication figure appears to contradict part of the stated conclusion. These are fixable with additional analysis and clearer reporting rather than a change of direction. If the authors provide a matched-protocol analysis and a corrected replication table, I would be willing to approve the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper gives the keyphrase generation field its first systematic look in the mirror. The authors compiled best scores from 50+ papers, documented metric inconsistencies, ran replication experiments showing normalization inflates F1 by 2-3.5 points, and released a BART-large baseline with code and weights. That is real value, especially the artifacts.\n\nThe soft spot is the headline claim that the five standard datasets are so similar that reporting results on more than one adds no value. That conclusion comes from Pearson correlations among best reported scores computed under heterogeneous evaluation protocols. The paper itself shows those protocols shift scores by more than two F1 points; if protocol decisions travel across datasets, the high correlations could be artifacts of shared practice rather than dataset content. Model quality is also a confound—stronger models tend to score higher on every benchmark. The direct evidence of document overlap is suggestive but weaker than the quantitative 'no value' claim. The paper also doesn't report how many models contribute scores to all five datasets, and the p-values ignore the multiple comparisons issue. I don't think the strong version survives; the moderate version—these datasets overlap substantially and need a controlled re-evaluation—does.\n\nThis is fixable. The released data and baseline allow a single-protocol comparison across all five datasets, which would directly test the redundancy claim. I'd ask for that before accepting the first finding as stated. The baseline itself is a useful reference point; the lack of significance tests on its results is a minor concern given its purpose.\n\nThe paper's limitations section is honest about scope and extraction ambiguity, though it doesn't flag the correlation confound. All things considered, it deserves a serious referee. A revision that tamps down the redundancy claim to match the evidence, or adds the controlled experiment, would make it a solid contribution. I'd send it out.","headline":"A useful, candid meta-analysis that gives the field a needed look in the mirror; the dataset-redundancy claim is plausible but over-strong given the correlation method.","tokens_in":22905,"tokens_out":4143,"would_cite":true,"duration_ms":41725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Keyphrase generation benchmarks are so similar that evaluating on more than one adds no value, and inconsistent metric computation inflates reported performance.","keywords":["keyphrase generation","benchmark datasets","dataset redundancy","evaluation metrics","reproducibility","pre-trained language models","KP20k","F1@k"],"falsifier":"Recompute the five-dataset correlation matrix under one standardized evaluation pipeline, with identical normalization, identical F1@k padding rules, identical present/absent matching, and the same released model outputs on all five datasets, and check whether the correlations remain above 0.9.","tokens_in":21901,"feed_emoji":"📊","tokens_out":11213,"duration_ms":112843,"temperature":0.7,"pith_summary":"This paper claims that the field's standard evaluation practice for keyphrase generation is broken in two specific ways. The five most-used benchmark datasets (KP20k, SemEval-2010, Inspec, Krapivin, NUS) are so similar both in content and in the model rankings they produce that reporting results on more than one of them adds no information. At the same time, inconsistent metric computation, above all a ground-truth normalization recipe inherited from the KP20k work and a convention of padding short outputs with dummy phrases, inflates reported scores and makes cross-paper comparisons unreliable. If these findings are right, the community should consolidate benchmarks, standardize metric calculation, and treat many published performance gaps as artifacts rather than real progress. The paper also releases a fine-tuned BART-large baseline trained without normalization, intended as a solid reference point for future work.","feed_headline":"One benchmark dataset is enough for keyphrase evaluation","feed_subtitle":"An analysis of 52 papers finds near-perfectly correlated scores and metric quirks that inflate reported performance.","key_machinery":"Three instruments carry the analysis. The first is a manually assembled collection of 826 score triples from 50 models across 26 datasets, recording each model's best reported score per dataset and metric along with contribution type, architecture, significance-testing practice, and code and weight availability. The second is a correlation matrix computed over those best scores across the five dominant datasets; correlations above 0.9 are the quantitative basis for the dataset-redundancy claim. The third is a replication study that recomputes F1 for three published models (catSeqTG-2RF1, ExHiRD-h, SetTrans) under a fixed protocol, toggling the normalization on and off, which isolates the metric-inflation effect. The released object is a BART-large baseline fine-tuned on KP20k in the ONE2MANY format, in which keyphrases are generated as a single delimiter-separated sequence; it is trained without preprocessing, selected by validation F1, and evaluated with greedy and beam-search decoding.","core_discovery":"Analyzing 52 papers published after the first neural keyphrase generation work, the paper establishes three claims. First, model scores on the five dominant benchmark datasets are almost perfectly correlated (Pearson $\\rho > 0.9$, p-value < 0.01 for the pairwise correlations), and several of these datasets share source documents from the same digital library, so using more than one of them contributes no additional signal. Second, two inconsistencies inflate results: some authors compute F1@k after padding short predictions with dummy phrases, and at least 60% of papers apply a normalization that strips abbreviations, tokenizes on non-letter characters, and replaces digits with a placeholder before scoring. The paper's replication experiments show this normalization adds up to 3.5 F1@5 points for present keyphrases. Third, although model architectures have moved from RNNs to Transformers to fine-tuned pre-trained language models, absolute progress is limited: only about 3.1 F1@M points separate the best 2019 and best 2024 models on present keyphrases, and absent keyphrase F1@M hovers near 11%. In response, the paper releases a BART-large model fine-tuned on KP20k in the ONE2MANY format without normalization, which beats most prior models and reaches state-of-the-art absent keyphrase F1@5.","pith_inferences":["A direct extension of the redundancy claim is that new datasets should be chosen for domain diversity rather than size, because one scientific-abstract benchmark plus one news or social media benchmark would carry more signal than five overlapping abstract sets.","A testable extension of the metric-inflation argument is that retroactively applying a single standardized evaluation protocol to released model outputs would reorder published leaderboards, since normalization and padding affect different models by different amounts.","The paper's correlation table shows KPTimes and DUC2001 behaving markedly differently from the five scientific datasets, suggesting that cross-domain evaluation is where new benchmarks can still add information."],"forward_implications":["The five standard scientific-abstract datasets can be treated as one evaluation signal; papers should pair one of them with a different-domain set such as KPTimes instead of reporting all five.","Results computed with ground-truth normalization are inflated relative to unnormalized results, so standardizing evaluation will lower some reported scores and can reorder model comparisons.","State-of-the-art keyphrase generation is now dominated by fine-tuned pre-trained language models, yet the net gain over early sequence-to-sequence models is small, around 3.1 F1@M points for present keyphrases from 2019 to 2024.","Absent keyphrase generation remains far behind present keyphrase generation, with best F1@M near 11%, and strict single-ground-truth matching makes those scores especially unreliable.","Only 8 of the 50 model papers release model weights; without weights, statistical significance testing and fair comparison remain rare, so releasing weights is a necessary condition for credible progress claims."],"supporting_citations":[{"why":"Introduced the KP20k dataset, the CopyRNN sequence-to-sequence model, and the ground-truth normalization procedure whose effect the paper measures.","marker":"(Meng et al., 2017)"},{"why":"Introduced dummy-phrase padding for F1@k and is one of the three models whose outputs are re-evaluated in the replication experiments.","marker":"(Chan et al., 2019)"},{"why":"Contributed the ExHiRD-h model re-evaluated to quantify score inflation from normalization.","marker":"(Chen et al., 2020)"},{"why":"Contributed the One2Set model re-evaluated in the replication study and a milestone in the model evolution timeline.","marker":"(Ye et al., 2021b)"},{"why":"Provided the ONE2MANY training format used by the released baseline model.","marker":"(Yuan et al., 2020)"},{"why":"BART-large is the pre-trained language model that the released baseline fine-tunes.","marker":"(Lewis et al., 2020)"},{"why":"Investigated PLMs for keyphrase generation, marking the architectural transition documented in the analysis.","marker":"(Kulkarni et al., 2022)"},{"why":"The LLM-filtering approach that produced the largest recent state-of-the-art jump, discussed as the newest direction in the field.","marker":"(Shao et al., 2024)"},{"why":"Supports the claim that releasing source code alone is insufficient for reproducibility, motivating the release of model weights.","marker":"(Arvan et al., 2022)"},{"why":"Provides the argument that statistical significance testing is necessary, supporting the paper's observation that only 20 of 50 models report it.","marker":"(Dror et al., 2018)"}],"fun_headline_variants":["Keyphrase benchmarks are too similar to add value","Metric quirks inflate keyphrase scores by up to 3.5 F1","52-paper review shows most keyphrase gains are illusory","Keyphrase research needs metric reform, says 52-paper analysis","One keyphrase dataset suffices—redundancy and inflation exposed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The redundancy claim assumes that correlations above 0.9 among best reported model scores reflect genuine dataset similarity, and would collapse if those correlations mostly come from shared evaluation quirks, small sample sizes, or the general tendency of better models to score higher on every benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Keyphrase benchmarks are too similar to add value","Metric quirks inflate keyphrase scores by up to 3.5 F1","52-paper review shows most keyphrase gains are illusory","Keyphrase research needs metric reform, says 52-paper analysis","One keyphrase dataset suffices—redundancy and inflation exposed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3251,"prompt_tokens":973,"completion_tokens":2278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2187}},"tokens_in":589,"tokens_out":2278,"duration_ms":20335,"temperature":1.0,"reasoning_tokens":2187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:28:39.333425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the five-dataset correlation matrix under one standardized evaluation pipeline, with identical normalization, identical F1@k padding rules, identical present/absent matching, and the same released model outputs on all five datasets, and check whether the correlations remain above 0.9.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LLM-filtering approach that produced the largest recent state-of-the-art jump, discussed as the newest direction in the field."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that releasing source code alone is insufficient for reproducibility, motivating the release of model weights."}],"review_version":1}