{"id":"edce645b-73d6-470b-910a-f32fb996fdde","arxiv_id":"2411.17863","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A long-document keyphrase extractor using Longformer, convolution-based n-gram embeddings, and max-pooling over occurrences outperforms prior extractors on LDKP and most zero-shot datasets.","lead":"LongKey is a keyphrase extraction system that uses a long-context encoder (Longformer) and a max-pooling module to identify keyphrases in documents far longer than typical 512-token inputs. It edges out earlier extractors on standard long-document benchmarks by roughly two points of measured accuracy, with gains shrinking or reversing on short documents and a few unseen domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline reimplementations are the linchpin: if JointKPE et al. were given their original encoders and tuning, the 1.5-2.5 point F1 advantage on LDKP3K could vanish.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: the central claim is empirical superiority, and that claim rests on a comparison table whose validity depends on the baselines being faithful and strong. The decision to reimplement all baselines under a shared Longformer configuration is a concrete threat to that validity, because the baselines' original designs and hyperparameters were tuned for BERT-style encoders. The paper's own ablation shows the max-pooling embedder is helpful, but that is an internal comparison, not evidence of superiority over well-tuned external methods. The absence of error bars and significance tests amplifies the risk: a 1.5–2.5 point F1 gap could easily be explained by suboptimal baseline configuration. The concrete test I propose would settle this directly by re-running the baselines with their native encoders and tuning. Until that test is done, the 'consistently outperforms' claim is conditional, which matches the reader's verdict. I also note the abstract's 'consistently outperforms' is contradicted by the paper's own results on SemEval2010, TMC, and the short-document datasets, but that is a wording issue rather than a flaw in the method's core mechanism. The baseline-fidelity issue is more fundamental because it threatens the entire comparison, so the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":16691,"tokens_out":7225,"duration_ms":65405,"concrete_test":"Download the official code and checkpoints for JointKPE, HyperMatch, SpanKPE, TagKPE, and RankKPE, fine-tune each on the LDKP3K training set using the hyperparameters reported in their original papers while keeping their native BERT/RoBERTa encoders, and evaluate on the LDKP3K test set with the same stemming and duplicate-cleaning. If any baseline reaches within 1.0 F1@5 of LongKey's 39.55, the claimed margin is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core empirical claim (Table I) depends on the authors' reimplementations of JointKPE, HyperMatch, SpanKPE, TagKPE, and RankKPE. Section III-B states all encoder-based supervised baselines were fine-tuned with Longformer and 'the same global attention mask as used in LongKey,' with no per-method hyperparameter tuning. These methods were originally designed and tuned with BERT/RoBERTa encoders; forcing them into Longformer with a single shared configuration can depress their performance. The paper's own data show encoder choice matters: BERT-LongKey beats Longformer-LongKey on TMC (16.69 vs 15.89 F1@5). Additionally, GELF is not re-run at all—its Table I entry is quoted from its paper without a K value, making that comparison uncontrolled. Main tables provide no error bars or significance tests, so the 1.5–2.5 point F1@5 advantage over JointKPE could shrink or vanish under a faithful baseline implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LongKey, a keyphrase extraction framework for long documents. LongKey encodes documents with Longformer (with positional embeddings extended to 8,192 tokens and optional chunking), represents each word by its first token embedding, forms n-gram candidate embeddings with 1-D convolutions, aggregates all occurrences of a candidate via max pooling, and scores candidates with a ranking loss plus a chunking loss. The method is evaluated on the LDKP3K and LDKP10K long-document benchmarks, six zero-shot unseen datasets, and two short-document datasets, with F1@K, F1@O, and F1@Best metrics. The authors report that LongKey achieves the best results on LDKP3K (e.g., F1@5 39.55% for LongKey8K) and LDKP10K (F1@5 41.81%), generally outperforms baselines on most unseen datasets, and include a component analysis of the pooling operator.","tokens_in":16912,"tokens_out":3038,"duration_ms":26696,"significance":"If the reported results are reliable, LongKey is a meaningful advance for long-document keyphrase extraction: it is simple, publicly available, and shows consistent gains of roughly two F1 points over strong supervised baselines on the LDKP benchmarks and several zero-shot datasets. The paper's strengths include a clean ablation of the max-pooling candidate embedder (Table IV, with means and standard deviations over five runs), open-source code, and evaluation across a broad set of domains. However, the central empirical claim rests heavily on the fairness and faithfulness of the baseline reimplementations, and the manuscript does not currently provide enough evidence that the baseline comparison is controlled. The exceptions on SemEval2010, TMC, and short-document datasets also weaken the abstract's 'consistently outperforms' claim, so the contribution is best assessed after the comparison protocol is clarified and the statistical uncertainty of the main tables is addressed.","major_comments":[{"comment":"The central claim of outperforming JointKPE, HyperMatch, and related methods depends on the baseline reimplementation protocol. Section III-B states that all encoder-based supervised baselines were fine-tuned with Longformer and 'the same global attention mask as used in LongKey,' without per-method hyperparameter tuning. These methods were originally developed with BERT or RoBERTa encoders, and the paper's own results show that encoder choice changes performance substantially (e.g., in Table III, BERT-LongKey reaches 16.69 F1@5 on TMC versus 15.89 for Longformer-LongKey). If the Longformer-based baselines are disadvantaged by this protocol, the 1.5-2.5 point F1@5 margins on LDKP3K could shrink or disappear. The authors should either provide a principled justification for the common-encoder protocol (e.g., per-method tuning under that protocol) or report results with each method's originally designed encoder and hyperparameters.","section":"Section III-B and Table I"},{"comment":"The comparison against GELF is uncontrolled: Table I footnotes that the GELF score was 'reported in its paper without a specific K value,' and Tables II and III use the same asterisk. GELF is listed among supervised methods and its F1@O values (27.10 on LDKP3K, 16.70 on SemEval2010, 21.50 on NUS) are quoted from the original paper rather than measured under the authors' evaluation pipeline, which applies Porter stemming, a fixed maximum keyphrase length, and specific K values. This is not a like-for-like comparison, and it should either be removed from the main tables or GELF should be re-run under the same protocol.","section":"Tables I-III, GELF rows"},{"comment":"The main comparative tables report single runs with no variance estimates or significance testing, whereas the component analysis in Table IV reports means and standard deviations over five runs. Given that the reported advantages over JointKPE on LDKP3K are only about 1.5-2.5 F1 points, it is possible that these differences are within run-to-run noise. The authors should provide multiple-seed results or significance tests (e.g., paired bootstrap over documents) for at least the LDKP3K and LDKP10K headline comparisons, and for the unseen datasets where the margin is small.","section":"Tables I-III and §IV-C"},{"comment":"The abstract claims that LongKey 'consistently outperforms existing unsupervised and language model-based keyphrase extraction methods,' but the paper's own results contradict 'consistently': on TMC, RankKPE and BERT-RankKPE exceed LongKey (Table III, e.g., 16.21 vs 15.89 F1@5 for the LDKP3K-trained models), and on SemEval2010, LongKey is not always the top performer (e.g., BERT-HyperMatch reaches 24.35 F1@5 vs 22.31 for LongKey). The authors acknowledge these exceptions in §IV-B, but the abstract and conclusion should be qualified accordingly, for instance by saying 'outperforms on most evaluated long-document datasets.'","section":"Abstract and §IV-B"}],"minor_comments":[{"comment":"Equation (13) contains a typographical error: 'p+ = Softmax(schunk)+' should likely be 'p+ = Softmax(schunk)' with the superscript plus removed, since the positive-class probability is a scalar.","section":"Equation (13)"},{"comment":"The description of extending positional embeddings by 'duplicating the same weights to the next 4,096 elements' should be clarified: if the same positional weights are reused for positions 4096-8191, positions with the same index modulo 4096 will have identical positional encodings, which may affect distant-token interactions; the paper should state whether this was intentional and whether any ablation was performed.","section":"Section II-A"},{"comment":"The sentence 'we employed the same global attention mask as used in LongKey' is ambiguous for the BERT-based variants, since BERT has no global attention mechanism; please clarify whether global attention was only applied to Longformer runs and how BERT variants were adapted to the 8,192-token chunking setup.","section":"Section III-B"},{"comment":"In Figure 1, the label 'Conv1D(N-1)-gram' is confusing because the figure also shows 'Conv1D N-gram'; the labels should be made consistent (e.g., list the kernel sizes explicitly) to avoid the impression that the n-gram and (N-1)-gram convolutions are distinct modules at the same position.","section":"Figure 1"},{"comment":"The sentence 'While LongKey trained on the LDKP3K dataset outperformed other models trained on the same dataset, it scored significantly lower when compared to its performance on the LDKP10K dataset' is difficult to parse: it appears to compare results on different test sets, and should be rewritten to clearly state which model is being compared on which test set.","section":"Section IV-A"},{"comment":"The tables report many K values but do not state the number of ground-truth keyphrases O for each dataset; providing these values or a reference to them would help readers interpret the F1@O column.","section":"Tables II and III"}],"recommendation":"major_revision","confidential_remarks":"The core idea—max-pooling over candidate occurrences for long-document KPE—is interesting and the code availability is a plus. However, the main tables do not yet establish that the reported gains over prior supervised methods are not artifacts of the reimplementation protocol. I would ask the authors to re-run the key baselines with their original encoders or thoroughly tune them under the Longformer protocol, and to add variance/significance estimates for the headline results before this is publishable. The GELF comparison should also be made controlled or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read LongKey. The core empirical claim is that max-pooling keyphrase candidate embeddings gives a consistent edge on long-document KPE, and the component analysis is the strongest part of the paper. That is what a colleague should know first: the ablation is controlled and shows max-pooling beating JointKPE by about 3 F1 points under the same training budget, while average or sum pooling does worse. That is real, reproducible evidence that the proposed mechanism, not just the Longformer encoder, drives the gains. The framework itself is a recombination of known parts—JointKPE's dual losses, Longformer for long contexts, conv1d n-gram embeddings—and the actual novelty is the max-pooling pooler. The paper is honest and does not oversell: it reports exceptions (SemEval2010, TMC, short-document datasets) and explicitly limits itself to 8K context and 5-word keyphrases. The zero-shot evaluation across six datasets is a plus. The soft spots are mostly in the experimental protocol. The main tables report single runs with no variance or significance tests; only the component analysis gives standard deviations. More importantly, all encoder-based supervised baselines were reimplemented with Longformer and a shared attention mask, whereas they were originally tuned with BERT-style encoders. That could disadvantage them, and the paper's own BERT-vs-Longformer comparisons show encoder choice matters. GELF is quoted from its original paper without a re-run, so that comparison is uncontrolled. However, the component analysis mitigates the baseline concern somewhat, because it compares LongKey and JointKPE under identical encoder settings. The 'consistently outperforms' language in the abstract is broader than the data support, but the core claim on long documents is plausible. For whom is this paper? It is for people working on keyphrase extraction, especially long-document settings, and it offers a cheap, effective trick that could be adopted in other extractors. I would send it out: the ablation is credible, code is available, and the main open question—whether the two-point gain survives faithfully tuned baselines—is exactly what a good referee can probe. I would not desk-reject it.","headline":"A solid long-document KPE system paper whose max-pooling pooler is the real contribution, and the ablation supports it—worth serious review despite baseline-reimplementation concerns.","tokens_in":17460,"tokens_out":2175,"would_cite":true,"duration_ms":21334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LongKey claims that pooling all occurrences of each keyphrase candidate with an element-wise max—instead of scoring each occurrence separately—is what lets a Longformer-based extractor outperform prior unsupervised and supervised methods…","keywords":["keyphrase extraction","long documents","Longformer","max pooling","candidate embedding","ranking loss","LDKP","zero-shot evaluation"],"falsifier":"Re-run JointKPE and the other supervised baselines in their originally tuned BERT configurations on the LDKP3K test set; if any reaches or exceeds LongKey's F1@5 of 39.55%, the claim that max pooling is responsible would be undercut. A more direct ablation would retrain LongKey with per-occurrence scoring instead of the max pooler and check whether F1@5 falls to roughly the JointKPE level.","tokens_in":16456,"feed_emoji":"🔑","tokens_out":7413,"duration_ms":59824,"temperature":0.7,"pith_summary":"The paper sets out to establish that keyphrase extraction from long, full-text documents can be materially improved by pooling all occurrences of each candidate keyphrase into one embedding with a max operation, rather than scoring occurrences individually. LongKey, the proposed framework, pairs a Longformer encoder with an extended positional embedding and chunking for documents beyond 8K tokens, then builds convolutional n-gram embeddings and jointly optimizes ranking and chunking losses. On the LDKP3K and LDKP10K benchmarks and in zero-shot tests on six unseen datasets, LongKey reports top F1@K and F1@O scores; the max-pooling pooler is isolated as the decisive component, since average or sum pooling drops F1@5 below the JointKPE baseline. The paper also documents the approach's limits: no consistent gain on short-context datasets and slightly lower inference throughput.","feed_headline":"Max-pooled keyphrases top prior extractors on long documents","feed_subtitle":"LongKey pools every occurrence of each candidate into one embedding, reaching 41.81% F1@5 on LDKP10K.","key_machinery":"The central object is the keyphrase embedding pooler (KEP): for each unique candidate keyphrase, all its span embeddings from convolutional n-gram layers are reduced by an element-wise max over occurrences, $C^l = \\max(\\{h^l_1, \\dots, h^l_{S_l}\\})$. This single vector feeds a linear ranking head, while a separate chunking head scores each occurrence. The max operation is load-bearing—replacing it with average or sum pooling erases LongKey's advantage over JointKPE.","core_discovery":"On its own terms, the paper claims that the keyphrase embedding pooler (KEP) with max aggregation is the mechanism that lets LongKey beat prior supervised extractors on long documents. Because a keyphrase can appear many times across a long text, taking the element-wise maximum over all its occurrence embeddings highlights the most salient contextual features and yields a single candidate vector that the ranking head can score. The component analysis backs this up: average pooling gives 29.15% F1@5, sum pooling 32.76%, the JointKPE baseline 36.00%, and max-pooled LongKey 38.94% on the validation split. The headline results are F1@5 of 39.55% and F1@O of 41.84% on LDKP3K (LongKey8K) and F1@5 of 41.81% on LDKP10K.","pith_inferences":["[Editorial extension] The max-over-occurrences principle could transfer to other span-level tasks on long texts, such as claim detection or entity linking, where repeated mentions supply complementary evidence.","[Editorial extension] The paper's BERT-chunked variant implies a cheaper alternative to Longformer; training the pooler on chunked BERT with more long-document data is a testable comparison.","[Editorial extension] TMC was the one unseen domain where LongKey did not lead; fine-tuning on informal or noisy text would test whether the pooler's advantage reappears after domain adaptation.","[Editorial extension] The pooler also lowers variance across runs (std dev 0.07 vs 0.50 for JointKPE), hinting that aggregation stabilizes training; a seed-sensitivity study could quantify this directly."],"forward_implications":["LongKey moves keyphrase extraction from abstracts to documents up to 96K tokens, using chunked Longformer inference with duplicated positional embeddings.","The max-pooling pooler is the decisive component: average and sum pooling both fall below JointKPE, while max pooling wins on the validation split.","Models trained on LDKP3K transfer to six unseen long-document domains in most cases, with informal chat logs (TMC) as the main exception.","On short-context datasets (KP20k, OpenKP), LongKey does not consistently beat JointKPE or RankKPE, suggesting the gain is tied to long documents.","LongKey trades a small amount of inference speed for accuracy, since the pooler adds a per-candidate reduction step."],"supporting_citations":[{"why":"Supplies the Longformer encoder that processes long documents with sliding and global attention; LongKey extends its positional embeddings to 8,192 tokens.","marker":"[14]"},{"why":"JointKPE is the baseline and structural template for LongKey's ranking and chunking losses; LongKey replaces its per-occurrence scoring with the pooling pooler.","marker":"[10]"},{"why":"Provides the LDKP dataset family of long scientific documents used for training and the primary benchmark.","marker":"[15]"},{"why":"KP20k is the source of LDKP3K and serves as the short-context comparison benchmark.","marker":"[16]"},{"why":"OAGKx is the source of the LDKP10K training and test samples.","marker":"[17]"},{"why":"GELF is the long-document keyphrase extraction baseline whose published F1@O score is cited without re-running.","marker":"[13]"},{"why":"HyperMatch is the supervised hyperbolic matching baseline that LongKey outperforms in the reported comparisons.","marker":"[11]"}],"fun_headline_variants":["LongKey: max-pooling keyphrase extraction beats baselines on long docs","Max-pooled keyphrase embeddings win on long-document extraction","LongKey pools keyphrase occurrences to top long-doc F1 scores","Max-aggregated keyphrase vectors lift long-document extraction","LongKey: max-pooling keyphrase extraction for long texts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the assumption that the reimplemented baselines are faithful, especially that switching every supervised competitor to the Longformer encoder with the same global attention mask, and taking GELF's published score without re-running it, does not disadvantage them.","fun_headline_variants_meta":{"raw":{"variants":["LongKey: max-pooling keyphrase extraction beats baselines on long docs","Max-pooled keyphrase embeddings win on long-document extraction","LongKey pools keyphrase occurrences to top long-doc F1 scores","Max-aggregated keyphrase vectors lift long-document extraction","LongKey: max-pooling keyphrase extraction for long texts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000807,"raw_usage":{"total_tokens":3508,"prompt_tokens":873,"completion_tokens":2635,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2541}},"tokens_in":489,"tokens_out":2635,"duration_ms":16285,"temperature":1.0,"reasoning_tokens":2541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:45:48.255062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run JointKPE and the other supervised baselines in their originally tuned BERT configurations on the LDKP3K test set; if any reaches or exceeds LongKey's F1@5 of 39.55%, the claim that max pooling is responsible would be undercut. A more direct ablation would retrain LongKey with per-occurrence scoring instead of the max pooler and check whether F1@5 falls to roughly the JointKPE level.","supporting_citations":[{"cited_title":"Capturing global informativeness in open domain keyphrase extraction,","cited_arxiv_id":null,"evidence_quote":"JointKPE is the baseline and structural template for LongKey's ranking and chunking losses; LongKey replaces its per-occurrence scoring with the pooling pooler."},{"cited_title":"Deep Keyphrase Generation","cited_arxiv_id":"1704.06879","evidence_quote":"KP20k is the source of LDKP3K and serves as the short-context comparison benchmark."},{"cited_title":"Two Huge Title and Keyword Generation Corpora of Research Articles","cited_arxiv_id":"2002.04689","evidence_quote":"OAGKx is the source of the LDKP10K training and test samples."},{"cited_title":"Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings","cited_arxiv_id":"2305.09316","evidence_quote":"GELF is the long-document keyphrase extraction baseline whose published F1@O score is cited without re-running."},{"cited_title":"Hyperbolic relevance matching for neural keyphrase extraction,","cited_arxiv_id":null,"evidence_quote":"HyperMatch is the supervised hyperbolic matching baseline that LongKey outperforms in the reported comparisons."}],"review_version":1}