REVIEW 4 major objections 6 minor 29 references
LongKey: Keyphrase Extraction for Long Documents
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read LongKey claims that pooling all occurrences of each keyphrase candidate with an element-wise max—instead of scoring each occurrence separately—is what lets a Longformer-based extractor outperform prior unsupervised and supervised methods…
desk verdict A solid long-document KPE system paper whose max-pooling pooler is the real contribution, and the ablation supports it—worth serious review despite baseline-reimplementation concerns. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the keyphrase embedding pooler (KEP): for each unique candidate keyphrase, all its span embeddings from convolutional n-gram layers are reduced by an element-wise max over occurrences, $C^l = \max(\{h^l_1, \dots, h^l_{S_l}\})$. This single vector feeds a linear ranking head, while a separate chunking head scores each occurrence. The max operation is load-bearing—replacing it with average or sum pooling erases LongKey's advantage over JointKPE.
What would settle it
Re-run JointKPE and the other supervised baselines in their originally tuned BERT configurations on the LDKP3K test set; if any reaches or exceeds LongKey's F1@5 of 39.55%, the claim that max pooling is responsible would be undercut. A more direct ablation would retrain LongKey with per-occurrence scoring instead of the max pooler and check whether F1@5 falls to roughly the JointKPE level.
Extended reading notes
Core claim
On its own terms, the paper claims that the keyphrase embedding pooler (KEP) with max aggregation is the mechanism that lets LongKey beat prior supervised extractors on long documents. Because a keyphrase can appear many times across a long text, taking the element-wise maximum over all its occurrence embeddings highlights the most salient contextual features and yields a single candidate vector that the ranking head can score. The component analysis backs this up: average pooling gives 29.15% F1@5, sum pooling 32.76%, the JointKPE baseline 36.00%, and max-pooled LongKey 38.94% on the validation split. The headline results are F1@5 of 39.55% and F1@O of 41.84% on LDKP3K (LongKey8K) and F1@5 of 41.81% on LDKP10K.
Load-bearing premise
The reported gains rest on the assumption that the reimplemented baselines are faithful, especially that switching every supervised competitor to the Longformer encoder with the same global attention mask, and taking GELF's published score without re-running it, does not disadvantage them.
Editorial extensions
If this is right
- LongKey moves keyphrase extraction from abstracts to documents up to 96K tokens, using chunked Longformer inference with duplicated positional embeddings.
- The max-pooling pooler is the decisive component: average and sum pooling both fall below JointKPE, while max pooling wins on the validation split.
- Models trained on LDKP3K transfer to six unseen long-document domains in most cases, with informal chat logs (TMC) as the main exception.
- On short-context datasets (KP20k, OpenKP), LongKey does not consistently beat JointKPE or RankKPE, suggesting the gain is tied to long documents.
- LongKey trades a small amount of inference speed for accuracy, since the pooler adds a per-candidate reduction step.
Reading between the lines
- [Editorial extension] The max-over-occurrences principle could transfer to other span-level tasks on long texts, such as claim detection or entity linking, where repeated mentions supply complementary evidence.
- [Editorial extension] The paper's BERT-chunked variant implies a cheaper alternative to Longformer; training the pooler on chunked BERT with more long-document data is a testable comparison.
- [Editorial extension] TMC was the one unseen domain where LongKey did not lead; fine-tuning on informal or noisy text would test whether the pooler's advantage reappears after domain adaptation.
- [Editorial extension] The pooler also lowers variance across runs (std dev 0.07 vs 0.50 for JointKPE), hinting that aggregation stabilizes training; a seed-sensitivity study could quantify this directly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LongKey, a keyphrase extraction framework for long documents. LongKey encodes documents with Longformer (with positional embeddings extended to 8,192 tokens and optional chunking), represents each word by its first token embedding, forms n-gram candidate embeddings with 1-D convolutions, aggregates all occurrences of a candidate via max pooling, and scores candidates with a ranking loss plus a chunking loss. The method is evaluated on the LDKP3K and LDKP10K long-document benchmarks, six zero-shot unseen datasets, and two short-document datasets, with F1@K, F1@O, and F1@Best metrics. The authors report that LongKey achieves the best results on LDKP3K (e.g., F1@5 39.55% for LongKey8K) and LDKP10K (F1@5 41.81%), generally outperforms baselines on most unseen datasets, and include a component analysis of the pooling operator.
Significance. If the reported results are reliable, LongKey is a meaningful advance for long-document keyphrase extraction: it is simple, publicly available, and shows consistent gains of roughly two F1 points over strong supervised baselines on the LDKP benchmarks and several zero-shot datasets. The paper's strengths include a clean ablation of the max-pooling candidate embedder (Table IV, with means and standard deviations over five runs), open-source code, and evaluation across a broad set of domains. However, the central empirical claim rests heavily on the fairness and faithfulness of the baseline reimplementations, and the manuscript does not currently provide enough evidence that the baseline comparison is controlled. The exceptions on SemEval2010, TMC, and short-document datasets also weaken the abstract's 'consistently outperforms' claim, so the contribution is best assessed after the comparison protocol is clarified and the statistical uncertainty of the main tables is addressed.
major comments (4)
- [Section III-B and Table I] The central claim of outperforming JointKPE, HyperMatch, and related methods depends on the baseline reimplementation protocol. Section III-B states that all encoder-based supervised baselines were fine-tuned with Longformer and 'the same global attention mask as used in LongKey,' without per-method hyperparameter tuning. These methods were originally developed with BERT or RoBERTa encoders, and the paper's own results show that encoder choice changes performance substantially (e.g., in Table III, BERT-LongKey reaches 16.69 F1@5 on TMC versus 15.89 for Longformer-LongKey). If the Longformer-based baselines are disadvantaged by this protocol, the 1.5-2.5 point F1@5 margins on LDKP3K could shrink or disappear. The authors should either provide a principled justification for the common-encoder protocol (e.g., per-method tuning under that protocol) or report results with each method's originally designed encoder and hyperparameters.
- [Tables I-III, GELF rows] The comparison against GELF is uncontrolled: Table I footnotes that the GELF score was 'reported in its paper without a specific K value,' and Tables II and III use the same asterisk. GELF is listed among supervised methods and its F1@O values (27.10 on LDKP3K, 16.70 on SemEval2010, 21.50 on NUS) are quoted from the original paper rather than measured under the authors' evaluation pipeline, which applies Porter stemming, a fixed maximum keyphrase length, and specific K values. This is not a like-for-like comparison, and it should either be removed from the main tables or GELF should be re-run under the same protocol.
- [Tables I-III and §IV-C] The main comparative tables report single runs with no variance estimates or significance testing, whereas the component analysis in Table IV reports means and standard deviations over five runs. Given that the reported advantages over JointKPE on LDKP3K are only about 1.5-2.5 F1 points, it is possible that these differences are within run-to-run noise. The authors should provide multiple-seed results or significance tests (e.g., paired bootstrap over documents) for at least the LDKP3K and LDKP10K headline comparisons, and for the unseen datasets where the margin is small.
- [Abstract and §IV-B] The abstract claims that LongKey 'consistently outperforms existing unsupervised and language model-based keyphrase extraction methods,' but the paper's own results contradict 'consistently': on TMC, RankKPE and BERT-RankKPE exceed LongKey (Table III, e.g., 16.21 vs 15.89 F1@5 for the LDKP3K-trained models), and on SemEval2010, LongKey is not always the top performer (e.g., BERT-HyperMatch reaches 24.35 F1@5 vs 22.31 for LongKey). The authors acknowledge these exceptions in §IV-B, but the abstract and conclusion should be qualified accordingly, for instance by saying 'outperforms on most evaluated long-document datasets.'
minor comments (6)
- [Equation (13)] Equation (13) contains a typographical error: 'p+ = Softmax(schunk)+' should likely be 'p+ = Softmax(schunk)' with the superscript plus removed, since the positive-class probability is a scalar.
- [Section II-A] The description of extending positional embeddings by 'duplicating the same weights to the next 4,096 elements' should be clarified: if the same positional weights are reused for positions 4096-8191, positions with the same index modulo 4096 will have identical positional encodings, which may affect distant-token interactions; the paper should state whether this was intentional and whether any ablation was performed.
- [Section III-B] The sentence 'we employed the same global attention mask as used in LongKey' is ambiguous for the BERT-based variants, since BERT has no global attention mechanism; please clarify whether global attention was only applied to Longformer runs and how BERT variants were adapted to the 8,192-token chunking setup.
- [Figure 1] In Figure 1, the label 'Conv1D(N-1)-gram' is confusing because the figure also shows 'Conv1D N-gram'; the labels should be made consistent (e.g., list the kernel sizes explicitly) to avoid the impression that the n-gram and (N-1)-gram convolutions are distinct modules at the same position.
- [Section IV-A] The sentence 'While LongKey trained on the LDKP3K dataset outperformed other models trained on the same dataset, it scored significantly lower when compared to its performance on the LDKP10K dataset' is difficult to parse: it appears to compare results on different test sets, and should be rewritten to clearly state which model is being compared on which test set.
- [Tables II and III] The tables report many K values but do not state the number of ground-truth keyphrases O for each dataset; providing these values or a reference to them would help readers interpret the F1@O column.
Circularity Check
No circularity: LongKey's performance is measured against held-out test subsets and unseen datasets, and the single self-citation (TMC formatting from prior work) is not load-bearing.
full rationale
This is an empirical systems paper, not a formal derivation, so the circularity burden is low. The central claim is that LongKey outperforms baselines on LDKP3K, LDKP10K, and six unseen datasets. The evaluation is externally anchored: models are trained on public LDKP training splits and scored with F1@K and F1@O computed from predicted keyphrases against ground-truth annotations; these metrics are not constructed from the training objective or from LongKey's own parameters. The proposed max-pooling keyphrase embedding pooler is tested by a component analysis (Table IV) that compares max, average, and sum aggregation against a JointKPE configuration, and the reported gain is an empirical measurement, not a definitional consequence. The only self-citation is reference [24], used to state that TMC documents and keyphrases follow 'the formatting from [24]'. This is data-preparation provenance and does not feed the scoring or ranking equations, so it is not load-bearing. The GELF baseline is quoted from its own paper rather than re-run; that is a fairness/control limitation, not circularity. Likewise, reimplementing all supervised baselines with Longformer and the same global attention mask may disadvantage methods originally tuned with BERT, but that is a comparison-quality concern rather than a circularity of LongKey's own derivation. No equation in the paper reduces to its inputs by construction, and no fitted parameter is renamed as a prediction. The paper contains one minor, non-load-bearing self-citation, which puts it at the very bottom of the scale; no true circular step is present.
Assumptions & free parameters
free parameters (5)
- max keyphrase length k=5 =
5 words
- chunk size and positional embedding extension =
8192 tokens
- global attention token choice =
[CLS] only
- training hyperparameters =
lr 5e-5, batch 16, 25k/78,125 iterations
- evaluation K per dataset =
dataset-specific (e.g., 4/5/6, oracle O, best up to 100)
assumptions (5)
- ad hoc to paper Longformer with duplicated positional embeddings and no cross-chunk attention preserves enough document context for keyphrase extraction.
- domain assumption LDKP ground-truth keyphrases are a valid supervision signal for full-text documents.
- domain assumption Stem-based exact matching between candidates and ground truth approximates semantic keyphrase equivalence.
- ad hoc to paper First-token word embeddings are sufficient word representations.
- ad hoc to paper Max-pooling over occurrences is the correct aggregation for candidate semantics.
Cite this review
Pith. "Pith review of LongKey: Keyphrase Extraction for Long Documents." pith.science (2026). https://pith.science/paper/NOVUSAVO
@misc{pith2026241117863,
author = {Pith},
title = {Pith review of: LongKey: Keyphrase Extraction for Long Documents},
year = {2026},
howpublished = {\url{https://pith.science/paper/NOVUSAVO}},
note = {Machine review of arXiv:2411.17863}
}
read the original abstract
In an era of information overload, manually annotating the vast and growing corpus of documents and scholarly papers is increasingly impractical. Automated keyphrase extraction addresses this challenge by identifying representative terms within texts. However, most existing methods focus on short documents (up to 512 tokens), leaving a gap in processing long-context documents. In this paper, we introduce LongKey, a novel framework for extracting keyphrases from lengthy documents, which uses an encoder-based language model to capture extended text intricacies. LongKey uses a max-pooling embedder to enhance keyphrase candidate representation. Validated on the comprehensive LDKP datasets and six diverse, unseen datasets, LongKey consistently outperforms existing unsupervised and language model-based keyphrase extraction methods. Our findings demonstrate LongKey's versatility and superior performance, marking an advancement in keyphrase extraction for varied text lengths and domains.
Figures
Reference graph
Works this paper leans on
-
[1]
Recent advances in natural language processing via large pre-trained language models: A survey,
B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Computing Surveys, vol. 56, no. 2, pp. 1–40, 2023
2023
-
[2]
A survey on recent advances in keyphrase extraction from pre-trained language models,
M. Song, Y . Feng, and L. Jing, “A survey on recent advances in keyphrase extraction from pre-trained language models,” Findings of the Association for Computational Linguistics: EACL 2023 , pp. 2153–2164, 2023
work page 2023
-
[3]
Keyword and keyphrase extraction tech- niques: a literature review,
S. Siddiqi and A. Sharan, “Keyword and keyphrase extraction tech- niques: a literature review,” International Journal of Computer Applica- tions, vol. 109, no. 2, 2015
2015
-
[4]
Using tf-idf to determine word relevance in document queries,
J. Ramos et al., “Using tf-idf to determine word relevance in document queries,” in Proceedings of the first instructional conference on machine learning, vol. 242, no. 1. Citeseer, 2003, pp. 29–48. TABLE VI RESULTS OBTAINED IN SHORT DOCUMENT DATASETS WITH MODELS TRAINED ON LDKP3K AND LDKP10K TRAINING SUBSETS . VALUES IN %. B EST SCORES , FOR EACH K AND D...
work page 2003
-
[5]
Automatic keyword extraction from individual documents,
S. Rose, D. Engel, N. Cramer, and W. Cowley, “Automatic keyword extraction from individual documents,” Text mining: applications and theory, pp. 1–20, 2010
2010
-
[6]
Textrank: Bringing order into text,
R. Mihalcea and P. Tarau, “Textrank: Bringing order into text,” in Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp. 404–411
work page 2004
-
[7]
Keybert: Minimal keyword extraction with bert
M. Grootendorst, “Keybert: Minimal keyword extraction with bert.”
-
[8]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018
arXiv 2018
Show all 29 references
-
[9]
Patternrank: Leveraging pre- trained language models and part of speech for unsupervised keyphrase extraction,
T. Schopf, S. Klimek, and F. Matthes, “Patternrank: Leveraging pre- trained language models and part of speech for unsupervised keyphrase extraction,” in Proceedings of the 14th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Man- age...
2022
-
[10]
Capturing global informativeness in open domain keyphrase extraction,
S. Sun, Z. Liu, C. Xiong, Z. Liu, and J. Bao, “Capturing global informativeness in open domain keyphrase extraction,” in Natural Lan- guage Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13–17, 2021, Proceedings, Part I...
2021
-
[11]
Hyperbolic relevance matching for neural keyphrase extraction,
M. Song, Y . Feng, and L. Jing, “Hyperbolic relevance matching for neural keyphrase extraction,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , M. Carpuat, M.-C. de Marneffe, a...
2022
-
[13]
Enhancing keyphrase extraction from long scientific documents using graph embeddings,
R. Martínez-Cruz, D. Mahata, A. J. López-López, and J. Portela, “Enhancing keyphrase extraction from long scientific documents using graph embeddings,” arXiv preprint arXiv:2305.09316 , 2023
2023 arXiv
-
[14]
Longformer: The long- document transformer,
I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,” arXiv preprint arXiv:2004.05150 , 2020
2004 arXiv
-
[15]
LDKP: A Dataset for Iden- tifying Keyphrases from Long Scientific Documents,
D. Mahata, N. Agarwal, D. Gautam, A. Kumar, S. Parekh, Y . K. Singla, A. Acharya, and R. R. Shah, “LDKP: A Dataset for Iden- tifying Keyphrases from Long Scientific Documents,” arXiv preprint arXiv:2203.15349, 2022
2022 arXiv
-
[16]
Deep keyphrase generation,
R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, and Y . Chi, “Deep keyphrase generation,” arXiv preprint arXiv:1704.06879 , 2017
2017 arXiv
-
[17]
Two huge title and keyword generation corpora of research articles,
E. Çano and O. Bojar, “Two huge title and keyword generation corpora of research articles,” arXiv preprint arXiv:2002.04689 , 2020
2002 arXiv
-
[18]
Large dataset for keyphrases extraction,
M. Krapivin, A. Autaeu, M. Marchese et al. , “Large dataset for keyphrases extraction,” 2009
2009
-
[19]
Semeval- 2010 task 5: Automatic keyphrase extraction from scientific articles,
S. N. Kim, O. Medelyan, M.-Y . Kan, and T. Baldwin, “Semeval- 2010 task 5: Automatic keyphrase extraction from scientific articles,” in Proceedings of the 5th International Workshop on Semantic Evaluation . Association for Computational Linguistics, 2010, pp. 21–26
2010
-
[20]
Keyphrase extraction in scientific publications,
T. D. Nguyen and M.-Y . Kan, “Keyphrase extraction in scientific publications,” in International conference on Asian digital libraries . Springer, 2007, pp. 317–326
2007
-
[21]
Domain-independent automatic keyphrase indexing with small training sets,
O. Medelyan and I. H. Witten, “Domain-independent automatic keyphrase indexing with small training sets,” Journal of the American Society for Information Science and Technology , vol. 59, no. 7, pp. 1026–1040, 2008
2008
-
[22]
The nlm indexing initiative
A. R. Aronson, O. Bodenreider, H. F. Chang, S. M. Humphrey, J. G. Mork, S. J. Nelson, T. C. Rindflesch, and W. J. Wilbur, “The nlm indexing initiative.” in Proceedings of the AMIA Symposium. American Medical Informatics Association, 2000, p. 17
2000
-
[23]
Text mining and cybercrime,
A. Kontostathis, L. Edwards, and A. Leatherman, “Text mining and cybercrime,” Text mining: Applications and theory , pp. 149–164, 2010
2010
-
[24]
Detecting relevant information in high- volume chat logs: Keyphrase extraction for grooming and drug dealing forensic analysis,
J. H. Alves, H. A. C. G. Pedroso, R. H. Venetikides, J. E. M. Köster, L. R. Grochocki, C. O. A. Freitas, and J. P. Barddal, “Detecting relevant information in high- volume chat logs: Keyphrase extraction for grooming and drug dealing forensic analysis,” in 2023 International C...
2023
-
[25]
Open domain web keyphrase extraction beyond language modeling,
L. Xiong, C. Hu, C. Xiong, D. Campos, and A. Overwijk, “Open domain web keyphrase extraction beyond language modeling,” arXiv preprint arXiv:1911.02671, 2019
1911 arXiv
-
[26]
Promptrank: Unsupervised keyphrase extraction using prompt,
A. Kong, S. Zhao, H. Chen, Q. Li, Y . Qin, R. Sun, and X. Bai, “Promptrank: Unsupervised keyphrase extraction using prompt,” arXiv preprint arXiv:2305.04490, 2023
2023 arXiv
-
[27]
One size does not fit all: Generating and evaluating variable number of keyphrases,
X. Yuan, T. Wang, R. Meng, K. Thaker, P. Brusilovsky, D. He, and A. Trischler, “One size does not fit all: Generating and evaluating variable number of keyphrases,” arXiv preprint arXiv:1810.05241, 2018
2018 arXiv
-
[28]
Nltk: The natural language toolkit,
E. Loper and S. Bird, “Nltk: The natural language toolkit,” arXiv preprint cs/0205028, 2002
2002 arXiv
-
[2019]
Available: http://arxiv.org/abs/1907.11692
[Online]. Available: http://arxiv.org/abs/1907.11692
1907 arXiv
-
[2020]
Available: https://doi.org/10.5281/zenodo.4461265
[Online]. Available: https://doi.org/10.5281/zenodo.4461265
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.