Pith. sign in

REVIEW 4 major objections 6 minor 4 references

Enhancing Keyphrase Extraction from Academic Articles Using Section Structure Information

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A section-by-section keyphrase extraction with frequency-based merging beats abstract-only and full-text inputs for a BERT-BiLSTM-CRF model, but gains are inconsistent across simpler models.

desk verdict Useful new corpus and a plausible integration heuristic, but the paper overstates its comparison with Zhang et al. and leaves the BERT length handling and the SSC split undefined. read the letter →

arxiv 2505.14149 v1 pith:OKFL7WJE submitted 2025-05-20 cs.CL cs.DLcs.IR

Chengzhi Zhang , Xinyi Yan , Lei Zhao , Yingyi Zhang This is my paper · ORCID
classification cs.CLcs.DLcs.IR
keywords academicsectionkeyphraseperformancearticlesstructureextractionfeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Keyphrase extraction is the task of picking the most important words or phrases from a paper, usually to help search engines and researchers find it. Most systems read only the title and abstract, because the full text contains too much unrelated material. The authors of this paper try a middle path: split the full text into sections (introduction, methods, results, etc.), extract keyphrases from each section separately, and then merge the shortlists into one final list. They test this on about 5,300 papers from biomedicine, computer science, and library science.

The merge is simple. From each section's candidate list, they keep a fixed number of top phrases, combine all the kept phrases, and sort them first by how often a phrase appeared across sections and then by its importance score. They call the final list CS. Across six extraction models, CS often beats using only the title and abstract, and usually beats feeding in the entire raw full text. The best result comes with a BERT-based sequence labeling model, where CS improves F1 by about 4.8% on average over the next best prior method.

The paper also studies whether the quality of the section labels matters. It compares corpora labeled by humans, by a model on clearly structured papers, and by a model on vaguely structured papers. The authors report that label quality affects performance, but the way they define clear versus vague structure is not spelled out. The numerical comparisons to prior work are also mixed: the CS approach does not beat a 2022 baseline for every model, even though the text claims it does.

Extended reading notes

Core claim

The keyphrase integration approach (CS), which merges section-level keyphrase lists by frequency and importance, yields the best overall KPE performance compared to abstract-only (AB) and full-text (FT) inputs, and outperforms prior methods on the BERT-BiLSTM-CRF model. The paper states in the abstract: 'The keyphrase integration approach yields the best performance.' Section 5.2.2 further claims that CS outperforms AB and FT across various domain datasets and evaluation metrics.

Load-bearing premise

The operationalization of 'section structure clarity' (SSC) used to split Corpus-MH from Corpus-ML is never defined. Section 3.2 asserts the split without specifying the scoring method or threshold, and Section 4.1 does not explain which classifier produced the model annotations for these corpora. If this split is arbitrary, the Section 5.3 finding that classification quality affects KPE is unsubstantiated.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes using section structure information from academic full texts to improve keyphrase extraction (KPE). It contributes a section-annotated corpus of 5,316 papers from biomedical science, computer science, and library/information science; a rule-based and machine-learning section classification pipeline; experiments that add seven structural features to SVM and CRF models; and a keyphrase integration algorithm (CS) that merges keyphrase lists from five section-based inputs. The authors report that structural features improve KPE, that CS outperforms abstract-only and full-text inputs, and that the quality of section structure recognition affects KPE performance. Code and data are promised at a GitHub repository.

Significance. If the claims are substantiated, the paper offers a practical recipe for section-aware KPE and a reusable multi-domain corpus. Strengths include the large corpus with a reported annotation Kappa of 0.938, evaluation across six KPE models and three datasets, statistical testing, and open code/data. However, several load-bearing details are currently missing: the definition of the high/low section-structure split, the source of model annotations, the handling of BERT's length limit, and the value of the integration parameter N. In addition, one headline comparison is contradicted by the paper's own Table 12. With revision, the empirical core is likely salvageable, but the current text does not fully support the paper's central claims.

major comments (4)
  1. [§5.2.2 / Table 12] Section 5.2.2 states that CS outperforms Zhang's KPE results across all models except for SVM, but Table 12 contradicts this. On TF*IDF, CS is below Zhang in five of nine cells (e.g., PMC F1@3 14.30 vs. 18.48), and on TextRank two IEEE cells are lower (e.g., F1@3 9.99 vs. 11.94). On SVM, CS is lower than Zhang in eight of nine cells (e.g., LIS F1@3 17.41 vs. 21.82), so SVM is not the only exception. The consistently superior cases are BiLSTM-CRF and BERT-BiLSTM-CRF. Please restrict the claim to the model/metric conditions that actually support it, or report pairwise significance tests for the full table.
  2. [§3.2 / §5.3] The split into Corpus-PH, Corpus-MH, and Corpus-ML relies on 'high SSC' and 'low SSC' articles, but the abbreviation SSC is never expanded and no scoring method, threshold, or operational definition is provided. Consequently, the construction in Table 5 and the Section 5.3 conclusion that 'the classification quality of section structure can affect KPE performance' are not traceable. In addition, Section 4.1 says that a machine learning approach is used for automatic annotation but does not state which of the models evaluated in Table 10 generated the annotations for Corpus-MH and Corpus-ML. Please define SSC and report the exact annotation procedure.
  3. [§4.2.3 / §5.2.2] For BERT-BiLSTM-CRF, the manuscript never states how the full text (FT) is encoded within BERT's 512-token limit, while section-level inputs are comparatively short. If FT is truncated to the first 512 tokens but the section-level inputs IN/RW/MD/ER/DC are processed separately and then integrated, the higher CS scores could reflect access to more of the document rather than the section structure itself. Section 6.2 does not mention this length-handling issue. Please specify the truncation/chunking policy and, if truncation was used, add a chunked-full-text control that applies the same integration algorithm to arbitrary chunks of the full text.
  4. [Table 9 / Table 11] Step 1 of the keyphrase integration algorithm selects 'N words in the top rank' but N is never stated anywhere in the manuscript. Also, Table 11 defines WS, SS, and CS as integrating 'according to the number of words, sentences and sections,' yet the algorithm in Table 9 mentions only word-frequency and importance ranking, not word/sentence/section counts. The method is therefore not reproducible without guessing the parameter and the weighting scheme. Please state N, define WS/SS/CS precisely, and report sensitivity of the headline result to N.
minor comments (6)
  1. [§2.1] The phrase 'state-of-the-art PLM-based PKE models' should read 'KPE models'; the abbreviation PKE is not used elsewhere.
  2. [Table 11] The row for RW is glossed as 'Title and Introduction of the article,' which duplicates IN; from Figures 4-6, RW should be 'Title and Related Work.'
  3. [§5.1] The text says 'Refer to Section 4.4 for the details on probability value calculations,' but the manuscript has no Section 4.4; the relevant description appears in Section 4.2.3.
  4. [References] The reference list gives Bao et al. (2025), but the text cites 'Bao et al. (2024)'; the year should be aligned.
  5. [§5.2.2] The sentence reporting that BERT-BiLSTM-CRF achieves 'an average improvement of 4.79%' does not state the baseline for that improvement; please specify whether it is relative to Zhang et al. (2022), to AB, or to another comparison.
  6. [§2.2] There is an incomplete citation in the sentence 'Most studies are dedicated to enhancing KPE performance through model optimization techniques (Nguyen & Luong, 2010; Zhang et al.)'; the Zhang et al. citation lacks a year.
Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's empirical claims rest on a hand-built section classifier, a heuristic merge rule, and a new corpus; no new physical or ontological entities are introduced.

free parameters (2)
  • N (top keyphrases per section in integration algorithm) = not reported
    Step 1 of the integration algorithm in Section 4.2.3 selects N top-ranked keyphrases from each section; the value of N is never stated in the paper.
  • High or low SSC split threshold = not defined
    Section 3.2 assigns papers to Corpus-MH versus Corpus-ML based on 'high SSC' and 'low SSC' without specifying the scoring function or threshold.
assumptions (3)
  • domain assumption Author-provided keywords are the gold standard for keyphrase evaluation
    Section 5.1 uses author keyphrases as Y_gold to compute F1@K; if these are noisy or incomplete, the evaluation favors methods that reproduce author indexing behavior.
  • domain assumption All three corpora conform to the IMRaD section structure
    Section 4.1 states the classification framework follows IMRaD and fits the corpus well; LIS articles may deviate, acknowledged in Section 5.2 as a reason for lower performance.
  • domain assumption Exact string matching after stemming is a sufficient similarity measure
    Section 5.1 evaluates with exact match on stems; semantically related keyphrases are counted as misses, which the case study in Section 5.4 tries to address qualitatively.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Keyphrase Extraction from Academic Articles Using Section Structure Information." pith.science (2026). https://pith.science/paper/OKFL7WJE

@misc{pith2026250514149,
  author       = {Pith},
  title        = {Pith review of: Enhancing Keyphrase Extraction from Academic Articles Using Section Structure Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKFL7WJE}},
  note         = {Machine review of arXiv:2505.14149}
}
read the original abstract

The exponential increase in academic papers has significantly increased the time required for researchers to access relevant literature. Keyphrase Extraction (KPE) offers a solution to this situation by enabling researchers to efficiently retrieve relevant literature. The current study on KPE from academic articles aims to improve the performance of extraction models through innovative approaches using Title and Abstract as input corpora. However, the semantic richness of keywords is significantly constrained by the length of the abstract. While full-text-based KPE can address this issue, it simultaneously introduces noise, which significantly diminishes KPE performance. To address this issue, this paper utilized the structural features and section texts obtained from the section structure information of academic articles to extract keyphrase from academic papers. The approach consists of two main parts: (1) exploring the effect of seven structural features on KPE models, and (2) integrating the extraction results from all section texts used as input corpora for KPE models via a keyphrase integration algorithm to obtain the keyphrase integration result. Furthermore, this paper also examined the effect of the classification quality of section structure on the KPE performance. The results show that incorporating structural features improves KPE performance, though different features have varying effects on model efficacy. The keyphrase integration approach yields the best performance, and the classification quality of section structure can affect KPE performance. These findings indicate that using the section structure information of academic articles contributes to effective KPE from academic articles. The code and dataset supporting this study are available at https://github.com/yan-xinyi/SSB_KPE.

Figures

Figures reproduced from arXiv: 2505.14149 by the authors.

Figure 1
Figure 1. Framework of this study The framework of this study comprised following four steps. Firstly, a section structure classification framework was established by referencing the IMRaD (Introduction, Methods, Results, and Discussion) (Sollaci & Pereira, 2004) structure of academic papers, which fit the corpus of our study well. In order to deeply examine the impact of the quality of section structure recognition on KPE’s … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 canonical work pages

  1. [1]

    Ajallouda, L., Zellou, A., Ettahiri, I., & Doumi, K. (2022). Keyphrases extraction: Approach Based on Document Paragraph Weights. 2022 International Conference on Computational Modelling, Simulation and Optimization (ICCMSO), 214–219. https://doi.org/10.1109/ICCMSO58359.2022.00051 Al-Zaidy, R. A., Caragea, C., & Lee Giles, C. (2019). Bi-LSTM-CRF sequence ...

    work page arXiv 2022

  2. [8]

    https://doi.org/10.14569/IJACSA.2020.0110857 Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 1746 –1751. https://doi.org/10.3115/v1/D14-1181 Kontoulis, C. G., Papagiannopoulou, E., & Tsoumakas, G. (2021). Keyphrase Extraction from Scienti...

    work page arXiv 2014

  3. [379]

    https://ieeexplore.ieee.org/abstract/document/6511911 Bennani-Smires, K., Musat, C., Hossmann, A., Baeriswyl, M., & Jaggi, M. (2018). Simple unsupervised keyphrase extraction using sentence embeddings. arXiv preprint arXiv:1801.04470. Bornmann, L., & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publicatio...

    work page arXiv 2018

  4. [1180]

    Zhang, C., Zhao, L., Zhao, M., & Zhang, Y. (2022). Enhancing keyphrase extraction from academic articles with their reference information. Scientometrics, 127(2), 703–731. https://doi.org/10.1007/s11192-021-04230- 4 Zhang, K., Xu, H., Tang, J., & Li, J. (2006). Keyword extraction using support vector machine. Lecture Notes in Computer Science (Including S...

    work page Pith review arXiv doi:10.48550/arxiv.2004.13826 2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.