{"id":"3883aaa3-129a-41eb-9d84-57d2e73798f8","arxiv_id":"2505.14149","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A section-by-section keyphrase extraction with frequency-based merging beats abstract-only and full-text inputs for a BERT-BiLSTM-CRF model, but gains are inconsistent across simpler models.","lead":"This paper tests whether splitting an academic article into sections and extracting keyphrases from each section separately, then merging the results, yields better keyphrases than using the abstract or the full text. On a new dataset of 5,316 papers across three fields, the merging approach beats the abstract baseline for a BERT-based model, but the gains are uneven across simpler models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CS-vs-FT advantage may be an artifact of BERT length handling: the paper never specifies how full text is encoded within the 512-token limit, so section splitting plus ensembling may simply be a better long-document strategy. A chunked-FT control is needed.","rationale":"The reader's weakest assumption about the undefined SSC split in Section 3.2 is a valid gap, but it mostly threatens Section 5.3, a secondary result about classification quality. The most load-bearing claim is the central CS-versus-FT comparison for BERT-BiLSTM-CRF, which is the basis of the abstract's 'integration yields best performance' statement. A standard BERT model cannot ingest a full academic article, and the paper is silent on how this is handled. If FT is truncated while section inputs are encoded separately, the central comparison is confounded by long-document segmentation rather than section structure. This is directly checkable from the released code and from a fixed-chunk control. The reader's conditional verdict remains appropriate; if the proposed test shows that a chunked-FT baseline matches CS, the verdict should move toward rejection or substantial weakening. I therefore keep the reader's CONDITIONAL verdict unchanged while identifying a different, more central concern than the SSC definition.","tokens_in":29644,"tokens_out":11256,"duration_ms":112160,"concrete_test":"Run BERT-BiLSTM-CRF on Corpus-PH with three settings: (1) the released GitHub code as-is, recording the actual max_seq_length and FT truncation behavior; (2) a chunked full-text baseline CT that splits each article into fixed 512-token chunks with 128-token overlap and applies the same Table 9 integration algorithm to per-chunk keyphrase lists; and (3) a long-context encoder for FT (e.g., Longformer or a sliding-window SciBERT) if feasible. If CT approximately matches CS on F1@3/5/10 across the three domains, the section-structure claim collapses to a segmentation effect. If CS significantly outperforms CT, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison (CS versus AB and FT) for the paper's best model, BERT-BiLSTM-CRF, is only valid if FT and section-level inputs are encoded equivalently. Academic full texts vastly exceed BERT's 512-token limit, yet Section 4.2.3 does not state whether FT is truncated, chunked, or processed with a long-context model. If FT is truncated to 512 tokens while each of the five section inputs is separately encoded and then merged by the Table 9 algorithm, CS has an unfair advantage: it effectively receives the whole document in five pieces, whereas FT receives only a prefix. The observed 'CS > FT' result would then demonstrate the benefit of input segmentation and ensembling, not of section structure per se. The WS and SS variants reuse the same section boundaries and therefore cannot serve as non-semantic chunking controls. Section 6.2 lists limitations but never mentions input-length handling, and only the released code would reveal the truncation policy. A second reproducibility gap is that N in Step 1 of Table 9 is never specified, so even a code-independent replication cannot currently be run without guessing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes using section structure information from academic full texts to improve keyphrase extraction (KPE). It contributes a section-annotated corpus of 5,316 papers from biomedical science, computer science, and library/information science; a rule-based and machine-learning section classification pipeline; experiments that add seven structural features to SVM and CRF models; and a keyphrase integration algorithm (CS) that merges keyphrase lists from five section-based inputs. The authors report that structural features improve KPE, that CS outperforms abstract-only and full-text inputs, and that the quality of section structure recognition affects KPE performance. Code and data are promised at a GitHub repository.","tokens_in":29911,"tokens_out":7969,"duration_ms":75641,"significance":"If the claims are substantiated, the paper offers a practical recipe for section-aware KPE and a reusable multi-domain corpus. Strengths include the large corpus with a reported annotation Kappa of 0.938, evaluation across six KPE models and three datasets, statistical testing, and open code/data. However, several load-bearing details are currently missing: the definition of the high/low section-structure split, the source of model annotations, the handling of BERT's length limit, and the value of the integration parameter N. In addition, one headline comparison is contradicted by the paper's own Table 12. With revision, the empirical core is likely salvageable, but the current text does not fully support the paper's central claims.","major_comments":[{"comment":"Section 5.2.2 states that CS outperforms Zhang's KPE results across all models except for SVM, but Table 12 contradicts this. On TF*IDF, CS is below Zhang in five of nine cells (e.g., PMC F1@3 14.30 vs. 18.48), and on TextRank two IEEE cells are lower (e.g., F1@3 9.99 vs. 11.94). On SVM, CS is lower than Zhang in eight of nine cells (e.g., LIS F1@3 17.41 vs. 21.82), so SVM is not the only exception. The consistently superior cases are BiLSTM-CRF and BERT-BiLSTM-CRF. Please restrict the claim to the model/metric conditions that actually support it, or report pairwise significance tests for the full table.","section":"§5.2.2 / Table 12"},{"comment":"The split into Corpus-PH, Corpus-MH, and Corpus-ML relies on 'high SSC' and 'low SSC' articles, but the abbreviation SSC is never expanded and no scoring method, threshold, or operational definition is provided. Consequently, the construction in Table 5 and the Section 5.3 conclusion that 'the classification quality of section structure can affect KPE performance' are not traceable. In addition, Section 4.1 says that a machine learning approach is used for automatic annotation but does not state which of the models evaluated in Table 10 generated the annotations for Corpus-MH and Corpus-ML. Please define SSC and report the exact annotation procedure.","section":"§3.2 / §5.3"},{"comment":"For BERT-BiLSTM-CRF, the manuscript never states how the full text (FT) is encoded within BERT's 512-token limit, while section-level inputs are comparatively short. If FT is truncated to the first 512 tokens but the section-level inputs IN/RW/MD/ER/DC are processed separately and then integrated, the higher CS scores could reflect access to more of the document rather than the section structure itself. Section 6.2 does not mention this length-handling issue. Please specify the truncation/chunking policy and, if truncation was used, add a chunked-full-text control that applies the same integration algorithm to arbitrary chunks of the full text.","section":"§4.2.3 / §5.2.2"},{"comment":"Step 1 of the keyphrase integration algorithm selects 'N words in the top rank' but N is never stated anywhere in the manuscript. Also, Table 11 defines WS, SS, and CS as integrating 'according to the number of words, sentences and sections,' yet the algorithm in Table 9 mentions only word-frequency and importance ranking, not word/sentence/section counts. The method is therefore not reproducible without guessing the parameter and the weighting scheme. Please state N, define WS/SS/CS precisely, and report sensitivity of the headline result to N.","section":"Table 9 / Table 11"}],"minor_comments":[{"comment":"The phrase 'state-of-the-art PLM-based PKE models' should read 'KPE models'; the abbreviation PKE is not used elsewhere.","section":"§2.1"},{"comment":"The row for RW is glossed as 'Title and Introduction of the article,' which duplicates IN; from Figures 4-6, RW should be 'Title and Related Work.'","section":"Table 11"},{"comment":"The text says 'Refer to Section 4.4 for the details on probability value calculations,' but the manuscript has no Section 4.4; the relevant description appears in Section 4.2.3.","section":"§5.1"},{"comment":"The reference list gives Bao et al. (2025), but the text cites 'Bao et al. (2024)'; the year should be aligned.","section":"References"},{"comment":"The sentence reporting that BERT-BiLSTM-CRF achieves 'an average improvement of 4.79%' does not state the baseline for that improvement; please specify whether it is relative to Zhang et al. (2022), to AB, or to another comparison.","section":"§5.2.2"},{"comment":"There is an incomplete citation in the sentence 'Most studies are dedicated to enhancing KPE performance through model optimization techniques (Nguyen & Luong, 2010; Zhang et al.)'; the Zhang et al. citation lacks a year.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript uses Zhang et al. (2022) as a baseline despite overlapping authorship. This is not inherently problematic, but the comparison should be framed as a replication/self-comparison and the shared authorship disclosed. The GitHub availability is a clear positive. The main risk to the central claim is the unstated BERT length handling; if FT was truncated while section inputs were not, the CS-versus-FT result may be an artifact of document segmentation rather than of section structure. The revision should make the encoding explicit and ideally add the chunked control."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline result—section-level keyphrase integration beats abstract- and full-text inputs—is plausibly real for the best model, but the paper overclaims its comparison with Zhang et al. and leaves three gaps that block a clean interpretation.\n\nWhat's genuinely useful: the new 5,316-paper section-annotated corpus (PMC, LIS, IEEE) with released code and data; the integration heuristic in Table 9 is simple and re-implementable; the structural feature ablation (F11–F17) is systematic; evaluation spans six KPE models across three domains and includes significance tests. That is solid, useful work for anyone doing KPE on scholarly text.\n\nSoft spots, in order of severity.\n\nFirst, the text says CS outperforms Zhang et al. “across all models except for SVM,” but Table 12 contradicts that. On TF*IDF, Zhang beats CS on five of nine cells; on TextRank, CS loses two of nine; on CRF, CS loses three of nine. The qualified claim—CS wins cleanly for BiLSTM-CRF and BERT-BiLSTM-CRF—is true, but the overstatement needs to be fixed.\n\nSecond, the high/low section-structure split (Corpus-MH vs Corpus-ML) is never operationalized. Section 3.2 says the split is based on “preliminary investigations” but gives no scoring method or threshold. Section 5.3's central conclusion—that section-structure classification quality affects KPE—rests entirely on that split. Without a definition, the finding is unsubstantiated.\n\nThird, the stress-test concern about BERT length handling is valid. The paper never states how full text is encoded within BERT's 512-token limit. If FT is truncated to 512 tokens while each of the five section inputs is separately encoded and then merged, the CS advantage could simply reflect input segmentation, not section semantics. WS and SS reuse the same section boundaries, so they are not non-semantic chunking controls. A chunked-full-text baseline is needed.\n\nMinor but related: N in Step 1 of the integration algorithm is never specified, so replication requires guessing; and there are no variance estimates or repeated runs.\n\nAudience: researchers working on keyphrase extraction from scholarly documents, and people building section-aware NLP resources. The corpus alone makes it worth engaging.\n\nRecommendation: send it to peer review, but require the authors to clarify the FT encoding, define the SSC split, correct the overclaim, and release N and variance. After those fixes, the core comparison for BERT-BiLSTM-CRF is likely to hold, and the corpus is a useful contribution regardless.","headline":"Useful new corpus and a plausible integration heuristic, but the paper overstates its comparison with Zhang et al. and leaves the BERT length handling and the SSC split undefined.","tokens_in":30415,"tokens_out":3611,"would_cite":true,"duration_ms":44605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:38:51.642591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}