{"id":"00a4ae11-6d76-425b-99ef-3bec0429de4a","arxiv_id":"1908.03313","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SemanticRank combines sentence position, word frequency, and semantic role labels in a PageRank-style graph and reports modest F1 gains over five unsupervised baselines on KDD, WWW, and Inspec datasets.","lead":"This paper builds a key phrase ranking system that adds semantic role labels to the traditional PageRank graph and tests it on three scientific article datasets. A smart generalist might read it to see whether grammatical roles like subject and object can improve automatic keyword extraction without labeled training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that semantic roles drive the gains lacks an ablation: removing or randomizing the SRL factor is never tested, and the cited sensitivity table is missing.","rationale":"The paper is an accept-shaped conditional: the idea is linguistically motivated and the reported improvements on KDD/WWW at N=6/8 are present in the tables. However, the causal role of the SRL feature is not established. My concern is not that the SRL labeler is noisy (the paper honestly reports 87%/77% precision), but that even perfect SRL labels would not explain the gains unless an ablation shows the SRL term affects rankings. Because S(vi) = SP(wi)/n * SRL(wi)/n, a constant SRL factor is a uniform rescaling of node weights, which does not change PageRank order. Therefore the only way to attribute the improvement to semantic roles is to remove the SRL signal and observe a drop. The paper instead presents a sensitivity analysis over role ranks (labeled Table ??, but missing) and claims the outcome is 'practically the same,' which, if true, shows the finer role ordering does not matter but does not test whether SRL contributes at all. The comparison to PositionRank is suggestive but confounded by multiple implementation differences (window size k is never reported, graph and edge-weight details differ). Thus the central claim is under-specified and currently supported only by an indirect comparison. I agree with the reader's conditional verdict: the work is promising and the artifact should be released, but publication-quality acceptance should require the constant-SRL ablation and the sensitivity table. Since the reader already recommended CONDITIONAL, I do not move the verdict.","tokens_in":6338,"tokens_out":6207,"duration_ms":63045,"concrete_test":"Implement SemanticRank as described and run it on the KDD and WWW datasets with the SRL factor set to a constant, SRL(wi,s_j) := 1 for every candidate word, keeping all other code identical. If the Top-6 and Top-8 F1 scores are unchanged (as they would be if SRL were merely a uniform multiplier), then the reported gains are not caused by semantic role information. As part of the same artifact release, report the missing 'Table ??' with SentSRL1 and SentSRL2 results; the decisive check, however, is the constant-SRL ablation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline assertion is that semantic role information is a new useful lexical feature. The only evidence is that SemanticRank beats five baselines on KDD/WWW at N=6/8. But no baseline isolates the SRL contribution. PositionRank is closest, yet it differs from SemanticRank in graph construction, position weighting, and edge weighting as well as in the absence of SRL, so an F1 improvement over PositionRank does not identify SRL as the cause. In fact, the SRL score SRL(wi,s_j) is a normalized value in [0,1], so it acts as a multiplier on the position-frequency term SP(wi). If SRL were replaced by a constant, the final node scores S(vi) would be scaled uniformly and PageRank rankings would be unchanged. The paper never runs this constant-SRL ablation. Moreover, Section 6 references 'Table ??' showing sensitivity of results to the role ranks, but the table is absent from the submission; the text says precision/recall/F1 are 'practically the same' for KDD and WWW under different role-rank assignments. That statement, if true, further undercuts the attribution: if the exact role ordering does not matter, the gains must come from something other than the semantic-role ranking. Finally, the abstract's 'overall improvement on all the data sets' is contradicted by Table 2: on Inspec SemanticRank matches PositionRank (36.5/6.2/10.3 vs 36.5/6.2/10.2 at Top2, 29.4/14.1/18.2 vs 29.3/14.1/18.1 at Top6) and at Top2 on KDD and WWW PositionRank and MultipartiteRank outperform it. The central claim therefore rests on an untested component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SemanticRank, an unsupervised keyphrase-ranking method that combines sentence-position and term-frequency scores with semantic role label (SRL) information as a node-weight multiplier in a PageRank graph. Candidate words are selected as nouns/adjectives, connected by co-occurrence within a window, and PageRank is initialized with S(vi) = (SP(wi)/n)·(SRL(wi)/n). Key phrases are generated as adjective-noun sequences and scored by summing PageRank scores of their words. The method is evaluated on KDD, WWW, and Inspec against TextRank, SingleRank, TopicRank, PositionRank, and MultipartiteRank, reporting precision, recall, F1, MRR, and MAP. The authors claim that semantic role information can serve as a new lexical feature and that the approach achieves overall improvement over state-of-the-art baselines on all datasets.","tokens_in":6678,"tokens_out":2847,"duration_ms":31461,"significance":"If the central claim is valid, the paper introduces a linguistically motivated, unsupervised feature for graph-based keyphrase ranking that is simple and complementary to position/frequency cues. The evaluation uses external standard benchmarks (KDD, WWW, Inspec) with author-labeled gold keyphrases, and the feature weights are set by linguistic priors rather than fitted to test labels, so the circularity burden is low. However, the evidence as presented does not yet isolate the contribution of the SRL component: there is no ablation removing or randomizing the SRL factor, the cited sensitivity table is missing, the headline improvement claim is contradicted by parts of Table 2, and several experimental parameters are unreported. The core idea is plausible and potentially publishable after substantial revision, but the current manuscript does not establish that semantic-role information is the cause of the reported gains.","major_comments":[{"comment":"The central claim that semantic role information drives the observed gains is not supported by an ablation. Because S(vi) = (SP(wi)/n)·(SRL(wi)/n) and the SRL score is a normalized value in [0,1], replacing SRL(wi) with any positive constant scales all PageRank initialization scores uniformly and, for fixed PageRank parameters, leaves the final ranking unchanged. The paper never reports this constant-SRL condition or a randomized-role condition, so the comparison with PositionRank cannot isolate the SRL contribution, especially since the graph construction, edge weighting, and position weighting also differ from PositionRank.","section":"§4.3, Node Weight Assignment"},{"comment":"The sensitivity analysis referenced as \"Table ??\" is absent from the submission, and the text states that precision, recall, and F1 are \"practically the same\" for KDD and WWW under different role-rank assignments. This missing table is load-bearing: the invariance claim, if true, directly weakens the attribution of the gains to the semantic-role ranking, and without the table the reader cannot assess how much the role-order choice matters or whether the reported numbers are reproducible.","section":"§5.3, Results"},{"comment":"The abstract's claim of \"an overall improvement on all the data sets\" is not supported by Table 2. On Inspec, SemanticRank essentially ties PositionRank at Top2 (36.5/6.2/10.3 vs 36.5/6.2/10.2) and Top6, and at Top2 on KDD and WWW PositionRank or MultipartiteRank achieves higher F1. The actual support is limited to N=6 and N=8 on KDD and WWW, with KDD significance at p=0.09/0.07; the scope of the improvement claim should be revised accordingly.","section":"Abstract and Table 2"},{"comment":"The co-occurrence window size k used to construct the graph is never reported, and no sensitivity to k is given. Combined with the unablated architectural differences from PositionRank, this leaves open the possibility that the gains come from the choice of k or from the graph-construction details rather than from the SRL feature. The authors should report k, justify its value, and ideally show results across a small range of k to rule out this confound.","section":"§4.1 and §5.3, Experimental Setup"}],"minor_comments":[{"comment":"The text mentions a \"Storwize dataset\" that does not appear anywhere else in the paper; this should be removed or the dataset should be introduced and described.","section":"§5.3, Sensitivity paragraph"},{"comment":"The MRR and MAP bar chart is difficult to read and does not report numeric values; the corresponding numbers should be given in the text or in a table so that the claims can be verified.","section":"Figure 2"},{"comment":"The statistical significance testing is not described: the authors should state which test was used, whether it is paired across documents, and how the p-values for KDD and WWW were computed, given that KDD values of 0.09 and 0.07 are only marginally significant.","section":"§5.2, Experimental Setups"},{"comment":"The limitations section correctly notes that SRL is less accurate on grammatically incomplete titles, but the paper does not quantify how often titles contribute key phrases or whether SemanticRank's dependence on well-formed sentences explains its weaker Inspec results; a brief quantitative note would strengthen the limitation discussion.","section":"§6, Conclusions and Future Work"},{"comment":"The phrase \"state-of-art\" should be \"state-of-the-art,\" and the claim of improvement should be made conditional on the specific datasets and N values that Table 2 actually supports.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"I do not think this should be rejected outright, because the core idea is simple, linguistically motivated, and evaluated on external benchmarks without circular use of test labels. However, the manuscript as submitted has a load-bearing missing artifact: the sensitivity table referenced as \"Table ??\" is absent, and the central attribution to SRL is not backed by an ablation. With a proper constant-SRL ablation, a reported sensitivity analysis, corrected claims about the scope of improvement, and fuller experimental reporting, the contribution could become acceptable for a shorter paper in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea is linguistically nice, but the formal description makes the semantic-role factor weightless. In the PageRank formula in Section 4.2, the teleportation term is uniform, and initializing PR with the node weights S(.) has no effect on the stationary distribution. So, as written, SRL cannot change the outcome. That is a load-bearing flaw, not a footnote.\n\nWhat is genuinely new: combining semantic role labels with a graph-based ranker is an interesting and inexpensive feature, and the paper is honest about building on TextRank/PositionRank. The evaluation uses three standard datasets and five baselines, with P/R/F1 and MRR/MAP, and the authors include a limitations paragraph acknowledging SRL precision issues. That is good practice.\n\nThe soft spots are bigger than the reader's report suggests. The missing sensitivity table (Table ??) is not just an omission; the text in Section 6 says results are 'practically the same' across role-rank assignments, which, combined with the initialization issue, suggests the SRL component is not doing the work. The abstract claims 'overall improvement on all the data sets,' but Table 2 shows ties on Inspec and losses at Top2 on KDD/WWW. The window size k is never reported, and the KDD significance values (0.09, 0.07) are weak.\n\nCredit where due: the authors report baselines fairly and do not hide the Inspec tie. The main flaw is conceptual. If the implementation actually used S(vi) as the teleportation vector, the formula needs to say so; as it stands, the method is not reproducible and the reported gains cannot be attributed to semantic roles. The stress-test note's constant-SRL ablation would be a good check, but it is even more decisive: any constant multiplier on the initial vector does not change the stationary distribution.\n\nWho is this for? Someone working on unsupervised keyphrase extraction might find the SRL idea worth revisiting, but not as presented. This needs a corrected description or a real ablation before it deserves referee time. I would desk reject this version.","headline":"The SRL weights are inert as described because PageRank initialization cannot influence the final rank; the paper's central claim rests on a misdescribed mechanism.","tokens_in":7255,"tokens_out":5229,"would_cite":false,"duration_ms":53020,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic role knowledge, folded into PageRank as node weights, improves unsupervised keyphrase ranking on scientific abstracts.","keywords":["key phrase extraction","key phrase ranking","semantic role labeling","PageRank","unsupervised learning","sentence position","scientific articles"],"falsifier":"Replace the semantic-role score $SRL(w_i)$ in SemanticRank's initialization with a constant while keeping sentence-position scores unchanged, and re-run on KDD and WWW; if F1 at top 8 stays at 12.8 and 15.6 rather than falling toward the position-only baseline, the semantic-role component contributes nothing.","tokens_in":6115,"feed_emoji":"🏷️","tokens_out":6503,"duration_ms":63184,"temperature":0.7,"pith_summary":"SemanticRank is an unsupervised keyphrase ranking method that merges two signals for each candidate word: how early it appears in sentences and which semantic role it plays (agent, patient, location, and so on). The paper's central claim is that semantic role information, injected as the starting weights of a PageRank graph, becomes a genuinely new lexical feature for ranking key phrases. On the KDD and WWW scientific-abstract datasets the method reports higher F1 than five unsupervised baselines at top 6 and top 8, and it matches the strongest baseline on Inspec. If the claim holds, graph-based extractors no longer have to rely on surface position alone; they can use the linguistic structure of who-did-what-to-whom to decide what matters.","feed_headline":"Semantic role labels lift key phrase ranking over 5 baselines","feed_subtitle":"A word's role as agent or adjunct seeds PageRank and beats position-only ranking on KDD and WWW.","key_machinery":"The load-bearing mechanism is the node-weight function $S(v_i)=SP(w_i)/n \\times SRL(w_i)/n$, in which $SP(w_i)$ sums inverse sentence position weighted by frequency and $SRL(w_i)$ sums the semantic-role ranks of the word over the sentences where it appears, with $n$ the number of sentences containing the word. Role ranks follow the numbered argument scheme, with Arg0 highest, Arg1 next, and adjuncts lowest, encoding the intuition that agents and patients are more keyphrase-worthy than temporal or locative modifiers. This product initializes PageRank, so the graph's score flow is modulated by role information rather than by connectivity alone.","core_discovery":"The central claim is that semantic roles carry ranking signal for key phrases and that this signal survives being folded into PageRank as node initialization. Concretely, each candidate word is scored by multiplying its normalized sentence-position score by its normalized semantic-role score, and that product seeds PageRank; phrase scores are sums of the PageRank scores of their words. On KDD at top 6 and top 8, SemanticRank reaches F1 of 12.2 and 12.8, above the best baseline scores of 11.6 and 12.1; on WWW it reaches 15.0 and 15.6, above the best baselines of 13.1 and 12.3; on Inspec it is comparable to PositionRank. The paper interprets these results as evidence that semantic role knowledge can act as a new lexical feature in ranking.","pith_inferences":["The paper's sensitivity test shows nearly identical results when Arg0 and Arg1 ranks are perturbed, which suggests the ordered role ranks may matter less than the coarse argument-versus-adjunct distinction; replacing ranks with a binary indicator would isolate the mechanism.","The same role-weighted initialization could transfer to other graph-scoring tasks such as extractive summarization or entity ranking, because the ingredient is a document graph plus a linguistic prior, not a keyphrase-specific pipeline.","A title-only evaluation would likely show smaller or negative gains, since semantic-role precision drops on elliptical text; comparing title-only against abstract-only on the same datasets would test whether the feature's benefit is confined to well-formed sentences."],"forward_implications":["Any graph-based keyphrase ranker can adopt the same trick: replace its uniform or position-only PageRank initialization with role-weighted scores, without retraining or supervision.","The method can be applied to any new document collection where a semantic-role parser is available, since no labeled keyphrases are needed.","On well-formed abstract prose the role signal helps; on terse or grammatically incomplete titles it is likely to hurt, so practical deployments should apply the feature selectively.","The reported parity with PositionRank on Inspec indicates the feature does not degrade ranking where position already dominates, making it a safe addition in mixed corpora."],"supporting_citations":[{"why":"supplies the SRL implementation that produces the semantic role labels used in the node weights.","marker":"(Roth and Woodsend, 2014)"},{"why":"provides the PositionRank baseline, the candidate phrase pattern (adjective)* (noun)+, and the KDD and WWW datasets.","marker":"(Florescu and Caragea, 2017)"},{"why":"contributes TextRank and the undirected co-occurrence graph construction that SemanticRank builds on.","marker":"(Mihalcea and Tarau, 2004)"},{"why":"provides SingleRank, including edge weighting by co-occurrence count.","marker":"(Wan and Xiao, 2008)"},{"why":"provides TopicRank, the topic-based baseline that SemanticRank is compared against.","marker":"(Liu et al., 2010)"},{"why":"provides MultipartiteRank, the strongest recent baseline and the source of the MRR and MAP evaluation setup.","marker":"(Boudin, 2018)"},{"why":"defines the PageRank recursion that SemanticRank uses, with damping factor 0.85.","marker":"(Page et al., 1999)"},{"why":"supplies the Inspec dataset with controlled and uncontrolled annotated keyphrases.","marker":"(Hulth, 2003)"}],"fun_headline_variants":["Semantic roles boost key phrase ranking in PageRank","Role labels improve key-phrase ranking, study finds","Semantic role knowledge ranks key phrases better","Unsupervised key phrase ranking improved by semantic roles","Semantic roles as new lexical feature for ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking advantage rests on the semantic-role labeler being accurate on the input abstracts, and the paper reports that labeler at 87% precision in-domain, 77% out-of-domain, with worse performance on grammatically incomplete sentences—the kind of text found in many of the titles in its own datasets.","fun_headline_variants_meta":{"raw":{"variants":["Semantic roles boost key phrase ranking in PageRank","Role labels improve key-phrase ranking, study finds","Semantic role knowledge ranks key phrases better","Unsupervised key phrase ranking improved by semantic roles","Semantic roles as new lexical feature for ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000463,"raw_usage":{"total_tokens":2231,"prompt_tokens":778,"completion_tokens":1453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":394,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":394,"tokens_out":1453,"duration_ms":10694,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:17:10.245327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the semantic-role score $SRL(w_i)$ in SemanticRank's initialization with a constant while keeping sentence-position scores unchanged, and re-run on KDD and WWW; if F1 at top 8 stays at 12.8 and 15.6 rather than falling toward the position-only baseline, the semantic-role component contributes nothing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the SRL implementation that produces the semantic role labels used in the node weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides SingleRank, including edge weighting by co-occurrence count."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides TopicRank, the topic-based baseline that SemanticRank is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Inspec dataset with controlled and uncontrolled annotated keyphrases."}],"review_version":1}