{"id":"80a85af9-c7ac-424c-a9bf-c8138af8830a","arxiv_id":"1908.07818","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The replication shows that log-odds ratio, grammatical, and positional features improve keyphrase extraction, but annotators favor the rarest terms over mid-frequency ones.","lead":"This report replicates the keyword extraction experiments from a 2012 study, applying them to a different corpus with crowd-sourced annotations. It confirms that multi-word phrases, log-odds features, and part-of-speech/positional features help, while finding that annotators prefer the rarest terms rather than mid-frequency terms.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hypothesis 2 was narrowed to log-odds after results were known, so the replication confirms a post-hoc hypothesis rather than the original probabilistic-features claim.","rationale":"The reader identified reconstruction uncertainty as the weakest assumption. My concern is different: even with perfect reconstruction, the second hypothesis was redefined after the experimental results were known, so the confirmation is partly an artifact of hypothesis selection. This is a load-bearing threat to the claim that 'all three hypotheses from the original study are true.' It does not overturn the paper entirely; Hypotheses 1 and 3 are supported, and the report is transparent about its refinements. However, it strengthens the case for a conditional verdict that explicitly requires the authors to distinguish original hypotheses from post-hoc refinements and to supply quantitative AUC comparisons. The reader's verdict of CONDITIONAL remains appropriate, hence UNCHANGED.","tokens_in":13401,"tokens_out":4942,"duration_ms":48171,"concrete_test":"Re-run the Section 4.2 comparison using the original broad hypothesis stated in Section 2: compare the group of probabilistic features (G2, BM25, weighted log-odds) against log(tf) and tfidf, reporting AUC values with standard errors from repeated negative-example sampling. If G2 and BM25 fail to beat the baselines, the original Hypothesis 2 is not confirmed, and the report should be re-verdict as a partial replication: Hypotheses 1 and 3 hold, but Hypothesis 2 in its original form does not.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that all three hypotheses replicate is least secure for Hypothesis 2, not primarily because of reconstruction guesses, but because the hypothesis itself was changed after seeing the data. Section 2 states the initial hypothesis as 'Probabilistic measures such as G2 outperform raw term frequency and tfidf,' and the report says it was 'refined and qualified ... in light of ... our experimental findings.' The final hypothesis replaces the class of probabilistic measures with a single measure: 'Log-odds Ratio outperforms raw term frequency and tfidf.' Section 4.2 then reports exactly that G2 and BM25 are 'much worse — sometimes even worse than log(tf),' while only weighted log-odds is competitive, and uses this to validate the second part of the final hypothesis. Thus the replication confirms a hypothesis that was selected because it held; it does not test the original broader claim. If the original claim is the target, the evidence actually points toward a partial refutation: of the three probabilistic measures tested, two do not outperform the baselines. This is a correctness risk for the headline conclusion, independent of whether the original feature definitions were reconstructed faithfully.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a replication report for the keyphrase-extraction component of Chuang, Manning, and Heer (2012). The author replaces the unavailable Stanford thesis abstracts with the SemEval 2010 training set, uses Inspec as the background corpus, collects keyphrase annotations from Amazon Mechanical Turk, builds logistic-regression classifiers with frequency, commonness, grammatical, and positional features, and evaluates them with precision-recall curves. The report states three final hypotheses: bigrams are the most frequent human-selected keyphrase length; weighted log-odds ratio outperforms raw term frequency and tf-idf; part-of-speech and positional features improve performance while parse-tree features add little. It also reports exploratory analyses of phrase length, extractive/abstractive behavior, first-sentence placement, and commonness, and it experiments with commonness binning. The paper concludes that all three parts of the final hypothesis are true on the author's dataset, annotations, features, and models.","tokens_in":13641,"tokens_out":7547,"duration_ms":70190,"significance":"If the claims were fully supported, this would be a useful replication case study: the original data and code are unavailable, and the report carefully documents reconstruction decisions, releases code and data, and honestly records discrepancies, including the failure of G2 and BM25 and the rarest-term commonness finding. The reproducibility-oriented presentation is a genuine strength. However, the significance is limited by two issues: the second hypothesis is refined after seeing the results, and the central comparative claims lack uncertainty quantification. Because of these, the report currently reads more as a new evaluation under a different protocol than as a confirmation of the original hypotheses. With appropriate reframing and additional statistics, the manuscript could be a valuable contribution to the replication literature.","major_comments":[{"comment":"Hypothesis 2 is changed after the results are known. The initial hypothesis states that 'probabilistic measures such as G2 outperform raw term frequency and tfidf', while the final hypothesis narrows this to 'Log-odds Ratio outperforms raw term frequency and tfidf'. Section 4.2 then reports that G2 and BM25 are 'much worse — sometimes even worse than log(tf)', so only the measure that survived is used to validate the final hypothesis. This is post-hoc hypothesis selection: the claim is not a test of the original probabilistic-features hypothesis but a statement that one selected measure works. In fact, the G2 and BM25 results partially refute the broader original claim. The authors should either test the original formulation as a family of measures with appropriate multiple-comparison handling, or explicitly reframe the contribution as identifying that weighted log-odds, but not the other probabilistic measures, improves on raw frequency and tf-idf.","section":"Section 2 and Section 4.2"},{"comment":"The core comparative claims are supported only by visual inspection of precision-recall curves. No numerical AUPRC values, confidence intervals, or significance tests are reported, and the negative examples are randomly sampled (Section 3.2), so the rankings could vary with the random seed. Without uncertainty quantification, statements such as 'log-odds ratio outperforms log(tf) and tfidf' and 'parse-tree-based features lead to very little additional improvement' cannot be evaluated. Please report AUPRC point estimates with standard errors or bootstrap confidence intervals, and preferably paired comparisons across the same train/test splits.","section":"Section 4.2, Figures 5–8"},{"comment":"The replication changes the foreground corpus (SemEval 2010 instead of Stanford thesis abstracts), the background corpus (Inspec instead of Stanford theses/Web1T), the annotator population (Mechanical Turk instead of the original judges), and several feature definitions are explicitly described as 'educated guesses' because 'many details were lost'. The text acknowledges this in Section 3.1, but Section 4 still states that the hypotheses are 'indeed true, on our dataset, annotations, features, and models.' The qualification is important, yet the title and conclusion frame the work as a replication of Chuang et al. The conclusions should be explicitly scoped as a new evaluation under a different protocol; as written, the manuscript overclaims confirmation of the original hypotheses.","section":"Section 3.1 and Section 4"},{"comment":"The first hypothesis is supported only weakly by the replicated data. The replication reports 24.5% bigrams, 21.4% unigrams, and 13.6% trigrams, compared with 43%, 25%, and 19% in the original study. The bigram peak is only 3.1 percentage points above the unigram peak, so without confidence intervals the claim that 'bigrams are most frequently chosen by humans' is fragile. The authors should quantify the uncertainty of these proportions (e.g., bootstrap over Turkers or assignments) and report whether the bigram-over-unigram ordering is stable.","section":"Section 4.1, Figure 1"}],"minor_comments":[{"comment":"The weighted log-odds ratio formula is garbled in the typeset version; the numerator appears to contain identical terms (t'_Doc/t'_Doc and T'_Doc/T'_Doc). Please provide an unambiguous notation, for example log((t_Doc+0.01)/(t_Ref+0.01)) − log((T_Doc+0.01)/(T_Ref+0.01)) divided by the appropriate standard error.","section":"Table 2"},{"comment":"The spurious-phrase filter is manually constructed from the responses. Although it removes only 19 phrases (17,640 to 17,621), the construction is subjective; please document the criteria more precisely and, if possible, show that the main results are robust to including or excluding this filter.","section":"Section 3.1 and Table 1"},{"comment":"The report infers that the original study used supervised ranking via a regression model rather than classification; the text says this 'could not be confirmed'. Please clarify whether this inference changes the interpretation of the term 'logistic regression' and whether it affects comparability with the original.","section":"Section 3.3"},{"comment":"The replicated histogram is plotted over assignments while the original is apparently plotted over documents; the text notes this, but the comparison of the five-phrase peak is still not apples-to-apples. Please plot both on the same unit or add a more prominent caveat to the figure caption.","section":"Section 4.1, Figure 2"},{"comment":"The statement that the original authors 'did miss a point' is too strong, because the difference between the commonness histograms is likely driven by the different background corpus (Inspec vs. Web1T) and by the Web1T frequency floor. Please soften this to a corpus-dependent observation.","section":"Section 4.1, commonness discussion"}],"recommendation":"major_revision","confidential_remarks":"This is an arXiv preprint/project report rather than a standard journal article, and the fit with a journal venue depends on whether replication reports are in scope. The author's own text acknowledges that many original details are unrecoverable and that several decisions were educated guesses; combined with the post-hoc refinement of Hypothesis 2, the editor may wish to weigh whether the contribution meets the journal's standard for empirical work. The code/data release and the honest reporting of negative results are strengths that argue for giving the authors a chance to revise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuinely careful replication study, and the author deserves credit for shipping code, data, and a frank account of every guess made along the way. But the headline claim that all three hypotheses hold needs a caveat: the second one is confirmed only after the hypothesis was narrowed to the specific measure that worked.\n\nWhat's new here is modest but real. On a different corpus (SemEval abstracts) and with MTurk annotators, the author re-examines the keyphrase length distribution, the value of probabilistic vs. frequency features, and the contribution of grammatical and positional features. Bigrams are still the most common choice, though the peak is lower; POS and positional features help; parse-tree features add little. The commonness analysis is interesting because it goes against the original: with the Inspec background corpus, annotators prefer the rarest terms, not mid-commonness, and the author shows this is partly an artifact of the Web1T frequency threshold. Binning commonness doesn't help. These are useful, reproducible observations.\n\nThe soft spots are also real. The biggest is the post-hoc fix. The initial hypothesis said probabilistic measures like G2 outperform raw term frequency and tfidf. The final hypothesis says log-odds ratio does. The paper admits this refinement came after seeing experimental findings, and the results show G2 and BM25 are worse than log(tf). So the replication validates a hypothesis selected post hoc; it does not confirm the original broad claim, and the evidence actually points toward partial refutation of that claim. Second, the original protocol was partly reconstructed from memory; the author is honest about this, but it limits how strongly we can read the comparison. Third, the spurious-phrase filter was built by hand from Turker responses, which introduces possible selection bias. Fourth, there are no confidence intervals or significance tests for the AUROC comparisons, so observed differences could be noise.\n\nFor all that, I think this paper deserves a serious referee. It is exactly the kind of replication we need more of: reproducible, transparent, and willing to report contradictions rather than smooth them over. The right verdict is probably 'revise' – make the post-hoc nature of Hypothesis 2 explicit in the abstract, add uncertainty measures where possible, and report the manual filter's effect. For a reader, it is worthwhile if you work on keyphrase extraction or on replication methodology in HCI/NLP. I would bring it to a reading group, but with the caveats above.","headline":"A transparent, carelessly framed replication that confirms the bigram and feature-ablation findings but only validates Hypothesis 2 by narrowing it after the results were known.","tokens_in":14101,"tokens_out":3322,"would_cite":false,"duration_ms":32325,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This replication study confirms, on a different corpus with crowdsourced labels, that bigrams are the most common keyphrases, log-odds ratio beats tfidf, and part-of-speech features help more than parse trees.","keywords":["keyphrase extraction","replication study","log-odds ratio","precision-recall curve","logistic regression","commonness","part-of-speech features","text visualization"],"falsifier":"Obtaining the original study's feature-extraction code, or a complete feature specification, and re-running the five logistic-regression models on the original corpus: if weighted log-odds ratio then fails to beat tfidf in area under the precision-recall curve, the paper's second hypothesis is false.","tokens_in":13225,"feed_emoji":"🔑","tokens_out":9304,"duration_ms":84284,"temperature":0.7,"pith_summary":"This report repeats the keyphrase-extraction study and asks whether its three findings hold when the experiment is run on a different corpus of scientific abstracts with crowd-sourced annotations. The report's answer is yes on all three: bigrams are the most frequent keyphrase length, weighted log-odds ratio outscores raw term frequency and tfidf, and part-of-speech and positional features improve ranking while parse-tree features add little. The practical stake is that a good keyphrase ranker can be built from cheap features and a modest background corpus. The report also argues that the original study's mid-commonness preference is an artifact of the large web corpus, since annotators on a smaller corpus concentrate on the rarest terms.","feed_headline":"Three keyphrase claims hold up in replication","feed_subtitle":"Bigrams, log-odds scoring, and part-of-speech features win again on a new corpus with crowdsourced labels.","key_machinery":"The load-bearing apparatus is a controlled feature-ablation comparison: five logistic-regression models trained on the same 7,816 positive examples and 78,160 down-weighted negative n-grams, adding one feature family at a time, with performance measured as area under the precision-recall curve. The feature families are frequency and probabilistic scores (including the weighted log-odds ratio, a smoothed contrast between a phrase's document frequency and its background frequency), commonness against a background corpus, binary grammatical features from part-of-speech patterns and parse trees, and three positional features. The relative ordering of the resulting curves is the identity that carries the argument: it is exactly what the second and third hypotheses assert.","core_discovery":"On the report's own terms, the discovery is that a full replication with different data reproduces the original study's three-part hypothesis. The author builds five logistic-regression models and compares them by area under the precision-recall curve. Weighted log-odds ratio outscores log(tf) and tfidf; adding part-of-speech features improves the curve; adding parse-tree features on top changes it little; adding positional features improves it further. In the crowd-sourced annotations, bigrams are the most common keyphrase length, followed by unigrams and then trigrams. The study also finds a divergence from the original: on the smaller background corpus, annotators prefer the rarest terms, and the original mid-commonness peak reappears only after rare terms are removed.","pith_inferences":["If the commonness finding generalizes, the original mid-commonness peak is an artifact of removing rare terms from a very large background corpus; a direct test would hold annotators fixed and vary only the background corpus.","Because negative examples are sampled at ten times the positive rate and down-weighted, the reported AUC differences mostly reflect ranking at low false-positive rates; re-running with different ratios would test whether the feature ordering is stable.","The small gain from parse trees suggests practical keyphrase systems can skip syntactic parsing, though the present report does not itself argue this for modern systems."],"forward_implications":["Weighted log-odds ratio is a stronger baseline for ranking keyphrases on scientific abstracts than raw term frequency or tfidf.","Grammatical features improve ranking mainly through part-of-speech information; parse-tree features can be dropped once part-of-speech features are present.","Candidate-generation strategies should emphasise bigrams because they are the most frequent human-chosen keyphrase length.","Positional features give a further boost to the best model, so first-occurrence information is worth keeping.","Commonness patterns in annotation data are background-corpus dependent, so claims about mid-commonness or rarity preferences must state the corpus they were measured on."],"supporting_citations":[{"why":"Defines the original descriptive-keyphrase study and its three hypotheses that this report sets out to replicate.","marker":"[3]"},{"why":"Supplies the foreground corpus of 144 scientific abstracts and the supervised keyphrase extraction setting.","marker":"[9]"},{"why":"Provides the background corpus used to compute commonness and corpus-level term statistics.","marker":"[8]"},{"why":"Supplies the parser used to derive part-of-speech and parse-tree grammatical features.","marker":"[10]"},{"why":"Provides sentence segmentation used for the presence-in-first-sentence positional feature.","marker":"[12]"},{"why":"Provides the R package used to generate the precision-recall curves and their area measures.","marker":"[18]"},{"why":"Provides the weighted logistic regression implementation used for all five models.","marker":"[19]"},{"why":"Documents the rarity cutoff in the large web corpus that explains the original commonness histogram shape.","marker":"[4]"}],"fun_headline_variants":["Three keyphrase hypotheses survive replication","Log-odds beats tf-idf in keyphrase replication","Bigrams dominate keyphrase length in replication","Keyphrase replication reveals rare-term twist"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The replication assumes that the feature definitions and experimental procedure were correctly reconstructed from the original paper, whose authors could not recover many details; if that reconstruction is wrong, the comparison to the original is not meaningful.","fun_headline_variants_meta":{"raw":{"variants":["Three keyphrase hypotheses survive replication","Log-odds beats tf-idf in keyphrase replication","Bigrams dominate keyphrase length in replication","Keyphrase replication reveals rare-term twist"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000346,"raw_usage":{"total_tokens":1847,"prompt_tokens":844,"completion_tokens":1003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":948}},"tokens_in":460,"tokens_out":1003,"duration_ms":9661,"temperature":1.0,"reasoning_tokens":948,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:13:37.796974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtaining the original study's feature-extraction code, or a complete feature specification, and re-running the five logistic-regression models on the original corpus: if weighted log-odds ratio then fails to beat tfidf in area under the precision-recall curve, the paper's second hypothesis is false.","supporting_citations":[{"cited_title":"Without the Clutter of Unimportant Words","cited_arxiv_id":null,"evidence_quote":"Defines the original descriptive-keyphrase study and its three hypotheses that this report sets out to replicate."},{"cited_title":"SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientific Articles","cited_arxiv_id":null,"evidence_quote":"Supplies the foreground corpus of 144 scientific abstracts and the supervised keyphrase extraction setting."},{"cited_title":"Improved Automatic Keyword Extraction Given More Linguistic Knowl- edge","cited_arxiv_id":null,"evidence_quote":"Provides the background corpus used to compute commonness and corpus-level term statistics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the parser used to derive part-of-speech and parse-tree grammatical features."},{"cited_title":"NLTK: The Natural Language Toolkit","cited_arxiv_id":null,"evidence_quote":"Provides sentence segmentation used for the presence-in-first-sentence positional feature."},{"cited_title":"ROCR: visual- izing classifier performance in R.Bioinformatics, 21(20):7881, 2005","cited_arxiv_id":null,"evidence_quote":"Provides the R package used to generate the precision-recall curves and their area measures."},{"cited_title":"R Foundation for Statistical Computing, Vienna, Austria, 2013","cited_arxiv_id":null,"evidence_quote":"Provides the weighted logistic regression implementation used for all five models."},{"cited_title":"Google Web 1T 5-grams Made Easy (but Not for the Computer)","cited_arxiv_id":null,"evidence_quote":"Documents the rarity cutoff in the large web corpus that explains the original commonness histogram shape."}],"review_version":1}