Replication of the Keyword Extraction part of the paper "'Without the Clutter of Unimportant Words': Descriptive Keyphrases for Text Visualization"

Shibamouli Lahiri

REVIEW 4 major objections 5 minor 21 references

Replication of the Keyword Extraction part of the paper "'Without the Clutter of Unimportant Words': Descriptive Keyphrases for Text Visualization"

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This replication study confirms, on a different corpus with crowdsourced labels, that bigrams are the most common keyphrases, log-odds ratio beats tfidf, and part-of-speech features help more than parse trees.

desk verdict A transparent, carelessly framed replication that confirms the bigram and feature-ablation findings but only validates Hypothesis 2 by narrowing it after the results were known. read the letter →

arxiv 1908.07818 v1 pith:JD76LY2S submitted 2019-08-15 cs.CL cs.DLcs.IR

Shibamouli Lahiri This is my paper · ORCID

classification cs.CLcs.DLcs.IR

keywords keyphraseextractionreplicationstudylog-oddsratioprecision-recallcurvelogisticregressioncommonnesspart-of-speechfeaturestextvisualization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This report repeats the keyphrase-extraction study and asks whether its three findings hold when the experiment is run on a different corpus of scientific abstracts with crowd-sourced annotations. The report's answer is yes on all three: bigrams are the most frequent keyphrase length, weighted log-odds ratio outscores raw term frequency and tfidf, and part-of-speech and positional features improve ranking while parse-tree features add little. The practical stake is that a good keyphrase ranker can be built from cheap features and a modest background corpus. The report also argues that the original study's mid-commonness preference is an artifact of the large web corpus, since annotators on a smaller corpus concentrate on the rarest terms.

What carries the argument

The load-bearing apparatus is a controlled feature-ablation comparison: five logistic-regression models trained on the same 7,816 positive examples and 78,160 down-weighted negative n-grams, adding one feature family at a time, with performance measured as area under the precision-recall curve. The feature families are frequency and probabilistic scores (including the weighted log-odds ratio, a smoothed contrast between a phrase's document frequency and its background frequency), commonness against a background corpus, binary grammatical features from part-of-speech patterns and parse trees, and three positional features. The relative ordering of the resulting curves is the identity that carries the argument: it is exactly what the second and third hypotheses assert.

What would settle it

Obtaining the original study's feature-extraction code, or a complete feature specification, and re-running the five logistic-regression models on the original corpus: if weighted log-odds ratio then fails to beat tfidf in area under the precision-recall curve, the paper's second hypothesis is false.

Watch

Extended reading notes

Core claim

On the report's own terms, the discovery is that a full replication with different data reproduces the original study's three-part hypothesis. The author builds five logistic-regression models and compares them by area under the precision-recall curve. Weighted log-odds ratio outscores log(tf) and tfidf; adding part-of-speech features improves the curve; adding parse-tree features on top changes it little; adding positional features improves it further. In the crowd-sourced annotations, bigrams are the most common keyphrase length, followed by unigrams and then trigrams. The study also finds a divergence from the original: on the smaller background corpus, annotators prefer the rarest terms, and the original mid-commonness peak reappears only after rare terms are removed.

Load-bearing premise

The replication assumes that the feature definitions and experimental procedure were correctly reconstructed from the original paper, whose authors could not recover many details; if that reconstruction is wrong, the comparison to the original is not meaningful.

Editorial extensions

If this is right

Weighted log-odds ratio is a stronger baseline for ranking keyphrases on scientific abstracts than raw term frequency or tfidf.
Grammatical features improve ranking mainly through part-of-speech information; parse-tree features can be dropped once part-of-speech features are present.
Candidate-generation strategies should emphasise bigrams because they are the most frequent human-chosen keyphrase length.
Positional features give a further boost to the best model, so first-occurrence information is worth keeping.
Commonness patterns in annotation data are background-corpus dependent, so claims about mid-commonness or rarity preferences must state the corpus they were measured on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the commonness finding generalizes, the original mid-commonness peak is an artifact of removing rare terms from a very large background corpus; a direct test would hold annotators fixed and vary only the background corpus.
Because negative examples are sampled at ten times the positive rate and down-weighted, the reported AUC differences mostly reflect ranking at low false-positive rates; re-running with different ratios would test whether the feature ordering is stable.
The small gain from parse trees suggests practical keyphrase systems can skip syntactic parsing, though the present report does not itself argue this for modern systems.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a genuinely careful replication study, and the author deserves credit for shipping code, data, and a frank account of every guess made along the way. But the headline claim that all three hypotheses hold needs a caveat: the second one is confirmed only after the hypothesis was narrowed to the specific measure that worked.

What's new here is modest but real. On a different corpus (SemEval abstracts) and with MTurk annotators, the author re-examines the keyphrase length distribution, the value of probabilistic vs. frequency features, and the contribution of grammatical and positional features. Bigrams are still the most common choice, though the peak is lower; POS and positional features help; parse-tree features add little. The commonness analysis is interesting because it goes against the original: with the Inspec background corpus, annotators prefer the rarest terms, not mid-commonness, and the author shows this is partly an artifact of the Web1T frequency threshold. Binning commonness doesn't help. These are useful, reproducible observations.

The soft spots are also real. The biggest is the post-hoc fix. The initial hypothesis said probabilistic measures like G2 outperform raw term frequency and tfidf. The final hypothesis says log-odds ratio does. The paper admits this refinement came after seeing experimental findings, and the results show G2 and BM25 are worse than log(tf). So the replication validates a hypothesis selected post hoc; it does not confirm the original broad claim, and the evidence actually points toward partial refutation of that claim. Second, the original protocol was partly reconstructed from memory; the author is honest about this, but it limits how strongly we can read the comparison. Third, the spurious-phrase filter was built by hand from Turker responses, which introduces possible selection bias. Fourth, there are no confidence intervals or significance tests for the AUROC comparisons, so observed differences could be noise.

For all that, I think this paper deserves a serious referee. It is exactly the kind of replication we need more of: reproducible, transparent, and willing to report contradictions rather than smooth them over. The right verdict is probably 'revise' – make the post-hoc nature of Hypothesis 2 explicit in the abstract, add uncertainty measures where possible, and report the manual filter's effect. For a reader, it is worthwhile if you work on keyphrase extraction or on replication methodology in HCI/NLP. I would bring it to a reading group, but with the caveats above.

Referee Report

4 major / 5 minor

Summary. This manuscript is a replication report for the keyphrase-extraction component of Chuang, Manning, and Heer (2012). The author replaces the unavailable Stanford thesis abstracts with the SemEval 2010 training set, uses Inspec as the background corpus, collects keyphrase annotations from Amazon Mechanical Turk, builds logistic-regression classifiers with frequency, commonness, grammatical, and positional features, and evaluates them with precision-recall curves. The report states three final hypotheses: bigrams are the most frequent human-selected keyphrase length; weighted log-odds ratio outperforms raw term frequency and tf-idf; part-of-speech and positional features improve performance while parse-tree features add little. It also reports exploratory analyses of phrase length, extractive/abstractive behavior, first-sentence placement, and commonness, and it experiments with commonness binning. The paper concludes that all three parts of the final hypothesis are true on the author's dataset, annotations, features, and models.

Significance. If the claims were fully supported, this would be a useful replication case study: the original data and code are unavailable, and the report carefully documents reconstruction decisions, releases code and data, and honestly records discrepancies, including the failure of G2 and BM25 and the rarest-term commonness finding. The reproducibility-oriented presentation is a genuine strength. However, the significance is limited by two issues: the second hypothesis is refined after seeing the results, and the central comparative claims lack uncertainty quantification. Because of these, the report currently reads more as a new evaluation under a different protocol than as a confirmation of the original hypotheses. With appropriate reframing and additional statistics, the manuscript could be a valuable contribution to the replication literature.

major comments (4)

[Section 2 and Section 4.2] Hypothesis 2 is changed after the results are known. The initial hypothesis states that 'probabilistic measures such as G2 outperform raw term frequency and tfidf', while the final hypothesis narrows this to 'Log-odds Ratio outperforms raw term frequency and tfidf'. Section 4.2 then reports that G2 and BM25 are 'much worse — sometimes even worse than log(tf)', so only the measure that survived is used to validate the final hypothesis. This is post-hoc hypothesis selection: the claim is not a test of the original probabilistic-features hypothesis but a statement that one selected measure works. In fact, the G2 and BM25 results partially refute the broader original claim. The authors should either test the original formulation as a family of measures with appropriate multiple-comparison handling, or explicitly reframe the contribution as identifying that weighted log-odds, but not the other probabilistic measures, improves on raw frequency and tf-idf.
[Section 4.2, Figures 5–8] The core comparative claims are supported only by visual inspection of precision-recall curves. No numerical AUPRC values, confidence intervals, or significance tests are reported, and the negative examples are randomly sampled (Section 3.2), so the rankings could vary with the random seed. Without uncertainty quantification, statements such as 'log-odds ratio outperforms log(tf) and tfidf' and 'parse-tree-based features lead to very little additional improvement' cannot be evaluated. Please report AUPRC point estimates with standard errors or bootstrap confidence intervals, and preferably paired comparisons across the same train/test splits.
[Section 3.1 and Section 4] The replication changes the foreground corpus (SemEval 2010 instead of Stanford thesis abstracts), the background corpus (Inspec instead of Stanford theses/Web1T), the annotator population (Mechanical Turk instead of the original judges), and several feature definitions are explicitly described as 'educated guesses' because 'many details were lost'. The text acknowledges this in Section 3.1, but Section 4 still states that the hypotheses are 'indeed true, on our dataset, annotations, features, and models.' The qualification is important, yet the title and conclusion frame the work as a replication of Chuang et al. The conclusions should be explicitly scoped as a new evaluation under a different protocol; as written, the manuscript overclaims confirmation of the original hypotheses.
[Section 4.1, Figure 1] The first hypothesis is supported only weakly by the replicated data. The replication reports 24.5% bigrams, 21.4% unigrams, and 13.6% trigrams, compared with 43%, 25%, and 19% in the original study. The bigram peak is only 3.1 percentage points above the unigram peak, so without confidence intervals the claim that 'bigrams are most frequently chosen by humans' is fragile. The authors should quantify the uncertainty of these proportions (e.g., bootstrap over Turkers or assignments) and report whether the bigram-over-unigram ordering is stable.

minor comments (5)

[Table 2] The weighted log-odds ratio formula is garbled in the typeset version; the numerator appears to contain identical terms (t'_Doc/t'_Doc and T'_Doc/T'_Doc). Please provide an unambiguous notation, for example log((t_Doc+0.01)/(t_Ref+0.01)) − log((T_Doc+0.01)/(T_Ref+0.01)) divided by the appropriate standard error.
[Section 3.1 and Table 1] The spurious-phrase filter is manually constructed from the responses. Although it removes only 19 phrases (17,640 to 17,621), the construction is subjective; please document the criteria more precisely and, if possible, show that the main results are robust to including or excluding this filter.
[Section 3.3] The report infers that the original study used supervised ranking via a regression model rather than classification; the text says this 'could not be confirmed'. Please clarify whether this inference changes the interpretation of the term 'logistic regression' and whether it affects comparability with the original.
[Section 4.1, Figure 2] The replicated histogram is plotted over assignments while the original is apparently plotted over documents; the text notes this, but the comparison of the five-phrase peak is still not apples-to-apples. Please plot both on the same unit or add a more prominent caveat to the figure caption.
[Section 4.1, commonness discussion] The statement that the original authors 'did miss a point' is too strong, because the difference between the commonness histograms is likely driven by the different background corpus (Inspec vs. Web1T) and by the Web1T frequency floor. Please soften this to a corpus-dependent observation.

Circularity Check

1 steps flagged · score 6.0 of 10

Hypothesis 2 is selected after seeing the results, so its confirmation is partly by construction.

fitted input called prediction [Section 2 (Research Hypothesis) and Section 4.2 (Model Performance)]
""We refined and qualified our initial hypothesis in light of our revelations from studying the paper in depth, and our experimental findings." The final hypothesis states: "Log-odds Ratio outperforms raw term frequency and tfidf in terms of area under the precision-recall curve, when logistic regression is used as classifier." Section 4.2 then says: "In Figure 5b, log-odds ratio outperforms log(tf) and tfidf in terms of area under the curve. This validates the second part of our final hypothesis.""

The initial Hypothesis 2 claimed a broad class: "Probabilistic measures such as G2 outperform raw term frequency and tfidf." After experiments, the authors "refined and qualified" this to a single measure, weighted log-odds ratio, which they had already observed to win. Section 4.2 then treats the same observed outperformance as "validat[ing]" the final hypothesis. The final claim is thus defined by the result it is said to support: the class was narrowed precisely to the measure that outperformed. The same figure shows G2 and BM25 are "much worse — sometimes even worse than log(tf)", so the data actually refutes the original broad claim; the final hypothesis avoids that refutation by construction. No independent data set or pre-registered test is used for this part.

full rationale

The paper is a transparent replication with largely self-contained evidence: fresh MTurk annotations (9,936 assignments), a different foreground corpus (SemEval 2010), an Inspec background corpus, explicit feature definitions, and precision-recall comparisons against the original figures. The self-citations (refs 11 and 14) are background references about existing datasets and annotation behavior, not load-bearing justifications of the hypotheses, so they do not create circularity. The one genuine circular element is Hypothesis 2: the initial broad probabilistic-features claim was replaced, after seeing experimental results, by a narrow claim about weighted log-odds ratio, and then the same results are cited as validation. That makes the second hypothesis partly self-confirming. Hypotheses 1 and 3 are not reduced to fitted inputs: the annotated distributions and feature-ablation comparisons are computed from the new data and compared with the original claims, including observed divergences. Overall, the report's main replication narrative is not circular, but the strongest "all three parts are true" claim overstates the support for Hypothesis 2 because that hypothesis was selected after the outcome was known.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several assumptions about the comparability of the replicated setup to the original study, the validity of crowd annotations, and the suitability of the background corpus.

free parameters (3)

negative example weight = 0.1
Negative examples were weighted 0.1, as in the original study, but the rationale was never explained (Section 3.2).
maximum keyphrase length = 5 words
Restricted to phrases of length 5 or shorter to keep with the original study; this cutoff affects the dataset (Section 3.1).
spurious phrase filter = manual list
A manually constructed list of spurious phrases was used to filter annotations (Table 1).

assumptions (3)

domain assumption Mechanical Turk annotations with minimal instructions are valid keyphrase judgments comparable to expert annotators
Used to build the training set; the paper notes Turkers may be less educated and more verbose than the original annotators (Section 3.1).
domain assumption Inspec abstracts are a suitable background corpus for computing frequency and commonness features for SemEval abstracts
The authors chose Inspec for similarity of content and length; results depend on this choice (Section 3.1).
ad hoc to paper The original feature definitions (e.g., binary grammatical features, weighted log-odds formula) were correctly interpreted
The paper states many details of the original study were lost and decisions were 'educated guesses' (Sections 3.1, 3.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Replication of the Keyword Extraction part of the paper "'Without the Clutter of Unimportant Words': Descriptive Keyphrases for Text Visualization"." pith.science (2026). https://pith.science/paper/JD76LY2S

@misc{pith2026190807818,
  author       = {Pith},
  title        = {Pith review of: Replication of the Keyword Extraction part of the paper "'Without the Clutter of Unimportant Words': Descriptive Keyphrases for Text Visualization"},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JD76LY2S}},
  note         = {Machine review of arXiv:1908.07818}
}

read the original abstract

"Keyword Extraction" refers to the task of automatically identifying the most relevant and informative phrases in natural language text. As we are deluged with large amounts of text data in many different forms and content - emails, blogs, tweets, Facebook posts, academic papers, news articles - the task of "making sense" of all this text by somehow summarizing them into a coherent structure assumes paramount importance. Keyword extraction - a well-established problem in Natural Language Processing - can help us here. In this report, we construct and test three different hypotheses (all related to the task of keyword extraction) that take us one step closer to understanding how to meaningfully identify and extract "descriptive" keyphrases. The work reported here was done as part of replicating the study by Chuang et al. [3].

Figures

Figures reproduced from arXiv: 1908.07818 by the authors.

**Figure 2.** Distribution of the number of human-assigned keyphrases. X-axis is the number [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗

**Figure 3.** Distribution of commonness. X-axis is commonness, and Y-axis is the percent [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Distribution of commonness on Inspec after the rarest terms are removed. X-axis is commonness, and Y-axis is the number of unique (human-selected) n-grams with that commonness. Note that as we remove more and more rare terms, the figures look more and more similar in s…

**Figure 5.** Figure 5: Performance of frequency and probabilistic features (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]

**Figure 6.** Figure 6: Adding term commonness to term frequency (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]

**Figure 7.** Figure 7: Adding grammatical features on top of term commonness (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]

**Figure 8.** Figure 8: Adding positional features on top of all other features (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]

**Figure 9.** Figure 9: Binning commonness (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]

**Figure 10.** Figure 10: Binning commonness (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]

**Figure 11.** Figure 11: Binning commonness (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]

**Figure 12.** Figure 12: Binning commonness (best viewed in color). [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages

[1]

SZTERGAK : Feature Engineering for Keyphrase Extraction

Gábor Berend and Richárd Farkas. SZTERGAK : Feature Engineering for Keyphrase Extraction. In Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden, July 2010. 33

work page 2010
[2]

Web 1T 5-gram corpus version 1

Thorsten Brants and Alex Franz. Web 1T 5-gram corpus version 1. Technical report, Google Research, 2006

work page 2006
[3]

Without the Clutter of Unimportant Words

Jason Chuang, Christopher D. Manning, and Jeﬀrey Heer. “Without the Clutter of Unimportant Words”: Descriptive Keyphrases for Text Visualization. ACM Trans. Comput.-Hum. Interact., 19(3):19:1–19:29, October 2012

work page 2012
[4]

Google Web 1T 5-grams Made Easy (but Not for the Computer)

Stefan Evert. Google Web 1T 5-grams Made Easy (but Not for the Computer). In Proceedings of the NAACL HLT 2010 Sixth Web As Corpus Workshop, WAC-6 ’10, pages 32–40, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics

work page 2010
[5]

Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling

Jenny Rose Finkel, Trond Grenager, and Christopher Manning. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. InProceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA, USA, 2005. Association for Computational Linguistics

work page 2005
[6]

Extracting Key Terms From Noisy and Multi-theme Documents

Maria Grineva, Maxim Grinev, and Dmitry Lizorkin. Extracting Key Terms From Noisy and Multi-theme Documents. In18th International World Wide Web Conference, pages 661–670, April 2009

work page 2009
[7]

Conundrums in Unsupervised Keyphrase Extrac- tion: Making Sense of the State-of-the-Art

Kazi Saidul Hasan and Vincent Ng. Conundrums in Unsupervised Keyphrase Extrac- tion: Making Sense of the State-of-the-Art. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages365–373.AssociationforCom- putational Linguistics, 2010

work page 2010
[8]

Improved Automatic Keyword Extraction Given More Linguistic Knowl- edge

Anette Hulth. Improved Automatic Keyword Extraction Given More Linguistic Knowl- edge. In Proceedings of the 2003 conference on Empirical methods in natural language processing, EMNLP ’03, pages 216–223, Stroudsburg, PA, USA, 2003. Association for Computational Linguistics

work page 2003

Show all 21 references

[9]

SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientiﬁc Articles

Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. SemEval-2010 Task 5: Automatic Keyphrase Extraction from Scientiﬁc Articles. InProceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pages 21–26, Stroudsburg, PA, USA, 2010. Association ...

2010
[10]

Dan Klein and Christopher D. Manning. Accurate Unlexicalized Parsing. InProceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423– 430, Sapporo, Japan, July 2003. Association for Computational Linguistics

2003
[11]

Keyword and Keyphrase Extraction Using Centrality Measures on Collocation Networks

Shibamouli Lahiri, Sagnik Ray Choudhury, and Cornelia Caragea. Keyword and Keyphrase Extraction Using Centrality Measures on Collocation Networks. CoRR, abs/1401.6571, 2014

2014 arXiv
[12]

NLTK: The Natural Language Toolkit

Edward Loper and Steven Bird. NLTK: The Natural Language Toolkit. InProceedings of the ACL-02 Workshop on Eﬀective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1, ETMTNLP ’02, pages 63–70, Stroudsburg, PA, USA, 2002. A...

2002
[13]

HUMB: Automatic Key Term Extraction from Scientiﬁc Articles in GROBID

Patrice Lopez and Laurent Romary. HUMB: Automatic Key Term Extraction from Scientiﬁc Articles in GROBID. InProceedings of the 5th International Workshop on Semantic Evaluation, SemEval ’10, pages 248–251, 2010

2010
[14]

Building a Dataset for Summarization and Keyword Extraction from Emails

Vanessa Loza, Shibamouli Lahiri, Rada Mihalcea, and Po-Hsiang Lai. Building a Dataset for Summarization and Keyword Extraction from Emails. In Nicoletta Cal- zolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion M...

2014
[15]

Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a Large Annotated Corpus of English: The Penn Treebank.Computational Linguistics, 19(2):313–330, June 1993

1993
[16]

TextRank: Bringing Order into Texts

Rada Mihalcea and Paul Tarau. TextRank: Bringing Order into Texts. In Dekang Lin and Dekai Wu, editors,Proceedings of EMNLP 2004, pages 404–411, Barcelona, Spain, July 2004. Association for Computational Linguistics. 35

2004
[17]

UsingTermInformativenessforNamedEntity Detection

JasonD.M.RennieandTommiJaakkola. UsingTermInformativenessforNamedEntity Detection. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 353–360. ACM, 2005

2005
[18]

ROCR: visual- izing classiﬁer performance in R.Bioinformatics, 21(20):7881, 2005

Tobias Sing, Oliver Sander, Niko Beerenwinkel, and Thomas Lengauer. ROCR: visual- izing classiﬁer performance in R.Bioinformatics, 21(20):7881, 2005

2005
[19]

R Foundation for Statistical Computing, Vienna, Austria, 2013

RCore Team.R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2013

2013
[20]

A Language Model Approach to Keyphrase Extraction

Takashi Tomokiyo and Matthew Hurst. A Language Model Approach to Keyphrase Extraction. In Proceedings of the ACL 2003 workshop on Multiword expressions: anal- ysis, acquisition and treatment-Volume 18, pages 33–40. Association for Computational Linguistics, 2003

2003
[21]

Automatic Keyword Extraction from Documents Using Conditional Random Fields

Chengzhi Zhang, Huilin Wang, Yao Liu, Dan Wu, Yi Liao, and Bo Wang. Automatic Keyword Extraction from Documents Using Conditional Random Fields. Journal of Computational Information Systems, 2008. 36

2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

SZTERGAK : Feature Engineering for Keyphrase Extraction

Gábor Berend and Richárd Farkas. SZTERGAK : Feature Engineering for Keyphrase Extraction. In Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden, July 2010. 33

work page 2010

[2] [2]

Web 1T 5-gram corpus version 1

Thorsten Brants and Alex Franz. Web 1T 5-gram corpus version 1. Technical report, Google Research, 2006

work page 2006

[3] [3]

Without the Clutter of Unimportant Words

Jason Chuang, Christopher D. Manning, and Jeﬀrey Heer. “Without the Clutter of Unimportant Words”: Descriptive Keyphrases for Text Visualization. ACM Trans. Comput.-Hum. Interact., 19(3):19:1–19:29, October 2012

work page 2012

[4] [4]

Google Web 1T 5-grams Made Easy (but Not for the Computer)

Stefan Evert. Google Web 1T 5-grams Made Easy (but Not for the Computer). In Proceedings of the NAACL HLT 2010 Sixth Web As Corpus Workshop, WAC-6 ’10, pages 32–40, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics

work page 2010

[5] [5]

Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling

Jenny Rose Finkel, Trond Grenager, and Christopher Manning. Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling. InProceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA, USA, 2005. Association for Computational Linguistics

work page 2005

[6] [6]

Extracting Key Terms From Noisy and Multi-theme Documents

Maria Grineva, Maxim Grinev, and Dmitry Lizorkin. Extracting Key Terms From Noisy and Multi-theme Documents. In18th International World Wide Web Conference, pages 661–670, April 2009

work page 2009

[7] [7]

Conundrums in Unsupervised Keyphrase Extrac- tion: Making Sense of the State-of-the-Art

Kazi Saidul Hasan and Vincent Ng. Conundrums in Unsupervised Keyphrase Extrac- tion: Making Sense of the State-of-the-Art. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages365–373.AssociationforCom- putational Linguistics, 2010

work page 2010

[8] [8]

Improved Automatic Keyword Extraction Given More Linguistic Knowl- edge

Anette Hulth. Improved Automatic Keyword Extraction Given More Linguistic Knowl- edge. In Proceedings of the 2003 conference on Empirical methods in natural language processing, EMNLP ’03, pages 216–223, Stroudsburg, PA, USA, 2003. Association for Computational Linguistics

work page 2003