ConExion: Concept Extraction with Large Language Models

Ebrahim Norouzi; Harald Sack; Sven Hertling

REVIEW 4 major objections 5 minor 30 references

ConExion: Concept Extraction with Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ConExion claims that a reproducible few-shot LLM prompting pipeline—asking a model to list keyphrases present in a document, then keeping only phrases that literally appear—extracts present domain concepts more accurately than prior…

desk verdict Useful empirical sweep, but the headline SOTA is a test-set-selected maximum and the evaluation conflates keyphrases with concepts. read the letter →

arxiv 2504.12915 v2 pith:OTJXSIMP submitted 2025-04-17 cs.CL cs.IR

Ebrahim Norouzi , Sven Hertling , Harald Sack This is my paper · ORCID

classification cs.CLcs.IR

keywords ConceptextractionKeyphraseLargelanguagemodelsFew-shotpromptingZero-shotOntologyevaluationUnsupervised

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models, used with carefully chosen prompts and no training, are the best current extractors of present concepts from scientific abstracts. The authors argue that concept extraction for ontology evaluation should return all concepts that occur in a document, not just top-k important keyphrases, and they build a pipeline that asks an LLM for keyphrases and filters the output to exact document substrings. On two standard benchmarks, the best configuration—one random example in the prompt with Llama3 70B—reaches F1 0.451 on Inspec and 0.311 on SemEval2017, higher than all compared traditional and embedding-based methods. They also show that prompt wording and a single in-context example matter more than model scale alone.

What carries the argument

The load-bearing mechanism is the chat prompt plus the present-concept filter. The prompt defines the extraction task through three slots—system, user, assistant—and is converted into each model's expected chat format by the tokenizer's chat template; the authors test five zero-shot search terms, four context-augmented zero-shot prompts, and few-shot prompts with fixed, random, or embedding-closest examples. The filter removes any generated phrase that is not an exact substring of the document, which implements the 'present' constraint and also strips conversational filler. Confidence scores come from the geometric mean of per-token generation probabilities under greedy decoding, which is what allows F1@5 and F1@10 comparisons.

What would settle it

Take a random sample of Inspec and SemEval2017 abstracts and have annotators mark every noun phrase that denotes a domain concept, regardless of importance; run ConExion's best configuration on the same documents and compute F1 against this mention-level gold standard. If that F1 is not clearly higher than the keyphrase-gold F1—or if a plain noun-phrase chunker matches it—the claim of extracting all present concepts, rather than important keyphrases, is not supported.

Watch

Extended reading notes

Core claim

The discovery is empirical: a simple, fully reproducible setup for present concept extraction outperforms the previous state of the art. The method uses an instruction-tuned LLM as an extractor, with a system prompt, user prompt, and assistant prompt, and the model is asked to output concepts separated by commas. Greedy decoding keeps the run deterministic, and each extracted phrase receives a confidence score equal to the geometric mean of its token probabilities, which supports ranking. The output is split on commas, semicolons, stars, and newlines, then filtered to phrases that occur verbatim in the input, guaranteeing that every reported concept is grounded in the document. On Inspec, the FS 1-Random prompt with Llama3 70B achieves precision 0.413, recall 0.575, and F1 0.451; on SemEval2017 it achieves precision 0.301, recall 0.372, and F1 0.311, the highest F1 among all compared models on both datasets.

Load-bearing premise

The evaluation assumes that the ground-truth keyphrases in Inspec and SemEval2017—annotations of what is important in a document—are a valid proxy for the full set of present domain concepts that ConExion claims to extract, so the reported F1 numbers measure the stated task only if that proxy holds.

Editorial extensions

If this is right

For ontology coverage evaluation, the method provides an unsupervised, training-free way to produce a grounded set of concepts from a corpus, with a confidence score for each concept, so no user-chosen threshold is required.
Prompt wording is a first-order performance factor: on Inspec, switching Llama3 70B from 'keywords' to 'keyphrases' raises F1 from 0.362 to 0.414, and adding a task description raises it further to 0.431.
A single in-context example is enough to teach the output format; beyond one example, adding more fixed, random, or closest examples yields little additional gain, and five examples can even lower F1.
LLM-based extraction is more precise but less comprehensive than graph-based extraction: PositionRank beats ConExion on recall, so the best near-term improvement is a hybrid that combines LLM precision with graph-based recall.
Because greedy decoding is used, the full pipeline is deterministic; fixing model checkpoints, tokenizer chat templates, and datasets means the reported numbers can be reproduced exactly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit is that the benchmark mismatch is directly testable: Inspec and SemEval2017 labels are important-keyphrase annotations, not exhaustive annotations of every present concept, so evaluating against a full mention-level concept annotation would likely change the reported recall and precision substantially.
The exact-substring filter is a strict lower bound on the model's semantic recall; allowing inflectional, hyphenation, or synonym variants would probably raise F1, since the paper itself notes that semantically accurate concepts are discarded.
The token-probability confidence score is a byproduct of greedy decoding, but it could be reused as a ranking feature or compared against scores from sampled decoding to test whether it truly orders concepts by importance.
The prompt sensitivity observed here suggests that published concept-extraction numbers may not transfer across LLM generations; a useful extension would be a small prompt and example-selection protocol that minimizes variance across model versions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Two things to know about arXiv:2504.12915. First, it is a genuinely useful empirical resource: eight LLMs, five search terms, a large set of zero- and few-shot prompt variations, run on two standard keyphrase datasets with code and pinned versions. Second, the headline claim — Llama3 70B with a few-shot prompt reaching 0.451 F1 on Inspec, far above classical baselines — is a selected maximum from a test-set sweep, not a pre-specified comparison. The same goes for the SemEval2017 number. The direction of the result is credible; the exact SOTA number is not.

What the paper does well: the sweep is broad, and results are reported with precision, recall, F1, top-k variants, and extraction count. The non-top-k F1 is the right metric for ontology coverage, and the confidence score via geometric mean of token probabilities is a sensible, reproducible trick. The comparison against 14 classical baselines is fair, and the code and data release is real. The most robust finding — which I buy — is that even a zero-shot "keyphrases" prompt with Llama3 70B beats the best classical unsupervised method (0.414 vs 0.287 F1 on Inspec). So the broad conclusion is not fragile.

Soft spots, in order of severity. (1) Model and prompt are chosen on the test set. Table 3 fixes the model, Table 4 fixes the prompt, then Table 5 reports the winner. No validation split, no multiple-comparison control, no significance tests. The gap between FS 1-Random (0.451) and the zero-shot default (0.414) is small enough that selection could be noise. Also, FS n-Random draws random training examples per test document, and no seed is reported; the headline might move with one random draw. (2) The abstract promises "all present concepts," but the evaluation uses keyphrase annotations that mark important concepts, and the prompts themselves say "keyphrases." So the measured task is keyphrase extraction, not the claimed concept extraction. This matters for ontology evaluation, where coverage of all domain concepts is the goal. (3) Exact-match filtering is acknowledged as a limitation, but the paper never shows how much LLM output gets discarded by it.

Who should read this: anyone working on practical keyphrase extraction for ontology engineering, or on prompt-based NLP recipes. It is not a theoretical contribution, but it is a valuable empirical resource. It deserves a serious referee, not because the SOTA claim holds as stated, but because the methodology is fixable: hold-out split for prompt selection, seeded random examples, and honest task framing. I would send it to peer review, likely at a workshop or short-paper venue, and ask for those easy fixes. I would cite the broad result, not the specific number.

Referee Report

4 major / 5 minor

Summary. The paper presents ConExion, a prompt-based method that uses large language models (LLMs) to extract present keyphrases/concepts from scientific abstracts. The approach builds a system/user/assistant prompt template, generates a comma-separated list with greedy decoding, splits the output by several delimiters, and filters to exact matches in the input document; confidence scores are computed as the geometric mean of token probabilities. The authors evaluate eight LLMs (Llama-2, Llama-3, Mistral, Mixtral, GPT-3.5-Turbo) on Inspec and SemEval2017 with zero-shot prompts over five search terms, four context-augmented prompts, and few-shot prompts with 1/3/5 fixed, random, and closest training examples. They compare against 14 unsupervised keyphrase-extraction baselines and report that FS 1-Random with Llama3 70B achieves the highest F1 on both datasets (0.451 on Inspec, 0.311 on SemEval2017), which is the central state-of-the-art claim.

Significance. If the reported result holds, the paper would make a useful empirical contribution: a simple, reproducible LLM-prompting recipe that outperforms a broad set of traditional unsupervised keyphrase extractors, with public code and fixed model/dataset versions. The zero-shot finding is noteworthy: even the ZS Keyphrases prompt with Llama3 70B reaches F1 0.414 on Inspec, above all 14 baselines in Table 5, so the qualitative conclusion that simple LLM prompting is competitive does not depend on the few-shot selection. However, the headline claim of state-of-the-art concept extraction is currently not established because the best configuration is selected on the test set, and because the benchmark measures keyphrase extraction rather than the paper's stated target of extracting all present domain concepts.

major comments (4)

[Section 4.4, Tables 3-5] The headline configuration is selected using test-set performance, making the SOTA claim an order statistic rather than a pre-specified comparison. The text explicitly states: "Based on the results of Table 3, the Llama3 70B model was fixed, and term keyphrases is selected," then Table 4 uses the same Inspec test set to select the prompt, and Table 5 reports the winning configuration as "ours." With 40 model-prompt combinations in Table 3 and 13 prompt variants in Table 4, the best F1 is the maximum over a large grid. There is no validation split, no multiple-comparison control, and no significance test. The authors should either introduce a validation split (or nested selection), report significance tests, or explicitly present the zero-shot result as the primary claim and the few-shot variant as exploratory.
[Section 3 and Section 4.5] The FS n-Random results are not reproducible as reported. The method draws random training examples for each test document, but Section 4.5 states that reproducibility is ensured without reporting any random seed. The headline F1 values (0.451 on Inspec, 0.311 on SemEval2017) may therefore depend on a single undocumented random draw. Please report the seed, or better, run multiple draws and report the mean and standard deviation.
[Abstract, Section 4.1, and Section 4.2] There is a mismatch between the task claimed and the task measured. The abstract and introduction frame the contribution as extracting "all present concepts related to the specific domain, not just the important ones," but the evaluation uses Inspec and SemEval2017, whose ground truth consists of keyphrases annotated for importance, and the metrics compare against those keyphrase annotations. The paper's own formal definition in Section 1 also describes concepts as "words that best represent the topics," which is keyphrase-like. Consequently, the reported F1 scores measure keyphrase extraction, not extraction of all present domain concepts. The authors should either reframe the claims to keyphrase extraction or evaluate on a dataset / with an annotation scheme that captures all present concepts; otherwise the central concept-extraction claim is not supported.
[Section 4.2, Eq. (R@k)] The recall-at-k formula is written as R@k = |C:k ∩ C'| / |C':k|. Since C' is the unordered ground-truth set, the denominator |C':k| is undefined as written and contradicts the preceding sentence stating that recall is measured against all ground-truth concepts. This should be corrected to |C'| (or C':k must be defined). This is a presentation issue but it affects the comparability of the reported top-k numbers.

minor comments (5)

[Section 4.5] The reproducibility statement "the versions of datasets and models from HuggingFace were fixed, ensuring that the results could be reproduced" is in tension with the undocumented randomness of FS n-Random prompts; please add the missing seed or clarify that only the deterministic parts are reproducible.
[Table 1] The FS n-Random row in Table 1 says "randomly selected examples" but does not state that the selection is per test document and whether it is seeded; the text in Section 3 provides this detail, so aligning the table with the text would improve clarity.
[Table 2] The header "Maxcon / Mincon / Avgcon" is a single merged column, which is hard to read; separate the columns so the statistics are clear.
[References] References [17] and [23] are the same paper (PromptRank); please merge or cite one entry.
[Throughout] The terms "concepts," "keyphrases," and "keywords" are sometimes used interchangeably (e.g., abstract vs. Section 1 vs. Table 5); a brief definition at first use would reduce ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark with externally fixed baselines; test-set model selection is a validity concern, not a derivation loop.

full rationale

The paper makes no claimed derivation from first principles and contains no fitted parameter that is later renamed as a prediction. Its central claim is an empirical comparison against fixed external datasets (Inspec, SemEval2017) and independently published baselines. The choice of Llama3 70B and the FS 1-Random prompt is determined by inspecting test-set F1 in Tables 3 and 4 and then reported in Table 5; this is test-set model/prompt selection and a multiple-comparisons concern, not a circular reduction, because the reported F1 values are produced by running the selected configuration rather than being implied by the selection criterion. The paper cites no load-bearing self-referential uniqueness theorem, and none of its definitions reduce the output to the input. The random few-shot example selection and unreported seed raise reproducibility questions but do not constitute circularity. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

All free parameters are hyperparameters selected using test-set performance rather than a validation split. The axioms concern the mapping from benchmark annotations to the paper's stated task and the evaluation metric's equivalence relation. No invented entities are introduced.

free parameters (4)

Search term in prompt = keyphrases
The authors tried keywords, keyphrases, concepts, entities, and topics, and selected the best term on the Inspec test set (Table 3).
Base LLM = Llama3 70B Instruct
The model that scored highest on the Inspec test set in Table 3 was fixed for all subsequent experiments.
Number of few-shot examples = 1
FS 1-Random gave the highest F1 on the test set in Table 4.
Few-shot example selection strategy = random
Chosen over fixed and closest-embedding strategies based on test-set F1 in Table 4.

assumptions (3)

domain assumption The annotated keyphrases in Inspec and SemEval2017 are a valid proxy for the full set of domain concepts present in the documents.
Sections 4.1 and 4.4 use these annotations as ground truth, but the stated task is to extract all present domain concepts, not only important keyphrases.
domain assumption Exact lexical or stemmed string matching is a sufficient equivalence test between extracted and true concepts.
Section 4.2 and the filtering step in Section 3 rely on exact string presence; the conclusion acknowledges this discards semantically equivalent concepts.
domain assumption LLM outputs can be decomposed into concepts by splitting on commas, semicolons, asterisks, and newlines, then filtering to n-grams present in the document.
The parsing method in Section 3 is assumed to recover the model's intended concept list; alternative output formats would lose concepts.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ConExion: Concept Extraction with Large Language Models." pith.science (2026). https://pith.science/paper/OTJXSIMP

@misc{pith2026250412915,
  author       = {Pith},
  title        = {Pith review of: ConExion: Concept Extraction with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTJXSIMP}},
  note         = {Machine review of arXiv:2504.12915}
}

read the original abstract

In this paper, an approach for concept extraction from documents using pre-trained large language models (LLMs) is presented. Compared with conventional methods that extract keyphrases summarizing the important information discussed in a document, our approach tackles a more challenging task of extracting all present concepts related to the specific domain, not just the important ones. Through comprehensive evaluations of two widely used benchmark datasets, we demonstrate that our method improves the F1 score compared to state-of-the-art techniques. Additionally, we explore the potential of using prompts within these models for unsupervised concept extraction. The extracted concepts are intended to support domain coverage evaluation of ontologies and facilitate ontology learning, highlighting the effectiveness of LLMs in concept extraction tasks. Our source code and datasets are publicly available at https://github.com/ISE-FIZKarlsruhe/concept_extraction.

Figures

Figures reproduced from arXiv: 2504.12915 by the authors.

**Figure 1.** Overall approach of ConExion. phrase bank by combining keyphrases from all documents into a corpus, considering both lexical and semantic similarities for selecting top candidate keyphrases for each input document. PromptRank [17] is an unsupervised approach based on a pre-trained language model (PLM) with an encoderdecoder architecture. PromptRank feeds the document into the encoder and calculates the probability … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

[1]

K. S. Hasan, V. Ng, Automatic keyphrase extraction: A survey of the state of the art, in: K. Toutanova, H. Wu (Eds.), Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Lin- guistics, Baltimore, Maryland, 2014, pp. 1262–1273. URL: https://aclanthology.org/P14-1119....

doi:10.3115/v1/p14-1119 2014
[2]

R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, Y. Chi, Deep keyphrase generation, in: R. Barzilay, M.-Y. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 582–592. URL: https://aclanthology.org/P17-1054. do...

doi:10.18653/v1/p17-1054 2017
[3]

Bezerra, F

C. Bezerra, F. Freitas, F. Santana, Evaluating ontologies with competency questions, in: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 3, 2013, pp. 284–285. doi:10.1109/WI-IAT.2013.199

doi:10.1109/wi-iat.2013.199 2013
[4]

J. Raad, C. Cruz, A survey on ontology evaluation methods, in: International conference on knowledge engineering and ontology development, volume 2, SciTePress, 2015, pp. 179–186

work page 2015
[5]

B. Xie, J. Song, L. Shao, S. Wu, X. Wei, B. Yang, H. Lin, J. Xie, J. Su, From statistical methods to deep learning, automatic keyphrase prediction: A survey, Information Processing & Management 60 (2023) 103382. URL: https://www.sciencedirect.com/science/article/pii/S030645732300119X. doi:https://doi.org/10.1016/j.ipm.2023.103382

arXiv 2023
[6]

Gómez-Pérez, Ontology evaluation, in: Handbook on ontologies, Springer, 2004, pp

A. Gómez-Pérez, Ontology evaluation, in: Handbook on ontologies, Springer, 2004, pp. 251–273

work page 2004
[7]

M. Song, Y. Feng, L. Jing, A survey on recent advances in keyphrase extraction from pre-trained language models, in: A. Vlachos, I. Augenstein (Eds.), Findings of the Association for Computational Linguistics: EACL 2023, Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 2153–2164. URL: https://aclanthology.org/2023.findings-eacl.161...

doi:10.18653/v1/2023 2023
[8]

K. S. Jones, A statistical interpretation of term specificity and its application in retrieval, J. Documentation 60 (2021) 493–502. URL: https://api.semanticscholar.org/CorpusID:2996187

work page 2021

Show all 30 references

[9]

Campos, V

R. Campos, V. Mangaravite, A. Pasquali, A. M. Jorge, C. Nunes, A. Jatowt, Yake! collection- independent automatic keyword extractor, in: G. Pasi, B. Piwowarski, L. Azzopardi, A. Hanbury (Eds.), Advances in Information Retrieval - 40th European Conference on IR Research, ECIR 2...

2018 doi
[10]

Mihalcea, P

R. Mihalcea, P. Tarau, Textrank: Bringing order into text, in: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing , EMNLP 2004, A meeting of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACL 2004, 25-26 July 2004, Bar...

2004
[11]

L. Page, S. Brin, R. Motwani, T. Winograd, The pagerank citation ranking : Bringing order to the web, in: The Web Conference, 1999. URL: https://api.semanticscholar.org/CorpusID:1508503

1999
[12]

Bougouin, F

A. Bougouin, F. Boudin, B. Daille, Topicrank: Graph-based topic ranking for keyphrase extraction, in: Sixth International Joint Conference on Natural Language Processing, IJCNLP 2013, Nagoya, Japan, October 14-18, 2013, Asian Federation of Natural Language Processing / ACL, 20...

2013
[13]

Florescu, C

C. Florescu, C. Caragea, Positionrank: An unsupervised approach to keyphrase extraction from scholarly documents, in: R. Barzilay, M. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - Aug...

2017 doi
[14]

Bennani-Smires, C

K. Bennani-Smires, C. Musat, A. Hossmann, M. Baeriswyl, M. Jaggi, Simple unsupervised keyphrase extraction using sentence embeddings, in: A. Korhonen, I. Titov (Eds.), Proceedings of the 22nd Conference on Computational Natural Language Learning, CoNLL 2018, Brussels, Belgium,...

2018 doi
[15]

H. M. Zahera, D. Vollmers, M. A. Sherif, A.-C. N. Ngomo, Multpax: Keyphrase extraction using language models and knowledge graphs, in: U. Sattler, A. Hogan, M. Keet, V. Presutti, J. P. A. Almeida, H. Takeda, P. Monnin, G. Pirrò, C. d’Amato (Eds.), The Semantic Web – ISWC 2022,...

2022
[16]

X. Shen, Y. Wang, R. Meng, J. Shang, Unsupervised deep keyphrase generation, Proceedings of the AAAI Conference on Artificial Intelligence 36 (2022) 11303–11311. URL: https://ojs.aaai.org/ index.php/AAAI/article/view/21381. doi:10.1609/aaai.v36i10.21381

2022 doi
[17]

A. Kong, S. Zhao, H. Chen, Q. Li, Y. Qin, R. Sun, X. Bai, PromptRank: Unsupervised keyphrase extraction using prompt, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

2023 doi
[18]

M. Song, X. Geng, S. Yao, S. Lu, Y. Feng, L. Jing, Large language models as zero-shot keyphrase extractors: A preliminary empirical study, 2024. arXiv:2312.15156

2024 arXiv
[19]

Y. Su, T. Lan, Y. Wang, D. Yogatama, L. Kong, N. Collier, A contrastive framework for neural text generation, Advances in Neural Information Processing Systems 35 (2022) 21548–21561

2022
[20]

Reimers, I

N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2019. URL: https://arxiv.org/abs/1908.10084

2019 arXiv
[21]

Hulth, Improved automatic keyword extraction given more linguistic knowledge, in: Pro- ceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, 2003, pp

A. Hulth, Improved automatic keyword extraction given more linguistic knowledge, in: Pro- ceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, 2003, pp. 216–223. URL: https://aclanthology.org/W03-1028

2003
[22]

Augenstein, M

I. Augenstein, M. Das, S. Riedel, L. Vikraman, A. McCallum, SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications, in: S. Bethard, M. Carpuat, M. Apidianaki, S. M. Mohammad, D. Cer, D. Jurgens (Eds.), Proceedings of the 11th Interna...

2017
[23]

A. Kong, S. Zhao, H. Chen, Q. Li, Y. Qin, R. Sun, X. Bai, Promptrank: Unsupervised keyphrase extraction using prompt, 2023. URL: https://arxiv.org/abs/2305.04490. arXiv:2305.04490

2023 arXiv
[24]

Lu, Pyate: Python implementation of term extraction algorithms, https://github.com/ kevinlu1248/pyate/tree/master, 2020

K. Lu, Pyate: Python implementation of term extraction algorithms, https://github.com/ kevinlu1248/pyate/tree/master, 2020. Accessed: 2024-07-03

2020
[25]

Boudin, Pke: an open source python-based keyphrase extraction toolkit, https://github.com/ boudinfl/pke, 2016

F. Boudin, Pke: an open source python-based keyphrase extraction toolkit, https://github.com/ boudinfl/pke, 2016. Accessed: 2024-07-03

2016
[26]

S. R. El-Beltagy, A. Rafea, Kp-miner: Participation in semeval-2, in: Proceedings of the 5th international workshop on semantic evaluation, 2010, pp. 190–193

2010
[27]

I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, C. G. Nevill-Manning, Kea: Practical automatic keyphrase extraction, in: Proceedings of the fourth ACM conference on Digital libraries, 1999, pp. 254–255

1999
[28]

Boudin, Unsupervised keyphrase extraction with multipartite graphs, arXiv preprint arXiv:1803.08721 (2018)

F. Boudin, Unsupervised keyphrase extraction with multipartite graphs, arXiv preprint arXiv:1803.08721 (2018)

2018 arXiv
[29]

X. Wan, J. Xiao, Collabrank: towards a collaborative approach to single-document keyphrase extraction, in: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), 2008, pp. 969–976

2008
[30]

Surfer, Rake: Rapid automatic keyword extraction algorithm, https://github.com/csurfer/ rake-nltk/tree/a80f633098dba19c409cb1778206189d8573696a, 2018

C. Surfer, Rake: Rapid automatic keyword extraction algorithm, https://github.com/csurfer/ rake-nltk/tree/a80f633098dba19c409cb1778206189d8573696a, 2018. Accessed: 2024-07-03

2018

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

K. S. Hasan, V. Ng, Automatic keyphrase extraction: A survey of the state of the art, in: K. Toutanova, H. Wu (Eds.), Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Lin- guistics, Baltimore, Maryland, 2014, pp. 1262–1273. URL: https://aclanthology.org/P14-1119....

doi:10.3115/v1/p14-1119 2014

[2] [2]

R. Meng, S. Zhao, S. Han, D. He, P. Brusilovsky, Y. Chi, Deep keyphrase generation, in: R. Barzilay, M.-Y. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 582–592. URL: https://aclanthology.org/P17-1054. do...

doi:10.18653/v1/p17-1054 2017

[3] [3]

Bezerra, F

C. Bezerra, F. Freitas, F. Santana, Evaluating ontologies with competency questions, in: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), volume 3, 2013, pp. 284–285. doi:10.1109/WI-IAT.2013.199

doi:10.1109/wi-iat.2013.199 2013

[4] [4]

J. Raad, C. Cruz, A survey on ontology evaluation methods, in: International conference on knowledge engineering and ontology development, volume 2, SciTePress, 2015, pp. 179–186

work page 2015

[5] [5]

B. Xie, J. Song, L. Shao, S. Wu, X. Wei, B. Yang, H. Lin, J. Xie, J. Su, From statistical methods to deep learning, automatic keyphrase prediction: A survey, Information Processing & Management 60 (2023) 103382. URL: https://www.sciencedirect.com/science/article/pii/S030645732300119X. doi:https://doi.org/10.1016/j.ipm.2023.103382

arXiv 2023

[6] [6]

Gómez-Pérez, Ontology evaluation, in: Handbook on ontologies, Springer, 2004, pp

A. Gómez-Pérez, Ontology evaluation, in: Handbook on ontologies, Springer, 2004, pp. 251–273

work page 2004

[7] [7]

M. Song, Y. Feng, L. Jing, A survey on recent advances in keyphrase extraction from pre-trained language models, in: A. Vlachos, I. Augenstein (Eds.), Findings of the Association for Computational Linguistics: EACL 2023, Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 2153–2164. URL: https://aclanthology.org/2023.findings-eacl.161...

doi:10.18653/v1/2023 2023

[8] [8]

K. S. Jones, A statistical interpretation of term specificity and its application in retrieval, J. Documentation 60 (2021) 493–502. URL: https://api.semanticscholar.org/CorpusID:2996187

work page 2021