{"id":"e1c028de-7751-4b3a-b721-86c3e12a560f","arxiv_id":"2504.12915","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Prompting Llama3 70B with one random in-context example achieves the best reported F1 on present-concept extraction for Inspec and SemEval2017, outperforming traditional unsupervised keyphrase extractors.","lead":"This paper tests whether large language models can extract domain concepts from scientific papers by prompting them, and compares many model and prompt combinations on two standard datasets. The best configuration, Llama3 70B with one random example, reports the highest F1 scores, beating classical keyphrase extraction baselines on both datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F1 is chosen on the test set; without a validation split the SOTA claim is a selected maximum, not a pre-specified comparison.","rationale":"The reader's weakest assumption (ground-truth keyphrases as a proxy for all present concepts) is real and should be acknowledged or fixed. However, the numerical SOTA claim is undercut more directly by the selection protocol: the model and prompt are chosen by looking at test-set F1. Because Table 3 alone compares 40 model-prompt pairs and Table 4 adds 13 prompt variants on the same test set, the reported best F1 is the maximum of a grid and will be optimistically biased even if every individual number is accurate. The small differences among few-shot prompts (e.g., FS 1-Fixed 0.442, FS 1-Random 0.451, FS 1-Closest 0.445) and the unseeded random example selection make the choice especially fragile. A validation-based re-selection is the minimal check that would let a reader trust the 'surpass SOTA' claim. The proxy/task mismatch remains a separate framing issue, so I keep the conditional verdict rather than accepting or rejecting outright. The paper's code and fixed model versions are a real strength; the concern is about the statistical protocol around the headline number, not about the individual experimental measurements.","tokens_in":13822,"tokens_out":7119,"duration_ms":73828,"concrete_test":"Treat the Inspec training split (976 abstracts) as validation. Run all prompt/model configurations of Tables 3 and 4 on this split; for FS n-Random, average over at least 10 random seeds. Select the single best configuration by validation F1. Freeze that configuration and its seed, then run it once on the official Inspec and SemEval2017 test sets, comparing with the same baselines. If the validation-selected configuration still beats PositionRank by the reported margins (or a bootstrap 95% CI excludes the baseline), the SOTA claim survives; if the winner on validation is a different prompt or its test F1 drops substantially, the headline is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result is an order statistic, not an estimate for a fixed system. Section 4.4 first uses test-set F1 to fix the model: \"Based on the results of Table 3, the Llama3 70B model was fixed, and term keyphrases is selected\". It then uses Table 4 on the same Inspec test set to select the prompt, and finally reports the best configuration (FS 1-Random) in Table 5 as \"ours\". No validation split, no multiple-comparison control, and no significance test separate this winner from at least 40 model×prompt combinations in Table 3 and 13 prompt variants in Table 4. Additionally, FS n-Random is defined as drawing random training examples per test document, and no random seed is reported in Section 4.5; the headline F1 values (0.451 Inspec, 0.311 SemEval2017) may depend on one random draw. Thus the central SOTA claim rests on a maximum over a noisy grid evaluated on the test set. This is more directly damaging than the keyphrase-vs-concept annotation mismatch: even if the datasets matched the intended task, the reported comparison would not establish SOTA as it stands.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ConExion, a prompt-based method that uses large language models (LLMs) to extract present keyphrases/concepts from scientific abstracts. The approach builds a system/user/assistant prompt template, generates a comma-separated list with greedy decoding, splits the output by several delimiters, and filters to exact matches in the input document; confidence scores are computed as the geometric mean of token probabilities. The authors evaluate eight LLMs (Llama-2, Llama-3, Mistral, Mixtral, GPT-3.5-Turbo) on Inspec and SemEval2017 with zero-shot prompts over five search terms, four context-augmented prompts, and few-shot prompts with 1/3/5 fixed, random, and closest training examples. They compare against 14 unsupervised keyphrase-extraction baselines and report that FS 1-Random with Llama3 70B achieves the highest F1 on both datasets (0.451 on Inspec, 0.311 on SemEval2017), which is the central state-of-the-art claim.","tokens_in":14066,"tokens_out":4071,"duration_ms":41359,"significance":"If the reported result holds, the paper would make a useful empirical contribution: a simple, reproducible LLM-prompting recipe that outperforms a broad set of traditional unsupervised keyphrase extractors, with public code and fixed model/dataset versions. The zero-shot finding is noteworthy: even the ZS Keyphrases prompt with Llama3 70B reaches F1 0.414 on Inspec, above all 14 baselines in Table 5, so the qualitative conclusion that simple LLM prompting is competitive does not depend on the few-shot selection. However, the headline claim of state-of-the-art concept extraction is currently not established because the best configuration is selected on the test set, and because the benchmark measures keyphrase extraction rather than the paper's stated target of extracting all present domain concepts.","major_comments":[{"comment":"The headline configuration is selected using test-set performance, making the SOTA claim an order statistic rather than a pre-specified comparison. The text explicitly states: \"Based on the results of Table 3, the Llama3 70B model was fixed, and term keyphrases is selected,\" then Table 4 uses the same Inspec test set to select the prompt, and Table 5 reports the winning configuration as \"ours.\" With 40 model-prompt combinations in Table 3 and 13 prompt variants in Table 4, the best F1 is the maximum over a large grid. There is no validation split, no multiple-comparison control, and no significance test. The authors should either introduce a validation split (or nested selection), report significance tests, or explicitly present the zero-shot result as the primary claim and the few-shot variant as exploratory.","section":"Section 4.4, Tables 3-5"},{"comment":"The FS n-Random results are not reproducible as reported. The method draws random training examples for each test document, but Section 4.5 states that reproducibility is ensured without reporting any random seed. The headline F1 values (0.451 on Inspec, 0.311 on SemEval2017) may therefore depend on a single undocumented random draw. Please report the seed, or better, run multiple draws and report the mean and standard deviation.","section":"Section 3 and Section 4.5"},{"comment":"There is a mismatch between the task claimed and the task measured. The abstract and introduction frame the contribution as extracting \"all present concepts related to the specific domain, not just the important ones,\" but the evaluation uses Inspec and SemEval2017, whose ground truth consists of keyphrases annotated for importance, and the metrics compare against those keyphrase annotations. The paper's own formal definition in Section 1 also describes concepts as \"words that best represent the topics,\" which is keyphrase-like. Consequently, the reported F1 scores measure keyphrase extraction, not extraction of all present domain concepts. The authors should either reframe the claims to keyphrase extraction or evaluate on a dataset / with an annotation scheme that captures all present concepts; otherwise the central concept-extraction claim is not supported.","section":"Abstract, Section 4.1, and Section 4.2"},{"comment":"The recall-at-k formula is written as R@k = |C:k ∩ C'| / |C':k|. Since C' is the unordered ground-truth set, the denominator |C':k| is undefined as written and contradicts the preceding sentence stating that recall is measured against all ground-truth concepts. This should be corrected to |C'| (or C':k must be defined). This is a presentation issue but it affects the comparability of the reported top-k numbers.","section":"Section 4.2, Eq. (R@k)"}],"minor_comments":[{"comment":"The reproducibility statement \"the versions of datasets and models from HuggingFace were fixed, ensuring that the results could be reproduced\" is in tension with the undocumented randomness of FS n-Random prompts; please add the missing seed or clarify that only the deterministic parts are reproducible.","section":"Section 4.5"},{"comment":"The FS n-Random row in Table 1 says \"randomly selected examples\" but does not state that the selection is per test document and whether it is seeded; the text in Section 3 provides this detail, so aligning the table with the text would improve clarity.","section":"Table 1"},{"comment":"The header \"Maxcon / Mincon / Avgcon\" is a single merged column, which is hard to read; separate the columns so the statistics are clear.","section":"Table 2"},{"comment":"References [17] and [23] are the same paper (PromptRank); please merge or cite one entry.","section":"References"},{"comment":"The terms \"concepts,\" \"keyphrases,\" and \"keywords\" are sometimes used interchangeably (e.g., abstract vs. Section 1 vs. Table 5); a brief definition at first use would reduce ambiguity.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is probably best received as an empirical study of LLM prompting for keyphrase extraction rather than as a demonstration of a new concept-extraction task. The zero-shot result is likely robust and could be a useful contribution, but the current test-set-based selection and the task mismatch need to be addressed before the SOTA claim is credible. I would not reject, because the issues are fixable by reframing and by adding a validation split or re-running with seeds, but they are central to the paper's main claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2504.12915. First, it is a genuinely useful empirical resource: eight LLMs, five search terms, a large set of zero- and few-shot prompt variations, run on two standard keyphrase datasets with code and pinned versions. Second, the headline claim — Llama3 70B with a few-shot prompt reaching 0.451 F1 on Inspec, far above classical baselines — is a selected maximum from a test-set sweep, not a pre-specified comparison. The same goes for the SemEval2017 number. The direction of the result is credible; the exact SOTA number is not.\n\nWhat the paper does well: the sweep is broad, and results are reported with precision, recall, F1, top-k variants, and extraction count. The non-top-k F1 is the right metric for ontology coverage, and the confidence score via geometric mean of token probabilities is a sensible, reproducible trick. The comparison against 14 classical baselines is fair, and the code and data release is real. The most robust finding — which I buy — is that even a zero-shot \"keyphrases\" prompt with Llama3 70B beats the best classical unsupervised method (0.414 vs 0.287 F1 on Inspec). So the broad conclusion is not fragile.\n\nSoft spots, in order of severity. (1) Model and prompt are chosen on the test set. Table 3 fixes the model, Table 4 fixes the prompt, then Table 5 reports the winner. No validation split, no multiple-comparison control, no significance tests. The gap between FS 1-Random (0.451) and the zero-shot default (0.414) is small enough that selection could be noise. Also, FS n-Random draws random training examples per test document, and no seed is reported; the headline might move with one random draw. (2) The abstract promises \"all present concepts,\" but the evaluation uses keyphrase annotations that mark important concepts, and the prompts themselves say \"keyphrases.\" So the measured task is keyphrase extraction, not the claimed concept extraction. This matters for ontology evaluation, where coverage of all domain concepts is the goal. (3) Exact-match filtering is acknowledged as a limitation, but the paper never shows how much LLM output gets discarded by it.\n\nWho should read this: anyone working on practical keyphrase extraction for ontology engineering, or on prompt-based NLP recipes. It is not a theoretical contribution, but it is a valuable empirical resource. It deserves a serious referee, not because the SOTA claim holds as stated, but because the methodology is fixable: hold-out split for prompt selection, seeded random examples, and honest task framing. I would send it to peer review, likely at a workshop or short-paper venue, and ask for those easy fixes. I would cite the broad result, not the specific number.","headline":"Useful empirical sweep, but the headline SOTA is a test-set-selected maximum and the evaluation conflates keyphrases with concepts.","tokens_in":14586,"tokens_out":2773,"would_cite":true,"duration_ms":29346,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConExion claims that a reproducible few-shot LLM prompting pipeline—asking a model to list keyphrases present in a document, then keeping only phrases that literally appear—extracts present domain concepts more accurately than prior…","keywords":["Concept extraction","Keyphrase extraction","Large language models","Few-shot prompting","Zero-shot prompting","Ontology evaluation","Unsupervised extraction"],"falsifier":"Take a random sample of Inspec and SemEval2017 abstracts and have annotators mark every noun phrase that denotes a domain concept, regardless of importance; run ConExion's best configuration on the same documents and compute F1 against this mention-level gold standard. If that F1 is not clearly higher than the keyphrase-gold F1—or if a plain noun-phrase chunker matches it—the claim of extracting all present concepts, rather than important keyphrases, is not supported.","tokens_in":13635,"feed_emoji":"🤖","tokens_out":7298,"duration_ms":66520,"temperature":0.7,"pith_summary":"The paper tries to establish that large language models, used with carefully chosen prompts and no training, are the best current extractors of present concepts from scientific abstracts. The authors argue that concept extraction for ontology evaluation should return all concepts that occur in a document, not just top-k important keyphrases, and they build a pipeline that asks an LLM for keyphrases and filters the output to exact document substrings. On two standard benchmarks, the best configuration—one random example in the prompt with Llama3 70B—reaches F1 0.451 on Inspec and 0.311 on SemEval2017, higher than all compared traditional and embedding-based methods. They also show that prompt wording and a single in-context example matter more than model scale alone.","feed_headline":"One-shot Llama3 70B prompt beats prior concept extractors","feed_subtitle":"A filtered LLM output hits F1 0.451 on Inspec and 0.311 on SemEval2017 without training.","key_machinery":"The load-bearing mechanism is the chat prompt plus the present-concept filter. The prompt defines the extraction task through three slots—system, user, assistant—and is converted into each model's expected chat format by the tokenizer's chat template; the authors test five zero-shot search terms, four context-augmented zero-shot prompts, and few-shot prompts with fixed, random, or embedding-closest examples. The filter removes any generated phrase that is not an exact substring of the document, which implements the 'present' constraint and also strips conversational filler. Confidence scores come from the geometric mean of per-token generation probabilities under greedy decoding, which is what allows F1@5 and F1@10 comparisons.","core_discovery":"The discovery is empirical: a simple, fully reproducible setup for present concept extraction outperforms the previous state of the art. The method uses an instruction-tuned LLM as an extractor, with a system prompt, user prompt, and assistant prompt, and the model is asked to output concepts separated by commas. Greedy decoding keeps the run deterministic, and each extracted phrase receives a confidence score equal to the geometric mean of its token probabilities, which supports ranking. The output is split on commas, semicolons, stars, and newlines, then filtered to phrases that occur verbatim in the input, guaranteeing that every reported concept is grounded in the document. On Inspec, the FS 1-Random prompt with Llama3 70B achieves precision 0.413, recall 0.575, and F1 0.451; on SemEval2017 it achieves precision 0.301, recall 0.372, and F1 0.311, the highest F1 among all compared models on both datasets.","pith_inferences":["An implication the paper leaves implicit is that the benchmark mismatch is directly testable: Inspec and SemEval2017 labels are important-keyphrase annotations, not exhaustive annotations of every present concept, so evaluating against a full mention-level concept annotation would likely change the reported recall and precision substantially.","The exact-substring filter is a strict lower bound on the model's semantic recall; allowing inflectional, hyphenation, or synonym variants would probably raise F1, since the paper itself notes that semantically accurate concepts are discarded.","The token-probability confidence score is a byproduct of greedy decoding, but it could be reused as a ranking feature or compared against scores from sampled decoding to test whether it truly orders concepts by importance.","The prompt sensitivity observed here suggests that published concept-extraction numbers may not transfer across LLM generations; a useful extension would be a small prompt and example-selection protocol that minimizes variance across model versions."],"forward_implications":["For ontology coverage evaluation, the method provides an unsupervised, training-free way to produce a grounded set of concepts from a corpus, with a confidence score for each concept, so no user-chosen threshold is required.","Prompt wording is a first-order performance factor: on Inspec, switching Llama3 70B from 'keywords' to 'keyphrases' raises F1 from 0.362 to 0.414, and adding a task description raises it further to 0.431.","A single in-context example is enough to teach the output format; beyond one example, adding more fixed, random, or closest examples yields little additional gain, and five examples can even lower F1.","LLM-based extraction is more precise but less comprehensive than graph-based extraction: PositionRank beats ConExion on recall, so the best near-term improvement is a hybrid that combines LLM precision with graph-based recall.","Because greedy decoding is used, the full pipeline is deterministic; fixing model checkpoints, tokenizer chat templates, and datasets means the reported numbers can be reproduced exactly."],"supporting_citations":[{"why":"Supplies the Inspec benchmark, one of the two test sets whose ground-truth keyphrases define the evaluation.","marker":"[21]"},{"why":"Supplies the SemEval2017 benchmark, the second test set used for comparison.","marker":"[22]"},{"why":"Provides the sentence embeddings used to retrieve closest training documents for the n-Closest few-shot prompts.","marker":"[20]"},{"why":"Provides the TextRank baseline, a graph-based unsupervised extractor the approach is compared against.","marker":"[10]"},{"why":"Provides the PositionRank baseline, which is identified as achieving higher recall than the proposed method.","marker":"[13]"},{"why":"Provides the EmbedRank baseline, an embedding-based unsupervised keyphrase extractor included in the comparison.","marker":"[14]"},{"why":"Provides the MultPAX baseline, a language-model and knowledge-graph keyphrase extraction method included in the comparison.","marker":"[15]"},{"why":"Supplies the prior study of LLMs as zero-shot keyphrase extractors that motivates the prompt-based direction and the gap this paper fills.","marker":"[18]"}],"fun_headline_variants":["LLM concept extraction hits top F1 on two benchmarks","Llama3 70B prompt extracts all concepts, not just keyphrases","Prompt-only concept extraction outperforms state of the art","F1 0.451 Inspec, 0.311 SemEval: prompt-only LLM wins","Grounding LLM output in source text boosts concept recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the ground-truth keyphrases in Inspec and SemEval2017—annotations of what is important in a document—are a valid proxy for the full set of present domain concepts that ConExion claims to extract, so the reported F1 numbers measure the stated task only if that proxy holds.","fun_headline_variants_meta":{"raw":{"variants":["LLM concept extraction hits top F1 on two benchmarks","Llama3 70B prompt extracts all concepts, not just keyphrases","Prompt-only concept extraction outperforms state of the art","F1 0.451 Inspec, 0.311 SemEval: prompt-only LLM wins","Grounding LLM output in source text boosts concept recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1215,"prompt_tokens":895,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":224}},"tokens_in":511,"tokens_out":320,"duration_ms":3758,"temperature":1.0,"reasoning_tokens":224,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:19:20.586128+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Inspec and SemEval2017 abstracts and have annotators mark every noun phrase that denotes a domain concept, regardless of importance; run ConExion's best configuration on the same documents and compute F1 against this mention-level gold standard. If that F1 is not clearly higher than the keyphrase-gold F1—or if a plain noun-phrase chunker matches it—the claim of extracting all present concepts, rather than important keyphrases, is not supported.","supporting_citations":[{"cited_title":"Hulth, Improved automatic keyword extraction given more linguistic knowledge, in: Pro- ceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, 2003, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the Inspec benchmark, one of the two test sets whose ground-truth keyphrases define the evaluation."},{"cited_title":"Augenstein, M","cited_arxiv_id":null,"evidence_quote":"Supplies the SemEval2017 benchmark, the second test set used for comparison."},{"cited_title":"Mihalcea, P","cited_arxiv_id":null,"evidence_quote":"Provides the TextRank baseline, a graph-based unsupervised extractor the approach is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MultPAX baseline, a language-model and knowledge-graph keyphrase extraction method included in the comparison."},{"cited_title":"Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Study","cited_arxiv_id":"2312.15156","evidence_quote":"Supplies the prior study of LLMs as zero-shot keyphrase extractors that motivates the prompt-based direction and the gap this paper fills."}],"review_version":1}