{"id":"a01ea40f-b76f-4bd4-9bd0-d486fba1702c","arxiv_id":"2506.16628","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs can retrieve relevant clinical snippets with high recall (0.98 to 0.99) and generate keywords that cover existing NER rules, but the evaluation lacks precision and baselines.","lead":"This paper tests whether large language models can help build rule-based clinical NLP systems by finding relevant snippets and suggesting keywords. The models achieved high recall for snippets but very low precision, and keyword coverage was measured without checking false positives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.0 keyword-extraction result is an artifact of an overgeneration-tolerant, circular metric: test snippets come from the original rules, precision is not measured, and the authors themselves concede generated keywords remain distant from the optimized rule set.","rationale":"The paper is an initial feasibility study; its central claim is that LLMs can reliably perform the first two development steps of a rule-based clinical NLP pipeline. I accept the snippet-identification recall numbers as credible evidence for Hypothesis 1, and I do not treat the low precision as disqualifying because the authors clearly state they prioritize recall for this collection stage and provide a qualitative error analysis. The load-bearing weakness is Hypothesis 2. The only quantitative support is a coverage rate measured against a test set constructed from the original EasyCIE rules, with no precision metric and no automated check against overgeneralization. The paper's own Discussion and Limitations sections concede that keyword generation is excessive and that the generated keywords remain distant from the optimized rule set. That makes the abstract's '1.0 in extracting key terms for NER' an overstatement: the metric can be satisfied by broad overgeneration, so it does not measure whether the keywords are useful for building NER rules. A simple trivial-baseline test would settle whether the coverage metric is vacuous. Because the study is preliminary, the qualitative examples show some promise, and the flaw is fixable with additional evaluation, the appropriate verdict remains conditional rather than reject. The reader's weakest-assumption analysis identified the same concern, so my assessment agrees with the reader's conditional verdict.","tokens_in":8372,"tokens_out":6883,"duration_ms":80266,"concrete_test":"Run a vacuity check on the coverage metric: submit each test snippet to a trivial baseline that outputs every content word (or every noun phrase) in the snippet as the 'keyword set,' and apply the paper's coverage evaluation to that baseline. If the trivial baseline also achieves 100% (or near-100%) snippet coverage, the metric cannot distinguish keyword quality and the reported 1.0 is vacuous; the abstract's keyword claim should then be revised or removed. If the baseline achieves substantially lower coverage, this concern is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim has two pillars: high-recall snippet identification and 1.0 coverage for keyword extraction. The first is credibly supported. The second is not. The evaluation in Methods (Tasks formulation; Evaluation) builds the keyword test set by running a subset of the existing EasyCIE NER rules and keeping only snippets that produced output; the metric then asks whether LLM-generated keywords 'cover' those same original rule outputs. Because coverage counts a snippet as covered if any generated keyword overlaps, a model that emits broad terms such as 'infection,' 'wound,' or 'antibiotic' will trivially reach 100%. No precision is measured, and the authors state explicitly in the Discussion: 'Not surprisingly, both models excessively generate keywords that covered the snippets of the original rules can match.' The Limitations section goes further: 'the keywords generated currently remain distant from the optimized rule set utilized in the NER component.' These admissions undermine the abstract's '1.0 in extracting key terms for NER.' Additionally, selecting test snippets via the original rules means the metric can only measure backward compatibility with the existing rule set; it cannot detect whether LLM keywords would help identify relevant text the original rules miss. The 1.0 therefore does not establish that LLM-generated keywords are NER-ready, and the central claim is not secured for the keyword-extraction step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using large language models (LLMs) during the development phase of a rule-based clinical NLP pipeline, rather than at runtime, and investigates the first two development steps: identifying clinically relevant snippets from patient notes and extracting keywords for a rule-based named entity recognition (NER) component. The authors evaluate their approach on a surgical site infection (SSI) detection task using the EasyCIE system, reporting high recall for snippet identification (Deepseek 0.98, Qwen 0.99) and 100% coverage for keyword extraction. They argue that these results provide initial evidence that LLMs can assist in semi-automating rule-based NLP development.","tokens_in":8653,"tokens_out":3058,"duration_ms":33482,"significance":"If the central feasibility claim were well supported, this would be a valuable contribution: it points to a way of combining the interpretability and efficiency of rule-based systems with the semantic capabilities of LLMs, using LLMs only at development time. The high recall for snippet identification is genuinely suggestive, and the transparent prompt design and error analysis are useful for the community. However, the keyword-extraction result, which is a key pillar of the abstract's claim, rests on an evaluation design that essentially guarantees high coverage and does not measure precision, so the current evidence does not establish that LLM-generated keywords are ready for NER rule construction.","major_comments":[{"comment":"The keyword extraction test set is constructed by running a subset of the existing EasyCIE NER rules and keeping only snippets that produced output, and the coverage metric asks whether any LLM-generated keyword overlaps with the original rule output. This design measures backward compatibility with the existing rules, not the quality or clinical utility of the generated keywords, and it makes 100% coverage trivially achievable by overgenerating broad terms. The authors explicitly acknowledge this in the Discussion ('Not surprisingly, both models excessively generate keywords that covered the snippets of the original rules can match') and in the Limitations ('the keywords generated currently remain distant from the optimized rule set utilized in the NER component'). The abstract's claim of '1.0 in extracting key terms for NER' is therefore not supported by the evidence.","section":"Methods – Tasks formulation; Methods – Evaluation"},{"comment":"The reported precision for snippet identification is 0.10 for Deepseek and 0.08 for Qwen, with F1 scores of 0.18 and 0.15. The paper argues that low precision is acceptable because many false positives are actually useful clinical cues, but the reclassification in Table 2 is based on the authors' own judgment and is not validated against an external gold standard or adjudicated by independent annotators. Without such validation, the claim that the false positives are 'largely informative' is not empirically established, and the high recall alone does not demonstrate that the LLMs reliably identify snippets useful for downstream rule development.","section":"Results – Table 1; Discussion – Snippet identification error analysis"},{"comment":"Because the keyword test snippets are selected by running the original EasyCIE rules, the evaluation cannot detect whether LLM-generated keywords would help identify relevant text that the original rules miss. The study therefore cannot support the broader claim that LLMs assist in extending or improving a rule-based system beyond the existing rule set, only that they can reproduce some of its outputs. This is a load-bearing limitation for the paper's central argument that LLMs can accelerate rule-based NLP development.","section":"Discussion – Findings of the keyword extraction task; Limitations"}],"minor_comments":[{"comment":"The abstract describes the recall as 'exceptional' without mentioning the very low precision and F1 scores; reporting these values in the abstract, or at least a balanced summary, would set more accurate expectations.","section":"Abstract; Results – Table 1"},{"comment":"The sentence 'Additionally, the expanded keywords instructions seem inspiring to make the rules more generalizable' appears twice in close succession, with slightly different wording, and should be reduced to one occurrence.","section":"Discussion – Findings of the keyword extraction task"},{"comment":"There is a typo 'if if such a snippet were merely part of a routine follow-up' that should be corrected.","section":"Discussion – Snippet identification error analysis"},{"comment":"The appendix contains 'dictonary' instead of 'dictionary' and uses an incomplete example template that may confuse readers; a careful proofread of the appendix is recommended.","section":"Appendix – Keywords extraction prompts"},{"comment":"The note 'Some snippets fall into multiple types' makes the counts in Table 2 ambiguous; presenting the number of snippets in each exclusive category, or the total number of snippets classified, would improve clarity.","section":"Results – Table 2"},{"comment":"No confidence intervals or statistical significance tests are reported for the recall, precision, or coverage estimates; given the small test sets likely involved, this would help assess the reliability of the results.","section":"Methods – Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The paper's abstract substantially overstates the keyword-extraction result, which is an artifact of a circular evaluation design. The snippet-recall result is more credible but is paired with extremely low precision, and the reclassification of false positives as useful lacks external validation. The authors' own Limitations section acknowledges the main weaknesses. I believe the work has potential but needs a fundamentally better evaluation of keyword quality, a more balanced presentation, and preferably an external gold standard for snippet informativeness before it can be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is the first I've seen that uses LLMs to help build rule-based clinical NLP components rather than replace them, and the snippet-recall result is credible. But the star claim — 1.0 keyword coverage — is an artifact of how the test set was built, and the authors' own limitations section basically concedes that. So: a conditional, not a reject, but the abstract oversells.\n\nWhat's new: applying existing prompting techniques (CoT, MoPE, few-shot) to the development workflow of a rule-based SSI detection system. That's a sensible niche. The evaluation of snippet identification is straightforward and the high recall (0.98, 0.99) looks supported by the reported counts. The error analysis is genuinely useful — distinguishing true false positives from informative \"over-inclusion\" is the right kind of discussion for this problem. Credit where due: they don't hide the low precision (0.10, 0.08) and they attempt to explain it.\n\nThe soft spot is the keyword extraction result. The test snippets were generated by running the original EasyCIE rules, and the metric asks whether LLM keywords \"cover\" the rule outputs. Since a single overlapping keyword counts as coverage, a model emitting broad terms like \"infection\" or \"wound\" hits 100% trivially. No precision is measured. And in the Discussion they write \"both models excessively generate keywords\" and in Limitations \"the keywords generated currently remain distant from the optimized rule set.\" That's an admission that the 1.0 doesn't mean NER-ready. The abstract's \"1.0 in extracting key terms for NER\" is simply not supported.\n\nLesser issue: the snippet precision is so low that calling the overall approach \"exceptional recall\" is fair, but the reclassification of most false positives as useful is done with an ad hoc scheme. It's plausible, but not validated.\n\nWho this is for: clinical NLP groups thinking about whether LLMs can shortcut the early, labor-intensive parts of rule-based development. They'll get a useful initial data point, especially the error typology. The keyword extraction part should be treated as a pilot, not a result.\n\nMy recommendation: send it to peer review, but with the expectation of revision. The keyword evaluation needs a real precision measure and a test set not derived from the original rules — or the claims need to be scaled back to \"suggested keywords are a promising starting point.\" As is, it's a conditional.","headline":"First clinical NLP study to use LLMs in the development loop of a rule-based system; the snippet recall result is credible, but the 1.0 keyword coverage is a circular artifact and the paper's own limitations concede it — conditional, not reject.","tokens_in":9126,"tokens_out":1819,"would_cite":false,"duration_ms":18798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can do the two hardest early steps of rule-based clinical NLP with near-perfect recall.","keywords":["LLM-assisted development","rule-based NLP","clinical NLP","surgical site infection surveillance","named entity recognition","snippet identification","keyword extraction","chain-of-thought prompting"],"falsifier":"Run the keyword-extraction prompt on a random sample of snippets from the annotated corpus that the existing EasyCIE rules do not match, then measure how many extracted keywords correspond to annotated concepts and how many would cause false matches in a rule-based NER pass. If coverage drops well below 100% or precision approaches zero, the keyword-extraction claim is an artifact of the evaluation design rather than a property of the method.","tokens_in":8188,"feed_emoji":"🩺","tokens_out":6861,"duration_ms":66451,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models can take over the two most labor-intensive early steps of building a rule-based clinical NLP system: picking out the clinically relevant snippets from patient notes and listing the key terms that become the rules for named entity recognition. The authors test this on EasyCIE, a rule-based surgical site infection surveillance system, using two instruction-tuned open-weight models. They report snippet-identification recall of 0.98 and 0.99, meaning the models found nearly all relevant snippets, and they report that the generated keywords covered 100% of the snippets the existing rules matched. The point of the approach is that the expensive LLM works only during development, while the finished system stays rule-based, transparent, and cheap to run.","feed_headline":"LLMs spot 98–99% of infection clues in surgical notes","feed_subtitle":"The LLM writes the tedious first steps; the finished surveillance system stays rule-based, fast, and auditable.","key_machinery":"The machinery is a two-pass prompt protocol wrapped around a rule-based pipeline. Each task has a reasoning prompt that embeds the NSQIP SSI manual as markdown and asks the model to walk through the relevant subtasks (signs and symptoms, treatment, anatomic sites, procedures, infection, wound care), then a verification prompt that embeds the annotation guideline, asks the model to critique the first pass, and requires a JSON output. Snippet identification is run five times and decided by majority vote; keyword extraction asks for both verbatim concepts and expanded synonyms, then prunes certainty, temporality, and experiencer modifiers so downstream rules can handle them. EasyCIE, the target system, is a modular rule-based NLP pipeline whose NER component matches keywords against note text, and the two LLM steps feed exactly that component.","core_discovery":"The paper's central claim is that the first two development steps of a rule-based clinical NLP pipeline can be delegated to an LLM without sacrificing the downstream system's rule-based character. Working from the EasyCIE surgical site infection pipeline, the authors cast snippet identification as binary classification: given a sentence-level snippet from a postoperative note, decide whether it carries any cue relevant to an SSI diagnosis. On the held-out test set, Deepseek R1 distilled Qwen 32B reached recall 0.98 and Qwen2.5 Coder 32B reached 0.99, with precision 0.10 and 0.08 respectively. A manual review of 100 errors per model found that most apparent false positives were snippets an annotator had skipped for contextual reasons rather than true noise. For keyword extraction, both models produced keyword sets that covered 100% of the snippets that the existing EasyCIE NER rules matched, and the authors show examples where generated keywords were broadened into more generalizable rule candidates.","pith_inferences":["A fairer test of the keyword claim would measure coverage and precision on snippets sampled independently of the existing EasyCIE rules; the reported 100% coverage is measured against snippets the old rules themselves produced, so it is an upper bound on how well the keywords would serve a fresh rule set.","If the development-phase-only pattern generalizes, the same workflow could apply in other regulated text-processing domains where explainability is mandatory, such as diagnostic coding, prior authorization, or adverse-event reporting.","An end-to-end comparison—building NER rules from the generated keywords, adding the downstream context detector, and comparing final SSI classification against EasyCIE—would settle whether the 100% keyword coverage actually translates into equal or better system performance.","The similar recall across two different model families suggests the bottleneck is the prompt-embedded domain knowledge rather than the specific model; a controlled ablation with and without the NSQIP manual in the prompt would test this."],"forward_implications":["If the two hypotheses hold, the dominant labor cost of rule-based NLP development—reading notes and drafting rules—can be shifted to an LLM, with clinicians editing candidates instead of starting from scratch.","Because the LLM is used only at build time, the deployed surveillance system retains the speed, low cost, and interpretability that make rule-based NLP attractive in clinical settings.","High recall in snippet identification means the downstream NER and context components receive nearly all relevant evidence; the low precision may be tolerable if later stages filter false positives.","The keyword sets, including expanded synonyms, give developers a concrete starting point for NER rules and can make the rules more generalizable than the original hand-built rules.","The prompt design, which embeds the manual and guideline directly, points to a route for semi-automated maintenance: when guidelines change, prompts can be regenerated rather than re-engineering rules by hand."],"supporting_citations":[{"why":"Supplies the EasyCIE system, the NSQIP-annotated dataset, and the existing NER rules used to define both evaluation tasks.","marker":"[11]"},{"why":"Provides the medspaCy sentence segmentation that turns annotated clinical notes into the snippets used in both tasks.","marker":"[16]"},{"why":"Supplies the chain-of-thought prompting strategy used in the reasoning step of both tasks.","marker":"[17]"},{"why":"Supplies the mixture-of-prompt-experts strategy that shapes the iterative prompt refinement.","marker":"[18]"},{"why":"Documents that rule-based information extraction systems remain widely deployed, which is the problem the authors aim to de-risk.","marker":"[1]"}],"fun_headline_variants":["LLM-assisted rules recall 99% of clinical cues","LLM drafts clinical rules: 98-99% recall, 100% keyword coverage","Rule-based NLP, LLM-built: recall 0.99","LLM finds 99% of clues; rules stay audit-friendly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The keyword result rests on the assumption that matching the snippets the old rules already catch is a good test of whether the generated keywords would support a new rule-based component; if that test is too easy, the 100% coverage figure does not carry the weight the paper gives it.","fun_headline_variants_meta":{"raw":{"variants":["LLM-assisted rules recall 99% of clinical cues","LLM drafts clinical rules: 98-99% recall, 100% keyword coverage","Rule-based NLP, LLM-built: recall 0.99","LLM finds 99% of clues; rules stay audit-friendly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000849,"raw_usage":{"total_tokens":3688,"prompt_tokens":932,"completion_tokens":2756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2678}},"tokens_in":548,"tokens_out":2756,"duration_ms":21280,"temperature":1.0,"reasoning_tokens":2678,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:21:23.330448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the keyword-extraction prompt on a random sample of snippets from the annotated corpus that the existing EasyCIE rules do not match, then measure how many extracted keywords correspond to annotated concepts and how many would cause false matches in a rule-based NER pass. If coverage drops well below 100% or precision approaches zero, the keyword-extraction claim is an artifact of the evaluation design rather than a property of the method.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EasyCIE system, the NSQIP-annotated dataset, and the existing NER rules used to define both evaluation tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the medspaCy sentence segmentation that turns annotated clinical notes into the snippets used in both tasks."},{"cited_title":"MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion","cited_arxiv_id":"2403.10568","evidence_quote":"Supplies the mixture-of-prompt-experts strategy that shapes the iterative prompt refinement."},{"cited_title":"& Reiss, F","cited_arxiv_id":null,"evidence_quote":"Documents that rule-based information extraction systems remain widely deployed, which is the problem the authors aim to de-risk."}],"review_version":2}