{"id":"523460f7-e99d-419c-aba9-7f5b97b6dae6","arxiv_id":"2506.11067","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM pipeline with novel attribution algorithm extracts ROS entities, negation status, and body systems from 24 clinical notes at up to 0.952 F1 using open-source models.","lead":"This paper describes an LLM pipeline that pulls Review of Systems sections from clinical notes, then uses few-shot prompting on open-source models to tag symptoms, their positive or negative status, and body systems. A new attribution step matches the model outputs back to the original text even when wording differs.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Small set of 24 notes with no reported sampling or IAA leaves F1=0.952 unanchored for reliability claims","rationale":"The reader correctly isolated the representativeness of the 24-note corpus as the weakest link. Because the headline performance numbers are the only quantitative support for the “practical, cost-efficient” conclusion, any uncertainty about how those 24 notes were chosen or how consistently they were labeled directly weakens the inference from observed F1 to deployable reliability. No other internal inconsistency (model choice, pipeline stages, or attribution logic) appears more load-bearing on the evidence presented.","tokens_in":1825,"tokens_out":339,"duration_ms":14549,"concrete_test":"Locate the methods or supplementary section describing note acquisition and annotation protocol; if sampling details or IAA are absent, re-annotate a fresh random sample of 100 notes from the same institution with two independent clinicians and recompute all three F1 scores; a drop >0.05 would indicate the original numbers are not yet reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that measured performance (entity extraction, negation, body-system classification) on the 340 annotations generalizes beyond the test collection. The paper evaluates exclusively on 24 general-medicine notes; without explicit selection criteria, stratification by note length, specialty, or patient demographics, and without inter-annotator agreement on the gold labels, it is impossible to know whether the reported F1 scores reflect stable entity boundaries or merely idiosyncrasies of this particular sample. The attribution algorithm’s reported gains are likewise conditioned on the same narrow reference set.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents an LLM-based pipeline for extracting Review of Systems (ROS) entities from clinical notes. SecTag identifies the ROS section, after which four open-source LLMs (llama3.1:8b, gemma3:27b, mistral3.1:24b, gpt-oss:20b) are few-shot prompted to detect entities, negation status, and body-system classification. A novel attribution algorithm aligns LLM outputs with source text to handle non-exact and synonymous matches. Evaluation on 24 general-medicine notes containing 340 annotated entities reports F1 scores up to 0.952, with consistent gains from the attribution algorithm across tasks.","tokens_in":1954,"tokens_out":560,"duration_ms":46048,"significance":"If the performance claims hold under broader validation, the work offers a practical, locally deployable, cost-efficient solution for automating ROS documentation using open-source models, which is valuable for resource-limited clinical settings. The attribution algorithm provides a methodological contribution for improving zero- and few-shot NER alignment. The emphasis on open-source LLMs and real-world applicability is a strength.","major_comments":[{"comment":"Materials and Methods / Evaluation: The test collection is limited to 24 general-medicine notes and 340 annotations. No sampling criteria, stratification by note length or specialty, annotation protocol, or inter-annotator agreement statistics are reported. This small, single-site sample is load-bearing for the central claim that the pipeline delivers reliable performance (F1 = 0.952) and that the attribution algorithm produces generalizable improvements.","section":"Materials and Methods"},{"comment":"Results: Performance is reported for three tasks (entity extraction, negation detection, body-system classification) but without statistical significance testing, confidence intervals, or error analysis. It is therefore unclear whether the observed gains from the attribution algorithm are robust or could be explained by the particular characteristics of the 24-note set.","section":"Results"}],"minor_comments":[{"comment":"Abstract and Methods: Model names (e.g., 'gpt-oss:20b') should be clarified with exact Hugging Face or Ollama identifiers for reproducibility.","section":"Abstract"},{"comment":"The few-shot prompting details (number of examples, selection criteria, and prompt templates) are not fully specified; adding them would improve replicability.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a clinical NLP journal. The primary concern is the narrow evaluation; addressing it would substantially strengthen the submission without changing its scope."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. The comments have prompted us to improve the transparency of our evaluation and the rigor of our statistical reporting. We address each major comment below and indicate the revisions made to the manuscript.","responses":[{"response":"We agree that greater detail on dataset construction is warranted. In the revised Materials and Methods, we now specify the sampling criteria (consecutive general-medicine admission notes selected from a single academic medical center's EHR during a defined 2023 period) and provide the full annotation protocol used by the board-certified internist who labeled the 340 entities. Stratification by note length or specialty was not applied because the study scope was restricted to typical general-medicine notes. We acknowledge the small, single-site sample as a genuine limitation and have expanded the Discussion to frame the work as a proof-of-concept study with explicit plans for future multi-site validation. Inter-annotator agreement statistics are unavailable because annotation was performed by a single expert; this is now stated as a limitation.","revision_made":"partial","referee_comment":"[Materials and Methods] Materials and Methods / Evaluation: The test collection is limited to 24 general-medicine notes and 340 annotations. No sampling criteria, stratification by note length or specialty, annotation protocol, or inter-annotator agreement statistics are reported. This small, single-site sample is load-bearing for the central claim that the pipeline delivers reliable performance (F1 = 0.952) and that the attribution algorithm produces generalizable improvements."},{"response":"We have strengthened the Results section by adding bootstrap-derived 95% confidence intervals for all reported F1 scores. Statistical significance of the attribution algorithm's improvements over the baseline prompting approach was evaluated with McNemar's test for paired binary outcomes; p-values are now reported for each of the three tasks. We have also inserted a dedicated error-analysis subsection that categorizes the remaining errors (boundary mismatches, negation-scope failures, and body-system misclassifications) and illustrates how the attribution step reduces each category. These additions indicate that the observed gains are consistent across error types rather than artifacts of the particular 24-note collection.","revision_made":"yes","referee_comment":"[Results] Results: Performance is reported for three tasks (entity extraction, negation detection, body-system classification) but without statistical significance testing, confidence intervals, or error analysis. It is therefore unclear whether the observed gains from the attribution algorithm are robust or could be explained by the particular characteristics of the 24-note set."}],"tokens_in":1526,"tokens_out":545,"duration_ms":46499,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Hey, the main thing here is a simple pipeline that pulls Review of Systems entities, negation status, and body systems from clinical notes using open-source LLMs, plus a new attribution step to match outputs back to the source text when the phrasing is not exact. On 24 notes they reach F1 scores up to 0.952, and the attribution method lifts performance for all four models tested, including the smaller Llama that uses less memory. They also keep everything local and low-cost, which matches real constraints in many clinics. That part is useful and straightforward to follow. The evaluation is the soft spot. The test set is only 24 general-medicine notes with 340 annotations, and there is no information on how the notes were chosen or whether annotators agreed on the labels. Without those details the high scores stay tied to this particular sample, so claims about reliability and scalability rest on thin ground. The stress-test concern about generalization holds up from what is shown. This is aimed at people who build or adapt tools for clinical documentation and want an off-the-shelf LLM approach that runs without cloud APIs. A reader working on practical medical text processing could borrow the pipeline layout and the alignment trick. I would send it to peer review so referees can check the annotation process and ask for more on sampling and error patterns.","headline":"Open-source LLM pipeline for ROS extraction shows practical results on a narrow test set but the small sample and missing annotation details limit how far the F1 scores can be trusted.","tokens_in":2483,"tokens_out":345,"would_cite":false,"duration_ms":48000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Open-source LLMs extract Review of Systems entities from clinical notes with high accuracy in a local pipeline.","keywords":["large language models","clinical notes","entity recognition","review of systems","open-source LLMs","named entity recognition","healthcare documentation","attribution algorithm"],"falsifier":"A substantial drop in F1 scores or accuracy when the pipeline is applied to a larger collection of clinical notes from multiple hospitals or different medical fields would indicate the results do not generalize.","tokens_in":2744,"feed_emoji":"🩺","tokens_out":643,"duration_ms":42361,"temperature":0.7,"pith_summary":"The paper develops a pipeline that uses open-source large language models to automatically pull out Review of Systems details like symptoms, diseases, their positive or negative status, and related body systems from doctors' notes. This approach starts by identifying the relevant section and then applies few-shot learning to the models, followed by a new algorithm to match the extractions back to the original text. A sympathetic reader would care because it offers a way to handle repetitive documentation tasks more efficiently without relying on expensive or cloud-based services, potentially freeing up time for patient care. The results on a small set of notes show strong performance, especially with the matching step improving outcomes for all tested models.","feed_headline":"Open-source LLMs reach 0.95 F1 on ROS entity extraction from notes","feed_subtitle":"Local pipeline with new alignment step cuts documentation effort using smaller models and no cloud dependency.","key_machinery":"The LLM-based pipeline that first isolates the Review of Systems section using SecTag headers, then employs few-shot prompting on open-source models to detect entities along with their status and body systems, and uses a new attribution algorithm to link outputs back to the original text.","core_discovery":"The authors establish that a pipeline combining section extraction with SecTag, few-shot prompting on open-source LLMs, and a novel attribution algorithm for aligning entities to source text enables effective recognition of ROS entities, negation status, and body systems, achieving a highest F1 score of 0.952 and consistent improvements across models including smaller ones.","pith_inferences":["This method could extend to other sections of clinical notes for broader automation of medical documentation.","Local open-source solutions address data privacy and cost barriers in adopting AI for clinical use.","Testing the pipeline on notes from varied medical specialties would help assess its broader applicability."],"forward_implications":["Larger models demonstrate robust performance across entity extraction, negation detection, and body system classification.","The attribution algorithm increases F1 score and accuracy while reducing error rate for all models.","The smaller Llama model delivers promising results with significantly lower VRAM usage.","The pipeline offers a scalable and locally deployable solution for reducing ROS documentation burden in healthcare."],"fun_headline_variants":["LLM pipeline hits 0.95 F1 on ROS entity extraction from notes","New attribution step enhances LLM ROS entity recognition","SecTag and few-shot LLMs boost ROS documentation accuracy","Local open LLMs classify negation and body systems in notes"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The small set of 24 general medicine notes with 340 annotations is representative of typical clinical notes and sufficient to support the reported performance levels.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline hits 0.95 F1 on ROS entity extraction from notes","New attribution step enhances LLM ROS entity recognition","SecTag and few-shot LLMs boost ROS documentation accuracy","Local open LLMs classify negation and body systems in notes"]},"model":"grok-4.3","cost_usd":0.006465,"raw_usage":{"total_tokens":2993,"prompt_tokens":760,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":64653000,"prompt_tokens_details":{"text_tokens":760,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2166,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":760,"tokens_out":67,"duration_ms":23104,"temperature":1.0,"reasoning_tokens":2166,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-19T11:29:48.646259+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A substantial drop in F1 scores or accuracy when the pipeline is applied to a larger collection of clinical notes from multiple hospitals or different medical fields would indicate the results do not generalize.","supporting_citations":[],"review_version":1}