{"id":"b87d6fa8-11a7-4f7c-8d05-0eee1744bb64","arxiv_id":"2605.26560","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Hybrid neural-symbolic pipeline extracts (action, date) pairs from clinical notes at 0.99 Pair F1 by using BioBERT tagging plus deterministic time normalization, outperforming LLMs on a synthetic benchmark with OOV actions.","lead":"The paper tests a hybrid system that tags medical actions and times in notes using BioBERT then links and normalizes dates with fixed rules, achieving near-perfect pairing accuracy on synthetic data. A generalist might read it to see how separating learned detection from deterministic calculation improves reliability over pure generative models for structured extraction tasks.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Near-perfect scores rest on untested assumption that synthetic corpus matches real-note linguistic variety and time ambiguity","rationale":"The load-bearing concern is identical to the reader's weakest_assumption. The benchmark results themselves contain no internal contradiction or calculation error; the limitation is external validity, which the paper already flags. Therefore the reader's UNVERDICTED verdict requires no adjustment.","tokens_in":1809,"tokens_out":312,"duration_ms":25348,"concrete_test":"Annotate 50–100 real de-identified outpatient notes with the same TestSpecification/TimeSpecification schema, run the published hybrid pipeline, and recompute note-level Pair F1 and MAE; if Pair F1 falls below 0.90 or MAE exceeds 1 day on the real set, the synthetic benchmark results do not support the reliability claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim (Test-Time Pair F1 0.997/0.986, 0.00-day MAE on 259-note seen/OOV splits) is obtained exclusively on a 2,000-note synthetic outpatient corpus whose generation process, action distribution, and time-expression coverage are not shown to replicate authentic EHR notes. The hybrid pipeline's deterministic normalization and 28-action ontology are tailored to this corpus; the paper itself states in the conclusion that \"Transfer to real EHR notes is the next validation.\" Without that transfer, the reported advantage over generative baselines cannot be taken as evidence of reliability in the intended clinical setting.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a hybrid neural-symbolic pipeline for extracting follow-up instructions as (action, time) pairs from outpatient clinical notes. It uses BioBERT for BIO tagging and biaffine linking to identify TestSpecification and TimeSpecification entities and ScheduledFor relations, followed by canonicalization with a 28-action ontology and deterministic time normalization to day offsets. Evaluated on a 2,000-note synthetic corpus with action-disjoint train/test splits (including OOV actions), the pipeline achieves Test-Time Pair F1 of 0.997 (seen) and 0.986 (OOV) with 0.00-day MAE, significantly outperforming zero-shot GPT-4o-mini and LoRA-fine-tuned LLaMA-3 8B baselines whose Pair F1 remains around 0.5-0.57 despite high action F1.","tokens_in":1915,"tokens_out":567,"duration_ms":27984,"significance":"If the synthetic corpus adequately represents real clinical notes, the result demonstrates that separating learned entity extraction from deterministic symbolic date handling can achieve near-perfect performance and generalization to held-out actions, where end-to-end generative models fail on linking and arithmetic. The bootstrap 95% CIs and explicit OOV splits provide reproducible evidence of robustness on the benchmark.","major_comments":[{"comment":"Methods (synthetic corpus generation and evaluation setup): The headline claims of Test-Time Pair F1 0.997/0.986 and 0.00-day MAE rest exclusively on the 2,000-note synthetic outpatient corpus with action-disjoint splits. The paper states in the Conclusion that transfer to real EHR notes is the next validation and references a first-pass realism check in Limitations, but no quantitative evidence is provided that the corpus replicates the linguistic variety, time-expression ambiguity, or action distribution of authentic clinical notes. This assumption is load-bearing for the Objective's claim of reliability in a clinical setting.","section":"Methods (corpus and evaluation)"},{"comment":"Results (baseline comparison): The non-overlapping CIs are a strength, but the generative baselines' low Pair F1 (0.51-0.57) is attributed to failures in linking and arithmetic; without ablation showing that the hybrid's deterministic normalization is the decisive factor (vs. the ontology or linker), it is unclear whether the gap would persist on more varied data.","section":"Results"}],"minor_comments":[{"comment":"The abstract and Methods should explicitly state the number of bootstrap samples and how note-level aggregation is performed for the 95% CIs to support reproducibility.","section":"Abstract / Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major point below, indicating planned revisions where appropriate.","responses":[{"response":"We agree the synthetic corpus is central and that no quantitative metrics comparing it to real notes are provided. The corpus was deliberately constructed with action-disjoint splits and controlled time expressions to isolate OOV generalization and deterministic normalization effects. The manuscript already states in Conclusion and Limitations that real-EHR transfer is future work. We will revise Limitations to more explicitly discuss potential gaps in linguistic variety and ambiguity relative to authentic notes and will temper claims about immediate clinical deployment.","revision_made":"partial","referee_comment":"[Methods (corpus and evaluation)] Methods (synthetic corpus generation and evaluation setup): The headline claims of Test-Time Pair F1 0.997/0.986 and 0.00-day MAE rest exclusively on the 2,000-note synthetic outpatient corpus with action-disjoint splits. The paper states in the Conclusion that transfer to real EHR notes is the next validation and references a first-pass realism check in Limitations, but no quantitative evidence is provided that the corpus replicates the linguistic variety, time-expression ambiguity, or action distribution of authentic clinical notes. This assumption is load-bearing for the Objective's claim of reliability in a clinical setting."},{"response":"The attribution follows directly from the baselines achieving high action F1 yet low Pair F1, pointing to linking and arithmetic as the failure modes. The hybrid architecture separates these steps by design. We acknowledge an explicit ablation isolating the deterministic normalizer is absent. We will add a clarifying sentence in Results/Discussion noting this limitation and that future work could include such ablations on more varied data.","revision_made":"partial","referee_comment":"[Results] Results (baseline comparison): The non-overlapping CIs are a strength, but the generative baselines' low Pair F1 (0.51-0.57) is attributed to failures in linking and arithmetic; without ablation showing that the hybrid's deterministic normalization is the decisive factor (vs. the ontology or linker), it is unclear whether the gap would persist on more varied data."}],"tokens_in":1602,"tokens_out":491,"duration_ms":30400,"standing_objections":["Quantitative evidence that the synthetic corpus replicates linguistic variety, time-expression ambiguity, or action distributions of real clinical notes"]},"desk_editor":{"model":"grok-4.3","letter":"The hybrid gets Test-Time Pair F1 of 0.997 on seen actions and 0.986 on OOV ones, with zero day-offset error, while the LLMs stay around 0.5 on the pair metric. That gap is the main takeaway.\n\nThey run BioBERT for BIO tagging, add a biaffine linker, canonicalize actions against a fixed 28-item ontology, and normalize times with deterministic rules. The evaluation uses action-disjoint splits on their 2,000-note synthetic outpatient set, so the OOV numbers actually test generalization to unseen actions. The generative baselines (zero-shot GPT-4o-mini and LoRA-tuned LLaMA-3) reach decent action F1 but lose the linking and arithmetic steps that the hybrid keeps explicit.\n\nThe comparison is clean: non-overlapping bootstrap CIs, held-out splits, and a clear failure-mode analysis for the generative side. The method is straightforward and the numbers are reproducible on the data they built.\n\nThe soft spot is the data itself. Everything rests on a synthetic corpus whose generation process, phrasing variety, and time-expression coverage are not shown to match real outpatient notes. The paper itself flags transfer to actual EHR as future work, so the current advantage is benchmark-specific rather than proven in the target setting.\n\nThis is worth a serious referee for clinical NLP groups that need reliable structured extraction. The empirical separation of learned and symbolic parts is useful even if the synthetic limitation keeps the claim narrow. I would send it to review rather than desk-reject.","headline":"Hybrid pipeline beats generative baselines on synthetic action-time extraction but the real EHR test is still missing.","tokens_in":2420,"tokens_out":380,"would_cite":false,"duration_ms":27578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A hybrid pipeline extracts clinical follow-up action-date pairs at 0.99 F1 by separating neural tagging from deterministic arithmetic.","keywords":["clinical follow-up extraction","hybrid neural-symbolic pipeline","action-date pairs","outpatient notes","time normalization","entity linking","synthetic corpus evaluation","generative model comparison"],"falsifier":"Running the same pipeline and generative baselines on a held-out collection of real EHR outpatient notes and comparing pair F1 and day MAE would show whether the reported advantage survives outside the synthetic corpus.","tokens_in":2690,"feed_emoji":"📋","tokens_out":737,"duration_ms":23470,"temperature":0.7,"pith_summary":"The paper establishes that a pipeline which uses neural methods only for identifying actions and times, then applies fixed rules for linking and date calculation, produces far more accurate (action, date) pairs than models that generate the full extraction in one step. On a synthetic set of outpatient notes it reaches near-perfect pair-level F1 and exact day accuracy even for actions never seen in training, while generative approaches lose most of the correct pairings. A reader would care because reliable extraction of scheduled follow-ups directly supports automated scheduling, compliance checks, and reduced manual review in outpatient care. The work shows the gain comes from making the arithmetic and linking steps explicit rather than implicit in decoding.","feed_headline":"Hybrid pipeline extracts clinical follow-up pairs at 0.99 F1","feed_subtitle":"Separating neural tagging from deterministic date arithmetic outperforms generation on seen and held-out actions in synthetic notes.","key_machinery":"ScheduledFor relation between TestSpecification and TimeSpecification entities, extracted neurally then normalized symbolically via ontology and deterministic day-offset conversion.","core_discovery":"The pipeline defines TestSpecification and TimeSpecification entities linked by a ScheduledFor relation; a neural tagger and linker identify them in text, after which an ontology canonicalizes the 28 possible actions and a deterministic converter turns time expressions into day offsets. On 259-note seen and OOV splits this yields Test-Time Pair F1 of 0.997 and 0.986 with 0.00-day mean absolute error. Generative baselines reach high action F1 yet only 0.51-0.57 pair F1, with non-overlapping confidence intervals.","pith_inferences":["The same split between learned recognition and rule-based calculation could be applied to other clinical extraction tasks that require precise temporal offsets.","High performance on OOV actions suggests the ontology plus deterministic normalizer removes the main source of date errors once entities are found.","If real notes contain more varied phrasing than the synthetic set, the neural tagging stage would become the new bottleneck rather than the arithmetic stage.","The zero-day MAE result indicates that once the ScheduledFor link is correct the rest of the pipeline introduces no additional timing error."],"forward_implications":["The method generalizes to actions held out from training, maintaining high pair F1.","Explicit separation of entity extraction from date arithmetic produces exact day offsets where generative decoding does not.","The approach surfaces concrete failure modes of pure generation on linking and arithmetic tasks.","The benchmark identifies transfer to real EHR notes as the required next validation step."],"fun_headline_variants":["Neural-symbolic pipeline extracts follow-up pairs at 0.99 F1","Hybrid extractor outperforms GPT on action-date pairs in notes","0.99 F1 via BioBERT tagging and deterministic time offsets","Pipeline generalizes to OOV actions with 0.986 pair F1"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 2,000-note synthetic outpatient corpus with action-disjoint splits captures the linguistic variety, time-expression ambiguity, and action distribution of real clinical notes.","fun_headline_variants_meta":{"raw":{"variants":["Neural-symbolic pipeline extracts follow-up pairs at 0.99 F1","Hybrid extractor outperforms GPT on action-date pairs in notes","0.99 F1 via BioBERT tagging and deterministic time offsets","Pipeline generalizes to OOV actions with 0.986 pair F1"]},"model":"grok-4.3","cost_usd":0.004139,"raw_usage":{"total_tokens":2158,"prompt_tokens":790,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":41387000,"prompt_tokens_details":{"text_tokens":790,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1294,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":790,"tokens_out":74,"duration_ms":12383,"temperature":1.0,"reasoning_tokens":1294,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T18:40:48.248597+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the same pipeline and generative baselines on a held-out collection of real EHR outpatient notes and comparing pair F1 and day MAE would show whether the reported advantage survives outside the synthetic corpus.","supporting_citations":[],"review_version":1}