{"id":"8a7b6e64-9397-425c-9408-53fb62f127fa","arxiv_id":"2606.19183","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A hybrid LLM-ML pipeline extracts features from clinical narratives and applies XGBoost for more stable pediatric appendicitis diagnosis than end-to-end LLMs.","lead":"The paper presents ClaMPAPP, a hybrid system using an LLM to extract structured clinical features from narrative notes and feeding them into an XGBoost classifier for pediatric appendicitis diagnosis. This separates language handling from prediction to improve stability and safety over direct LLM use.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Generated narratives from structured EHRs via templates/LLM rewriting may not proxy real free-text clinical notes","rationale":"Reader's weakest_assumption matches the load-bearing methodological choice exactly; full-text availability does not remove the synthetic-input limitation, so the UNVERDICTED stance remains appropriate pending real-note validation.","tokens_in":1765,"tokens_out":308,"duration_ms":9534,"concrete_test":"Obtain (or simulate) a small set of authentic free-text notes for the same or matched pediatric appendicitis cases; run the full ClaMPAPP pipeline (LLM extraction + plausibility checks + XGBoost) and compare AUROC/sensitivity at the reported operating point against the generated-narrative results; >10% drop in sensitivity would indicate the proxy does not support the headline claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (ClaMPAPP superiority on internal/external validation, especially minimizing missed appendicitis) rests on evaluations where all test inputs are template-rendered structured data further rewritten by constrained LLM and permuted. Section 3 (Methods) and the abstract describe this construction explicitly to 'preserve ground truth while testing free-text input.' If real physician notes exhibit different lexical distributions, missingness patterns, abbreviations, or narrative flow not reproducible by the template+LLM pipeline, both the schema-constrained extraction step and the XGBoost feature inputs could be unrealistically clean, inflating reported metrics relative to end-to-end LLM baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents ClaMPAPP, a hybrid system for pediatric appendicitis diagnosis that employs an LLM solely as an interface to extract schema-constrained features from note-like narratives, applies deterministic plausibility checks, and routes validated features to a trained XGBoost classifier. Narratives are constructed from structured EHR data via template rendering, constrained LLM rewriting, and sentence-order permutation to enable free-text testing while preserving ground truth. The system is evaluated on two independent German pediatric cohorts (internal and external validation) and compared against end-to-end LLM baselines; the central claim is that ClaMPAPP achieves the strongest overall diagnostic performance while minimizing missed appendicitis cases, with greater robustness to narrative reordering than direct LLM approaches.","tokens_in":1885,"tokens_out":440,"duration_ms":14985,"significance":"If the generated narratives serve as a valid proxy, the work provides concrete evidence for an LLM-as-interface design that separates natural-language handling from stable predictive inference, offering improved auditability and safety over end-to-end LLM use in clinical triage. The explicit comparison on independent cohorts and focus on missed-case minimization address a key practical concern in acute care.","major_comments":[{"comment":"Section 3 (Methods) and abstract: The central performance claims (superiority on internal/external validation, minimized missed cases) rest entirely on test inputs that are template-rendered structured data further rewritten by constrained LLM and permuted. If real physician notes exhibit different lexical distributions, abbreviations, missingness patterns, or narrative flow, both the schema-constrained extraction and downstream XGBoost inputs could be unrealistically clean, inflating metrics relative to true free-text baselines. This assumption is load-bearing for the claim that the hybrid system outperforms end-to-end LLMs on free-text clinical documentation.","section":"Section 3"}],"minor_comments":[{"comment":"Abstract: No quantitative metrics, confidence intervals, or baseline details are provided despite the strong performance claims; these should be summarized with effect sizes to allow readers to assess the magnitude of improvement.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for highlighting this important methodological consideration. We address the concern point by point below.","responses":[{"response":"We agree that the generated narratives constitute a controlled proxy rather than authentic physician notes and that this limits direct claims about performance on arbitrary real-world free-text. The design was chosen to retain exact ground-truth labels from the original structured EHR while enabling systematic testing of LLM extraction robustness (via constrained rewriting) and positional sensitivity (via permutation). Real free-text notes would require separate manual annotation to establish equivalent ground truth, which was outside the scope of the available datasets. We will revise the Methods, abstract, and Discussion to state this limitation more explicitly, qualify the generalizability claims, and note that future work should include prospective evaluation on raw physician documentation.","revision_made":"yes","referee_comment":"[Section 3] Section 3 (Methods) and abstract: The central performance claims (superiority on internal/external validation, minimized missed cases) rest entirely on test inputs that are template-rendered structured data further rewritten by constrained LLM and permuted. If real physician notes exhibit different lexical distributions, abbreviations, missingness patterns, or narrative flow, both the schema-constrained extraction and downstream XGBoost inputs could be unrealistically clean, inflating metrics relative to true free-text baselines. This assumption is load-bearing for the claim that the hybrid system outperforms end-to-end LLMs on free-text clinical documentation."}],"tokens_in":1437,"tokens_out":337,"duration_ms":12672,"standing_objections":["Direct empirical comparison on a corpus of authentic, physician-authored free-text notes with independently verified ground truth labels."]},"desk_editor":{"model":"grok-4.3","letter":"The core idea is to keep the LLM in a supporting role: it pulls structured features from note-like text, runs some checks, and passes the results to an XGBoost model for the final risk score. They built ClaMPAPP around this split and tested it on two German pediatric appendicitis cohorts, showing better stability than end-to-end LLMs when the input text is reordered.\n\nThe work is new in its concrete application to this setting with both internal and external validation. The comparison highlights how pure LLMs can swing on sensitivity-specificity and degrade under simple changes like sentence order, while the hybrid stays steadier. That separation of concerns is a reasonable design choice for a safety-critical triage problem.\n\nThe soft spot is the data construction. All test narratives come from structured EHR records via templates and constrained LLM rewriting, followed by permutations. This preserves labels but creates inputs that are likely cleaner and more consistent than actual physician notes, which often contain abbreviations, omissions, and idiosyncratic phrasing. If the feature extractor performs well only because the text is artificially tidy, the reported gains on missed cases could shrink in deployment.\n\nThe abstract itself gives no numbers, confidence intervals, or baseline details, so the size of the improvement is hard to judge from the summary alone. The full paper presumably supplies those, but the synthetic-note limitation still needs direct scrutiny.\n\nThis is for people working on hybrid clinical systems who want a worked example in pediatrics rather than a general framework. It deserves a serious referee because the safety focus and the LLM-versus-ML split are worth testing in review, even with the data-generation caveat.","headline":"The hybrid pipeline looks practical for this narrow task but its performance edge rests on synthetic notes that may not match real clinical writing.","tokens_in":2357,"tokens_out":397,"would_cite":false,"duration_ms":19045,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"ClaMPAPP uses an LLM only to extract features from clinical narratives and an XGBoost classifier for prediction, outperforming end-to-end LLMs in pediatric appendicitis while minimizing missed cases.","keywords":["ClaMPAPP","pediatric appendicitis","LLM as interface","hybrid LLM-ML system","XGBoost classifier","clinical decision support","feature extraction","diagnostic performance"],"falsifier":"ClaMPAPP performance falling below that of end-to-end LLMs when both are tested on actual unstructured physician-written notes rather than generated narratives would falsify the claimed advantage of the hybrid design.","tokens_in":2663,"feed_emoji":"🩺","tokens_out":656,"duration_ms":16677,"temperature":0.7,"pith_summary":"The paper shows that large language models work best as interfaces that turn free-text clinical notes into structured features rather than as direct diagnostic oracles. ClaMPAPP applies an LLM to pull schema-constrained variables from note-like text, runs deterministic plausibility checks, and hands the validated data to a trained XGBoost model that combines clinical, lab, and ultrasound inputs. Tested on two independent pediatric cohorts from German hospitals, the hybrid system delivered the strongest overall performance and the lowest rate of missed appendicitis, the critical safety failure in acute triage. End-to-end LLMs, by contrast, produced unstable sensitivity-specificity balances and lost accuracy when sentence order changed. A sympathetic reader cares because the design keeps the strengths of natural-language handling while relying on stable machine-learning inference for the final risk score.","feed_headline":"Hybrid LLM-ML system outperforms end-to-end models on appendicitis","feed_subtitle":"ClaMPAPP extracts features with an LLM then uses XGBoost for the diagnosis, cutting missed cases across two pediatric cohorts.","key_machinery":"The ClaMPAPP pipeline, which uses an LLM strictly as a constrained feature extractor and interface while delegating prediction to an XGBoost classifier after deterministic checks.","core_discovery":"ClaMPAPP extracts schema-constrained clinical features from note-like narratives, applies deterministic plausibility checks, and passes validated features to an XGBoost classifier trained on clinical, laboratory, and ultrasound variables. On two independent pediatric appendicitis cohorts it achieved the strongest overall diagnostic performance in both internal and external validation while minimizing missed appendicitis cases. End-to-end LLM baselines showed unstable sensitivity-specificity trade-offs and greater degradation under narrative reordering.","pith_inferences":["The same interface-plus-predictor split could be tested on other acute conditions that rely on free-text notes.","Real-world deployment would require measuring how often the deterministic plausibility checks reject LLM-extracted features.","The method may reduce the need for prompt engineering by moving all inference stability into the downstream classifier."],"forward_implications":["The hybrid design minimizes missed appendicitis cases, the key safety concern in acute triage.","End-to-end LLMs exhibit unstable sensitivity-specificity trade-offs and degrade more under narrative reordering.","Separating natural-language usability from predictive inference yields a more auditable pathway for clinical decision support.","The approach integrates narrative clinical workflows with tabular machine-learning inputs without requiring direct tabular entry."],"fun_headline_variants":["LLM interface enables stable ML prediction for appendicitis","ClaMPAPP outperforms end-to-end LLMs across two cohorts","Hybrid system minimizes missed appendicitis better than LLMs","LLM extracts validated features for XGBoost appendicitis model"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The template-rendered and constrained-LLM-rewritten narratives, including sentence-order permutations, serve as a valid proxy for real free-text clinical documentation without introducing artifacts that bias feature extraction or model performance.","fun_headline_variants_meta":{"raw":{"variants":["LLM interface enables stable ML prediction for appendicitis","ClaMPAPP outperforms end-to-end LLMs across two cohorts","Hybrid system minimizes missed appendicitis better than LLMs","LLM extracts validated features for XGBoost appendicitis model"]},"model":"grok-4.3","cost_usd":0.005227,"raw_usage":{"total_tokens":2568,"prompt_tokens":739,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":52274500,"prompt_tokens_details":{"text_tokens":739,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1764,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":739,"tokens_out":65,"duration_ms":17921,"temperature":1.0,"reasoning_tokens":1764,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T21:06:36.888358+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"ClaMPAPP performance falling below that of end-to-end LLMs when both are tested on actual unstructured physician-written notes rather than generated narratives would falsify the claimed advantage of the hybrid design.","supporting_citations":[],"review_version":1}