{"id":"699af37b-17a3-441a-9326-5e1337e0e479","arxiv_id":"2601.09072","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HACHI, an iterative human-AI loop that turns clinical notes into small interpretable yes/no-concept risk scores, outperforms standard instruments in two retrospective UCSF case studies.","lead":"HACHI is an iterative framework in which an LLM agent proposes yes/no clinical concepts from medical notes while a human clinical team reviews and redirects each round, producing five-to-ten-term interpretable risk scores. The paper reports that on two UCSF tasks (childhood TBI and post-surgical AKI) the resulting models beat standard instruments, though the evidence is retrospective and mostly single-center.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TBI arm lacks an independent test set: iterative human feedback was tuned on the same validation data, so the reported site-generalizability gains may reflect overfitting rather than true improvement.","rationale":"The reader correctly flagged annotation accuracy as an unmeasured assumption, but I see an even more load-bearing problem: the TBI evaluation is circular. Every model decision in the four rounds was informed by validation-set performance, and the final reported numbers are on that same validation set. This directly threatens the headline claim of improved site generalizability, which is one of the two pillars of the abstract. The AKI arm provides a genuine temporally disjoint test and is the strongest part of the evidence, so a full rejection would be too harsh; however, the paper should either present an independent TBI test set or explicitly downgrade the TBI claims to 'validation performance.' The concrete test I propose is feasible with the existing dataset and code (the repository is promised open-source), and it would settle whether the reported gains are real. I agree with the reader's overall CONDITIONAL verdict because the framework is plausible, the AKI arm is solid, and the code is open; but the specific conditions should include an independent test for TBI and, ideally, a BC-LLM comparison to isolate the human contribution. I am not raising annotation accuracy as the primary concern because, even if the LLM annotations were perfectly accurate, the lack of a held-out test set in the TBI arm would still invalidate the generalizability claim as stated.","tokens_in":21612,"tokens_out":2505,"duration_ms":26006,"concrete_test":"Before running any HACHI rounds, set aside a test set from the TBI encounters (e.g., all encounters after December 2024, or a random 20% held out and never inspected). Run the complete 4-round protocol on the training portion only, with the same prompts and stopping rules. Then evaluate the final Round 4 model on the untouched test set, reporting overall AUC and campus-specific AUCs. If the test AUCs do not approximate the reported 0.91 / 0.93 / 0.80 — or if the Mission Bay AUC remains near 0.71 — the generalizability claim is unsupported. The same test set should also be used for the PECARN and OpenEvidence comparators to ensure fair comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HACHI 'improves model generalizability across clinical sites and time periods' rests heavily on the TBI case study (Sec. 2.1), but that arm has no genuinely held-out test set. All four rounds used the same dataset; the clinical AI team modified prompts, removed cases, and introduced sample weights after observing validation AUCs, and the reported final AUCs (0.91 overall; 0.93 and 0.80 by campus) are computed on that same validation partition. Round 4's reweighting was explicitly chosen to fix the site disparity seen in Round 3's validation results, so the reported improvement is circular and may be optimistic. The AKI arm has a temporally disjoint Period 2 test set and is stronger evidence, but the abstract claims generalizability without caveating that the TBI site results are validation-only. The paper also omits a comparison to BC-LLM (the closest AI-only baseline), so the human co-design benefit is not isolated; however, the missing independent test set is the more fundamental threat because it undermines the comparison to PECARN and OpenEvidence as well. Annotation accuracy (the reader's weakest assumption) is acknowledged in the Discussion and is a real reliability concern, but even perfect annotations would not fix the circular evaluation in the TBI arm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HACHI, an iterative human-in-the-loop framework in which an LLM-based AI agent proposes, annotates, and selects yes/no clinical concepts from unstructured notes, while a clinical AI team reviews each round and provides feedback through prompt edits, dataset modifications, or changes to the learning procedure. The framework is evaluated on two retrospective UCSF case studies: pediatric traumatic brain injury (TBI) and postoperative acute kidney injury (AKI). The authors report that HACHI outperforms standard instruments (PECARN, Kheterpal) and a one-shot OpenEvidence-based brainstorming baseline, and that it improves generalizability across sites (TBI) and across time periods (AKI). The paper includes an open-source implementation and a web-based audit interface.","tokens_in":21880,"tokens_out":3376,"duration_ms":31823,"significance":"If the evaluation concerns are resolved, HACHI is a meaningful contribution to interpretable clinical prediction modeling: it operationalizes the iterative human-expert/LLM co-design loop, makes the learned concepts inspectable, and publishes reusable code. The AKI arm provides a genuine temporally disjoint test set (Period 2), which is a real strength, and the paper is transparent about many limitations, including the retrospective design and dependence on LLM extraction accuracy. However, the load-bearing claim of 'improved generalizability across clinical sites' rests almost entirely on the TBI arm, which lacks a true held-out test set; the site-level results are validation-set numbers obtained after iterative feedback and reweighting on the same data. The absence of a direct comparison to BC-LLM, the closest AI-only baseline, also leaves the specific benefit of human feedback unquantified. These issues are fixable and the core framework is plausible, but the current evidence is not sufficient to support the abstract's strong claims.","major_comments":[{"comment":"The TBI arm has no independent test set. The four HACHI rounds are evaluated on the same validation partition used for concept selection, and Round 4's campus reweighting was introduced after observing the Round 3 campus AUC gap (0.93 vs 0.71). The reported final AUCs (0.91 overall; 0.93 and 0.80 by campus) are therefore validation-set numbers, not held-out estimates. This makes the abstract's claim that HACHI 'improves model generalizability across clinical sites' unsupported for the site dimension. The AKI arm's temporally disjoint Period 2 set is the right template; the TBI arm should be re-analyzed with a true holdout (e.g., temporal split or nested validation) or the claim must be explicitly downgraded to validation-only, exploratory evidence.","section":"Sec. 2.1, Fig. 1b, Table 1"},{"comment":"The closest baseline, BC-LLM [28], is described as the most related method but is never empirically compared. HACHI's central contribution is the human-in-the-loop component over an AI-only agentic loop. Without a BC-LLM comparison or an ablation with human feedback removed, the observed improvements cannot be attributed to the co-design process rather than to the underlying LLM-driven greedy concept search. Please add this comparison or narrow the claim to 'HACHI, including its human feedback, outperforms...'.","section":"Sec. 2.1, Sec. 2.2, Sec. 3"},{"comment":"The framework treats LLM yes/no concept annotations as ground-truth features for fitting and selecting the CPMs, but annotation accuracy is not measured on either dataset. The Discussion acknowledges this dependency, but a concrete reliability check is needed: for example, clinician audit of a random sample of concept annotations with agreement statistics, or a sensitivity analysis using a different extraction model. Without this, both the reported AUCs and the claim that HACHI 'surfaces new clinically relevant concepts' remain vulnerable to annotation noise or bias.","section":"Sec. 4.1 Step 3, Discussion limitations"}],"minor_comments":[{"comment":"The text says 'Repeat steps 2-4 until convergence'; the loop should say 'repeat steps 2-3' since step 4 is the iteration itself.","section":"Sec. 4.1, Step 4"},{"comment":"Typo: 'SOF A' should be 'SOFA'.","section":"Introduction, reference [1]"},{"comment":"The figure uses 'Hospital 1' and 'Hospital 2' while the text and Table 1 refer to Oakland and Mission Bay campuses. Use consistent labels for readability.","section":"Fig. 1b"},{"comment":"Several confidence intervals appear implausible (e.g., sensitivity 0.942 with CI (0.942, 1.000); specificity lower bounds of 0.000). Recalculate or omit the CIs if they are derived from a small bootstrap.","section":"Table A2"},{"comment":"The statement 'In ablation studies, we did not find significant improvement...' is not accompanied by any results or supplement reference. Provide the ablation or cite the supplement.","section":"Sec. 4.1, Implementation details"}],"recommendation":"major_revision","confidential_remarks":"The TBI evaluation problem is the main obstacle; the AKI arm shows the authors can do proper held-out evaluation, so this is fixable. The missing BC-LLM comparison is also important because it bears directly on the paper's novelty claim. I would not reject, but the abstract and conclusions need to be recalibrated or the experiments strengthened before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HACHI is a genuinely useful framework: it wraps an LLM agent that hunts for yes/no concepts in clinical notes with an outer loop where a clinical AI team reviews results and steers the search. The paper's real content is that outer loop, and the worked examples show how the team catches data leakage, spurious correlations, vague concept definitions, and site imbalance in 3–4 rounds. The writing is clear and the limitations are discussed more honestly than most clinical ML papers.\n\nThe AKI arm is the credible piece. It has a temporally disjoint Period 2 test set, sensible comparators (Kheterpal, OpenEvidence brainstorming), and the improvement from about 0.60 to 0.73 on the validation period and to 0.77 on the true test period is meaningful if it holds. The code is promised open-source, which matters for reproducibility.\n\nThe TBI arm is the main problem. Everything—concept selection, the sign-constraint modification, the post-hoc campus reweighting, and the final AUCs—is done on the same validation partition. The authors are transparent that Round 4's reweighting came after seeing the site gap, but that transparency doesn't fix the circularity. As it stands, the reported site-generalizability gain (0.93/0.71 → 0.93/0.80) cannot support the abstract's claim that HACHI 'improves model generalizability across clinical sites.' This is load-bearing, because the abstract sells the method on that claim.\n\nA secondary issue: BC-LLM, the closest AI-only baseline, is missing from both case studies. Without it, the human-feedback benefit isn't isolated; the gains could just come from having a fresh set of concepts. A third, minor concern is that LLM annotation accuracy is never audited on these datasets. The authors acknowledge this in the Discussion, and it's a real reliability issue, but it's not the fatal one—even perfect annotations wouldn't fix the TBI evaluation.\n\nThis paper is for clinical ML researchers and health-AI practitioners who care about interpretable models. It's not a foundational algorithmic advance, but it is a useful demonstration of a human-in-the-loop workflow. The AKI arm alone justifies refereeing; the TBI arm needs an independent test set and a BC-LLM comparison before the generalizability claim can be taken seriously. I'd send it to peer review, but expect a major revision.","headline":"A well-written, plausible framework for human-LLM co-design of interpretable clinical prediction models, with strong AKI evidence but a circular TBI evaluation that undercuts the headline generalizability claim.","tokens_in":22427,"tokens_out":1939,"would_cite":false,"duration_ms":21911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an iterative human-in-the-loop framework, HACHI, lets clinicians steer a language-model agent to find yes/no concepts in notes that yield interpretable models outperforming standard scores and one-shot brainstorming.","keywords":["clinical prediction models","concept bottleneck","large language models","human-in-the-loop","electronic health records","clinical notes","interpretability","traumatic brain injury"],"falsifier":"Take one learned concept such as 'normal gait' or 'sleep apnea,' have two clinicians independently annotate a random sample of 100-200 notes, compare their answers with the LLM's, and check whether disagreements are associated with the outcome; low agreement or outcome-correlated errors would indicate the reported AUC gains are artifacts of annotation bias rather than true predictive signal.","tokens_in":21460,"feed_emoji":"🩺","tokens_out":7150,"duration_ms":61271,"temperature":0.7,"pith_summary":"The paper claims that an iterative collaboration between clinicians and a large-language-model agent can turn unstructured clinical notes into small, transparent prediction models that outperform standard instruments. The workflow, called HACHI, repeatedly has the agent propose candidate concepts as yes/no questions, annotate notes, and fit a linear model, while clinicians review results each round and refine prompts, data, and objectives. In two retrospective tasks—pediatric head trauma and postoperative kidney injury—the loop raised accuracy, surfaced predictors absent from common models, and improved performance across sites and time periods at a cost of roughly 1-2 hours of expert time per round. A sympathetic reader would care because this is a concrete path from black-box LLM automation to interpretable bedside tools.","feed_headline":"Clinician-feedback rounds lift simple risk models past standard scores","feed_subtitle":"In two clinical tasks, three to four rounds of expert feedback raised accuracy and surfaced new risk factors.","key_machinery":"The central mechanism is a concept bottleneck defined by yes/no questions: each concept is a single human-readable question, and the model is a linear combination of those binary answers. The AI agent runs a greedy hill-climbing loop—extract keyphrases from notes, fit a bag-of-words logistic model to find outcome-associated phrases, ask the LLM to propose candidate yes/no questions, have the LLM annotate every note for each candidate, evaluate validation AUC, and replace weak concepts—while the human team steers the next round by editing the prompts, the dataset, the coefficient-sign constraint, or the sample weights. This division of labor is what carries the argument: the agent explores an","core_discovery":"On the paper's own terms, the central discovery is that human oversight in the loop changes what the AI agent learns, not just how well it performs: the clinicians caught a leaked 'brain bleed' concept that inflated the first model, removed transferred patients with prior CT results, forced concept wording to refer to patient attributes rather than note style, required coefficient signs to match clinical priors, and reweighted sites to fix a 3:1 imbalance. The final TBI model used five yes/no questions (loss of consciousness, altered mental status, headache, head trauma, normal gait) and improved cross-campus AUCs from 0.93/0.71 to 0.93/0.80 while outperforming the standard rule at matched s","pith_inferences":["Editorial inference: a natural ablation would separate the value of the human feedback loop from the value of the agent's search—if clinicians only approve or reject the final concept list rather than steering prompts, performance should drop measurably if the human input is doing the real work.","Editorial inference: the design treats the LLM's yes/no annotations as ground truth, so chart-reviewing a subsample of annotations and checking whether disagreements correlate with the outcome would clarify how much of the reported AUC advantage is genuine signal versus annotation bias.","Editorial inference: the same outer/inner loop should transfer to other high-stakes domains with unstructured text, such as eligibility decisions, discharge planning, or social-work notes, where a small expert team could steer concept discovery in the same way.","Editorial inference: the claimed human cost of 1-2 hours per round excludes dataset preparation, team meetings, and code updates, so a fuller time accounting is needed before comparing HACHI's efficiency to traditional model-building pipelines."],"forward_implications":["If HACHI works as reported, a small clinical team with EHR notes and a compliant LLM can produce a locally valid, fully interpretable prediction model in about 3-4 rounds and 1-2 expert hours per round.","The learned concepts—e.g., normal gait in pediatric head trauma, or minimally invasive surgery and sleep apnea in AKI—become candidate predictors for prospective validation in larger, multi-site cohorts.","At inference, a HACHI model requires only a clinician answering 5-10 yes/no questions during an encounter; no LLM is needed at the bedside, which lowers deployment barriers.","Because the learning loop exposes the human team to the agent's intermediate annotations and errors, it also exposes data leakage, documentation-style artifacts, and site imbalances that pure automated feature discovery would miss.","HACHI's procedure is not tied to its defaults: swapping the lasso model for sparse integer scoring or rule lists is a stated extension, which would preserve interpretability while potentially improving calibration."],"fun_headline_variants":["Human feedback catches leaked data, sharpens AI clinical models","Expert rounds improve simple risk models in two clinical tasks","Human-AI loop boosts interpretable models, catches data leakage","Three rounds of expert feedback lift AUC and spot biased data","Co-design loop surfaces new risk factors, boosts generalizability"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the language model's yes/no answers about the notes are accurate enough to serve as the ground-truth labels used to fit and choose concepts; if those annotations are noisy or biased, the learned concepts and reported AUCs are not trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Human feedback catches leaked data, sharpens AI clinical models","Expert rounds improve simple risk models in two clinical tasks","Human-AI loop boosts interpretable models, catches data leakage","Three rounds of expert feedback lift AUC and spot biased data","Co-design loop surfaces new risk factors, boosts generalizability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3271,"prompt_tokens":846,"completion_tokens":2425,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":2344}},"tokens_in":590,"tokens_out":2425,"duration_ms":15850,"temperature":1.0,"reasoning_tokens":2344,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:42:38.144118+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one learned concept such as 'normal gait' or 'sleep apnea,' have two clinicians independently annotate a random sample of 100-200 notes, compare their answers with the LLM's, and check whether disagreements are associated with the outcome; low agreement or outcome-correlated errors would indicate the reported AUC gains are artifacts of annotation bias rather than true predictive signal.","supporting_citations":[],"review_version":1}