{"id":"630c3c2f-23f2-4f0a-9a65-3cd5f72456f0","arxiv_id":"2508.19096","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TrustEHRAgent, an EHR assistant with step-by-step confidence checks, scores 44% and 25% accuracy on MIMIC-III and eICU when only answers it is 70% confident in are counted, while baseline methods score 0%.","lead":"This paper introduces a reliability-focused scoring method for medical AI assistants and an EHR assistant that can say 'I don't know' when unsure. In tests on two hospital databases, the assistant proved more accurate than existing systems when only high-confidence answers were allowed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HCAcc@k% is maximized over τ on the same test set, so the reported 44.23p/25.34p gaps may reflect in-sample threshold overfitting rather than a deployable reliability advantage.","rationale":"The reader's weakest_assumption identified exactly the in-sample threshold selection problem, and I agree it is the most load-bearing concern. The paper's central contribution is a reliability-aware metric and an agent that purportedly excels under strict reliability constraints. If the reported HCAcc@k% values are computed via oracle threshold selection on the test set, they represent an upper bound that may not hold when the threshold must be chosen in advance. This does not invalidate the metric as a descriptive tool for a method's best possible reliability-accuracy envelope, but it does undermine the comparative claim that TrustEHRAgent beats baselines by 44.23p at HCAcc@70%, because that comparison is performed at each method's individually oracle-optimized operating point. The concern is concrete and testable; the fix (validation-based threshold selection, error bars) is straightforward. I therefore recommend keeping the conditional verdict: the paper should be accepted only if the held-out evaluation reproduces the qualitative advantage. I also considered the normalization issue in Eq. 2, but that affects calibration quality rather than the central comparative claim; the threshold-selection issue is more fundamental because it biases the main quantitative result.","tokens_in":12953,"tokens_out":2573,"duration_ms":32883,"concrete_test":"Perform a validation-test split on each dataset (e.g., 290/290 stratified questions). Use only the validation split to select the threshold τ* that maximizes OA(τ) subject to HR(τ) ≤ 0.30 (the HCAcc@70% operating point). Then freeze τ* and report OA(τ*), HR(τ*), and HCAcc on the held-out test split. Repeat with k-fold cross-validation or bootstrapping to obtain means and CIs. If TrustEHRAgent's held-out HCAcc@70% falls to near 0 or becomes statistically indistinguishable from the baselines (which would also be re-evaluated with the same protocol), then the claimed 44.23p/25.34p advantage is an artifact of test-set threshold optimization. Additionally, report the full precision-recall style curve with threshold varied on validation to show stability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim hinges on HCAcc@k% (Eq. 5), which defines HCAcc@k% as the maximum overall accuracy achievable subject to a hallucination-rate constraint, with the maximization performed over the confidence threshold τ. As implemented, this maximum is computed on the exact same test set used for reporting (Sec. 5.1, Table 1). This in-sample threshold selection is an oracle evaluation: the operating point is chosen after seeing test labels. In deployment, the threshold must be set a priori or on a labeled validation set, and with only 580 questions per dataset, the in-sample maximum can substantially overestimate the achievable accuracy at a fixed reliability level. This is especially acute at high k% (e.g., 70-90%), where the set of accepted questions Qτ is small and the chance of finding a threshold that happens to admit mostly correct answers is high. The fact that TrustEHRAgent posts 44.23% and 25.34% HCAcc@70% while both confidence-augmented baselines achieve 0.0% could therefore be driven by differences in the shape of the confidence distribution (e.g., baselines may produce more continuous or less conveniently ordered scores) rather than by a genuinely superior accuracy-reliability trade-off. The paper does not report error bars, split-half stability, or validation-based threshold selection, so the headline gaps are not yet established as reproducible under a realistic protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hallucination-Controlled Accuracy at k% (HCAcc@k%), a metric that measures the maximum overall accuracy achievable while keeping the hallucination rate below a specified threshold, and introduces TrustEHRAgent, a confidence-aware EHR question-answering agent with step-wise confidence estimation. On the MIMIC-III and eICU QA datasets, the agent is reported to achieve HCAcc@70% of 44.23% and 25.34%, respectively, while confidence-augmented SQLAgent and PythonAgent baselines achieve 0%, and it retains near-baseline accuracy at HCAcc@0% (62.13% and 59.82%). The paper argues that traditional accuracy metrics hide reliability differences that matter for clinical safety.","tokens_in":13331,"tokens_out":5309,"duration_ms":60555,"significance":"If the reported results hold, the paper makes two useful contributions: a reliability-aware evaluation metric for selective answering systems and an agent design that integrates step-wise confidence signals. The full HCAcc curves in Figure 3 and the availability of code are strengths, and the comparison against multiple baselines is informative. However, the central quantitative claims are not yet established because HCAcc@k% is maximized over the confidence threshold on the same test set used for reporting, and the confidence score in Eq. (2) is not a properly normalized probability. These issues directly affect the headline 44.23p/25.34p gaps, so the current evidence supports the approach as promising but not yet as a validated deployable advantage.","major_comments":[{"comment":"HCAcc@k% is defined as max_τ OA(τ) subject to HR(τ) ≤ (100−k)/100, and the reported values are obtained by optimizing τ on the same 580-question test sets used for reporting. This is an oracle evaluation: the operating point is chosen after seeing test labels. Especially at k=70–90%, the accepted set Q_τ is small, so the maximum over τ can be substantially higher than the accuracy at any fixed or validation-selected threshold. The headline claim that TrustEHRAgent achieves 44.23%/25.34% while baselines achieve 0.0% therefore needs to be re-evaluated with τ selected on a validation split (or via nested cross-validation), with the chosen τ values reported. Bootstrap confidence intervals should also be provided, since n=580 per dataset makes the high-k values unstable.","section":"Sec. 3.4, Eq. (5); Sec. 5.1, Table 1"},{"comment":"The confidence score is defined as C = (Σ i·P(S=i))/Smax with P(S=i)=exp(logprob_i). This is not a normalized probability distribution: summing exp(logprob_i) over the five confidence tokens does not sum to 1, and the resulting C is not guaranteed to lie in [0,1]. The missing probability mass is ignored, so the score is not a well-defined expected level. This matters because C is the quantity compared against τ; an unnormalized, token-set-dependent weighting can change the ordering of examples and hence the reported HCAcc@k% curves. Please define P(S=i) via a softmax over the five levels (or explicitly normalize by the sum), and clarify whether logprob_i are log-probabilities or logits.","section":"Sec. 3.2, Eq. (2)"},{"comment":"No uncertainty quantification is provided for any HCAcc@k% value. The central comparison is between 44.23% and 0.0% at a single operating point, but at k=70% the baselines' 0.0% could arise from a single unfavorable threshold while the proposed method's value is the best threshold on the test set. Bootstrap resampling over questions or multiple evaluation splits is needed to establish that the gap is stable and not an artifact of in-sample threshold selection. This is particularly important because the paper's own Limitations section notes the need for better calibration methods.","section":"Sec. 5.1, Figure 3"}],"minor_comments":[{"comment":"The rows without confidence estimation appear to list only two numbers; please specify which HCAcc@k% columns are reported for each baseline. Currently the layout is ambiguous.","section":"Table 1"},{"comment":"The query text 'sc 20 fe ng intake' appears garbled. If this is from the actual EHR data, please add a gloss or footnote; otherwise this looks like a rendering error.","section":"Figure 2"},{"comment":"The phrase 'improvements of 44.23%p' is imprecise: Table 1 reports an absolute HCAcc@70% value of 44.23% for TrustEHRAgent and 0% for baselines. The percentage-point improvement is 44.23p, not a relative improvement. Please clarify.","section":"Abstract and Conclusion"},{"comment":"The confidence-aware baselines are said to use 'the same step-wise confidence prompting and final confidence estimator.' It would be useful to state explicitly whether SQLAgent and PythonAgent actually received the step-wise confidence rating instruction during reasoning, or only the final estimator. This affects whether the comparison isolates the agent architecture or the confidence mechanism.","section":"Appendix D"},{"comment":"The text says 'N (= 5) discrete levels (S ∈ {0, 1, ..., Smax})' and later sets Smax=4. This is consistent, but the notation is confusing; consider writing S ∈ {0,1,2,3,4} explicitly.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the paper is worth pursuing, but the evaluation protocol currently selects the confidence threshold on the test set, which is an oracle procedure. I would ask the authors to re-run the main experiments with a validation-based threshold and to provide confidence intervals. The unnormalized confidence definition in Eq. (2) is a straightforward fix but should be corrected before the reported values are taken at face value. I do not see a fundamental flaw in the metric itself; the issues are in the evaluation protocol and the confidence estimator's definition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: the 44.23%/25.34% headline gains at HCAcc@70% are computed by taking the maximum over the confidence threshold on the test set itself. That is an oracle evaluation. With only 580 questions, the in-sample maximum can easily find a threshold that happens to admit mostly correct answers, especially at high k% where the accepted set is small. So the magnitude of the claimed advantage over the 0.0% baselines is not yet established under a realistic protocol. This is the paper's biggest soft spot.\n\nWhere credit is due: the paper is honest about its limitations, the full HCAcc@k% curves are in the figures, and the ablation in Figure 4 shows that step-wise confidence estimation helps at high thresholds. The integration of verbalized step-wise confidence with token-log-probability weighting is a reasonable design, and the experiments include confidence-augmented baselines, which is more than many papers do.\n\nThe metric itself is not new. HCAcc@k% is a re-parameterization of the standard accuracy-coverage trade-off with a precision constraint from the selective prediction literature (Chow, El-Yaniv and Wiener, etc.), and that literature is not cited. Eq. 2 is also sloppy: P(S=i) = exp(logprob_i) is not normalized, so the \"weighted average\" is not a proper expectation over a probability distribution. That's a minor issue for monotonicity, but it should be fixed.\n\nNo error bars are reported, and the test set is a single split of 580 questions. A validation-based threshold, or a fixed a priori threshold, plus some variance estimate, would make the empirical claims much stronger.\n\nBottom line: the core research question is important—how to make EHR agents abstain when uncertain—and the step-wise confidence approach is plausible and worth testing. But the headline numbers are likely to shrink when the evaluation protocol is cleaned up. This paper deserves a serious referee, but with a clear request to recompute the numbers under a non-oracle threshold selection, fix the normalization, and report variance. If those changes are made, I could see this being a useful reference for the clinical LLM reliability community.","headline":"The headline gains are partly an artifact of test-set-optimal threshold selection, but the step-wise confidence mechanism and the empirical comparison are worth serious attention.","tokens_in":13775,"tokens_out":3077,"would_cite":false,"duration_ms":32190,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An EHR question-answering agent that estimates confidence step-by-step can stay accurate where reliability is strictly required, while baseline agents that answer everything collapse to zero.","keywords":["confidence estimation","electronic health records","hallucination control","selective answer abstention","LLM agent","text-to-SQL","reliability metric","clinical decision support"],"falsifier":"Run the same experiments with the confidence threshold fixed on a validation split, never touched on the test set, then compute HCAcc@70% on MIMIC-III and eICU; if TrustEHRAgent's margin over baseline agents drops substantially, the central claim fails. A second check: recompute the confidence score with true softmax normalization over the five log-probabilities instead of raw exp(logprob) sums; if the ranking of answers changes, the reported thresholds are not interpretable as stated.","tokens_in":12861,"feed_emoji":"🩺","tokens_out":6764,"duration_ms":64123,"temperature":0.7,"pith_summary":"This paper proposes a way to judge medical database assistants by both accuracy and willingness to abstain: Hallucination-Controlled Accuracy at k% (HCAcc@k%) reports the best accuracy an agent can achieve while keeping the share of wrong answers among provided answers at or below (100-k)%. On MIMIC-III and eICU EHR benchmarks, the authors' TrustEHRAgent reaches 44.23% and 25.34% accuracy at HCAcc@70%, where confidence-augmented baseline agents score 0.0% because every answer they give above the confidence threshold is wrong. The agent estimates confidence at each reasoning step and again at the final answer, using token log-probabilities to form a weighted confidence score. The central claim is that step-wise confidence estimation, not raw reasoning ability, is what lets a clinical agent remain useful under strict patient-safety constraints.","feed_headline":"EHR agent answers only when sure, stays accurate where others fall to 0%","feed_subtitle":"At a 70% reliability bar, the stepwise confidence agent keeps 44% accuracy on MIMIC-III while baseline agents provide nothing.","key_machinery":"HCAcc@k% = the maximum over thresholds τ of Overall Accuracy subject to Hallucination Rate ≤ (100-k)/100, where Hallucination Rate is the fraction of wrong answers among provided answers and Overall Accuracy is correct answers divided by all questions. The carrying mechanism is the stepwise confidence estimator: the agent verbalizes a 0-10 confidence at each reasoning step, then a separate LLM-based estimator takes query, reasoning history, and answer, computes P(S=i)=exp(logprob_i) over five confidence levels, and returns the weighted average normalized by Smax. That continuous C gives the agent an abstention rule C ≥ τ, and the metric rewards methods that can set τ to filter errors.","core_discovery":"TrustEHRAgent couples a code-based EHR reasoning loop with stepwise confidence ratings and a final confidence estimator, producing a score C in [0,1] for each candidate answer. During execution, the agent writes a 0-10 confidence comment at each code step; after the final answer, a separate confidence-estimator call, conditioned on the question, the enriched reasoning history, and the answer, maps five discrete confidence levels into a continuous score via a log-probability weighted average. Answers are emitted only if C passes threshold τ. With τ chosen to satisfy a hallucination-rate cap, the agent achieves HCAcc@70% of 44.23% on MIMIC-III and 25.34% on eICU, while the same confidence pipe","pith_inferences":["The HCAcc@k% values presume the threshold τ is selected on the same test set used for reporting; a deployment protocol that fixes τ on a validation split would likely produce lower numbers, and the paper does not report that protocol.","Because HCAcc@k% does not depend on EHR content, it transfers to any selective question-answering setting where false answers are costly, such as legal or financial document querying.","The stepwise log-probability estimator could be combined with better calibration techniques (temperature scaling, ensembles) to push the HCAcc@90% numbers up, a direction the paper names as future work.","A direct replication on an open-weight model with accessible token probabilities would test whether the gap over baselines persists when the backbone is not the proprietary model used here."],"forward_implications":["Clinical EHR assistants can be deployed under an explicit safety budget: choose a risk tolerance, set k, and the agent answers only when its confidence clears the corresponding bar, otherwise it abstains.","Accuracy-only comparisons can mislead: two agents with close scores at HCAcc@0% behave very differently under strict reliability constraints, so high-stakes evaluations should report HCAcc curves.","Stepwise confidence signals improve the final confidence estimate; removing them degrades performance specifically at high reliability thresholds.","At HCAcc@90% even TrustEHRAgent drops to 3.27% (MIMIC-III) and 2.24% (eICU), so operating at extreme reliability remains unsolved."],"supporting_citations":[{"why":"Supplies the EHR QA dataset and the code-based reasoning agent that TrustEHRAgent extends and compares against.","marker":"Shi et al., 2024"},{"why":"Provides the EHRSQL text-to-SQL benchmark built on MIMIC-III and eICU that the evaluation dataset is derived from.","marker":"Lee et al., 2022"},{"why":"Grounds the stepwise verbalized confidence approach used to collect confidence ratings during reasoning.","marker":"Yang et al., 2024"},{"why":"Supports the weighted token log-probability scoring method over discrete confidence levels.","marker":"Liu et al., 2023"},{"why":"Provides the SQLAgent and PythonAgent baseline implementations that are augmented with confidence estimation for comparison.","marker":"Roucher et al., 2025"},{"why":"Motivates the need for reliability-aware evaluation by documenting limitations of LLMs in clinical decision-making.","marker":"Hager et al., 2024"},{"why":"Contextualizes calibration of LLM confidence for question answering, which the proposed estimator builds on.","marker":"Jiang et al., 2020"}],"fun_headline_variants":["EHR agent only answers when sure, beats baselines by 44% points","Confidence-aware EHR agent stays accurate where baselines collapse","New metric HCAcc@70%: agent achieves 44.23% while baselines fail","Stepwise confidence estimation lets EHR agent answer reliably at 70%","TrustEHRAgent: silent when unsure, 44% more accurate than baselines"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported HCAcc@k% numbers depend on two assumptions: that a confidence threshold chosen by scanning the test set will behave the same in deployment, and that the confidence score from Eq. 2 is a properly normalized probability; if either fails, the large gaps over baselines at k=70% may shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["EHR agent only answers when sure, beats baselines by 44% points","Confidence-aware EHR agent stays accurate where baselines collapse","New metric HCAcc@70%: agent achieves 44.23% while baselines fail","Stepwise confidence estimation lets EHR agent answer reliably at 70%","TrustEHRAgent: silent when unsure, 44% more accurate than baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1447,"prompt_tokens":703,"completion_tokens":744,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":656}},"tokens_in":447,"tokens_out":744,"duration_ms":20058,"temperature":1.0,"reasoning_tokens":656,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:55:52.489892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiments with the confidence threshold fixed on a validation split, never touched on the test set, then compute HCAcc@70% on MIMIC-III and eICU; if TrustEHRAgent's margin over baseline agents drops substantially, the central claim fails. A second check: recompute the confidence score with true softmax normalization over the five log-probabilities instead of raw exp(logprob) sums; if the ranking of answers changes, the reported thresholds are not interpretable as stated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EHR QA dataset and the code-based reasoning agent that TrustEHRAgent extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the EHRSQL text-to-SQL benchmark built on MIMIC-III and eICU that the evaluation dataset is derived from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the need for reliability-aware evaluation by documenting limitations of LLMs in clinical decision-making."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contextualizes calibration of LLM confidence for question answering, which the proposed estimator builds on."}],"review_version":1}