{"id":"46ede6c5-6140-4fa2-a040-d05bc286d918","arxiv_id":"2608.01361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Re-analyzing a consumer triage system's failures, the paper shows much of the under-triage comes from an implicit resource-conserving decision policy, and that utility prompts can steer capable models along the safety-resource curve.","lead":"AI health apps that decide whether you need the emergency room may fail not because they misread symptoms, but because they quietly value avoiding unnecessary visits more than catching emergencies. This paper shows that explaining the desired trade-off in plain language pushes capable AI models to catch far more true emergencies, so the priorities behind high-stakes AI decisions should be made explicit.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deployed tool lies below GPT-5-mini's ROC curve at its own false-alarm rate, so the paper's caveat contradicts the claim that 'much' of ChatGPT Health's under-triage is explained by implicit utility; the vertical gap is unquantified.","rationale":"The reader's weakest assumption was the fidelity of elicited probabilities to the probabilities used at decision time. That is an important methodological concern, but the paper's self-consistency check (Extended Data Figure 3) partially mitigates it by showing decisions are well explained by a threshold on the elicited probabilities. A more direct threat to the specific central claim about ChatGPT Health is the paper's own admission that the deployed tool's operating point lies below the ROC curve generated from GPT-5-mini's probabilities: at the tool's observed false-alarm rate, it catches fewer emergencies than the model's rank ordering would predict. This gap cannot be explained by any cost-ratio threshold, because the tool is not on the frontier; it must reflect properties of the deployed system beyond utility. The paper acknowledges this but does not quantify the gap, making 'much' unsupported. The proposed test—reading the ROC curve at the tool's FPR—would settle whether the threshold explanation accounts for most or only a fraction of the under-triage. This concern does not overturn the paper's broader decision-analytic framing or the utility-prompting results for the models themselves, so the conditional verdict remains appropriate, but it should be conditioned on this quantitative check rather than only on indirectness and missing confidence intervals.","tokens_in":921,"tokens_out":817,"duration_ms":110663,"concrete_test":"Using the released data, compute GPT-5-mini (high reasoning) ROC curve on the primary 576-case endpoint, read TPR at FPR = 0.10 (ChatGPT Health's reported FPR), and compare with the reported 48% TPR. Repeat for the expanded 1,248-case endpoint. Report the vertical gap and the fraction of the sensitivity deficit (from the tool's TPR to 100%) that is closed by moving along the curve at fixed FPR. If the fraction is < 50%, the claim should be reworded from 'much' to 'some' and the residual attributed to the deployed system; if > 50%, the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('Much of the reported under-triage can therefore be explained by an implicit preference against over-referral rather than by limited discriminatory performance alone') is undercut by the paper's own first caveat in Results: 'even at the same false-alarm rate, the tool identifies fewer emergencies than would be expected on the basis of the model's probability estimates.' If ChatGPT Health operates below the ROC frontier defined by GPT-5-mini's elicited probabilities, then no single cost-ratio threshold over those probabilities can reproduce the tool's behavior; a residual gap remains that is attributable to the deployed system itself (safeguards, post-processing, or a different internal model), not to a utility trade-off. The size of this gap is never reported, and 'much' is never quantified. Since the AUROCs are 0.95–0.99, the ROC curve at FPR ≈ 0.10 likely yields TPR well above the tool's ~48%; if so, the threshold explanation accounts for only a minority of the sensitivity shortfall on the primary endpoint. The paper concedes that the discrepancy is 'significantly lower' on the expanded set, so the strength of the claim is endpoint-dependent. This is an internal inconsistency between the headline result and the admitted caveat, not merely an external limitation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reanalyzes the ChatGPT Health emergency-triage failures reported by Ramaswamy et al. (2026). The authors elicit probability estimates and triage decisions separately from several frontier language models, fit a decision-theoretic model to recover the implicit cost ratio (false-negative cost relative to false-positive cost), and show that prompting models with explicit cost ratios moves their operating points along ROC curves derived from the elicited probabilities. They argue that the deployed tool's low sensitivity is largely attributable to an implicit resource-conserving utility rather than to poor discriminative ability, and they propose decision-analytic steering and evaluation as a general paradigm for high-stakes LM deployment.","tokens_in":13257,"tokens_out":7465,"duration_ms":78107,"significance":"The decision-analytic framing is valuable and timely: it separates probabilistic competence from utility-based decision policy in LM evaluation, a distinction that is often collapsed in safety evaluations. The paper is unusually transparent: it releases data and code, reports self-consistency checks (Extended Data Figure 3), validates the choice of GPT-5-mini as the reference model (Extended Data Figure 4), and explicitly states its caveats. If the central claim is sustained, the work has practical implications for how triage systems and other high-stakes LM applications should be prompted, evaluated, and deployed. However, the headline claim about the deployed tool is currently supported only qualitatively: the paper concedes that the tool lies below the GPT-5-mini ROC curve at its observed false-alarm rate, and it never quantifies how much of the under-triage is explained by utility versus residual inconsistency. That gap is the main barrier to accepting the paper's central conclusion.","major_comments":[{"comment":"The central claim, 'Much of the reported under-triage can therefore be explained by an implicit preference against over-referral rather than by limited discriminatory performance alone,' is undercut by the paper's own first caveat and is never quantified. The text states that at the same false-alarm rate the ChatGPT Health tool identifies fewer emergencies than expected from GPT-5-mini's probability estimates; that is, the tool lies below the ROC curve. No cost-ratio threshold over the elicited probabilities can reproduce such behavior, so the threshold explanation cannot account for the full shortfall. The manuscript should report the GPT-5-mini TPR at FPR ≈ 0.10 (the tool's operating region), quantify the vertical residual gap, and state what fraction of the 48% sensitivity shortfall is attributable to threshold choice versus remaining inconsistency. The claim is also endpoint-dependen","section":"Decoding a model's decision priorities (caveats paragraph)"},{"comment":"The recovery procedure is under-specified. The manuscript says a 'standard discrete-choice logistic model' is fit to decisions and probabilities and that the implied cost ratio is derived, but it does not give the model equation, the link function, the covariate (raw probability or logit), or the formula converting the fitted parameters into the reported cost ratio. This matters because Figures 3 and 4, and the recovered default thresholds throughout, depend directly on this mapping. The paper should specify, for example, P(refer) = σ(α + β·p) or P(refer) = σ(α + β·logit(p)), and state whether the threshold is α = −β·t or the equivalent. Without this, the quantitative recovered ratios are not reproducible.","section":"Methods, 'Recovering the cost ratio implied by a model's behavior'"},{"comment":"The claim that 'the same underlying predictions can support markedly different decision policies' assumes that the probabilities elicited in the neutral, utility-free query are the probabilities the model uses when making decisions under utility prompts. This is not directly tested. If a safety-prioritized prompt also inflates the model's internal probability estimates, the recovered cost ratio would absorb that change and the steering result would conflate probability shift with utility shift. The self-consistency check (Extended Data Figure 3) shows that decisions are well described by a threshold on the elicited probabilities within each regime, but it does not establish cross-regime invariance. The authors should either re-elicit probabilities under the utility-prompt conditions in separate queries and demonstrate invariance, or temper the wording so that 'same underlying predictions","section":"Methods, 'Eliciting probabilities'; Abstract"}],"minor_comments":[{"comment":"Typo: 'and the we suggest improving the precision' should read 'and we suggest improving the precision.'","section":"Discussion"},{"comment":"The phrase 'increases by roughly 50% without increasing unnecessary referrals' should state whether the increase is in percentage points or relative, and should reconcile with the fact that along a single ROC curve a higher TPR at the same FPR is not achievable by a threshold shift. If FPR did increase slightly, report the exact change.","section":"Decoding a model's decision priorities"},{"comment":"The red star coordinates and the corresponding ROC TPR at the same FPR should be reported numerically in the text or caption. This would directly address the unquantified vertical gap and make the 'much' claim assessable.","section":"Figure 1 / Figure 2"},{"comment":"Standardize the terminology: the text alternates between 'Extended Figure 1' and 'Extended Data Figure 1'; the supplementary section should be consistent.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent and the core methodology is defensible, but the main risk is the unquantified 'much' claim. If the authors provide a quantitative decomposition of the sensitivity shortfall into threshold-driven and residual components, and either confirm or soften the 'same underlying predictions' claim, the paper could be publishable. The current version requires major revision to make the headline claim commensurate with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The most striking result here is not the ROC curves but the demonstration that you can recover a model's implicit triage utility from its decisions and then steer it with plain-language cost-ratio prompts. Across 18 model configs, GPT and DeepSeek with reasoning track the requested trade-off closely; Claude is damped; small models without reasoning stay stuck. The released data and code make this reproducible. That part of the paper is solid and worth engaging with.\n\nThe soft spot is the claim about ChatGPT Health's under-triage. The paper says \"much\" of the deployed tool's poor sensitivity can be explained by an implicit resource-conserving utility, but its own caveat 1 says the tool sits below the ROC curve implied by GPT-5-mini's elicited probabilities at the same false-alarm rate. If the tool is below the curve, no single cost-ratio threshold over those probabilities reproduces its behavior. The ROC curves look good (AUC 0.95–0.99), so at 10% FPR the expected TPR is likely well above the tool's 48%. That means the threshold story can only account for a minority of the sensitivity shortfall on the primary endpoint; the rest is something about the deployed system itself—post-processing, safeguards, or a different internal model. The paper acknowledges this but never quantifies the gap, and the abstract still says \"much.\" That is an overstatement, not an internal contradiction, because the paper is transparent about the caveat. But it should be fixed: either estimate the gap or soften the claim.\n\nThe other concerns are minor: the reference-model assumption is validated only by decision agreement, and headline effect sizes lack CIs. Neither changes the core finding that capable models are steerable.\n\nThis paper is for people who design or evaluate clinical LLM decision support. It deserves a serious referee, but the revision should quantify or appropriately caveat the deployed-tool decomposition.\n\nRecommendation: send to peer review with a request to address the gap quantification.","headline":"Strong steering results and a clean decision-theoretic decomposition, but the headline claim about ChatGPT Health's under-triage overstates what the data support given the tool sits below the ROC curve.","tokens_in":13783,"tokens_out":4051,"would_cite":true,"duration_ms":34085,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language-model triage failures are better explained by unstated cost preferences than by poor clinical knowledge, the paper argues, so evaluations and deployments should make utilities explicit.","keywords":["emergency triage","language models","decision analysis","utility elicitation","revealed preferences","cost ratios","ChatGPT Health","calibration"],"falsifier":"Check the yardstick assumption directly: on a fixed case set, elicit probabilities and decisions in both orders (probability first vs. decision first) across repeated samples. If the recovered cost ratio shifts with query order, or if a model states a high emergency probability while withholding referral under a recovered safety-prioritized ratio whose implied threshold is much lower, then the separate probability estimate is not the belief driving the decision, and the utility attribution fails. For the deployed-tool claim, the decisive check is access to ChatGPT Health's internal probability","tokens_in":12849,"feed_emoji":"🚑","tokens_out":13039,"duration_ms":117148,"temperature":0.7,"pith_summary":"The paper tries to establish that emergency triage by a language model is a decision problem, not just a prediction problem, and that the well-known under-triage of ChatGPT Health is largely a policy failure rather than a knowledge failure. Eliciting a model's probability estimate separately from its referral decision, the authors recover the implicit cost ratio—the weight a model places on a missed emergency relative to an unnecessary referral—that best explains its choices. On clinician-adjudicated vignettes, the elicited probabilities rank emergencies very well, yet the deployed tool's default operating point sits at the resource-conserving end of the trade-off curve. Prompting capable models with explicit cost ratios moves their operating points up the curve, reaching near-100% sensitivity. The upshot is that evaluating and steering high-stakes language-model systems requires specifying utilities explicitly and reporting performance across the whole trade-off curve.","feed_headline":"Under-triage is a hidden preference, not a knowledge gap","feed_subtitle":"Splitting probability from preference shows explicit cost-ratio prompts can push model sensitivity toward 100%.","key_machinery":"The central object is the cost ratio $\\Gamma = c_{\\mathrm{FN}}/c_{\\mathrm{FP}}$: the relative cost of a missed emergency versus an unnecessary referral. Under a cost-minimizing decision rule it implies a break-even referral probability $p^* = c_{\\mathrm{FP}}/(c_{\\mathrm{FP}}+c_{\\mathrm{FN}})$, so recovering the threshold that best explains a model's binary decisions from its own elicited probabilities reveals its implicit utility. The machinery is black-box: separate queries elicit the probability and the decision, and a discrete-choice logistic fit recovers the implied cost ratio. This decomposes triage errors into discrimination (AUROC) and decision policy (location on the ROC curve), and","core_discovery":"Much of ChatGPT Health's under-triage comes from an implicit preference against over-referral, not from limited discrimination. On the 576-case endpoint, elicited probabilities rank emergencies well (AUROC 0.95–0.99), yet the tool sits at the resource-conserving end of the curve, catching about 48% of emergencies at about 10% false alarms. Prompting the same model family with explicit 5:1 or 10:1 safety-prioritized cost ratios moves it up the curve, adding roughly 50% sensitivity with little added over-referral. Default utilities vary by model and reasoning level, and steering fidelity depends on capability. High-stakes LM systems should be treated as probabilistic decision systems governed","pith_inferences":["The paper leaves implicit that the same revealed-preference procedure could run prospectively as a pre-deployment safety check, producing a documented default cost ratio for any triage system.","An unstated consequence of the reported variation across reasoning settings: changing inference-time reasoning effort could silently alter a model's triage policy, so safety reports should list operating points per reasoning configuration.","A natural testable extension is to turn this into a benchmark: on decision tasks with known asymmetric error costs, compare a model's recovered cost ratio with the instructed ratio to measure steering fidelity domain by domain."],"forward_implications":["Stating a cost ratio in plain language can move a capable model's operating point up the ROC curve to near-100% sensitivity without substantially increasing unnecessary referrals.","Default LM behavior encodes an implicit, model-specific and reasoning-level-specific trade-off that should not be treated as clinically appropriate without explicit specification.","Single-point evaluations can misclassify steerable models as unsafe or poorly controlled models as safe; reporting the full trade-off curve and prompted operating points corrects this.","A target cost ratio can be deployed either by prompting the model or by applying an external threshold to its elicited probabilities, and the threshold approach offers some robustness to miscalibration.","The distinction between probability and utility generalizes beyond triage to any high-stakes setting with asymmetric error costs."],"supporting_citations":[{"why":"Supplies the clinician-authored vignettes, gold-standard labels, and the documented ChatGPT Health under-triage result that the paper reinterprets.","marker":"[8]"},{"why":"Provides the revealed-preference framework used to recover the cost ratio implicit in a model's decisions from its elicited probabilities.","marker":"[19]"},{"why":"Validates that probability estimates elicited separately from decisions can serve as a trustworthy yardstick for interpreting those decisions.","marker":"[18]"},{"why":"Supports the reliability of language-model probability elicitation, cited by the authors as prior validation of their elicitation procedure.","marker":"[23]"},{"why":"Supplies the conditional-logit discrete-choice model used to fit decisions to elicited probabilities and derive the implied cost ratio.","marker":"[24]"},{"why":"Provides the discrete-choice estimation practice the cost-ratio recovery follows.","marker":"[25]"}],"fun_headline_variants":["Preference, not ignorance, drives LM under-triage in emergencies","Explicit cost ratios steer LM triage toward safety, not over-referral","Emergency triage: LMs are decision systems, not just predictors","Making triage utilities explicit boosts LM sensitivity by 50%"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the probability a model reports in a separate, cost-free question is the same probability it acts on when making the triage recommendation, and that the tested small model variant is a faithful stand-in for the deployed ChatGPT Health system.","fun_headline_variants_meta":{"raw":{"variants":["Preference, not ignorance, drives LM under-triage in emergencies","Explicit cost ratios steer LM triage toward safety, not over-referral","Emergency triage: LMs are decision systems, not just predictors","Making triage utilities explicit boosts LM sensitivity by 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000599,"raw_usage":{"total_tokens":2634,"prompt_tokens":742,"completion_tokens":1892,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1815}},"tokens_in":486,"tokens_out":1892,"duration_ms":14011,"temperature":1.0,"reasoning_tokens":1815,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:16:09.878640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the yardstick assumption directly: on a fixed case set, elicit probabilities and decisions in both orders (probability first vs. decision first) across repeated samples. If the recovered cost ratio shifts with query order, or if a model states a high emergency probability while withholding referral under a recovered safety-prioritized ratio whose implied threshold is much lower, then the separate probability estimate is not the belief driving the decision, and the utility attribution fails. For the deployed-tool claim, the decisive check is access to ChatGPT Health's internal probability","supporting_citations":[{"cited_title":"Can Revealed Preferences Clarify LLM Alignment and Steering?","cited_arxiv_id":"2605.08556","evidence_quote":"Provides the revealed-preference framework used to recover the cost ratio implicit in a model's decisions from its elicited probabilities."},{"cited_title":"When agents say one thing and do another: Validating elicited beliefs from llms,","cited_arxiv_id":null,"evidence_quote":"Validates that probability estimates elicited separately from decisions can serve as a trustworthy yardstick for interpreting those decisions."},{"cited_title":"Conditional logit analysis of qualitative choice behavior.Frontiers in Econometrics, pages 105–142, 1973","cited_arxiv_id":null,"evidence_quote":"Supplies the conditional-logit discrete-choice model used to fit decisions to elicited probabilities and derive the implied cost ratio."},{"cited_title":"refer to emergency care whenever the model’s elicited probability exceeds the corresponding decision threshold","cited_arxiv_id":null,"evidence_quote":"Provides the discrete-choice estimation practice the cost-ratio recovery follows."}],"review_version":1}