{"id":"77aa0d03-b456-4dfd-9976-63e3d5c38572","arxiv_id":"2608.09080","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs keep high reported confidence when medical questions become ambiguous or unanswerable, producing more confident wrong answers instead of abstaining.","lead":"Five LLMs were tested on 500 medical questions under prompt ambiguity and with the correct answer removed, and confidence stayed high even as accuracy dropped. The paper matters because it quantifies how overconfident AI answers could mislead clinicians when information is incomplete.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'insensitivity' claim lacks a direct information-present vs information-absent confidence comparison, and self-reported scores are unvalidated as measures of model uncertainty.","rationale":"The reader's strongest claim is that confidence signals from current LLMs cannot be relied on as calibrated indicators of correctness in medical QA. The paper's evidence for this is mostly absolute high self-reported confidence; however, a confidence signal that is unvalidated and not compared within-question cannot support an 'insensitivity' claim. The log-prob discrepancy in Table III indicates c_i is at least partly prompt compliance rather than epistemic uncertainty. This concern is not a disagreement with field consensus; it is an internal evidence gap. The proposed test would either strengthen the paper (if log-prob also stays flat and AUROC is meaningfully above chance) or force a more limited claim about verbalized confidence. The reader's weakest assumption identified self-reported confidence validity, which matches my concern, but I extend it: even before validity, no paired comparison of information presence versus absence is reported. The numerical inconsistency in Table IV further underscores the need for corrected, released data and code. Since the concern is addressable by additional analysis, the conditional verdict remains appropriate; I would keep the verdict conditional and unchanged from the reader's CONDITIONAL.","tokens_in":12213,"tokens_out":7314,"duration_ms":75212,"concrete_test":"Recompute the two experiments on the same 500 MedMCQA questions and produce a paired sensitivity test: for each question, compare self-reported confidence and token-logit confidence (Eq. 6-7) between the C0 condition (correct answer present) and the Setting B condition (correct answer removed, abstention option added). Also compute token-logit confidence under C0-C3 in Setting A. Report mean paired differences and a Wilcoxon signed-rank test with Holm-Bonferroni correction. Additionally, validate c_i as an uncertainty measure by computing AUROC of confidence against correctness within each condition; if AUROC is near 0.5 or token-logit confidence drops under C0-C3 while c_i does not, the conclusion should be restricted to verbalized confidence. Finally, recompute Table IV with hallucination = 1 - abstention and verify unsafe <= hallucination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V claims LLM confidence is largely insensitive to clinically meaningful information loss, but the experimental design never directly measures sensitivity. In Setting A (uncertainty framing), Eq. 3 transforms only the prompt wording; the option set and clinical content are unchanged, so a calibrated model need not change its posterior confidence. Flat self-reported confidence across C0-C3 is therefore not itself evidence of miscalibration. In Setting B, the correct answer is removed (Eq. 4), but the paper reports only absolute confidence in that condition; it does not pair each question with its C0 baseline, so 'insensitivity' is asserted rather than measured. The only confidence signal used in Setting A is the self-reported numeric score c_i from Section III-C, which the paper's own Table III shows can be radically disconnected from token-logit confidence (LLaMA-3: 0.970 vs 0.280; Mistral-7B: 0.998 vs 0.657). Thus the central claim may reduce to 'verbally elicited confidence scores are high and stable,' not 'model confidence fails to track information loss.' Separately, Table IV is internally inconsistent (LLaMA-3 abstention 0.156 implies hallucination 0.844, not 0.798, and unsafe 0.828 cannot exceed hallucination 0.798), so the numeric basis for the conclusion needs verification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a behavioral evaluation framework for LLM confidence in medical multiple-choice QA under two forms of uncertainty: (i) graded linguistic uncertainty cues added to otherwise unchanged prompts (conditions C0–C3, Eq. 3 and Table I), and (ii) removal of the correct answer option with an explicit abstention option (Eq. 4). Five models (GPT-5, GPT-4o, GPT-4o-mini, LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3) are evaluated on 500 sampled MedMCQA questions using accuracy, self-reported and logit-based confidence, ECE, UCER, abstention, and hallucination rates. The central claim is that model confidence remains largely insensitive to clinically meaningful information loss, leading to unsafe confident errors, and that models rarely abstain when no correct answer is available.","tokens_in":12480,"tokens_out":5038,"duration_ms":48573,"significance":"If the central claim holds, the paper would provide a useful warning about the reliability of verbalized confidence in medical decision support. The study has several strengths: it uses a public benchmark, evaluates both proprietary and open-weight models, defines quantitative metrics (ECE, UCER), reports deterministic inference settings, and includes a limitations section. The qualitative pattern of overconfidence in open-weight models is visible in Table II. However, as detailed below, the evidence currently does not directly support the insensitivity claim because the experimental design lacks the necessary paired comparisons, the promised statistical tests for confidence are not reported, and one key table contains internally inconsistent numbers.","major_comments":[{"comment":"Setting A keeps the answer options A_i and ground-truth labels y_i unchanged across C0–C3; only the prompt wording is transformed. A calibrated model whose posterior over the same options does not change has no reason to lower its confidence. Flat self-reported confidence under C0–C3 is therefore expected and cannot, by itself, support the claim that confidence is insensitive to information loss. The authors should provide a direct measure of whether confidence should move under their framing, or compare calibration metrics such as ECE/UCER against a C0 baseline rather than interpreting stability in raw confidence as miscalibration.","section":"Section III-B, Eq. (3), Table I"},{"comment":"The methods section promises paired Wilcoxon signed-rank tests for self-reported confidence comparisons, with Holm–Bonferroni correction, but Section V-C reports only McNemar tests for model accuracy. No p-values, effect sizes, or corrected significance statements for confidence are presented anywhere. Consequently, the claim in Section V-A that 'uncertainty framing does not lead to any meaningful reduction in confidence' is not backed by the stated inferential analysis. Please report the confidence comparisons (at minimum C0 vs. C3 for each model) with appropriate multiple-testing correction.","section":"Section III-D3 and Section V-C"},{"comment":"In Setting B the correct answer is removed, but the paper reports only aggregate abstention, hallucination, and mean confidence in the removal condition. There is no paired per-question comparison with a condition in which the correct answer is present (e.g., C0). Thus the claim that confidence fails to decrease when information is removed is asserted rather than measured. The authors should provide per-question paired differences in confidence between the intact condition and the removal condition, and ideally model abstention conditional on the original question, to test sensitivity directly.","section":"Section III-C, Eq. (4), Tables III–IV"},{"comment":"Eq. (10) defines Hallucination = 1 − Abstention, so an abstention rate of 0.156 for LLaMA-3 should yield a hallucination rate of 0.844, not 0.798 as reported. Additionally, the Unsafe rate of 0.828 exceeds the hallucination rate of 0.798, which is impossible under the definition in Section V-B2 that unsafe predictions are a subset of hallucinated (incorrect) predictions. The MeanConf for LLaMA also differs between Table III (0.970) and Table IV (0.963). These inconsistencies must be corrected and the affected safety conclusions re-verified.","section":"Table IV, LLaMA-3 row"},{"comment":"The paper itself shows that self-reported and logit-based confidence diverge dramatically (e.g., LLaMA-3: 0.970 vs. 0.280; Mistral-7B: 0.998 vs. 0.657) and concludes that self-reported confidence 'cannot be reliably interpreted as a measure of model uncertainty.' Yet the abstract and Section V use self-reported confidence as evidence about 'model confidence.' The central claim should either be explicitly restricted to verbalized confidence, or the paper should provide evidence that the self-reported score is a valid uncertainty measure in this setting.","section":"Section V-B3, Table III"}],"minor_comments":[{"comment":"Several rows contain values that are run together (e.g., '10.8020.904' and '30.8360.798'), making the table difficult to read; add explicit column separators.","section":"Table II"},{"comment":"The caption says 'across Uncertainty Framing Conditions (C0–C3)' but the figure reports the answer-removal experiment; the caption should be corrected.","section":"Fig. 5 caption"},{"comment":"The UCER threshold τ is defined in Eq. (8) but its value τ=0.8 is introduced only later in Section V-B2; state and justify the threshold in the metrics section.","section":"Section III-D2, Eq. (8)"},{"comment":"The text describes log-probability confidence as 'the probability of each answer option,' but Eq. (6) computes a softmax over single answer tokens; clarify that this is a token-level proxy rather than a full response probability.","section":"Section III-C, Eq. (6)"},{"comment":"The abstention option position is said to be randomized for each question, but no random seed is reported for this randomization, which weakens the stated reproducibility.","section":"Section IV-B"},{"comment":"Temperature 0 is reported for all models, but API-based models do not guarantee deterministic outputs; please add a caveat about possible nondeterminism in the proprietary models.","section":"Section IV-C"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a clean empirical core: five models, 500 MedMCQA questions, two uncertainty manipulations, and a sensible metric suite (ECE, UCER, abstention). The observation that self-reported confidence and token-logit confidence diverge wildly (LLaMA: 0.970 vs 0.280) is a useful reminder that verbalized confidence is not the same as internal probability. Those measurements are new and could be a solid base for a good paper.\n\nBut the central claim — that LLM confidence is largely insensitive to clinically meaningful information loss — is not actually tested by the experimental design. In Setting A, the choice set and clinical content are unchanged; you only add a generic warning phrase. A calibrated model need not lower its confidence when the task is objectively the same, so flat confidence across C0–C3 is neither surprising nor evidence of miscalibration. In Setting B, the correct answer is removed, but the paper never reports a paired comparison of confidence on the same questions with and without the correct option. It only gives absolute confidence in the degraded condition. So \"insensitivity\" is asserted, not measured. If the goal is to show models fail to reduce confidence when the answer disappears, you need the same questions in both conditions, not an aggregate across different sets.\n\nThe missing statistics are a real problem too. Section III-D3 promises Wilcoxon signed-rank tests for confidence comparisons, but the results section reports only McNemar tests on accuracy. No confidence p-values or effect sizes appear anywhere. That is a major omission for a paper whose thesis is about confidence. And Table IV has an internal inconsistency: LLaMA-3's abstention of 0.156 implies a hallucination rate of 0.844, but the table says 0.798; the unsafe rate of 0.828 also needs reconciliation with the hallucination rate. These are checkable numbers, and they should be fixed before anyone trusts the aggregate figures.\n\nI also want to flag the reliance on self-reported confidence as the main measure. The paper's own Table III shows that this can be radically disconnected from logit-based probability. That doesn't invalidate the study, but it means the claim is really about verbalized confidence, not model uncertainty more broadly. The paper should either validate the self-reported scores or restrict the conclusion accordingly.\n\nWho is this for? People building or evaluating medical QA systems, especially those interested in abstention and calibration. The framework is a useful starting point, and the empirical patterns are plausible, but the analysis needs substantial revision. I would send it to peer review rather than desk reject, because the core idea is timely and the measurements are genuinely new. The authors need to add the direct confidence comparisons, report the Wilcoxon results, fix Table IV, and soften or re-justify the central claim. A serious referee can push them to do that.","headline":"Worth reading for the setup, but the 'confidence is insensitive' claim isn't supported by the experiments as reported.","tokens_in":12997,"tokens_out":3073,"duration_ms":30268},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM confidence stays high when clinical information is missing or ambiguous, so self-reported confidence is not a reliable safety signal in medical QA.","keywords":["medical question answering","large language models","confidence calibration","overconfidence","uncertainty awareness","abstention","hallucination","unsafe confident errors"],"falsifier":"A concrete test: run the same 500-question protocol with a model whose self-reported confidence drops steadily from C0 to C3 and whose abstention rate exceeds 90% when the correct answer is removed; that would directly contradict the paper's insensitivity claim. A cheaper check on the removal set is to see whether mean confidence on hallucinated answers falls below 0.5 or the Unsafe Confident Error Rate approaches zero.","tokens_in":12028,"feed_emoji":"⚠️","tokens_out":6975,"duration_ms":53625,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models, despite strong accuracy on medical multiple-choice questions, do not lower their confidence when the clinical information they receive becomes uncertain or incomplete. Using 500 questions from the MedMCQA benchmark, the authors inject graded linguistic uncertainty cues into prompts and, in a second setting, remove the correct answer and replace it with an abstention option. Accuracy stays roughly flat or drops while reported confidence remains high, and the rate of wrong answers delivered with confidence above 0.8 rises sharply. The authors conclude that current LLM confidence scores cannot be trusted as indicators of answer correctness in clinical settings, and that prompt-level warnings do not repair this. A confidently wrong suggestion from a decision-support system could mislead a clinician, so the finding matters for any plan to deploy these models in medical workflows.","feed_headline":"LLM confidence ignores missing clinical information","feed_subtitle":"Five models keep high-confidence answers even after the correct option is removed, failing safe abstention.","key_machinery":"The central object is a two-setting evaluation framework built on the MedMCQA benchmark. Setting A prepends graded linguistic uncertainty statements — C0 (none), C1, C2, C3 — to otherwise identical questions, leaving the answer set unchanged, so any behavioral shift can be attributed to the framing. Setting B removes the ground-truth option and replaces it with an abstention option, so the only safe behavior is selecting 'Insufficient information to determine the correct answer.' Confidence is collected two ways: a self-reported numeric score elicited from the model and a logit-based probability computed by softmax over the answer-token logits. The metrics — accuracy, calibration gap, ECE, UCER, abstention and hallucination rates — convert the setup into a test of whether confidence tracks information loss.","core_discovery":"The paper's central discovery is a systematic failure mode in LLM confidence behavior. When uncertainty cues are added to clinical questions, or when the correct answer is removed and an explicit 'insufficient information' option is offered, the models' self-reported confidence remains largely insensitive to the information loss. In the framing setting, mean confidence stays near or above 0.85 for most models even at the strongest cue level, with calibration gaps and UCER staying roughly constant; in the removal setting, the best abstention rate is only 38.2%, and hallucinated answers are frequently produced with confidence above 0.8. The paper also finds that open-weight models show a large split between self-reported confidence and logit-based probability, meaning their stated confidence overestimates what their own output distribution implies. The stated conclusion is that current LLMs lack reliable uncertainty awareness in medical QA, with the failure strongest in open-weight models but present across proprietary ones.","pith_inferences":["One consequence the authors leave implicit is that the same accuracy–confidence decoupling should be expected in other high-stakes domains whose benchmarks assume a correct answer always exists, since the proposed cause is the training objective rather than medicine-specific content.","The large self-reported versus logit-based confidence gap suggests a concrete, testable mitigation that the paper does not propose: replace or rank by token-logit probabilities when the API exposes them, rather than trusting elicited scores.","Because the paper's uncertainty cues are inserted prompts, a stronger test would use naturally occurring ambiguous clinical cases; if confidence stays high there too, the insensitivity claim would be more robust to the concern that the cues are simply too weak.","A boundary condition worth probing is whether some cue strength or instruction (for example, explicitly instructing the model to abstain when unsure) eventually breaks the insensitivity; the paper only shows that the tested prompts do not."],"forward_implications":["If the paper is right, a high confidence score from an LLM in medical QA carries little information about whether the answer is actually correct once information is incomplete.","Prompt-based uncertainty cues like C1–C3 are not a sufficient safeguard: they do not meaningfully reduce confidence or increase abstention.","Open-weight models such as LLaMA-3 and Mistral-7B would be particularly unsafe as clinical decision-support tools under current calibration, with near-maximal confidence on hallucinated answers.","Even the better-calibrated proprietary models abstain less than 40% of the time when the correct answer is absent, so closed-world benchmarks overestimate reliability.","Medical LLM evaluation should include abstention and unsafe-error metrics rather than accuracy alone if it is meant to reflect real clinical conditions."],"supporting_citations":[{"why":"Supplies the MedMCQA dataset from which the 500 medical multiple-choice questions are sampled.","marker":"[4]"},{"why":"Provides the 'confidence paradox' framing that verbalized confidence can diverge from a model's internal state, motivating the paper's confidence measures.","marker":"[7]"},{"why":"Defines calibration and the ECE metric with the 10-bin partition used throughout the evaluation.","marker":"[9]"},{"why":"Prior work on using token probabilities to mitigate overconfidence in medical QA, which the paper contrasts with self-reported confidence.","marker":"[12]"},{"why":"Supports the strategy of eliciting numeric confidence scores from language models by prompting.","marker":"[14]"},{"why":"Benchmark showing that LLMs frequently fail to abstain, extended here to clinical missing-information settings.","marker":"[18]"},{"why":"Paired McNemar test used for statistical comparisons of model accuracy.","marker":"[23]"},{"why":"Wilcoxon signed-rank test used for comparing bounded self-reported confidence values.","marker":"[24]"}],"fun_headline_variants":["LLMs overlook missing clinical info, staying overconfident","When certainty is wrong: LLM confidence fails medical tests","Overconfident LLMs: even without the correct answer, they guess","Medical QA: LLM confidence ignores information loss","Abstention fails: LLMs hallucinate answers with high confidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the confidence score a model writes when asked to 'provide a confidence score' reflects its internal uncertainty, and that the inserted linguistic cues are a valid proxy for clinically meaningful information loss; if either fails, the claim that confidence is insensitive does not follow.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overlook missing clinical info, staying overconfident","When certainty is wrong: LLM confidence fails medical tests","Overconfident LLMs: even without the correct answer, they guess","Medical QA: LLM confidence ignores information loss","Abstention fails: LLMs hallucinate answers with high confidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1471,"prompt_tokens":1001,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":617,"tokens_out":470,"duration_ms":200949,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:56:55.053960+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-daily-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A concrete test: run the same 500-question protocol with a model whose self-reported confidence drops steadily from C0 to C3 and whose abstention rate exceeds 90% when the correct answer is removed; that would directly contradict the paper's insensitivity claim. A cheaper check on the removal set is to see whether mean confidence on hallucinated answers falls below 0.5 or the Unsafe Confident Error Rate approaches zero.","supporting_citations":[{"cited_title":"MedMCQA: A large- Scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the MedMCQA dataset from which the 500 medical multiple-choice questions are sampled."},{"cited_title":"The Confidence Paradox: Can LLM Know When It’s Wrong?","cited_arxiv_id":null,"evidence_quote":"Provides the 'confidence paradox' framing that verbalized confidence can diverge from a model's internal state, motivating the paper's confidence measures."},{"cited_title":"On Calibration of Modern Neural Networks,","cited_arxiv_id":null,"evidence_quote":"Defines calibration and the ECE metric with the 10-bin partition used throughout the evaluation."},{"cited_title":"Token Probabilities to Mitigate Large Language Models Overcon- fidence in Answering Medical Questions: Quantitative Study,","cited_arxiv_id":null,"evidence_quote":"Prior work on using token probabilities to mitigate overconfidence in medical QA, which the paper contrasts with self-reported confidence."},{"cited_title":"Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback,","cited_arxiv_id":null,"evidence_quote":"Supports the strategy of eliciting numeric confidence scores from language models by prompting."},{"cited_title":"Do LLMs Know When to Not Answer? Investigating Abstention Abilities of Large Language Models,","cited_arxiv_id":null,"evidence_quote":"Benchmark showing that LLMs frequently fail to abstain, extended here to clinical missing-information settings."},{"cited_title":"Note on the Sampling Error of the Difference between Correlated Proportions or Percentages,","cited_arxiv_id":null,"evidence_quote":"Paired McNemar test used for statistical comparisons of model accuracy."},{"cited_title":"Individual Comparisons by Ranking Methods,","cited_arxiv_id":null,"evidence_quote":"Wilcoxon signed-rank test used for comparing bounded self-reported confidence values."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}