{"id":"d14314ca-2c67-4f29-8cca-04453e244b25","arxiv_id":"2607.20526","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frontier LLMs' verbalized confidence is often miscalibrated: the most accurate model is not the best-calibrated, and several models score worse than a calibrated random baseline.","lead":"ConfidenceBench is a 200-question private multiple-choice benchmark that measures whether LLMs' stated confidence matches their actual accuracy. Fifteen frontier models were tested; the best-calibrated model had a Brier score of 0.103, while several models scored worse than a calibrated random guesser.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Brier scores contradict the paper's own refusal-handling rule: Section 3.2 says refusals are scored as 25% + random answer, but Table 2's n<600 shows refusals were dropped, biasing cross-model comparisons.","rationale":"The reader's weakest assumption was training-data contamination and label auditing. That is a real but speculative risk for any private benchmark. I find a more concrete and immediately verifiable flaw: the paper's stated refusal-scoring protocol is contradicted by the Table 2 results. The central claim depends on Brier scores being computed consistently across models; if refusals were excluded, then models with high refusal rates receive scores that are not comparable to models with zero refusals. This is not a hypothetical—the table's own n column shows non-refused sample sizes. However, I do not think this forces rejection: the top two models have n=600, and simple back-of-envelope inclusion of refusals at the expected random Brier of 0.1875 does not obviously overturn the qualitative pattern of 'calibration is distinct from accuracy.' The conditional verdict is therefore appropriate, with the concrete test needed to verify whether the exact numbers and the underperformer count are robust. My assessment thus agrees with the reader's CONDITIONAL verdict, and I would keep that verdict; I only differ on which concern is singled out as most load-bearing.","tokens_in":11994,"tokens_out":8443,"duration_ms":95132,"concrete_test":"Recompute all metrics from the per-response logs using the Section 3.2 protocol: every refusal or unparseable response gets p=0.25 and a uniformly random selected answer, and all 600 responses per model are included. Compare the resulting Brier, ECE, and calibration gap to Table 2. Check specifically whether (a) any model with reported n<600 crosses the 0.1875 baseline, (b) the identity/count of models underperforming the baseline changes, and (c) the best-calibrated model or the top-tier ranking changes. If none of these change materially, the current conditional verdict can stand; if any change, the central numerical claims require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that calibration is a distinct axis of reliability rests on the Brier-score rankings in Table 2. Section 3.2 explicitly states that refusals and unparseable responses are 'scored as 25% confidence with a randomly selected answer rather than being excluded,' because exclusion 'would inflate apparent calibration.' Yet Table 2's header defines n as 'pooled non-refused responses across 3 runs' and reports n<600 for every model with nonzero refusals (e.g., GPT-5 Nano n=488, GPT-5 Mini n=527). This means refused responses were excluded from the Brier computation, not scored as specified. If refusals concentrate on hard questions, the excluded models' Brier scores are artificially optimistic, and comparisons across models are not made on a common response set. The paper's own warning says this is a bias, and it is precisely what the reported n values indicate. Preliminary arithmetic shows the top two n=600 models are unaffected, so the qualitative conclusion may survive: for GPT-5 Nano, including 112 refusals at expected Brier 0.1875 would raise its Brier from 0.133 to about 0.143. But the exact values, the 'five of fifteen underperform baseline' count, and the model ordering in Table 2 are not reliable as reported. This is an internal inconsistency, not a speculation about contamination, and it is directly testable from the evaluation logs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ConfidenceBench introduces a private 200-question multiple-choice benchmark across four categories (spatial reasoning, high-precision mathematics, word lookup, unknowable questions) and uses verbalized confidence from 15 frontier LLMs over three runs. Calibration is measured with the Brier score as the primary metric, supplemented by ECE and a signed calibration gap. The paper reports that Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve the lowest Brier scores (0.103), that the most accurate model is not the best-calibrated, and that five of fifteen models underperform a calibrated-random baseline (0.1875). A human tester with internet access is used as a baseline. The central claim is that verbalized confidence calibration is a distinct, practically important axis of LLM reliability, complementary to accuracy.","tokens_in":12338,"tokens_out":11254,"duration_ms":104365,"significance":"If the reported measurements are correct, the paper provides a useful, compact stress test of verbalized confidence and supports the view that accuracy alone does not capture reliability. Strengths include manual authorship of the question set to reduce contamination, use of a proper scoring rule, three-run repeated evaluation, category-level diagnostics, and an explicit protocol for handling refusals. The main qualitative conclusion is plausible and may survive a corrected reanalysis. However, the reported quantitative results are compromised by an internal inconsistency in refusal handling, and the human baseline is not directly comparable to the model condition. With a careful reanalysis and fuller disclosure, this could be a valuable benchmark contribution.","major_comments":[{"comment":"The stated protocol is that refusals and unparseable responses are scored as 25% confidence with a randomly selected answer, because exclusion would inflate apparent calibration. Table 2 defines n as 'pooled non-refused responses' and reports n<600 for every refusal-prone model (e.g., GPT-5 Nano n=488, GPT-5 Mini n=527; Figure 9 shows GPT-5 Nano refusal rate 18.7%, i.e., 112/600). Thus refusals were excluded from the reported Brier, accuracy, and ECE, not scored as specified. Recomputing GPT-5 Nano's Brier with the stated rule (112 refusals at expected Brier 0.1875) gives about 0.143 instead of 0.133, and its accuracy drops from 70.5% conditional to about 62% overall; GPT-5 Nano and GPT-5 Low would swap order. This internal inconsistency directly affects the headline rankings, the 'five of fifteen underperform baseline' enumeration, and Figures 1, 7, and 8. The results must be recomputed","section":"Section 3.2 vs. Table 2 and Figure 9"},{"comment":"The human baseline is not directly comparable to the models. The human tester completed the questions with internet access and one minute per question, whereas the models were API queries with no internet access. The claim in Section 6 that 'two models achieve lower Brier scores than the human tester' is therefore not an apples-to-apples comparison; internet access likely helps on Word Lookup and Unknowable items. To support the human-comparison claim, the human should be evaluated under the same information constraints, or the claim should be removed or clearly qualified.","section":"Section 4 / Section 5.1 / Section 6"},{"comment":"The benchmark's validity rests on the correctness and non-contamination of 200 private questions, but the set is withheld and no independent audit, contamination check, or inter-rater verification is reported. The Limitations paragraph acknowledges the audit limitation, but this is load-bearing: a few mislabeled or leaked items could change rankings. Please provide at least a public sample with detailed verification, an independent audit of labels, a contamination analysis, or item-level statistics that would reveal anomalies. At minimum, state the exact protocol used to confirm every answer and report per-category model performance to aid external assessment.","section":"Section 3.1 and Limitations"}],"minor_comments":[{"comment":"The figure labels contain escaped unicode sequences (e.g., '/uni00000030/...') instead of readable text, making the figure effectively unreadable in the current rendering. Please fix.","section":"Figure 3"},{"comment":"Model naming is inconsistent: 'Gemini 3.1 Flash-Lite' in Table 1 and text vs 'Gemini 3.1 Flash-Lite Preview' in Figures 1, 5, and 9. Standardize names.","section":"Throughout"},{"comment":"The claim that word-lookup refusals account for the majority of refusals and are mostly copyright-related is not supported by a per-category refusal breakdown. Add such a breakdown.","section":"Section 5.3"},{"comment":"The paper does not include the full question set or any public subset, nor code for the scoring protocol. Since the benchmark is a central artifact, providing sample questions per category and the scoring script would substantially aid interpretability and reproducibility.","section":"Appendix / Availability"},{"comment":"The human baseline is a single volunteer with no run-to-run variance. State explicitly that this is an illustrative baseline, not a population estimate, and avoid over-interpreting small differences.","section":"Section 4 / Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central direction is worth publishing after a corrected reanalysis. The refusal-handling inconsistency is the main technical obstacle; it is fixable by recomputing all metrics under Section 3.2's stated rule and reporting the sensitivity to exclusion. The private-set issue is a legitimate limitation but not, by itself, grounds for rejection if the authors add transparency measures. I would ask for the corrected tables/figures, per-category refusal counts, and a clear statement of how refusals were actually handled in the reported results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gives you a genuinely usable compact calibration benchmark, and the main empirical claim—that verbalized calibration is partly independent of accuracy—is plausible and consistent with prior work. But the paper contradicts its own refusal-handling protocol in the headline table, so the exact Brier values and rankings are provisional until that is addressed.\n\nWhat's new: 200 private multiple-choice questions across four deliberately chosen epistemic regimes (spatial reasoning, math, word recall, unknowable), 15 frontier models, a human baseline, and Brier score as the proper scoring rule. The category-level results are the most interesting part: unknowable questions expose underconfidence, word lookup triggers copyright refusals, and within-family comparisons (GPT-5 reasoning effort, Claude/Gemini scale) look sensible. The writing is clear and does not oversell.\n\nSoft spots, in proportion. The stress-test note is right. Section 3.2 explicitly says refusals are scored as 25% confidence with a random answer because exclusion inflates apparent calibration. Table 2's header defines n as pooled non-refused responses and reports n<600 for refusal-prone models, so refusals were dropped, not scored. That is exactly the bias the paper says it avoids, and it affects the Brier scores anchoring the accuracy-vs-calibration thesis. A quick back-of-envelope check for GPT-5 Nano (112 refusals at expected Brier 0.1875) moves its Brier from 0.133 to about 0.143. The top two models have n=600, so the broad story probably survives, but \"five of fifteen underperform baseline\" and the exact ordering in Table 2 are not reliable as reported.\n\nSecondary issues: the question set is private with no code or data release, so contamination and label errors cannot be independently checked. The paper acknowledges this, but it caps what a referee can verify. There are also no significance tests for the worse-than-baseline claims; three runs give means and standard deviations but not pairwise comparisons. Fixable, not fatal.\n\nWho this is for: people building or using black-box confidence elicitation, and model-routing/abstention practitioners. It deserves a serious referee, not a desk reject—but only after the authors either rerun with the stated refusal protocol or explain why dropping refusals is defensible in a way Section 3.2 itself rejects.","headline":"ConfidenceBench is a useful compact calibration benchmark whose broad claim is plausible, but its headline numbers contradict the paper's own refusal-handling protocol and are provisional until that's fixed.","tokens_in":12814,"tokens_out":1988,"would_cite":true,"duration_ms":21122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that how confidently an LLM says it knows an answer is a distinct, measurable reliability property: on a 200-question private benchmark, two frontier models come close to human calibration while five score worse than a mode","keywords":["calibration","Brier score","verbalized confidence","large language models","uncertainty quantification","benchmark","proper scoring rule","confidence elicitation"],"falsifier":"Independently audit the 200-question set: have a second party re-verify every label against primary sources and search for near-duplicates in public training corpora. If a substantial number of labels are wrong or leaked into training data, the reported Brier scores and the claimed divergence between accuracy and calibration collapse.","tokens_in":11871,"feed_emoji":"🎯","tokens_out":25211,"duration_ms":225491,"temperature":0.7,"pith_summary":"ConfidenceBench asks whether large language models can report how likely they are to be right, not just whether they happen to be right. It gives 15 frontier models 200 private four-choice questions spanning spatial reasoning, high-precision math, word lookup, and genuinely unknowable facts, and scores each model's stated probability with the Brier score. The central result is that accuracy and calibration come apart: the most accurate model is not the best calibrated, and five models do worse than a calibrated random guesser despite decent accuracy. The paper concludes that verbalized confidence is a practically important axis of LLM reliability that should be evaluated directly, especially when models are used to route, abstain, or request human review.","feed_headline":"Five of 15 AI models are worse than random at judging their accuracy","feed_subtitle":"A 200-question benchmark shows stated confidence and accuracy diverge; two models match human calibration.","key_machinery":"The load-bearing mechanism is the Brier score applied to verbalized confidence: for each question the model is prompted to answer A–D and give a 0–100 probability, then each stated probability p is scored as (p − y)² against correctness y. Because the Brier score is a proper scoring rule, it rewards truthful probability reports and punishes both over- and underconfidence on the same scale; the calibrated-random baseline of 0.1875 for a four-choice question makes the scores interpretable. The private 200-question set, split into four categories that isolate distinct failure modes, is what lets the benchmark attribute miscalibration to mental-simulation gaps, arithmetic slips, recall over-gues","core_discovery":"On the paper's own terms, the discovery is that a compact, private, prompt-only calibration benchmark can separate models by whether their stated confidence tracks the truth, in a way accuracy does not. Across three runs on 200 questions, Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve Brier scores of 0.103, close to a human tester's 0.105 and well below the calibrated-random baseline of 0.1875, while Gemini 3.1 Flash-Lite scores 0.367. The same data show the most accurate model (Gemini 3.1 Pro Preview at 82.7%) is not the best calibrated, and five of fifteen models underperform the calibrated-random baseline despite reasonable accuracy. The paper reads this as evidence that verbalized co","pith_inferences":["Because the 200-question set is private and small, the exact rankings are probably unstable across question samples; a natural extension is a continuously refreshed public calibration set that rotates questions to prevent contamination.","The paper leaves implicit that if verbalized confidence partly reflects instruction-following rather than true uncertainty, calibration could be improved by prompt or training design without changing the underlying model.","The four-category split suggests a cheap, logit-free monitoring recipe: sample a few dozen questions per failure mode in a production setting and compute a Brier score from the model's own stated probabilities.","The manuscript includes an unusual block after the Limitations paragraph (the '/uni000000...' passage) whose content is not cleanly legible in the provided text; it should be treated as in-scope evidence and audited against a clean copy before relying on the paper's final conclusions."],"forward_implications":["If calibration is distinct from accuracy, then deployment decisions such as abstention, escalation, and human review should be based on Brier-style scores rather than accuracy alone.","Five of fifteen models scoring worse than a calibrated random guesser implies their confidence statements actively mislead, so models with reasonable accuracy should not be assumed safe to trust at face value.","Two models matching a human tester's Brier score shows that frontier verbalized confidence can reach a human-comparable calibration level in this multiple-choice format.","Reasoning effort and model scale generally improve calibration within a family, so calibration quality could be reported as a product attribute alongside accuracy.","The 25% 'I don't know' floor on unknowable questions creates underconfidence, a safer error mode than overconfidence for routing and abstention."],"fun_headline_variants":["5 of 15 AI models worse than random at judging accuracy","Best-calibrated AI model isn't the most accurate one","AI confidence on 200 tests: accuracy and calibration diverge","Two AI models match human calibration on a 200-question test","Verbalized AI confidence reveals a distinct failure axis"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 200 private questions are free from training-data contamination and are correctly labeled; if either fails, every headline Brier score and ranking in the paper is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["5 of 15 AI models worse than random at judging accuracy","Best-calibrated AI model isn't the most accurate one","AI confidence on 200 tests: accuracy and calibration diverge","Two AI models match human calibration on a 200-question test","Verbalized AI confidence reveals a distinct failure axis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1086,"prompt_tokens":792,"completion_tokens":294,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":223}},"tokens_in":536,"tokens_out":294,"duration_ms":4038,"temperature":1.0,"reasoning_tokens":223,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T07:40:52.458472+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently audit the 200-question set: have a second party re-verify every label against primary sources and search for near-duplicates in public training corpora. If a substantial number of labels are wrong or leaked into training data, the reported Brier scores and the claimed divergence between accuracy and calibration collapse.","supporting_citations":[],"review_version":1}