Pith. sign in

REVIEW 3 major objections 5 minor 14 references

The paper argues that how confidently an LLM says it knows an answer is a distinct, measurable reliability property: on a 200-question private benchmark, two frontier models come close to human calibration while five score worse than a mode

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:40 UTC pith:YF2QJH3G

load-bearing objection ConfidenceBench is a useful compact calibration benchmark whose broad claim is plausible, but its headline numbers contradict the paper's own refusal-handling protocol and are provisional until that's fixed. the 3 major comments →

arxiv 2607.20526 v1 pith:YF2QJH3G submitted 2026-07-10 cs.AI cs.LGstat.ML

ConfidenceBench: Evaluating Confidence Calibration in Large Language Models

Matthew ffrench-Constant , Daniel Yang , Xinmeng Huang , Sanyam Kapoor This is my paper
classification cs.AI cs.LGstat.ML
keywords calibrationBrier scoreverbalized confidencelarge language modelsuncertainty quantificationbenchmarkproper scoring ruleconfidence elicitation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ConfidenceBench asks whether large language models can report how likely they are to be right, not just whether they happen to be right. It gives 15 frontier models 200 private four-choice questions spanning spatial reasoning, high-precision math, word lookup, and genuinely unknowable facts, and scores each model's stated probability with the Brier score. The central result is that accuracy and calibration come apart: the most accurate model is not the best calibrated, and five models do worse than a calibrated random guesser despite decent accuracy. The paper concludes that verbalized confidence is a practically important axis of LLM reliability that should be evaluated directly, especially when models are used to route, abstain, or request human review.

Core claim

On the paper's own terms, the discovery is that a compact, private, prompt-only calibration benchmark can separate models by whether their stated confidence tracks the truth, in a way accuracy does not. Across three runs on 200 questions, Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve Brier scores of 0.103, close to a human tester's 0.105 and well below the calibrated-random baseline of 0.1875, while Gemini 3.1 Flash-Lite scores 0.367. The same data show the most accurate model (Gemini 3.1 Pro Preview at 82.7%) is not the best calibrated, and five of fifteen models underperform the calibrated-random baseline despite reasonable accuracy. The paper reads this as evidence that verbalized co

What carries the argument

The load-bearing mechanism is the Brier score applied to verbalized confidence: for each question the model is prompted to answer A–D and give a 0–100 probability, then each stated probability p is scored as (p − y)² against correctness y. Because the Brier score is a proper scoring rule, it rewards truthful probability reports and punishes both over- and underconfidence on the same scale; the calibrated-random baseline of 0.1875 for a four-choice question makes the scores interpretable. The private 200-question set, split into four categories that isolate distinct failure modes, is what lets the benchmark attribute miscalibration to mental-simulation gaps, arithmetic slips, recall over-gues

Load-bearing premise

The load-bearing premise is that the 200 private questions are free from training-data contamination and are correctly labeled; if either fails, every headline Brier score and ranking in the paper is unsupported.

What would settle it

Independently audit the 200-question set: have a second party re-verify every label against primary sources and search for near-duplicates in public training corpora. If a substantial number of labels are wrong or leaked into training data, the reported Brier scores and the claimed divergence between accuracy and calibration collapse.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If calibration is distinct from accuracy, then deployment decisions such as abstention, escalation, and human review should be based on Brier-style scores rather than accuracy alone.
  • Five of fifteen models scoring worse than a calibrated random guesser implies their confidence statements actively mislead, so models with reasonable accuracy should not be assumed safe to trust at face value.
  • Two models matching a human tester's Brier score shows that frontier verbalized confidence can reach a human-comparable calibration level in this multiple-choice format.
  • Reasoning effort and model scale generally improve calibration within a family, so calibration quality could be reported as a product attribute alongside accuracy.
  • The 25% 'I don't know' floor on unknowable questions creates underconfidence, a safer error mode than overconfidence for routing and abstention.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the 200-question set is private and small, the exact rankings are probably unstable across question samples; a natural extension is a continuously refreshed public calibration set that rotates questions to prevent contamination.
  • The paper leaves implicit that if verbalized confidence partly reflects instruction-following rather than true uncertainty, calibration could be improved by prompt or training design without changing the underlying model.
  • The four-category split suggests a cheap, logit-free monitoring recipe: sample a few dozen questions per failure mode in a production setting and compute a Brier score from the model's own stated probabilities.
  • The manuscript includes an unusual block after the Limitations paragraph (the '/uni000000...' passage) whose content is not cleanly legible in the provided text; it should be treated as in-scope evidence and audited against a clean copy before relying on the paper's final conclusions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ConfidenceBench introduces a private 200-question multiple-choice benchmark across four categories (spatial reasoning, high-precision mathematics, word lookup, unknowable questions) and uses verbalized confidence from 15 frontier LLMs over three runs. Calibration is measured with the Brier score as the primary metric, supplemented by ECE and a signed calibration gap. The paper reports that Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve the lowest Brier scores (0.103), that the most accurate model is not the best-calibrated, and that five of fifteen models underperform a calibrated-random baseline (0.1875). A human tester with internet access is used as a baseline. The central claim is that verbalized confidence calibration is a distinct, practically important axis of LLM reliability, complementary to accuracy.

Significance. If the reported measurements are correct, the paper provides a useful, compact stress test of verbalized confidence and supports the view that accuracy alone does not capture reliability. Strengths include manual authorship of the question set to reduce contamination, use of a proper scoring rule, three-run repeated evaluation, category-level diagnostics, and an explicit protocol for handling refusals. The main qualitative conclusion is plausible and may survive a corrected reanalysis. However, the reported quantitative results are compromised by an internal inconsistency in refusal handling, and the human baseline is not directly comparable to the model condition. With a careful reanalysis and fuller disclosure, this could be a valuable benchmark contribution.

major comments (3)
  1. [Section 3.2 vs. Table 2 and Figure 9] The stated protocol is that refusals and unparseable responses are scored as 25% confidence with a randomly selected answer, because exclusion would inflate apparent calibration. Table 2 defines n as 'pooled non-refused responses' and reports n<600 for every refusal-prone model (e.g., GPT-5 Nano n=488, GPT-5 Mini n=527; Figure 9 shows GPT-5 Nano refusal rate 18.7%, i.e., 112/600). Thus refusals were excluded from the reported Brier, accuracy, and ECE, not scored as specified. Recomputing GPT-5 Nano's Brier with the stated rule (112 refusals at expected Brier 0.1875) gives about 0.143 instead of 0.133, and its accuracy drops from 70.5% conditional to about 62% overall; GPT-5 Nano and GPT-5 Low would swap order. This internal inconsistency directly affects the headline rankings, the 'five of fifteen underperform baseline' enumeration, and Figures 1, 7, and 8. The results must be recomputed
  2. [Section 4 / Section 5.1 / Section 6] The human baseline is not directly comparable to the models. The human tester completed the questions with internet access and one minute per question, whereas the models were API queries with no internet access. The claim in Section 6 that 'two models achieve lower Brier scores than the human tester' is therefore not an apples-to-apples comparison; internet access likely helps on Word Lookup and Unknowable items. To support the human-comparison claim, the human should be evaluated under the same information constraints, or the claim should be removed or clearly qualified.
  3. [Section 3.1 and Limitations] The benchmark's validity rests on the correctness and non-contamination of 200 private questions, but the set is withheld and no independent audit, contamination check, or inter-rater verification is reported. The Limitations paragraph acknowledges the audit limitation, but this is load-bearing: a few mislabeled or leaked items could change rankings. Please provide at least a public sample with detailed verification, an independent audit of labels, a contamination analysis, or item-level statistics that would reveal anomalies. At minimum, state the exact protocol used to confirm every answer and report per-category model performance to aid external assessment.
minor comments (5)
  1. [Figure 3] The figure labels contain escaped unicode sequences (e.g., '/uni00000030/...') instead of readable text, making the figure effectively unreadable in the current rendering. Please fix.
  2. [Throughout] Model naming is inconsistent: 'Gemini 3.1 Flash-Lite' in Table 1 and text vs 'Gemini 3.1 Flash-Lite Preview' in Figures 1, 5, and 9. Standardize names.
  3. [Section 5.3] The claim that word-lookup refusals account for the majority of refusals and are mostly copyright-related is not supported by a per-category refusal breakdown. Add such a breakdown.
  4. [Appendix / Availability] The paper does not include the full question set or any public subset, nor code for the scoring protocol. Since the benchmark is a central artifact, providing sample questions per category and the scoring script would substantially aid interpretability and reproducibility.
  5. [Section 4 / Table 2] The human baseline is a single volunteer with no run-to-run variance. State explicitly that this is an illustrative baseline, not a population estimate, and avoid over-interpreting small differences.

Circularity Check

0 steps flagged

No circular derivation: the headline Brier results are direct empirical measurements (Eq. 1), and the two co-author self-citations are contextual, not load-bearing.

full rationale

ConfidenceBench is an empirical measurement paper, not a derivation chain. Every headline number — the 0.103 Brier scores, the 0.1875 calibrated-random baseline, 'five of fifteen models underperform the baseline' — is a direct function of elicited probabilities and correctness indicators via Eq. (1) (B = 1/n Σ(pi − yi)^2) with no fitted parameter renamed as a prediction and no definition that presupposes the conclusion. The 0.1875 baseline is the textbook expectation for p=0.25 random guessing on four options (0.25·0.5625 + 0.75·0.0625), computable independently of all model outputs, so the 'better/worse than calibrated-random' comparisons are not artifacts of construction. The two co-author self-citations (Kapoor et al. 2024 on RLHF distorting probabilities; Yang et al. 2024 on prompt design affecting verbalized confidence) appear only as background in Section 2; neither is invoked to justify the benchmark design, the Brier metric, the private question set, or the rankings, so they are not load-bearing, and per the rubric this yields a low score. One in-scope consistency issue is flagged explicitly though it is not circularity: Section 3.2 states that refusals 'are scored as 25% confidence with a randomly selected answer rather than being excluded' (because exclusion 'would inflate apparent calibration'), yet Table 2 defines n as 'pooled non-refused responses across 3 runs' and reports n<600 for every model with refusals (e.g., GPT-5 Nano n=488 at an 18.7% refusal rate per Figure 9). The reported Brier scores therefore dropped refused responses contrary to the stated rule, which can bias cross-model comparisons and exact rankings, but this does not make any claim reduce to its own inputs. The central claim — that calibration is distinct from accuracy — is supported directly by the measurements in Figure 1 and Table 2 and would survive independent reimplementation.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claims are empirical measurements rather than derivations, so there are no fitted model parameters. The load-bearing assumptions are the integrity of the private question set and the comparability of the elicitation protocol. The refusal-scoring rule is a hand-chosen design constant that is, in the table, not applied as described.

free parameters (2)
  • Refusal scoring floor (p = 0.25) = 0.25
    Chosen as the calibrated-random baseline for 4-choice questions; §3.2 states refusals are scored at 25% with a random answer, but Table 2 actually excludes refusals from reported metrics.
  • ECE bin count B = 10 = 10
    Equal-width bin count chosen for ECE; authors note ECE is sensitive to this choice (§3.3). Secondary metric only.
axioms (3)
  • domain assumption The 200 private questions are free from training-data contamination and have correct labels.
    If any questions were in training data or labels are wrong, all Brier/accuracy measurements are invalid. The set is not released, so this cannot be audited (§3.1).
  • domain assumption Verbalized probability from a JSON prompt reflects the model's epistemic uncertainty rather than instruction-following behavior.
    The paper's own Limitations section says reported probabilities may partly reflect instruction-following or prompt framing, not underlying uncertainty.
  • domain assumption A single human volunteer with internet access is a directly comparable baseline to API-only models.
    The human had internet and was evaluated once with a time limit, whereas models were queried via API without internet; N=1 gives no variance estimate (§4).

pith-pipeline@v1.3.0-alltime-deepseek · 11747 in / 15560 out tokens · 143620 ms · 2026-08-02T07:40:52.458472+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly. In these settings, accuracy alone is insufficient: models must also know when they are likely to be wrong. We present ConfidenceBench, a calibration benchmark that evaluates verbalized confidence estimates in 15 frontier LLMs using the Brier score, a proper scoring rule that incentivises truthful probability reporting. Confidence is elicited via prompting, requiring no access to model logits and making the framework applicable to both closed-source and open-source systems. The benchmark comprises 200 private multiple-choice questions across four categories: spatial reasoning, high-precision mathematics, word lookup, and unknowable questions. Across three independent runs, Claude Opus 4.6 and Gemini 3.1 Pro Preview achieve the lowest Brier scores, both reported as 0.103. Both substantially outperform the calibrated-random baseline of 0.1875, while Gemini 3.1 Flash-Lite scores 0.367, indicating severe miscalibration. Accuracy and calibration diverge substantially across model families: the most accurate model is not the best-calibrated, and several models perform worse than the calibrated-random Brier baseline despite reasonable accuracy. These results show that verbalized confidence calibration is a distinct and practically important axis of LLM reliability, complementary to standard accuracy-based evaluation.

Figures

Figures reproduced from arXiv: 2607.20526 by Daniel Yang, Matthew ffrench-Constant, Sanyam Kapoor, Xinmeng Huang.

Figure 1
Figure 1. Figure 1: Brier score vs. accuracy (3-run means). Error bars show standard deviation in both [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy by question type for selected models and the human baseline (3-run means). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Calibration gap (mean stated confidence minus actual accuracy) by question type for [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Stated probability distributions for Claude Opus 4.6 (left; Brier [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Kernel-smoothed reliability diagrams [Błasiok and Nakkiran, 2023]. Dashed diagonal = [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy by model (3-run means, error bars show std). Gemini 3.1 Pro Preview leads on [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mean Brier score by model (3 runs, 200 questions per model). Error bars show standard [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Mean Brier score by question type for selected models (3 runs, means). Dashed line: [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Total refusal rate per model (3-run means). Word-lookup refusals dominate across OpenAI [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 7 linked inside Pith

  1. [1]

    Błasiok and P

    J. Błasiok and P. Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing.arXiv preprint arXiv:2309.12236,

    Pith/arXiv arXiv

  2. [3]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Pith/arXiv arXiv 2009

  3. [10]

    Xiong, Z

    M. Xiong, Z. Hu, X. Lu, Y . Li, J. Fu, J. He, and B. Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.arXiv preprint arXiv:2306.13063,

    Pith/arXiv arXiv

  4. [11]

    Yang, Y .-H

    D. Yang, Y .-H. H. Tsai, and M. Yamada. On verbalized confidence scores for llms.arXiv preprint arXiv:2412.14737,

    Pith/arXiv arXiv

  5. [12]

    Y . Yang, H. Yoo, and H. Lee. Maqa: Evaluating uncertainty quantification in llms regarding data uncertainty. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 5846–5863,

    2025

  6. [13]

    Z. Yin, Q. Sun, Q. Guo, J. Wu, X. Qiu, and X.-J. Huang. Do large language models know what they don’t know? InFindings of the association for Computational Linguistics: ACL 2023, pages 8653–8665,

    2023

  7. [14]

    n = pooled non-refused responses across 3 runs

    10 A Full Results and Run-to-Run Stability Table 2: Full results sorted by Brier score. n = pooled non-refused responses across 3 runs. Mean and standard deviation computed across 3 independent runs; human tester evaluated once (no run-to-run variance). Accuracy in [0,1] . Best Brier (Claude Opus 4.6, first by unrounded score), best Accuracy (Gemini 3.1 P...

    2000

  8. [2017]

    Fadeeva, R

    E. Fadeeva, R. Vashurin, A. Tsvigun, A. Vazhentsev, S. Petrakov, K. Fedyanin, D. Vasilev, E. Gon- charova, A. Panchenko, M. Panov, et al. Lm-polygraph: Uncertainty estimation for language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 446–461,

    2023

  9. [2019]

    Lafage, O

    A. Lafage, O. Laurent, F. Gabetni, and G. Franchi. Torch-uncertainty: A deep learning framework for uncertainty quantification.arXiv preprint arXiv:2511.10282,

    arXiv

  10. [2020]

    Kadavath, T

    S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,

    Pith/arXiv arXiv

  11. [2023]

    X. Wang, Z. Zhang, G. Chen, Q. Li, B. Luo, Z. Han, H. Wang, Z. Li, H. Gao, and M. Hu. Ubench: Benchmarking uncertainty in large language models with multiple choice questions. InFindings of the Association for Computational Linguistics: ACL 2025, pages 8076–8107,

    2025

  12. [2024]

    Kirichenko, M

    P. Kirichenko, M. Ibrahim, K. Chaudhuri, and S. J. Bell. Abstentionbench: Reasoning llms fail on unanswerable questions.arXiv preprint arXiv:2506.09038,

    Pith/arXiv arXiv

  13. [2025]

    9 S. Lin, J. Hilton, and O. Evans. Teaching models to express their uncertainty in words.arXiv preprint arXiv:2205.14334,

    Pith/arXiv arXiv

  14. [2026]

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442,

    2023