Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage prompting method that asks LLMs for confidence before answers reduces overconfidence and makes confidence track task difficulty more closely, especially for GPT-4o.

desk verdict AFCE cuts overconfidence on hard QA, but the 'human-like sensitivity' claim rests on a confounded cross-task comparison and only shows up clearly in GPT-4o. read the letter →

arxiv 2506.00582 v2 pith:NNSANYQS submitted 2025-05-31 cs.AI

Chenjun Xu , Bingbing Wen , Bin Han , Robert Wolfe , Lucy Lu Wang , Bill Howe This is my paper
classification cs.AI
keywords confidencecalibrationLLMoverconfidenceanswer-freeestimationverbalizedpersonabiastaskdifficultysensitivityexpectederrorrole-playingLLMs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models, like humans, tend to be overconfident on hard questions and underconfident on easy ones, but the paper argues that the resemblance stops there: model confidence is far less sensitive to task difficulty, and it shifts with the persona the model is asked to adopt even when accuracy barely changes. The central proposal is Answer-Free Confidence Estimation (AFCE), which asks the model first to estimate how many of ten questions it can answer correctly and only later asks for the answers. On expert-level science questions from MMLU and GPQA, AFCE lowers Expected Calibration Error across three models and, for GPT-4o, produces a steeper calibration slope that the authors read as closer to human calibration patterns. The stakes are practical: if verbalized confidence can be detached from the act of answering, then calibration can be improved without model-internal access, and persona-driven confidence biases become a measurable risk for role-playing applications.

What carries the argument

The load-bearing device is Answer-Free Confidence Estimation (AFCE), a two-stage prompting scheme. In the confidence stage the model reads ten questions and estimates how many it can answer correctly, choosing an integer from 0 to 10, without producing answers; in the accuracy stage it answers the same questions by option letter only. The paper hypothesizes that generating factual answers is an epistemically intensive process that dominates reasoning and defaults to a high baseline confidence, so removing answer generation lets the model attend to confidence and task difficulty itself. The evaluation machinery is Expected Calibration Error (ECE) with ten bins, used to compare AFCE against vanilla verbalized confidence, top-k prompting, quiz-like prompting, sampling-based confidence, and first-token probability.

What would settle it

Have humans and the same model answer an identically matched set of items, compute the confidence-accuracy regression slope under vanilla prompting and AFCE, and compare slopes; if AFCE's slope is not steeper, or if the gap to human slopes does not close on the matched set, the 'more human-like sensitivity' claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that answer generation and confidence estimation are mediated by different mechanisms in LLMs, and that fusing them together is what produces overconfidence. AFCE separates the two: one prompt asks the model to read a set of questions and estimate how many (from 0 to 10) it can answer correctly; a separate prompt asks only for the option letters. Across LLaMA-3-70B, Claude-3-Sonnet, and GPT-4o on high-school, college, and expert-level physics, chemistry, and biology questions, this separation lowers Expected Calibration Error on the hardest tasks and, for GPT-4o, yields a regression slope closer to the ideal calibration line and to human slopes reported in the psychology literature. The paper also claims that models overplace confidence toward expert personas, underplace it toward layman personas, and express stereotypically lower confidence for demographic groups such as female, Asian, or older personas, while their accuracy stays roughly constant.

Load-bearing premise

The claim that AFCE is more human-like assumes that human calibration slopes from a different set of questions can be fairly compared with model slopes on MMLU and GPQA, even though the original human questions were not available.

Editorial extensions

If this is right

  • Verbalized confidence methods should stop assuming human-like behavior; if answer generation and confidence estimation are decoupled, calibration should be treated as a separate stage rather than bundled into one prompt.
  • AFCE gives black-box API models a prompt-only route to better calibration on hard tasks: on GPT-4o's expert-level questions average ECE drops substantially relative to vanilla, quiz-like, and sampling baselines.
  • Persona-based simulation studies inherit the models' stereotyped confidence biases: an 'expert' persona inflates confidence and a 'layman' persona deflates it with little change in accuracy, so downstream social-science simulations should separate role-played confidence from role-played performance.
  • The method extends beyond multiple-choice questions: on open-ended NQ-open and SimpleQA sets, AFCE also lowers ECE relative to baselines, and it is robust to question order and group size.
  • Models like LLaMA-3-70B and Claude-3 show a relatively flat confidence curve across difficulty, suggesting they fall back on a standard confidence level; this flatness would be a useful diagnostic for when verbalized confidence cannot be trusted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if decoupling confidence from execution is the mechanism, the same separation should reduce overconfidence in agentic or tool-use settings where the model both plans and acts; this is testable by comparing joint and split confidence elicitation in multi-step tasks.
  • Beyond the paper: the 'more human-like' slope claim depends on the comparability of human calibration data collected on different questions; a direct matched-question study with the same item set for humans and models would either strengthen or dissolve that comparison.
  • Beyond the paper: the demographic confidence biases likely mirror stereotypes in training text rather than real group differences; if so, demographic-aware calibration or counterfactual prompting could be evaluated by checking whether confidence gaps persist when the model is told the persona has identical knowledge.
  • Beyond the paper: the paper's ECE improvement on hard tasks might come partly from AFCE lowering confidence overall rather than from sharper discrimination between easy and hard items; inspecting item-level regression slopes rather than aggregate ECE would distinguish these.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies confidence calibration in three LLMs (Llama-3-70B, Claude-3-Sonnet, GPT-4o) on MMLU and GPQA, comparing observed confidence–accuracy patterns with human over/underconfidence findings from Moore and Healy (2008). The authors propose Answer-Free Confidence Estimation (AFCE), a two-stage prompting method that elicits confidence scores separately from answers, and report that AFCE reduces Expected Calibration Error (ECE) on expert-level tasks, especially for GPT-4o, and yields a steeper calibration slope that they interpret as more human-like sensitivity to task difficulty. The paper also reports that persona prompts induce overplacement for expert personas and underplacement for layman personas, and that demographic personas produce stereotypical confidence shifts without corresponding accuracy changes. The manuscript includes prompt templates, dataset statistics, and a code repository.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: AFCE is a simple, black-box-compatible alternative to verbalized confidence that appears to reduce overconfidence on difficult multiple-choice QA, and the persona/demographic findings are relevant to the growing use of LLMs in computational social science and role-playing applications. The paper is generally careful to provide prompt templates, dataset splits, and appendix results across seven models, which aids reproducibility. However, the central 'human-like sensitivity' claim rests on an uncontrolled cross-dataset comparison, and the 'significantly reduces overconfidence' claim is not backed by any statistical inference; the observed effects are plausible but currently supported only by point estimates.

major comments (4)
  1. [§4.1, §4.2; Table 1] The headline claim that AFCE 'delivers more human-like sensitivity to task difficulty' is not established by the evidence presented. The human regression slopes in Figure 3 come from Moore and Healy (2008), whose items and difficulty distributions differ from MMLU and GPQA; the paper itself notes in the §3 footnote that the original questions were not available. A regression slope is not invariant to the range and distribution of the predictor, so differences between LLM slopes and human slopes could reflect task mismatch rather than a genuine difference in confidence behavior. The claim is further weakened because the steeper-slope effect is reported only for GPT-4o; for Llama-3-70B and Claude-3, Section 4.2 and Figure 3 show relatively flat confidence curves even with AFCE. To support the human-like claim, the authors should either collect human calibration data on the same item sets, reanalyze Moore–Healy data on a matched difficulty range, or explicitly reframe the conclusion as improved sensitivity to task difficulty without claiming human equivalence.
  2. [§4.2, Appendix Table 5] The word 'significantly' in the abstract and in Section 4.2 is not supported by any statistical test, confidence interval, or bootstrap. ECE is computed from a small number of 10-question blocks per condition: for example, the 170-question high-school physics condition yields only 17 blocks, and 10-bin ECE over such a small sample will have large variance. The observed differences in Table 1, such as GPT-4o expert-physics ECE of 16.1 (AFCE) versus 21.1 (top-k), may be real, but the paper provides no measure of uncertainty. The authors should add error bars or significance tests (e.g., bootstrap over question blocks) before claiming that AFCE 'significantly reduces overconfidence'.
  3. [§7, Table 10] The claim that AFCE 'consistently outperforms' all baselines is contradicted by the appendix results on additional models. For example, in Table 5, Mixtral-8x7B high-school physics shows AFCE ECE 48.7 versus Top-K 47.7 and Vanilla 49.5, and LLaMA-3-8B high-school chemistry shows AFCE ECE 39.3 versus Vanilla 31.7. The main-text Table 1 also shows cases where AFCE is not the best on easier subtasks (e.g., GPT-4o high-school biology: AFCE ECE 6.1 versus Vanilla 2.9). The authors should restrict the superiority claim to expert-level tasks or provide a quantitative summary (e.g., mean rank with significance) rather than 'consistent' across all conditions.
  4. [§10] The ablation section states that 'no significant differences' were observed under variations in question order and group size, but no significance test is reported. Table 10 shows point estimates such as GPQA-biology ECE changing from 10.0 with 10 questions to 7.1 with 5 questions, and high-school physics ECE changing from 14.7 to 22.0 under 5 questions. Without confidence intervals or paired tests, the robustness claim is unsupported. The authors should either provide the statistical evidence or soften the wording to 'no large differences were observed in these point estimates'.
minor comments (5)
  1. [References] References Hase et al. 2024a and 2024b are the same paper and should be merged; this duplication is a presentation issue that should be fixed in revision.
  2. [Table 1] The header structure of Table 1 is difficult to parse: the 'AvE' column appears to be placed under 'High School Physics' in the text rendering, and the caption describes it as 'Average ECE' but the row values are repeated. The table should be reformatted so that each difficulty level has clearly labeled Acc/ECE subcolumns and the average ECE column is visually separated.
  3. [Figure 4 caption] The overplacement formula in the Figure 4 caption uses terms such as 'Confidence Estimate Others' and 'Accuracy Others' that are not formally defined in the text; adding a short definition of these quantities in Section 5.1 would improve reproducibility.
  4. [§3] The footnote stating that the original Moore and Healy (2008) questions are unavailable is a crucial caveat for the central comparison; it should be moved into the main text and discussed explicitly in the limitations section, which currently only mentions ECE limitations and prompt dependence.
  5. [§6.2] The statement that GPT-4o 'demonstrates balanced confidence estimations across races, ages, and genders' is stronger than Figure 5 shows; GPT-4o still shows nonzero D_Demographic values for some groups. A more cautious wording, such as 'smaller disparities than the other models,' would match the figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AFCE is evaluated on held-out accuracy and ECE against external baselines; the human-calibration comparison is an external benchmark, and self-citations are background only.

full rationale

The paper's central claims are empirical measurements rather than derivations. AFCE is defined operationally as two-stage prompting (confidence without answer, then answer-only), and its improvement is measured by ECE computed from the model's stated confidence and independently measured accuracy. No parameter is fitted to the reported outcomes; no equation defines AFCE in terms of ECE or in terms of the Moore–Healy regression slope. The 'more human-like sensitivity' statement is an interpretation of a slope overlay with human data from Moore and Healy (2008); even if that comparison is confounded by different question sets (a limitation the paper notes in the Section 3 footnote), confounding is a validity concern, not circularity, because the human slope is external input rather than an output of this paper's construction. The paper's self-citations (Wen et al. 2024a, 2024b, 2025) appear only in background and related-work passages—e.g., supporting that LLM simulations can replicate social-science experiments or that overconfidence has consequences—and none is load-bearing for the AFCE result or the calibration measurements. No uniqueness theorem, ansatz, or fitted quantity is imported from the authors' prior work. Per the rubric, this is a no-significant-circularity finding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on the validity of comparing self-reported confidence to accuracy across different tasks, on the reliability of ECE computed over small blocks, and on the assumption that persona prompts isolate the persona's ability rather than the model's stereotypes. These are domain assumptions, not unfalsifiable entities; no new particles or mediators are introduced.

free parameters (3)
  • Number of questions per prompt = 10
    Chosen to match Moore and Healy (2008)'s design of 10 questions per block; affects the granularity and noise of accuracy and ECE estimates.
  • Number of ECE bins = 10
    Standard choice but ECE is sensitive to bin definitions (the paper acknowledges ECE limitations); with roughly 10-34 blocks per condition, 10 bins leaves very few observations per bin.
  • Sampling temperature and top-p = temperature=0, top-p=1
    Deterministic setting chosen to reduce variability, but it does not eliminate prompt-order or API nondeterminism.
assumptions (4)
  • domain assumption Moore and Healy (2008)'s human calibration data is a valid benchmark for comparing LLM confidence patterns.
    Used in Figure 3 and the 'human-like sensitivity' claim, but collected on different question banks (see §3 footnote).
  • domain assumption Verbalized confidence on a 0-10 scale is a meaningful measure of a model's expected accuracy.
    AFCE and all baselines rely on self-reported numbers; the paper does not validate that these numbers correlate with internal probabilities beyond the ECE results.
  • domain assumption Persona prompts cause the model to simulate the stated persona's knowledge rather than merely applying stereotypes from pretraining.
    The overplacement and demographic experiments assume the prompt 'You are an expert' or 'A is an expert' elicits the persona's ability estimate; the observed stereotypical shifts may instead reflect the model's priors about social groups.
  • standard math ECE is an adequate metric for calibration comparison.
    Used throughout; the paper itself notes ECE's sensitivity to binning in §10.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs." pith.science (2026). https://pith.science/paper/NNSANYQS

@misc{pith2026250600582,
  author       = {Pith},
  title        = {Pith review of: Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNSANYQS}},
  note         = {Machine review of arXiv:2506.00582}
}
read the original abstract

Psychology research has shown that humans are poor at estimating their performance on tasks, tending towards underconfidence on easy tasks and overconfidence on difficult tasks. We examine three LLMs, Llama-3-70B-instruct, Claude-3-Sonnet, and GPT-4o, on a range of QA tasks of varying difficulty, and show that models exhibit subtle differences from human patterns of overconfidence: less sensitive to task difficulty, and when prompted to answer based on different personas -- e.g., expert vs layman, or different race, gender, and ages -- the models will respond with stereotypically biased confidence estimations even though their underlying answer accuracy remains the same. Based on these observations, we propose Answer-Free Confidence Estimation (AFCE) to improve confidence calibration and LLM interpretability in these settings. AFCE is a self-assessment method that employs two stages of prompting, first eliciting only confidence scores on questions, then asking separately for the answer. Experiments on the MMLU and GPQA datasets spanning subjects and difficulty show that this separation of tasks significantly reduces overconfidence and delivers more human-like sensitivity to task difficulty.

Figures

Figures reproduced from arXiv: 2506.00582 by the authors.

Figure 1
Figure 1. Comparative Analysis of Confidence Calibra [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. AFCE gathers confidence levels for a set of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. AFCE reduces overconfidence across models and improves sensitivity to task difficulty for GPT-4o. For [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overplacement Score quantifies the degree of overplacement, calculated as: (ConfidenceEstimate Others − AccuracyOthers)−(ConfidenceSelf-Estimate −AccuracySelf). All models exhibit overplacement towards Expert persona and underplacement towards Layman persona. GPT-4o an…
Figure 5
Figure 5. Figure 5: ∆Demographic measures the gap between confidence and accuracy (Confidence-Accuracy) across various demographic groups (gender, race, or age). GPT-4o demonstrates more balanced confidence estimations while LLaMA-3-70B and Claude-3 show consistent underconfidence across …
Figure 6
Figure 6. Figure 6: Confidence estimation using our method on tasks with various levels of difficulty. GPT-4o appears [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The difference between confidence and accuracy (Confidence - Accuracy) for three models across different [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: ∆Demographic: The difference between confidence and accuracy (Confidence − Accuracy) for four models across gender, race, and age groups. Gemma2-9B and Mixtral-8x7B exhibit higher ∆Demographic scores across all demographic categories, indicating greater bias. In contra…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Confidence Calibration in Large Language Models

    cs.AI 2026-04 conditional novelty 6.0 of 10

    LLMs show average overconfidence moderated by a strong hard-easy effect, quantified across tasks and via the new LifeEval actuarial-probability benchmark.

  2. Can Large Language Models Integrate Spatial Data? Empirical Insights into Reasoning Strengths and Computational Weaknesses

    cs.AI 2025-08 conditional novelty 6.0 of 10

    LLMs only become competitive at spatial data integration when given pre-computed geometric features; a review-and-refine prompt then exceeds hand-tuned heuristics.

Reference graph

Works this paper leans on

10 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [4]

    Do LLMs exhibit human-like response biases? A case study in survey design

    Role play with large language models.Nature, 623(7987):493–498. Carter Teplica, Yixin Liu, Arman Cohan, and Tim GJ Rudner. 2025. Sciurus: Shared circuits for inter- pretable uncertainty representations in language mod- els. InMINT: Foundation Model Interventions. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelse...

    work page Pith review arXiv 2025

  2. [6]

    ArXiv, abs/2302.13439

    Navigating the grey area: Expressions of overconfidence and uncertainty in language models. ArXiv, abs/2302.13439. Caleb Ziems, William B. Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large language models transform computational so- cial science?Computational Linguistics, 50:237– 291. A Appendix / supplemental material A.1 Datase...

    arXiv 2023

  3. [16]

    Do not provide any explanation

    <Only Option Letter > ”’ //Prompt for confidence calculation prompt = f"""Read the question, and estimate how many questions you can answer correctly (choose a number from 0-10). Do not provide any explanation. Questions: Q1: <> Q2: <> Q3: <> Q4: <> Q5: <> Q6: <> Q7: <> Q8: <> Q9: <> Q10: <> Estimate how many questions you can answer correctly: <choose a ...

    work page

  4. [26]

    Read the ques- tion, and after considering A’s ability, estimate how many questions A can answer correctly (choose a num- ber from 0-10)

    <Only Option Letter > ”’ //Prompt for confidence calculation prompt = “‘A is an {role} in {subject}. Read the ques- tion, and after considering A’s ability, estimate how many questions A can answer correctly (choose a num- ber from 0-10). Do not provide any explanation. Questions: Q1: <> Q2: <> Q3: <> Q4: <> Q5: <> Q6: <> Q7: <> Q8: <> Q9: <> Q10: <> Esti...

    work page

  5. [35]

    <Only Option Letter >

    work page

  6. [36]

    Read the question, estimate how many questions you can answer correctly (choose a number from 0-10)

    <Only Option Letter > ”’ //Prompt for confidence calculation prompt = “‘You are a {race} {gender} in the {age} age group. Read the question, estimate how many questions you can answer correctly (choose a number from 0-10). Do not provide any explanation. Questions: Q1: <> Q2: <> Q3: <> Q4: <> Q5: <> Q6: <> Q7: <> Q8: <> Q9: <> Q10: <> Estimate how many qu...

    work page

  7. [2021]

    i’m not sure, but

    Measuring massive multitask language under- standing. InInternational Conference on Learning Representations. Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. 2006. A fast learning algorithm for deep belief nets.Neural Computation, 18:1527–1554. Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. 2023. Personallm: Investig...

    arXiv 2006

  8. [2023]

    Yoshua Bengio and Yann LeCun

    Out of one, many: Using language mod- els to simulate human samples.Political Analysis, 31(3):337–351. Yoshua Bengio and Yann LeCun. 2007. Scaling learn- ing algorithms towards AI. InLarge Scale Kernel Machines. MIT Press. Eta S. Berner and Mark L Graber. 2008. Overconfi- dence as a cause of diagnostic error in medicine.The American journal of medicine, 1...

    arXiv 2007

Show all 10 references
  1. [2024]

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto

    Uncertainty is fragile: Manipulating uncer- tainty in large language models.arXiv preprint arXiv:2407.11282. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto

    arXiv

  2. [2025]

    fifty shades of bias

    The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks.arXiv preprint arXiv:2502.08235. Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2024. Applicability of large language mod- els and generative models for legal case judgement summarizatio...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.