Pith. sign in

REVIEW 12 references

Gauging Overprecision in LLMs: An Empirical Study

T0 review · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs are overprecise in numerical interval estimation: hit rates fall far below imposed confidence levels, and interval length does not track the requested confidence.

arxiv 2504.12098 v2 pith:YZIWWRIW submitted 2025-04-16 cs.CL

Adil Bahaj , Hamed Rahimi , Mohamed Chetouani , Mounir Ghogho This is my paper · ORCID
classification cs.CL
keywords confidencellmsanswersgenerationoverprecisionphasepromptgenerate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors ask two language models, GPT-3.5-turbo and GPT-4o-mini, to answer numerical questions from finance, medical, and general-knowledge datasets by giving a lower bound and an upper bound. The prompt tells the model to be, for example, 90% sure that the true answer lies inside the interval. The key measurement is the hit rate: the fraction of questions where the true answer actually falls inside the proposed interval. If the model understood the instruction perfectly, a 90% interval should contain the answer about 90% of the time.

The results show large gaps. At a stated 90% confidence, hit rates ranged from about 6% on financial questions to about 80% on general knowledge, far below the target in most settings. Raising the requested confidence from 60% to 95% barely changed the interval widths, and the correlation between interval length and confidence was close to zero. The models did produce wider intervals for harder tasks, so they seem to sense difficulty, but they do not convert the stated confidence into a matching interval.

The paper also tests two refinement ideas: averaging several sampled intervals, and letting the model critique its own answers. Averaging sometimes helped, while self-refinement usually did not and often made intervals worse. The authors conclude that verbalized confidence may be less reliable than imposed-confidence prompts, but the study leaves open whether the underlying problem is missing internal confidence knowledge or poor instruction following.

Extended reading notes

Core claim

The load-bearing claim is that LLMs are overprecise on numerical interval estimation: at each imposed confidence level, the actual hit rate of the generated intervals is far below that level, and interval length does not correlate with the imposed confidence (Section 5.2, Table 4). The paper states: 'All models exhibit overprecision to varying degrees of severity, as evidenced by the lack of calibration between the imposed confidence levels and the actual hit rates of the LLMs' (Section 5.1). If correct, this means LLM interval outputs are not calibrated statistical statements even when the confidence is supplied in the prompt.

Load-bearing premise

The measurement assumes that sampling questions from all dataset splits, including training splits, gives an unbiased view of LLM interval behavior. The data processing section states: 'Due to the limited number of numerical answers in the test splits of these datasets, we sampled questions from all splits during the process' (Section 4). If the LLMs memorized some of these public benchmarks during pretraining, their intervals around memorized answers could be narrower or more accurate, biasing the hit rates, the interval-length statistics, and thus the overprecision conclusions. This is a data-validity assumption that the paper does not test or control for.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted in this study. The confidence levels (60%, 70%, 80%, 90%, 95%), number of trials, and aggregation weights are specified by the protocol, not estimated from data. All reported statistics are computed directly from LLM outputs and benchmark ground truths.

assumptions (3)
  • domain assumption The LLM's generated interval is a meaningful confidence interval at the imposed level.
    The framework interprets (x, y) from Eq. 1 as the model's c% interval; if the model ignores or misunderstands the probabilistic instruction, hit@c measures instruction-following rather than internal confidence. The paper acknowledges this ambiguity in Sections 5.2 and 6, but the central conclusions rely on this interpretation.
  • domain assumption Benchmark ground-truth answers are correct and the MCQ-to-direct conversion preserves question meaning.
    The evaluation in Section 3.3 compares intervals to dataset answers; if conversion from multiple-choice to direct answer creates ambiguous questions, hit rates would be artificially low and overprecision inflated.
  • domain assumption The sampled questions from all dataset splits are representative and free of pretraining memorization effects.
    Section 4 states questions were sampled from all splits, but no analysis of potential benchmark contamination from LLM pretraining is provided. This assumption is load-bearing for the quantitative hit rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gauging Overprecision in LLMs: An Empirical Study." pith.science (2026). https://pith.science/paper/YZIWWRIW

@misc{pith2026250412098,
  author       = {Pith},
  title        = {Pith review of: Gauging Overprecision in LLMs: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZIWWRIW}},
  note         = {Machine review of arXiv:2504.12098}
}
read the original abstract

Recently, overconfidence in large language models (LLMs) has garnered considerable attention due to its fundamental importance in quantifying the trustworthiness of LLM generation. However, existing approaches prompt the \textit{black box LLMs} to produce their confidence (\textit{verbalized confidence}), which can be subject to many biases and hallucinations. Inspired by a different aspect of overconfidence in cognitive science called \textit{overprecision}, we designed a framework for its study in black box LLMs. This framework contains three main phases: 1) generation, 2) refinement and 3) evaluation. In the generation phase we prompt the LLM to generate answers to numerical questions in the form of intervals with a certain level of confidence. This confidence level is imposed in the prompt and not required for the LLM to generate as in previous approaches. We use various prompting techniques and use the same prompt multiple times to gauge the effects of randomness in the generation process. In the refinement phase, answers from the previous phase are refined to generate better answers. The LLM answers are evaluated and studied in the evaluation phase to understand its internal workings. This study allowed us to gain various insights into LLM overprecision: 1) LLMs are highly uncalibrated for numerical tasks 2) there is no correlation between the length of the interval and the imposed confidence level, which can be symptomatic of a a) lack of understanding of the concept of confidence or b) inability to adjust self-confidence by following instructions, {3) LLM numerical precision differs depending on the task, scale of answer and prompting technique 4) Refinement of answers doesn't improve precision in most cases. We believe this study offers new perspectives on LLM overconfidence and serves as a strong baseline for overprecision in LLMs.

Figures

Figures reproduced from arXiv: 2504.12098 by the authors.

Figure 1
Figure 1. An outline of the precision elicitation framework and an example. Given an input question, a confidence level is first specified, a prompt strategy is then chosen, and the confidence level is integrated into the prompt. Next, the sampling strategy and the number of samples are determined to control the amount and diversity of outputs of the same prompt. After that, an aggregator combines the different answers to pro… view at source ↗
Figure 2
Figure 2. Scale affect on precision: These figures show the distribution of the hit average for different answers in the vanilla prompt setting for different models on different datasets. The figures demonstrate that the performance Is affected by the prompting strategy, the scale of the answer, and the task. agg strategy CWA LWM MIA Union iLWM mean std mean std mean std mean std mean std dataset FinQA 19.58 0.37 21.47 0.34 1… view at source ↗
Figure 3
Figure 3. The hit average metric as a function of the number of examples provided in the self-refinement prompt. The titles of the subfigures are organized as follows: [setting][dataset][kind]. The setting can either be Single or mixed (refer to the experimental protocol for more detail). The kind can either be ”chosen” for answers that were selected by the LLM to be the most correct. The kind can also be ”proposed” for the a… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The figures show the distribution of the average DS metric across confidence levels for different datasets, in different models for vanilla and CoT prompts. GPT-3.5 is short for GPT-3.5-turbo, and GPT-4o is short for GPT-4o-mini. The DS values are lowest for MMLU, high…
Figure 5
Figure 5. Figure 5: The figures show the distribution of the average ILS metric across confidence levels for different datasets, in different models for vanilla and CoT prompts. GPT-3.5 is short for GPT-3.5-turbo, and GPT-4o is short for GPT-4o-mini. The figures show that interval lengths…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    and Valdenegro-Toro, M

    Groot, T. and Valdenegro-Toro, M. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), pp. 145–171,

    work page 2024

  3. [9]

    ’instruction’ represents the abbreviation used in the paper for a particular set of instructions

    Sets of instructions that are used in the prompts. ’instruction’ represents the abbreviation used in the paper for a particular set of instructions. ’Text’ is the instruction text. ’objective’ is the purpose of having that set of instructions. 12 Gauging Overprecision in LLMs: An Empirical Study hit-avg hit@95% hit@90% hit@60% hit@70% hit@80% corr mean st...

    work page 1955

  4. [11]

    L., and Howe, B

    Wen, B., Xu, C., Bin, H., Wolfe, R., Wang, L. L., and Howe, B. Mitigating overconfidence in large language models: A behavioral lens on confidence estimation and 10 Gauging Overprecision in LLMs: An Empirical Study calibration. In NeurIPS 2024 Workshop on Behavioral Machine Learning. Xiong, M., Hu, Z., Lu, X., LI, Y ., Fu, J., He, J., and Hooi, B. Can llm...

    work page 2024

  5. [1982]

    AsthmaBot: Multi-modal, Multi-Lingual Retrieval Augmented Generation For Asthma Patient Support

    Bahaj, A. and Ghogho, M. Asthmabot: Multi-modal, multi- lingual retrieval augmented generation for asthma patient support. arXiv preprint arXiv:2409.15815,

    work page Pith review arXiv

  6. [1997]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations. Huang, Y ., Liu, Y ., Thirukovalluru, R., Cohan, A., and Dhingra, B. Calibrating long-form generations from large language models. In Findings of the Associati...

    work page 2024

  7. [1999]

    SelfCheckGPT: Zero-resource black-box hallucination detection for gen- erative large language models

    Manakul, P., Liusie, A., and Gales, M. SelfCheckGPT: Zero-resource black-box hallucination detection for gen- erative large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004–9017, Singapore, December

    work page 2023

  8. [2004]

    Cognitive bi- ases in large language models: A survey and mitigation experiments

    Sumita, Y ., Takeuchi, K., and Kashima, H. Cognitive bi- ases in large language models: A survey and mitigation experiments. arXiv preprint arXiv:2412.00323,

    arXiv

Show all 12 references
  1. [2010]

    R., et al

    Chen, Z., Chen, W., Smiley, C., Shah, S., Borova, I., Lang- don, D., Moussa, R., Beane, M., Huang, T.-H., Routledge, B. R., et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Proces...

    2021

  2. [2021]

    Shifting attention to relevance: Towards the uncertainty estimation of large language models

    Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv. org. Geng, J., Cai, F., Wang, Y ., Koeppl, H., Nakov, P., and Gurevych, I. A survey of c...

    2024

  3. [2023]

    doi: 10.18653/v1/2023

    Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.557. URL https://aclanthology. org/2023.emnlp-main.557/. Mielke, S. J., Szlam, A., Dinan, E., and Boureau, Y .-L. Reducing conversational agents’ overconfidence through linguistic calibration. Transac...

    2023 doi

  4. [2024]

    Llamas know what gpts don’t show: Surrogate models for confidence estimation

    Shrivastava, V ., Liang, P., and Kumar, A. Llamas know what gpts don’t show: Surrogate models for confidence estimation. arXiv preprint arXiv:2311.08877,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.