Pith. sign in

REVIEW 4 major objections 6 minor 38 references

Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Confidence in an LLM answer is best obtained not from the LLM itself but from a small auxiliary model that sees several LLMs' responses, is trained on judge-scored correctness, and is optimized with focal loss.

desk verdict A broad, useful empirical sweep of calibration factors with a plausible but not fully proven central claim; the unvalidated judge model and a data-volume confound weaken the headline. read the letter →

arxiv 2501.03991 v1 pith:6U4M2BEU submitted 2025-01-07 cs.CL

Yuxi Xia , Pedro Henrique Luz de Araujo , Klim Zaporojets , Benjamin Roth This is my paper · ORCID
classification cs.CL
keywords LLMcalibrationconfidenceestimationresponseagreementfocallosspromptstylesauxiliarymodelexpectederrorverbalized
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a calibrated confidence score for an LLM's answer can be extracted more reliably by a text-only auxiliary model than by the LLM's own probabilities, its verbalized confidence, or existing post-hoc scaling, and it identifies which ingredients matter. The proposed Calib-n framework feeds a small transformer the question paired with answers from one or several LLMs and trains it to predict a judge model's binary correctness verdict; pooling several answers lets the estimator see inter-model agreement. Across 12 LLMs, 4 QA datasets, and 4 prompt styles, the paper reports that focal loss and response agreement improve calibration over BCE and AUC losses and over baselines including verbalized confidence, Platt-scaled probabilities, and APRICOT, with the focal Calib-1 variant winning most comparisons overall. It also finds that few-shot prompts help most, and that auxiliary-model confidence stays stable across accuracy levels while LLM internal probabilities and verbalized confidence become overconfident as accuracy falls. If correct, this gives a black-box-friendly route to calibrated confidence for LLM-based systems.

What carries the argument

The load-bearing object is the Calib-n auxiliary estimator: a small transformer (BERT-base-sized) with a 768-to-n classification head and sigmoid that maps n joint strings of the question plus each answer to per-answer probabilities, trained to match judge-provided binary correctness. Response agreement enters by giving the model all n answers at once, so it can learn that disagreement signals uncertainty. Three training losses are compared: binary cross-entropy, focal loss (which up-weights hard, high-loss examples, with the paper's default focal parameters), and an AUC surrogate loss that maximizes separation between correct and incorrect answers. The paper's conclusions rest on aggregate win counts across four calibration metrics and many model-dataset-prompt combinations.

What would settle it

Have human annotators provide correctness labels on a sample of the test questions and recompute all method rankings on those labels; if focal Calib-1 and Calib-n no longer beat Platt-scaled LLM probabilities and APRICOT, or if the ordering changes when a different judge model is used, the central claim does not hold.

Watch

Extended reading notes

Core claim

The central discovery is that calibration of LLM answers can be improved by an auxiliary confidence model trained on response agreement: concatenate the question with each candidate answer, run a BERT-base-sized transformer with a sigmoid head, and train it to predict whether a judge model, Prometheus-8x7b-v2.0, marked the answer correct. When responses from several LLMs are pooled, the model can detect low consensus and assign lower confidence; when the training loss is focal loss, the single-model variant (FL)Calib-1 is the best overall method by aggregate win counts across ECE, temperature-scaled ECE, Brier score, and selective-accuracy AUC. The paper also establishes that prompt style matters: few-shot prompts give the largest calibration gains for auxiliary models, and that the auxiliary methods' calibration is approximately flat across accuracy levels, unlike LLM probabilities and verbalized confidence, whose errors rise sharply as accuracy drops.

Load-bearing premise

The load-bearing premise is that the judge model's binary semantic-equivalence labels are correct; if that judge is noisy or biased, every ECE, Brier, and AUC comparison is miscalibrated, and the paper's Limitations section concedes the optimal correctness-assessment method remains underexplored.

Editorial extensions

If this is right

  • Confidence can be obtained for black-box LLMs with no access to weights or logits: only the text answers and a judge are needed.
  • Focal loss should be the default for training auxiliary confidence estimators; BCE and AUC losses trail it, and Platt scaling does not generalize to improve Calib-n.
  • Calibration studies should vary prompt style, because few-shot prompts are materially best for auxiliary methods and fixed-prompt conclusions can mislead.
  • A single static confidence method is not optimal across accuracy ranges: different methods win in low, mid, and high accuracy regimes.
  • Auxiliary confidence estimators stay calibrated when accuracy changes, making them safer for deployment on hard or out-of-distribution inputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the correctness labels come from a judge model, part of Calib-n's edge may be learning the judge's notion of correctness rather than ground truth; a human-labeled subset would separate these.
  • The accuracy-dependent winner pattern suggests an adaptive system could estimate its own accuracy and switch between FL Calib-1, Calib-n with Platt scaling, and Platt-scaled LLM probabilities, a direct testable extension.
  • Since the auxiliary model consumes only text, the same Calib-n design should transfer to API-only or proprietary LLMs and to heterogeneous model mixes, and to open-ended tasks beyond the four QA datasets.
  • If few-shot prompts help mostly by increasing response agreement, one could ablate agreement directly by feeding identical answers versus varied answers to verify the proposed mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies factors influencing LLM calibration: inter-model response agreement, loss functions (BCE, focal, AUC surrogate), and prompt styles (Verbalized, Zero-shot, CoT, Few-shot). It proposes Calib-n, an auxiliary BERT-based model that takes joint question-answer strings from n target LLMs as input and is trained on judge-provided binary correctness labels, with Calib-1 as a single-LLM variant. Experiments cover 12 LLMs (2B-72B) and four open-ended QA datasets, comparing against LLM probabilities, Platt scaling, verbalized confidence, and APRICOT across ECE, ECE-t, Brier, and AUC. The headline claims are that Calib-n outperforms state-of-the-art baselines, that focal loss improves calibration and (FL)Calib-1 is best overall, that few-shot prompts are most effective for auxiliary methods, and that auxiliary models are robust to accuracy variations.

Significance. If the central claims hold, the paper makes a useful empirical contribution by broadening calibration evaluation to 12 LLMs and four prompt styles and by demonstrating a practical auxiliary-model recipe (response agreement plus focal loss plus few-shot prompts). The experimental surface is large: the paper ships extensive tables, covers five model families, and explicitly compares against a recent state-of-the-art method. However, the headline conclusions rest on win-count aggregations without statistical support, and the correctness labels that ground all metrics come from an unvalidated judge model. The claims are therefore not yet established at the level of certainty the paper suggests, though they are plausible and testable.

major comments (4)
  1. [§3.2, Eq. (1), §7] All calibration metrics in Tables 1–8 and the win counts in Figs. 2–6 are computed against binary correctness labels c_i produced by Prometheus-8x7b-v2.0. The paper provides no validation of this judge on the four datasets or against human semantic-equivalence judgments, and Section 7 states that the optimal correctness-assessment method is underexplored. If the judge is noisy or biased (e.g., by answer length, prompt style, or model family), then the apparent superiority of Calib-* methods—which are trained to predict these very labels—over LLM probabilities and verbalized confidence could be inflated. Please add a human-annotated validation sample with judge agreement, and ideally a robustness check using exact-match or ROUGE-based labels for the calibration metrics.
  2. [§4.3, §5.1, Figs. 2–3] The headline conclusions ('Calib-n outperforms the state-of-the-art baselines', '(FL)Calib-1 exhibits the best overall performance') are based entirely on win-count aggregation without error bars, confidence intervals, or significance tests. Since the underlying ECE/Brier values differ by small margins in many cells (e.g., Table 1, Gemma2-27b few-shot: (BCE)Calib-n ECE 0.088 vs (FL)Calib-n ECE 0.085), a single metric or split can change the count. Please report paired significance tests or bootstrap CIs over the 12 models × 4 prompts × 4 datasets, and make clear which aggregation choices drive each claim.
  3. [§3.3, §4.2, §5.1] The comparison between Calib-n and Calib-1 is confounded by training-set size. Calib-n is trained on n joint strings per question (one per target LLM), giving n·k training instances, whereas Calib-1 is trained on k instances for a single LLM. Any improvement from Calib-n could therefore reflect more training data rather than inter-model response agreement per se. Please include a controlled comparison in which Calib-1 is trained on the same number of instances (e.g., by replicating or augmenting single-LLM examples), or otherwise show that the agreement signal, not data volume, drives the result.
  4. [§4.3] The definition of ECE is non-standard and potentially inconsistent with the reported values: the text says ECE is a 'weighted ... average of the squared difference' between bin accuracy and confidence, whereas the standard ECE (Guo et al., 2017b) uses the absolute difference. If the squared variant was actually used, the ECE values are not comparable to prior work and the ranking across methods could differ. Please provide the exact binning formula and confirm whether squared or absolute differences were used, ideally with a reference to the implementation.
minor comments (6)
  1. [§4.1] The phrase 'four open-ended quenstion-answering datasets' contains a typo; it should be 'question-answering'.
  2. [§3.3] The sentence starting 'Givek questions, we minimize' appears to be missing a space and a word; it should read 'Given k questions, we minimize'.
  3. [Table 2, Phi3-7b row] The Brier score entry for Verbalized % under the Verb. prompt is reported as 113.606, which is outside the valid [0,1] range for a Brier score and appears to be a data-entry error.
  4. [Fig. 1 caption] The caption contains the typo 'response aggrement' and should read 'response agreement'.
  5. [Appendix A.4] Two sub-figure rows are both described as 'The second row of Fig. 6'; the dataset-specific panel is actually the third row.
  6. [Abstract and §1] The footnote 'Code and data will be released upon acceptance' prevents independent verification in the meantime; please provide an anonymous code repository or supplementary material link for review.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core comparison is a held-out supervised learning evaluation against an externally sourced judge label, not a derivation whose conclusion is fixed by its inputs.

full rationale

The paper's derivation chain is an empirical supervised-learning pipeline, not a formal derivation whose conclusion is encoded in its assumptions. Correctness labels c_i are produced by an external judge model Prometheus-8x7b-v2.0 (Eq. 1, Section 3.2), and the auxiliary model f is trained to predict those labels on a training split using BCE, focal, and AUC surrogate losses (Eqs. 3, 5, 6, Section 3.3-3.4). All claimed results — ECE, ECE-t, Brier, AUC, and win counts — are computed on held-out test splits against the same label source (Eq. 4, Section 4.3). No parameter is fitted to the test predictions and then reported as a prediction; the auxiliary model's test performance is genuine out-of-sample generalization. The paper does not invoke a uniqueness theorem, and none of the load-bearing citations (e.g., Kim et al. 2023 for ensemble agreement, Lin et al. 2020 for focal loss, Yuan et al. 2021 for AUC surrogate loss, Ulmer et al. 2024 for APRICOT) are self-citations by the present authors, nor do they smuggle in the paper's conclusions. The remaining concern — that Prometheus judge labels may be noisy or biased, making all calibration numbers relative to the judge rather than to true semantic equivalence — is a measurement-validity caveat, not circularity; it is explicitly and honestly acknowledged in Section 7 ('While the optimal solution is underexplored, we choose the more commonly used and cost-efficient method'). The headline claims are therefore supported by an independent empirical comparison rather than by construction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. Calib-n is a model architecture, not an invented entity. The paper postulates a new capability (response agreement) but provides no independent falsifiable entity beyond the framework itself. The main loaded assumptions are the judge model's correctness labels and the standard calibration metrics.

free parameters (5)
  • focal loss alpha = 0.25 (default, not fitted to data)
    Chosen as default from Lin et al. (2020); the central focal-loss improvement claim depends on these values.
  • focal loss gamma = 2.0 (default, not fitted to data)
    Chosen as default from Lin et al. (2020); the central focal-loss improvement claim depends on these values.
  • learning rate = 1e-5
    Selected via grid search (Appendix A.3); affects auxiliary model training.
  • batch size = 16
    Selected via grid search (Appendix A.3); affects auxiliary model training.
  • training epochs = 5
    Selected via grid search (Appendix A.3); affects auxiliary model training.
assumptions (3)
  • domain assumption Prometheus-8x7b-v2.0 binary judgments are a reliable proxy for semantic equivalence and answer correctness.
    All correctness labels ci in Equation (1) come from this judge; the paper's Limitations section concedes the optimal correctness-assessment method is underexplored.
  • domain assumption BERT-base representations are sufficient to learn confidence from question-answer strings.
    The auxiliary model is BERT-base with a 768-dimensional classification head; no evidence is provided that this architecture transfers across all tested LLMs, datasets, and prompt styles.
  • standard math Standard calibration metrics (ECE, ECE-t, Brier, AUC) adequately capture calibration quality.
    These metrics are widely used, but the Limitations section notes they may not capture user-perceived confidence or task-specific utility.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles." pith.science (2026). https://pith.science/paper/6U4M2BEU

@misc{pith2026250103991,
  author       = {Pith},
  title        = {Pith review of: Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6U4M2BEU}},
  note         = {Machine review of arXiv:2501.03991}
}
read the original abstract

Calibration, the alignment between model confidence and prediction accuracy, is critical for the reliable deployment of large language models (LLMs). Existing works neglect to measure the generalization of their methods to other prompt styles and different sizes of LLMs. To address this, we define a controlled experimental setting covering 12 LLMs and four prompt styles. We additionally investigate if incorporating the response agreement of multiple LLMs and an appropriate loss function can improve calibration performance. Concretely, we build Calib-n, a novel framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement. To optimize calibration, we integrate focal and AUC surrogate losses alongside binary cross-entropy. Experiments across four datasets demonstrate that both response agreement and focal loss improve calibration from baselines. We find that few-shot prompts are the most effective for auxiliary model-based methods, and auxiliary models demonstrate robust calibration performance across accuracy variations, outperforming LLMs' internal probabilities and verbalized confidences. These insights deepen the understanding of influence factors in LLM calibration, supporting their reliable deployment in diverse applications.

Figures

Figures reproduced from arXiv: 2501.03991 by the authors.

Figure 1
Figure 1. Overview of calibration training of Calib-n ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Calib-1 and Calib-n methods [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The winning comparison results of different methods and prompts: [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The correlation between accuracies achieved [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Reliability diagrams for our different methods using 10 bins each for Llama3.1-70b on NQ. The color and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance Comparison results of different methods against baselines across three setting configurations [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Model performance ranking across different datasets, performance is averaged over four prompt styles. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Model performance across different prompts and datasets [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Reliability diagrams for Llama3.1-70b on TriviaQA with Verb. prompts. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Reliability diagrams for Llama3.1-70b on Sciq with Verb. prompts. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Reliability diagrams for Llama3.1-70b on WikiQA with Verb. prompts. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 7 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Glenn W. Brier. 1950. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 Verification of forecasts expressed in terms of probability . Monthly Weather Review, 78(1):1 -- 3

    doi:10.1175/1520-0493(1950)078 1950

  3. [3]

    Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2024. https://arxiv.org/abs/2310.14735 Unleashing the potential of prompt engineering in large language models: a comprehensive review . Preprint, arXiv:2310.14735

    arXiv 2024

  4. [4]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    arXiv 2018

  5. [5]

    Yonatan Geifman and Ran El-Yaniv. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/4a8423d5e91fda00bb7e46540e2b0cf1-Paper.pdf Selective classification for deep neural networks . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc

    work page 2017

  6. [6]

    Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577--6595

    2024

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    arXiv 2024

  8. [8]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017 a . On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1321–1330. JMLR.org

    work page 2017

Show all 38 references
  1. [9]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017 b . On calibration of modern neural networks. In International conference on machine learning, pages 1321--1330. PMLR

    2017

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  3. [11]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962--977

    2021

  4. [12]

    Matt Gardner Johannes Welbl, Nelson F. Liu. 2017. Crowdsourcing multiple choice science questions. arXiv:1707.06209v1

    2017 arXiv

  5. [13]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2017

  6. [14]

    Jaeyoung Kim, Dongbin Na, Sungchul Choi, and Sungbin Lim. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.40 Bag of tricks for in-distribution calibration of pretrained transformers . In Findings of the Association for Computational Linguistics: EACL 2023, pages 551--563,...

    2023 doi

  7. [15]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language models ....

    2024 arXiv

  8. [16]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  9. [17]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research

    2022

  10. [18]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2020. https://doi.org/10.1109/TPAMI.2018.2858826 Focal loss for dense object detection . IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318--327

    2020

  11. [19]

    Xin Liu, Muhammad Khalifa, and Lu Wang. 2024. https://arxiv.org/abs/2310.19208 Litcab: Lightweight language model calibration over short- and long-form responses . In Proceedings of the Twelfth International Conference on Learning Representations. ICLR

    2024 arXiv

  12. [20]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing...

    2022

  13. [21]

    Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. 2020. https://proceedings.mlr.press/v119/moon20a.html Confidence-aware learning for deep neural networks . In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine...

    2020

  14. [22]

    Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. 2020. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288--15299

    2020

  15. [23]

    Shiyu Ni, Keping Bi, Lulu Yu, and Jiafeng Guo. 2024. https://arxiv.org/abs/2408.09773 Are large language models more honest in their probabilistic or verbalized confidence? Preprint, arXiv:2408.09773

    2024 arXiv

  16. [24]

    John Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61--74

    1999

  17. [25]

    Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedin...

    2020

  18. [26]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  19. [27]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language mod...

    2023 doi

  20. [28]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  21. [29]

    Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Oh. 2024. https://doi.org/10.18653/v1/2024.acl-long.824 Calibrating large language models using their generations only . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024 doi

  22. [30]

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023

  23. [31]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  24. [32]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. https://arxiv.org/abs/2306.13063 Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms . Preprint, arXiv:2306.13063

    2024 arXiv

  25. [33]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  26. [34]

    Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. https://doi.org/10.18653/v1/D15-1237 W iki QA : A challenge dataset for open-domain question answering . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013--2018, Lisbon, Portu...

    2015 doi

  27. [35]

    Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. 2021. https://doi.org/10.1109/ICCV48922.2021.00303 Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification . In 2021 IEEE/CVF International Conference on Compute...

    2021

  28. [36]

    Min Zhang, Jianfeng He, Taoran Ji, and Chang-Tien Lu. 2024. https://doi.org/10.18653/v1/2024.acl-long.652 Don ' t go to extremes: Revealing the excessive sensitivity and calibration limitations of LLM s in implicit hate speech detection . In Proceedings of the 62nd Annual Meet...

    2024 doi

  29. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.