Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM uncertainty scores don't make LLMs safer for users, a 40-method survey argues.

desk verdict A well-argued position paper with a useful but under-audited benchmark annotation; the qualitative case holds, but the 40/37 count and Table 1 inconsistencies need fixing before the evidence is treated as definitive. read the letter →

arxiv 2506.07461 v1 pith:S75Z52Z5 submitted 2025-06-09 cs.CL

Siddartha Devic , Tejas Srinivasan , Jesse Thomason , Willie Neiswanger , Vatsal Sharan This is my paper
classification cs.CL
keywords LLMuncertaintyquantificationcalibrationecologicalvalidityhuman-LLMcollaborationaleatoricdistributionshiftbenchmarkevaluationhumanuplift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the way uncertainty quantification (UQ) for large language models (LLMs) is currently evaluated does not serve the stated goal of helping human users decide when to trust model outputs. Reviewing 40 LLM UQ methods and the 22 benchmarks those methods rely on, it identifies three systemic problems: the benchmarks are not representative of real-world tasks, they ignore aleatoric uncertainty and distribution shift, and the metrics being optimized (like ECE and Brier score) have not been shown to improve human decision making. The paper's position is that the field should shift from hill-climbing calibration scores to a human-centered approach that tests whether uncertainty information actually gives users measurable benefit. This matters because LLMs are already deployed as decision aids in high-stakes settings, where a well-calibrated confidence score that does not change user behavior offers little protection.

What carries the argument

The argument is carried by a systematic annotation exercise: the authors take 40 LLM UQ papers from two recent surveys, extract the 22 benchmarks used by at least two papers, and score each benchmark against four criteria (C1–C4) adapted from explainable-AI research: connection to a real task, realistic inputs, genuine difficulty for people, and potential for harm from bad decisions. They also classify each benchmark by uncertainty type (epistemic, aleatoric, distributional) and each method by supervision and evaluation metric. This yields the quantitative counts, such as 15 of 22 benchmarks being multiple choice and only 2 of 22 satisfying C1–C3, that anchor the three claimed barriers.

What would settle it

A human-subject study in which presenting calibrated LLM confidence scores to users produces a measurable, consistent improvement in decision accuracy on a realistic task, while uncalibrated scores do not, would directly contradict the claim that calibration metrics are unconnected to downstream utility. Alternatively, an independent re-annotation of the 22 benchmarks that placed the majority in different C1 or C2 categories would weaken the ecological-validity counts.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the NLP community's prevailing evaluation practices for LLM UQ methods are insufficient to benefit human users in real-world settings. The paper supports this with a survey of 40 UQ method papers, finding that most benchmarks are factual QA or commonsense tasks, that only two benchmarks satisfy all the adopted ecological-validity criteria (Natural Questions and NQ-Open), that only one benchmark intentionally contains aleatoric uncertainty (AmbigQA), that fewer than half of supervised methods are tested under distribution shift, and that almost all papers report calibration metrics without any human-uplift study. It concludes that better ECE scores on these benchmarks do not automatically translate into better human-LLM collaboration.

Load-bearing premise

The paper's quantitative evidence depends on the authors' subjective annotations in Table 1 being correct (for example, that Natural Questions passes C1 while TriviaQA does not) and on the 40 sampled methods being representative of the broader LLM UQ literature.

Editorial extensions

If this is right

  • LLM UQ papers should report results on tasks with intentional aleatoric uncertainty, such as datasets with multiple gold labels or conflicting outcomes, rather than only single-answer QA datasets.
  • Supervised UQ methods should be tested under distribution shift as a standard requirement, since calibrated parameters learned on one distribution may not transfer to new query distributions.
  • Calibration metrics like ECE and Brier score should be supplemented or replaced by metrics validated against human decision performance, and smECE should be preferred over ECE for evaluation sets smaller than about 5,000 examples.
  • The community should invest in human-uplift studies that compare decisions made with and without uncertainty information, including non-numeric presentation schemes such as hedged language and anthropomorphic expressions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the paper is right, the same critique likely applies to other AI-assistance fields that optimize calibration metrics without measuring user outcomes, so the C1–C4 screen could be reused as a pre-filter for any human-AI collaboration benchmark.
  • A testable extension would be to correlate ECE improvements with human uplift across the few existing studies that include human evaluations, to see whether any threshold of calibration quality actually predicts better joint decisions.
  • The paper's own recommendation implies a concrete measurable goal: an LLM UQ method should be judged by the increase in joint human-AI decision accuracy per unit of user trust expenditure, not by calibration error alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This position paper argues that the NLP community's prevailing practices for evaluating LLM uncertainty quantification (UQ) are not sufficient for building UQ that benefits human users in real-world decision-making. The authors support this with a survey of 40 LLM UQ method papers and 22 benchmarks, and they identify three barriers: benchmarks with low ecological validity, a narrow focus on epistemic uncertainty, and metrics not tied to downstream utility. For each barrier, they propose concrete recommendations and research directions, and they discuss two alternative views in Section 6. The paper's main empirical contribution is the structured annotation of benchmarks against four criteria (C1–C4) adapted from human-centered XAI work, together with frequency counts such as 'only 2 benchmarks satisfy C1–C3' and '10 out of 19 supervised parameter-learning methods tested under distribution shift.'

Significance. If the empirical survey is corrected and made auditable, the paper could serve as a useful programmatic statement for the LLM UQ community. Its strengths are that it makes falsifiable quantitative claims about the field's evaluation practices, imports concrete criteria from the XAI literature rather than relying on vague objections, and pairs each critique with actionable recommendations. The paper does not contain fitted parameters or circular derivations; it leans on external published results, including two from the authors' own group, which is appropriate for a position paper. The central qualitative message—that calibration improvements on QA benchmarks do not automatically translate into better human-LLM collaboration—is well supported by the cited classical UQ and HCI literature.

major comments (4)
  1. [§2 and Appendix B] The paper states that it analyzes 40 LLM UQ method papers and that Appendix B provides an annotation of each selected paper, but Appendix B contains only 37 entries (P1–P37). This discrepancy is load-bearing because Section 4.2's '10 out of 19 supervised, parameter-learning methods' statistic and Appendix A's '16 papers used ECE' count are computed on the survey corpus. The authors should either add the missing three annotations or revise the stated corpus size to 37 and recompute all dependent statistics; otherwise the prevalence claims cannot be reproduced or checked for selection bias.
  2. [§3.2 and Table 1] The text says that exactly two benchmarks satisfy C1–C3 and names Natural Questions and NQ-Open, but Table 1 marks NQ-Open's C3 as ✗. Under the stated criterion, only Natural Questions satisfies C1–C3. I verified that the C3 count in Table 1 is 9, matching the text, so that specific discrepancy is not present; however, the NQ-Open flag contradicts the 'only 2 benchmarks' sentence, which is the headline ecological-validity result. The authors should correct the table or the text and ensure that the 27.3% C1 percentage and the 'only 2' claim are computed from the final annotation.
  3. [§3.2] The C1–C4 annotations are the sole quantitative basis for the ecological-validity claims, but the paper provides no annotation protocol, no inter-annotator agreement, and no released annotation artifact. Several judgments are borderline and contestable (e.g., WebQA receives C1 but not C2; NQ-Open is treated as satisfying C3 in the text but not in the table). Without a rubric or a second annotator, the 27.3% and 'only 2' statements are not independently checkable. The authors should provide the annotation guidelines, ideally with agreement statistics or a public artifact.
  4. [Footnote 2 and §3] Restricting the benchmark pool to benchmarks used by at least two of the surveyed papers may bias the analysis toward established QA and commonsense benchmarks and away from newer, more ecological tasks. For example, FolkTexts is cited in Section 4.1 as a recommended aleatoric-uncertainty benchmark but does not appear in Table 1. Because the paper's claim that 'the majority of LLM UQ methods are evaluated on only factual QA or commonsense reasoning' is computed on this restricted set, the authors should justify the two-paper rule with a sensitivity analysis or report the full pool of benchmarks.
minor comments (8)
  1. [§1.1] There is a missing space in 'community’sprevailing practices'.
  2. [§3.3] The phrase 'estimate uncertain when the set of possible decisions is not enumerated' should be 'estimate uncertainty when'.
  3. [§4.1] The sentence 'the clinician is still required to making a treatment decision' contains a subject-verb error; it should be 'required to make a treatment decision'.
  4. [§5.1 and References] The author name 'Vodrahalli' is typeset as 'V odrahalli' in the text and references, and the reference format is inconsistent with the rest of the bibliography.
  5. [§5.1 and References] The citation style for the same research group is inconsistent: 'Corvelo Benz and Rodriguez (2023)' in the text but 'Corvelo Benz and Gomez Rodriguez (2025)' elsewhere, with the reference list using both forms.
  6. [Table 1 and Appendix B] Several dataset names contain spacing artifacts ('SW AG', 'HellaSW AG') that should be 'SWAG' and 'HellaSwag' for consistency with the cited papers.
  7. [§5.2] In recommendation R7, 'Brier scorecorrelate' is missing a space; it should read 'Brier score correlate'.
  8. [§3.1] The sentence 'We argue that the LLM UQ methods primarily evaluate on benchmarks with low ecological validity' should read 'the LLM UQ methods are primarily evaluated'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's survey-based argument is not derived from its own conclusion or from load-bearing self-citations.

full rationale

This is a position paper whose evidence is an annotated survey and an application of externally attributed criteria, not a mathematical derivation. The central claim—'the NLP community's prevailing practices for evaluating LLM UQ methods are insufficient to benefit human users in real-world settings'—is supported by (i) a corpus of 40 LLM UQ method papers assembled from two external surveys (Shorinwa et al., 2024; Huang et al., 2024), (ii) benchmark annotations using criteria explicitly adopted from Chaleshtori et al. (2024), and (iii) external human-AI collaboration studies. No fitted parameter is later renamed as a prediction, no quantity is defined in terms of the desired conclusion, and no uniqueness theorem or ansatz is imported from the authors' prior work. The self-citations (Hansen et al. 2024 in Appendix A; Srinivasan and Thomason 2025 in Section 4.2) support auxiliary points about smECE and trust erosion but are not load-bearing for the survey counts or benchmark annotations. Two audit concerns, distinct from circularity, are that the text says 40 methods while Appendix B annotates only 37 (P1–P37), and Section 3.2's claim that NQ-Open satisfies C1–C3 conflicts with Table 1's ✗ for NQ-Open's C3; these affect reproducibility but do not make the argument circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric parameters are fit; the paper's evidence is an annotation exercise. The central argument rests on the transferability of XAI validity criteria, the representativeness of the surveyed papers, and the reliability of the benchmark classification, none of which are formally established.

assumptions (4)
  • domain assumption An LLM UQ method's value should be judged by whether it improves real human decision-making, as stated in Section 1.1.
    The entire critique uses downstream human utility as the yardstick for UQ methods; this is a normative premise, not an empirical fact.
  • domain assumption Ecological validity criteria C1-C4 from Chaleshtori et al. (2024) are applicable to LLM UQ benchmarks.
    The paper adopts these XAI explanation criteria for UQ benchmarks without validating that they transfer; Section 3.2.
  • domain assumption The 40 methods collected from two prior surveys are a representative sample of user-motivated LLM UQ papers.
    The paper does not report what fraction of the surveys' methods were included or excluded, or how the 40 were selected; Section 2.
  • domain assumption The benchmark annotations in Table 1 are correct and reproducible.
    The C1-C4 checks and crosses are subjective judgments with no inter-annotator agreement or audit trail; Section 3.2 and Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered." pith.science (2026). https://pith.science/paper/S75Z52Z5

@misc{pith2026250607461,
  author       = {Pith},
  title        = {Pith review of: From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S75Z52Z5}},
  note         = {Machine review of arXiv:2506.07461}
}
read the original abstract

Large Language Models (LLMs) are increasingly assisting users in the real world, yet their reliability remains a concern. Uncertainty quantification (UQ) has been heralded as a tool to enhance human-LLM collaboration by enabling users to know when to trust LLM predictions. We argue that current practices for uncertainty quantification in LLMs are not optimal for developing useful UQ for human users making decisions in real-world tasks. Through an analysis of 40 LLM UQ methods, we identify three prevalent practices hindering the community's progress toward its goal of benefiting downstream users: 1) evaluating on benchmarks with low ecological validity; 2) considering only epistemic uncertainty; and 3) optimizing metrics that are not necessarily indicative of downstream utility. For each issue, we propose concrete user-centric practices and research directions that LLM UQ researchers should consider. Instead of hill-climbing on unrepresentative tasks using imperfect metrics, we argue that the community should adopt a more human-centered approach to LLM uncertainty quantification.

Figures

Figures reproduced from arXiv: 2506.07461 by the authors.

Figure 1
Figure 1. Category-wise distribu￾tion of LLM UQ benchmarks. Our first observation, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PaTAS: A Framework for Trust Propagation in Neural Networks Using Subjective Logic

    cs.AI 2025-11 conditional novelty 5.0 of 10

    PaTAS propagates Subjective Logic trust opinions through every neuron of a network and updates parameter trust from gradient evidence, yielding per-prediction trust scores intended to flag poisoned or low-reliability inputs.

Reference graph

Works this paper leans on

30 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aichberger, L., Schweighofer, K., Ielanskyi, M., and Hochreiter, S. (2025). Improving uncertainty estimation through semantically diverse language generation. In The Thirteenth International Conference on Learning Representations. Alufaisan, Y ., Marusich, L. R., Bakdash, J. Z., Zhou, Y ., and Kantarcioglu, M. (2021). Does explainable artificial intellige...

    arXiv 2025

  2. [3]

    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J

    OpenReview.net. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. (2010). A theory of learning from different domains. Machine learning, 79:151–175. Błasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. (2023). A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Comp...

    work page 2010

  3. [4]

    Brennan, T., Dieterich, W., and Ehret, B. (2009). Evaluating the predictive validity of the compas risk and needs assessment system. Criminal Justice and behavior, 36(1):21–40. Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American psychologist, 32(7):513. Buçinca, Z., Lin, P., Gajos, K. Z., and Glassman, E. L. (2020). Pr...

    arXiv 2009

  4. [6]

    Donahue, K., Chouldechova, A., and Kenthapadi, K. (2022). Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1639–1656. Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2023). Shifting attent...

    arXiv 2022

  5. [8]

    Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., and Kamar, E. (2022). Toxigen: A large- scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27,

    work page 2022

  6. [10]

    V ., Zhang, Y ., Luss, R., Doshi-Velez, F., and Dhurandhar, A

    14 Liao, Q. V ., Zhang, Y ., Luss, R., Doshi-Velez, F., and Dhurandhar, A. (2022). Connecting algorithmic research and usage contexts: a perspective of contextualized evaluation for explainable ai. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 10, pages 147–159. Lin, B. Y ., Deng, Y ., Chandu, K., Ravichander, A., P...

    work page 2022

  7. [11]

    Hou, Y ., Li, J., He, Z., Yan, A., Chen, X., and McAuley, J

    Association for Computational Linguistics. Hou, Y ., Li, J., He, Z., Yan, A., Chen, X., and McAuley, J. (2024). Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03952. Hu, Z., Liu, C., Feng, X., Zhao, Y ., Ng, S.-K., Luu, A. T., He, J., Koh, P. W. W., and Hooi, B. (2024). Uncertainty of thoughts: Uncertainty-aware pl...

    arXiv 2024

  8. [13]

    Kumar, A., Morabito, R., Umbet, S., Kabbara, J., and Emami, A. (2024). Confidence under the hood: An investigation into the confidence-probability alignment in large language models. arXiv preprint arXiv:2405.16282. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (...

    work page Pith review arXiv 2024

Show all 30 references
  1. [15]

    Lin, Z., Trivedi, S., and Sun, J. (2024). Generating with confidence: Uncertainty quantification for black-box large language models. Trans. Mach. Learn. Res.,

    2024

  2. [16]

    Liu, L., Pan, Y ., Li, X., and Chen, G. (2024a). Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993. Liu, O., Fu, D., Yogatama, D., and Neiswanger, W. (2024b). Dellma: A framework for decision making under uncertai...

    2024 arXiv

  3. [17]

    and Wu, Z

    Liu, T. and Wu, Z. S. (2024). Multi-group uncertainty quantification for long-form text generation. arXiv preprint arXiv:2407.21057. Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., and Wei, H. (2025). Uncertainty quantification and confidence calibration in large language models...

    2024 arXiv

  4. [18]

    Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero-resource black-box hallucina- tion detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10,

    2023

  5. [19]

    A., Kirschbaum, E., Kasiviswanathan, S., and Ramdas, A

    Manggala, P., Mastakouri, A. A., Kirschbaum, E., Kasiviswanathan, S., and Ramdas, A. (2025). Qa-calibration of language model confidence scores. In The Thirteenth International Conference on Learning Representations. Marusich, L., Bakdash, J. Z., Zhou, Y ., and Kantarcioglu, M...

    2025

  6. [20]

    J., Szlam, A., Dinan, E., and Boureau, Y .-L

    Mielke, S. J., Szlam, A., Dinan, E., and Boureau, Y .-L. (2022). Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. ...

    2022

  7. [21]

    Rong, Y ., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V ., Seidel, T., Kasneci, G., and Kasneci, E

    OpenReview.net. Rong, Y ., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V ., Seidel, T., Kasneci, G., and Kasneci, E. (2023). Towards human-centered explainable ai: A survey of user studies for model explanations. IEEE transactions on pattern analysis and machi...

    2023 arXiv

  8. [23]

    and Stoykova, R

    16 Valdenegro-Toro, M. and Stoykova, R. (2024). The dilemma of uncertainty estimation for general purpose ai in the eu ai act. arXiv preprint arXiv:2408.11249. Vasconcelos, H., Bansal, G., Fourney, A., Liao, Q. V ., and Wortman Vaughan, J. (2025a). Generation probabilities are...

    2024 arXiv

  9. [25]

    Xing, Z., Yuan, X., and Mostafa, J. (2022). Age-related difference in conversational search behavior: Preliminary findings. In CHIIR ’22: ACM SIGIR Conference on Human Information Interaction and Retrieval, Regensburg, Germany, March 14 - 18,

    2022

  10. [26]

    Xiong, M., Hu, Z., Lu, X., Li, Y ., Fu, J., He, J., and Hooi, B. (2023). Can llms express their uncer- tainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063. Yaldiz, D. N., Bakman, Y . F., Buyukates, B., Tao, C., Ramakrishna, A., Di...

    2023 arXiv

  11. [27]

    H., and Yamada, M

    Yang, D., Tsai, Y .-H. H., and Yamada, M. (2024c). On verbalized confidence scores for llms.arXiv preprint arXiv:2412.14737. 17 Ye, Z., Melo, L. C., Kaddar, Y ., Blunsom, P., Staton, S., and Gal, Y . (2025). Uncertainty-aware step-wise verification with generative reward model...

    2024 arXiv

  12. [28]

    Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y ., and Deng, Y . (2024). Wildchat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  13. [29]

    Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M

    OpenReview.net. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. (2024). Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. 18 A ECE and Measuring Calibration in Practice There may be other...

    2024

  14. [30]

    Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. (2021). Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics. Gichoya, J. W., Thomas, K., Celi, L. A., ...

    2021 arXiv

  15. [31]

    • Trainable / learnable response scoring function for LLMs

    (a) Type of uncertainty quantification (short / long form generation, conformal, etc.). • Trainable / learnable response scoring function for LLMs. Follow-up work on (Bakman et al., 2024). (b) Datasets used to evaluate proposed LLM UQ method. • TriviaQA NaturalQA WebQA GSM8K 2...

    2024

  16. [63]

    Springer. Sung, Y . Y ., Fleisig, E., Hou, Y ., Upadhyay, I., and Boyd-Graber, J. L. (2025). Grace: A gran- ular benchmark for evaluating model calibration against human calibration. arXiv preprint arXiv:2502.19684. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2019). Co...

    2025 arXiv

  17. [1070]

    P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., and Morency, L.-P

    Xiao, Y ., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., and Morency, L.-P. (2022). Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04714. Xie, J., Chen, A. S., Lee, Y ., Mitchell, E., and F...

    2022 arXiv

  18. [2021]

    Heo, J., Xiong, M., Heinze-Deml, C., and Narain, J. (2024). Do llms estimate uncertainty well in instruction-following? arXiv preprint arXiv:2410.14582. Hershcovich, D., Frank, S., Lent, H. C., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Piqueras, L. C., Chalkidis,...

    2024 arXiv

  19. [2022]

    He, G., Chen, J., and Zhu, J. (2023a). Preserving pre-trained features helps calibrate fine-tuned language models. arXiv preprint arXiv:2305.19249. He, J., Yang, R., Yu, L., Li, C., Jia, R., Chen, F., Jin, M., and Lu, C.-T. (2024). Can we trust the performance evaluation of un...

    2023 arXiv

  20. [2023]

    i’m not sure, but

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July...

    2017 arXiv

  21. [2024]

    Balepur, N., Rudinger, R., and Boyd-Graber, J. L. (2025). Which of these best describes multiple choice evaluation with llms? a) forced b) flawed c) fixable d) all of the above. arXiv preprint arXiv:2502.14127. Band, N., Li, X., Ma, T., and Hashimoto, T. (2024). Linguistic cal...

    2025 arXiv

  22. [2025]

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J

    OpenReview.net. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cole, J. R., Zhang, M. ...

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.