Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SMARTCAL shows that LLM tool-use agents overconfidently misuse tools, and that a self-evaluation plus a confidence–accuracy lookup table can reduce this, improving QA accuracy and calibration.

desk verdict A transparent, early-stage calibration method for tool-use LLMs; the headline ECE gains rest on a questionable zero-fill convention that the paper itself flags. read the letter →

arxiv 2412.12151 v1 pith:H6MBDCVD submitted 2024-12-11 cs.LG cs.AIcs.SE

Yuanhao Shen , Xiaodan Zhu , Lei Chen This is my paper · ORCID
classification cs.LGcs.AIcs.SE
keywords largelanguagemodelstoolusecalibrationExpectedErrorverbalizedconfidenceself-evaluationretrieval-augmentedgenerationagentreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models given tools to answer questions routinely fall into 'tool-abuse': they call tools that do not help, and they do so with confidence scores that overstate the chance that their reasoning is correct. The same pattern appears across model families and persists as models get more capable. To counter it, the paper proposes SMARTCAL, a fully prompt-based loop with three parts—self-evaluation that restricts which tools may be used, a confidence–accuracy lookup table built from a small held-out dev set, and an augmented reasoning step in which a calibration model edits step-wise confidence scores according to that table. Across two tool-use frameworks, three datasets, and three model families, the loop raises exact-match QA accuracy by an average of 8.6 percentage points and reduces Expected Calibration Error by an average of 21.6% relative to baselines.

What carries the argument

The load-bearing mechanism is the confidence–accuracy lookup table produced by the Confidence Prior Collection step. Verbalized confidence scores from a held-out dev set are averaged per reasoning trace, binned, and paired with the empirical accuracy of each bin; during Augmented Reasoning, a calibration model is prompted to replace each step's confidence score with the accuracy that the table associates with its confidence bin. The self-evaluation (SE) step supplies the complementary machinery: a teacher model issues task-familiarity and example-similarity verdicts that are compiled into tool-use instructions, telling the agent which tools to use, which to avoid, and when to rely on internal knowledge. Together the table and the instructions turn a single-agent tool-use loop into a multi-agent pipeline with explicit calibration feedback.

What would settle it

Run SMARTCAL with a deliberately shuffled confidence–accuracy table (pairing each confidence bin with the accuracy of a different bin). If the reported ~21.6% Expected Calibration Error reduction is still reproduced, the calibration model is not genuinely using the prior, and the claimed mechanism collapses.

Watch

Extended reading notes

Core claim

The central discovery is that LLM tool-use agents systematically overconfident-misuse tools: in over 20% of reasoning steps a model calls at least one tool that does not help, and more than 90% of stated confidence falls in bins where reported confidence exceeds actual accuracy. SMARTCAL addresses this by constraining tool selection through a teacher-model self-evaluation, grounding verbalized confidence in an empirical confidence–accuracy table computed on a small dev set, and then having a calibration model edit each step's confidence score toward the accuracy associated with its bin. The paper reports that this three-stage loop is compatible with two existing frameworks—ART and DSP—and yields an average 8.6% increase in QA accuracy and a 21.6% decrease in ECE against baselines, with ablations showing both the self-evaluation and the confidence prior contribute.

Load-bearing premise

The load-bearing premise is that a confidence–accuracy lookup table computed from 200 to 500 held-out dev questions accurately represents the confidence–accuracy relationship on the test set, and that the calibration model's prompt-following edits faithfully apply that table to test confidence scores.

Editorial extensions

If this is right

  • SMARTCAL raises exact-match QA accuracy by an average of 8.6 percentage points over baseline ART and DSP frameworks.
  • It lowers Expected Calibration Error by an average of 21.6% compared to verbalized-confidence baselines, so agents' stated confidence better tracks their true correctness.
  • The gains appear across three model families and three datasets, suggesting the calibration loop is model- and framework-agnostic rather than a one-off prompt tuning.
  • Ablations show self-evaluation alone contributes about 2.9% accuracy and 21.6% lower ECE, while the confidence prior alone lowers ECE by about 39.4%, implying both components work partly independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dev-set confidence–accuracy table is a lightweight, model-specific prior; a natural extension is to refresh it online as new confident answers accumulate, which would make SMARTCAL adaptive to distribution shift.
  • The same three-stage loop—constrain tool choice, build an empirical confidence prior, and edit step-wise scores—could apply to other agentic settings such as code execution or web navigation, where overconfident tool calls also erode trust.
  • The paper's finding that calibration error stays roughly flat as model capability grows implies that simply scaling models will not fix overconfidence; calibration-aware prompting or training may need to be a separate axis of improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies tool-use calibration in LLM-based agents. It first reports that LLMs exhibit 'tool-abuse' — using unnecessary tools with overconfident verbalized confidence scores — and that calibration errors remain high across model families. It then proposes SMARTCAL, a multi-agent framework with three components: Self-Evaluation (SE), which generates tool-use instructions from task familiarity and example similarity; Confidence Prior Collection (CPC), which builds a confidence-accuracy lookup table on a heldout dev set; and Augmented Reasoning (AR), in which a calibration model edits step-wise confidence scores according to that table. The authors report that SMARTCAL improves QA accuracy by an average of 8.6 percentage points and reduces ECE by 21.6% across two tool-use frameworks (ART, DSP), three datasets (Mintaka, PopQA, Entity Questions), and three model families (gpt-3.5-turbo, gpt-4, llama-3-70b-instruct). An ablation study on Mintaka isolates the contributions of the SE and CPC components.

Significance. If the reported improvements are robust, SMARTCAL addresses a real and understudied problem: LLM agents often use tools inappropriately and with poorly calibrated confidence. The framework is methodologically transparent — the authors release code and data, provide detailed prompts in the appendix, and present ablations of each component. The use of two tool-use frameworks and three model families is a strength, as is the focus on a heldout confidence prior rather than an in-sample fit. However, the paper's central calibration claim is currently weakened by a known artifact in the ECE computation (zero-filling unextracted confidence scores) and by the absence of a deterministic recalibration baseline. With a recomputation of ECE and a stronger baseline comparison, the contribution would be a useful step toward self-aware tool-use evaluation.

major comments (4)
  1. [A.3.2, Table 3] The ECE results are contaminated by the convention, stated in Section A.3.2, of defaulting unextracted confidence scores to zero. Under ECE, all such cases fall into the lowest confidence bin; if those cases are also wrong, the model appears nearly perfectly calibrated. This is exactly what happens for gpt-3.5-turbo on PopQA, where refusals are common (A.3.1): ART(V) receives ECE 0.010 while SMARTCAL receives 0.087, meaning SMARTCAL looks worse precisely because it avoids the zero-fill artifact. The 21.6% average ECE reduction in Section 4.2 is computed over a table that includes this cell, so the headline calibration improvement is not trustworthy. Please recompute ECE without the zero-fill convention (e.g., excluding unextracted cases, or treating them as missing) and report whether the average reduction persists.
  2. [4.2, Appendix A.2.2] The paper does not compare SMARTCAL's calibration editing to a deterministic recalibration baseline. Since the CPC module produces an empirical confidence-accuracy table, a natural baseline is to replace each step's verbalized confidence with the corresponding bin's empirical accuracy (or to apply standard histogram binning / isotonic regression on the dev set). Without such a baseline, it is unclear whether the improvement comes from the table itself or from the calibration model's prompt-driven edits. Please add a deterministic recalibration baseline to Table 3 and to the ablation in Table 5.
  3. [1, 4.1, 4.3] The introductory statistics on tool-abuse — 'over 20% of total reasoning steps' involve tool misuse and 'more than 90%' of stated confidence falls in an overconfident bin — are not backed by a dedicated result table. Section 4.1 gives aggregate accuracy, and Figure 4 shows tool-frequency for one model/dataset, but there is no quantitative definition of 'misuse' or a per-model, per-dataset table supporting the over-20% and over-90% claims. Similarly, the claim in Figure 3 that calibration error remains stable regardless of model capability is based on a small set of points with no error bars or significance test. Please add a table with precise metrics (e.g., fraction of steps where the chosen tool is not among the teacher-selected useful tools; fraction of confidence scores where confidence exceeds bin accuracy) across all models and datasets.
  4. [2.2, A.1] The confidence-accuracy lookup table is the core of the calibration mechanism, but it is constructed from only 200–500 dev questions and uses coarse confidence bins (stepsize 10, as shown in the CPC block of Figure 1). The paper does not report the number of samples per bin or any uncertainty estimate (e.g., bootstrap confidence intervals). Given that the table is used to edit test confidences, the reader cannot assess how sensitive the ECE results are to bin-boundary choices or to small bin counts. Please report per-bin sample sizes and bootstrap intervals, and discuss how the dev distribution transfers to the test set.
minor comments (6)
  1. [Abstract, 4.1] The abstract says '8.6 percent increase in QA performance'; the underlying numbers in Table 2 are absolute percentage-point increases (e.g., from 45.4% to 53.0%). Please clarify whether the claim is relative or absolute, and use consistent terminology.
  2. [Table 6] The DSP (V) prompt in Table 6 contains a typo: 'confidence socre' should be 'confidence score'.
  3. [Table 11] The calibration prompt in Table 11 contains a typo: 'resaoning process' should be 'reasoning process'.
  4. [Table 2 caption] The caption says 'accessed between Feburary 2024 to June 2024'; 'Feburary' is a typo for 'February'.
  5. [2.3] In the AR description, the phrase 'the calibration model h(x, d) interacts with both tool-use agent result and confidence prior' would benefit from stating explicitly which model (gpt-3.5-instruct-0914) is used, as is done in A.2.1, to avoid ambiguity.
  6. [4.3, Figure 5] In the Figure 5 caption and its referring text, the zero-confidence interval is attributed to 'regular expressions failed to extract a valid confidence score'; this should be stated in the main text as well, since it directly affects interpretation of the calibration curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SMARTCAL's ECE and accuracy gains are heldout transfer results from a disjoint dev set to test, not constructions equivalent to their inputs.

full rationale

Walking the derivation chain: SE generates instructions from a teacher model; CPC builds a confidence-accuracy lookup table on a heldout dev set; AR uses a calibration model to edit test-split confidence scores toward that table. The dev and test sets are disjoint (Mintaka: 200 dev vs 151 test; PopQA: 200 dev from training vs 2,349 test; Entity Questions: 500 dev from training vs test), so the reported ECE reduction is an empirical transfer measurement rather than a fitted parameter renamed as a prediction. The paper explicitly acknowledges in A.3.2 that zero-filling unextracted confidence scores can produce misleading 'perfectly calibrated' ECE values in low-accuracy refusal cases; this is a metric-artifact correctness caveat affecting the aggregate 21.6% claim, but it is not circular because the SMARTCAL pipeline is not defined in terms of the test ECE it reports. Self-citations in the paper are background references and are not load-bearing for the central framework: no uniqueness theorem, ansatz, or defining identity is imported from the authors' prior work. Therefore no step reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework's confidence prior and self-evaluation steps rest on a small heldout dev set and a teacher model, but these are standard empirical assumptions rather than new physical postulates. No new entities are introduced.

free parameters (3)
  • confidence bin stepsize = not specified exactly; Figure 1 shows intervals [5, 45, 55, 65, 75, 85]
    Used to bin dev-set confidence scores into a lookup table; the binning choice directly affects the calibration table and the resulting ECE.
  • low-popularity threshold = log pageviews less than 2
    Defines the test subset for all datasets; changing this threshold changes the distribution and the measured tool-use behavior.
  • dev set size = 200 for Mintaka and PopQA, 500 for Entity Questions
    The confidence prior is estimated from these small samples; the choice is arbitrary and affects the reliability of the lookup table.
assumptions (3)
  • domain assumption The confidence-accuracy distribution on the heldout dev set approximates the distribution on the test set.
    Required for the CPC lookup table to transfer. Stated in Section 2.2 as 'approximation of the underlying confidence-accuracy distribution on the test dataset'.
  • domain assumption Verbalized confidence scores extracted via regular expressions are faithful to the model's step-wise certainty.
    The method relies on extracting tool names and confidence from reasoning text. Appendix A.3.2 admits that regex failures are defaulted to zero, which can distort the calibration estimate.
  • domain assumption gpt-4-turbo as teacher provides accurate self-evaluation of task familiarity and example similarity for all tested models.
    The SE instructions are generated by gpt-4-turbo and assumed to be correct and transferable; no human validation or cross-model check is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration." pith.science (2026). https://pith.science/paper/H6MBDCVD

@misc{pith2026241212151,
  author       = {Pith},
  title        = {Pith review of: SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H6MBDCVD}},
  note         = {Machine review of arXiv:2412.12151}
}
read the original abstract

The tool-use ability of Large Language Models (LLMs) has a profound impact on a wide range of industrial applications. However, LLMs' self-control and calibration capability in appropriately using tools remains understudied. The problem is consequential as it raises potential risks of degraded performance and poses a threat to the trustworthiness of the models. In this paper, we conduct a study on a family of state-of-the-art LLMs on three datasets with two mainstream tool-use frameworks. Our study reveals the tool-abuse behavior of LLMs, a tendency for models to misuse tools with overconfidence. We also find that this is a common issue regardless of model capability. Accordingly, we propose a novel approach, \textit{SMARTCAL}, to mitigate the observed issues, and our results show an average of 8.6 percent increase in the QA performance and a 21.6 percent decrease in Expected Calibration Error (ECE) compared to baseline models.

Figures

Figures reproduced from arXiv: 2412.12151 by the authors.

Figure 1
Figure 1. Comparison between ART (Paranjape et al., 2023), ART (V), and SMARTCAL on the complex QA task. ART (V) introduces verbalized confidence elicitation. SMARTCAL includes three steps to mitigate tool-abuse. answer type]” as the useful tools from the fil￾tered tool-use examples in the similarity evaluation, and the familiarity evaluation results encourage the tool-use model f(x) to incorporate “[Internal Knowledge]” as a… view at source ↗
Figure 2
Figure 2. Distribution of entity popularity for Mintaka, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. ECE and QA accuracy trend comparison on Mintaka dataset. ECE scores remain stable despite in￾creasing model capability. perspectives as detailed below. SMARTCAL improves performance by mitigat￾ing tool-misuse. Previous work has shown the necessity of retrieval under low popularity context (Mallen et al., 2023). We further show that tool￾misuse may also exert a negative effect on the an￾swering accuracy [PITH_FULL_I… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Tracking of tool usage from Entity Questions [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: provides a comparison of the reliance plot of gpt-4 on Entity Questions data. Note that the zero confidence interval represents the questions where regular expressions failed to extract a valid confidence score from the agent’s reasoning history. A full comparison of c…
Figure 6
Figure 6. Figure 6: Examples of gpt-3.5-turbo reasoning his￾tory on PopQA dataset using ART. A.3.2 Tool-Use Calibration Following the calibration performance in [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Tool-use comparison between ART and SMARTCAL [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: ECE plot comparison between ART (V) and SMARTCAL [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.org

    cs.AI 2025-12 reject novelty 6.0 of 10

    An open-source agentic materials-design platform shows tool access can help or hurt accuracy depending on the property, but its headline memorization-resistant test results are not presented.

Reference graph

Works this paper leans on

44 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  4. [4]

    Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2024. https://openreview.net/forum?id=qV83K9d5WB Large language models as tool makers . In The Twelfth International Conference on Learning Representations

    2024

  5. [5]

    Smith, and Tao Yu

    Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. https://openreview.net/forum?id=lH1PV42cbF Binding language models in symbolic languages . In The Eleventh International Conference on Learning Representations

    2023

  6. [6]

    Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. https://openreview.net/forum?id=3Pf3Wg6o-A4 Selection-inference: Exploiting large language models for interpretable logical reasoning . In The Eleventh International Conference on Learning Representations

    work page 2023

  7. [7]

    Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.findings-acl.203 DARA : Decomposition-alignment-reasoning autonomous language agent for question answering over knowledge graphs . In Findings of the Association for Computational Linguistics ACL 2024, pages 3406--3432, Bangkok, Thailand and virtual meeting. Association ...

    doi:10.18653/v1/2024.findings-acl.203 2024

  8. [8]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321--1330. PMLR

    2017

Show all 44 references
  1. [9]

    Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/8fd1a81c882cd45f64958da6284f4a3f-Paper-Conference.pdf Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings . In Advances in Neur...

    2023

  2. [10]

    Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. 2023. Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901

    2023 arXiv

  3. [11]

    Jinhao Jiang, Kun Zhou, zican Dong, KeMing Ye, Xin Zhao, and Ji-Rong Wen. 2023. https://openreview.net/forum?id=R635gF7lXD Struct GPT : A general framework for large language model to reason over structured data . In The 2023 Conference on Empirical Methods in Natural Language...

    2023

  4. [12]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  5. [13]

    Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. 2023. https://doi.org/10.18653/v1/2023.acl-long.361 LAMBADA : Backward chaining for automated reasoning in natural language . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  6. [14]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. https://doi.org/10.48550/arXiv.2310.03714 Dspy: Compiling ...

    2023 doi arXiv

  7. [15]

    Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, M...

    2023

  8. [16]

    Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-industry.39 Are C hat GPT and GPT -4 general-purpose solvers for financial text analytics? a study on several typical tasks . In Proceed...

    2023 doi

  9. [17]

    Yihao Li, Ru Zhang, and Jianyi Liu. 2024. An enhanced prompt-based llm reasoning scheme via knowledge graph-integrated collaboration. In Artificial Neural Networks and Machine Learning -- ICANN 2024, pages 251--265, Cham. Springer Nature Switzerland

    2024

  10. [18]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://openreview.net/forum?id=8s8K2UZGTZ Teaching models to express their uncertainty in words . Transactions on Machine Learning Research

    2022

  11. [19]

    Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2024. https://doi.org/10.18653/v1/2024.naacl-long.308 Retrieval helps or hurts? a deeper dive into the efficacy of retrieval augmentation to language models . In Proceedings of the 2024 Conference of the North Ame...

    2024 doi

  12. [20]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023

  13. [21]

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682--15694

    2021

  14. [22]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29

    2015

  15. [23]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022...

    2022 arXiv

  16. [24]

    Stephen Obadinma, Hongyu Guo, and Xiaodan Zhu. 2021. Class-wise Calibration : A Case Study on COVID -19 Hate Speech . In Canadian AI 2021 . Canadian Artificial Intelligence Association (CAIAC). Https://caiac.pubpub.org/pub/vd3v9vby

    2021

  17. [25]

    Stephen Obadinma, Xiaodan Zhu, and Hongyu Guo. 2024. https://openreview.net/forum?id=TXzz9xwdpv Calibration attacks: A comprehensive study of adversarial attacks on model confidence . Submitted to Transactions on Machine Learning Research. Under review

    2024

  18. [26]

    OpenAI. 2023. https://chat.openai.com/chat Chatgpt (mar 14 version) [large language model] . Technical report

    2023

  19. [27]

    Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2023. https://doi.org/10.18653/v1/2023.acl-long.68 T hink S um: Probabilistic reasoning over sets using large language models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023 doi

  20. [28]

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014

    2023 arXiv

  21. [29]

    Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom Mitchell, and Estevam Hruschka. 2024. Reasoning capacity in multi-agent systems: Limitations, challenges and human-centered solutions. arXiv preprint arXiv:2402.01108

    2024 arXiv

  22. [30]

    Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, and Iryna Gurevych. 2024. http://arxiv.org/abs/2401.10065 Code prompting elicits conditional reasoning abilities in text+code llms . Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    2024 arXiv

  23. [31]

    Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing Jungwon Byun, Maggie Appleton, and Andreas Stuhlm \"u ller. 2023. Iterated decomposition: Improving science q&a by supervising reasoning processes. arXiv preprint arXiv:2301.01751

    2023 arXiv

  24. [32]

    Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.437 How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages...

    2020 doi

  25. [33]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.496 Simple entity-centric questions challenge dense retrievers . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page...

    2021 doi

  27. [35]

    Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. https://aclanthology.org/2022.coling-1.138 Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering . In Proceedings of the 29th International Conference on Computational Linguistics, pages 1...

    2022

  28. [36]

    Hua Shen, Tiffany Knearem, Reshmi Ghosh, Kenan Alkiek, Kundan Krishna, Yachuan Liu, Ziqiao Ma, Savvas Petridis, Yi-Hao Peng, Li Qiwei, et al. 2024. Towards bidirectional human-ai alignment: A systematic review for clarifications, framework, and future directions. arXiv preprin...

    2024

  29. [37]

    Lei Sun and Youdi Li. 2024. https://doi.org/10.11517/pjsai.JSAI2024.0_3Xin2111 Pyramid of thought: A novel approach for enhancing chain-of-thought reasoning in large language models . Proceedings of the Annual Conference of JSAI, JSAI2024:3Xin2111--3Xin2111

    2024 doi

  30. [38]

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. https://openreview.net/forum?id=WPZ2yPag4K Fine-tuning language models for factuality . In The Twelfth International Conference on Learning Representations

    2024

  31. [39]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. https://openreview.net/forum?id=g3faCfrwm7 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models...

    2023

  32. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  34. [42]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. https://openreview.net/forum?id=gjeQKFxFpZ Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s . In The Twelfth International Conference on Learn...

    2024

  35. [43]

    Hui Yang, Sifu Yue, and Yunzhong He. 2023. http://arxiv.org/abs/2306.02224 Auto-gpt for online decision making: Benchmarks and additional opinions . arXiv preprint arXiv:2306.02224

    2023 arXiv

  36. [44]

    Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.296 Re3: Generating longer stories with recursive reprompting and revision . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, page...

    2022 doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.