Pith. sign in

REVIEW 3 major objections 4 minor 60 references

MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Model-internal confidence estimators (MICE) decode every transformer layer, compare each layer's provisional answer to the final tool call, and use those similarities to produce calibrated confidence scores that increase expected…

desk verdict A careful empirical paper on layerwise-internal confidence for tool-calling agents; the utility metric is useful and the experiments are solid, but exact-match labels on a single synthetic dataset leave the headline gains vulnerable to a semantic-equivalence artifact. read the letter →

arxiv 2504.20168 v1 pith:LYBEBT63 submitted 2025-04-28 cs.CL cs.AIcs.LG

Nishant Subramani , Jason Eisner , Justin Svegliato , Benjamin Van Durme , Yu Su , Sam Thomson This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords confidencecalibrationtool-usingagentsmodelinternalslogitlensBERTScoreexpectederrortool-callingutilitylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tool-using agents need to know when to trust the API call they just generated, because a wrong call can be cheap or catastrophic. This paper proposes MICE, a family of confidence estimators that read the model's own internal trajectory: it decodes a preliminary answer from each transformer layer, compares each layer's answer to the final one with BERTScore, and feeds those similarities plus the model's raw token confidence into a small trained classifier. On the simulated trial-and-error tool-calling dataset with three Llama3 models, MICE matches or beats histogram and kernel recalibration baselines on smooth expected calibration error and, more importantly, improves a new utility metric that charges a cost for incorrect calls. The advantage appears as the cost of a wrong call rises: at medium and high risk levels, executing a call only when calibrated confidence exceeds a Bayes-optimal threshold yields higher expected utility than the baselines. A random-forest version is the strongest, and it retains its edge with only a few hundred training examples and generalizes to unseen APIs.

What carries the argument

The load-bearing object is the per-layer decoding feature: each transformer layer's hidden state is mapped through the unembedding matrix to produce a provisional token, so the whole layer stack yields a sequence of provisional strings $y^{(1)}, \dots, y^{(\ell)}$ that gradually converge to the final output $y$ (logit lens). BERTScore, a string-similarity measure built from a separate BERT-family embedder, turns each provisional string into a number measuring how close that layer's answer already is to the final answer. The hypothesis the method tests is that a tool call whose answer was already stable across the later layers is more likely to be correct than one that snaps into place only at the final layer, even when both end with identical token probabilities; the learned logistic-regression or random-forest classifier is what reads that trajectory.

What would settle it

Re-annotate the STE test set with semantic-equivalence judgments (for example, two annotators or a judge model decide whether each generated call would accomplish the same action as the gold call), then retrain and re-evaluate MICE against HRE and NWKR on that relabeled set. If the expected-tool-calling-utility advantage at medium and high risk disappears or shrinks to insignificance, the central claim is an artifact of exact-match labeling rather than a property of internal-layer confidence.

Watch

Extended reading notes

Core claim

At the paper's core is the claim that the way a language model's predictions evolve across its layers tells you whether its final tool call can be trusted. MICE operationalizes this by taking, for each layer $i$, the layerwise hidden state $h^{(i)}_{t-1}$, multiplying by the unembedding matrix $W_{\mathrm{out}}$ to get logits, argmax-decoding a preliminary string $y^{(i)}$, and scoring the similarity of each $y^{(i)}$ to the final string $y$ with BERTScore. A trained classifier maps these layer-similarity features, together with the product of non-formatting token probabilities, to a probability that the call is correct. The paper's central empirical finding is that when these probabilities drive a minimum-Bayes-risk call/no-call decision, they beat both a histogram regressor and a Nadaraya–Watson kernel regressor on expected tool-calling utility at medium risk for all three tested models and at high risk for two of them, while staying statistically tied on smooth ECE. The random-forest version is the strongest configuration; it is also sample-efficient, matching the full-data kernel baseline with about one-fifth of the training set, and zero-shot transfer to held-out APIs.

Load-bearing premise

The entire experiment depends on the STE dataset's binary label, which marks a generated tool call correct only if it exactly equals the gold call string; if many correct calls differ in wording, formatting, or argument order, the labels are noisy and the measured utility gains may partly reflect surface-form matching rather than true tool-calling ability.

Editorial extensions

If this is right

  • With MICE confidence and the risk-adjusted threshold of Section 3.2, an agent can abstain from executing a tool call when confidence is low, improving expected utility in medium- and high-risk settings relative to recalibrated raw confidence.
  • Because MICE matches the calibration-focused baselines on smooth ECE while improving utility, calibration alone is not sufficient: the paper's ETCU metric rewards estimators that separate correct from incorrect calls, and MICE does so better.
  • MICE is sample-efficient: the random-forest variant reaches or exceeds the full-data kernel regressor with roughly 300 training examples, so the internal-layer features are informative even with small labeled sets.
  • MICE generalizes across tools: models trained on 49 of the 50 STE APIs perform indistinguishably from baselines trained on all APIs when evaluated on the held-out API.
  • Raw token confidence is the weakest estimator in high-risk regimes and can yield negative utility; MICE's core value is recovering safe behavior from an overconfident base generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the paper does not pursue is using MICE-style features for continual calibration at deployment time, where the classifier is updated online as new tool calls arrive; the sample-efficiency result suggests this could work.
  • A sharper test of the mechanism would be to artificially perturb only the final layers of a model, making the decoded trajectory disagree with the final output; MICE should then withhold confidence even when raw confidence is high, which would isolate whether the layer-trajectory signal is causal rather than correlational.
  • Because MICE requires hidden states, an implicit consequence for closed models is that similar estimators would need to be trained on logprobs or other exposed signals; the gap between MICE and raw confidence suggests what closed APIs lose by hiding internals.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes MICE, a family of confidence estimators for tool-calling LLM agents. For each generated tool call, MICE decodes per-token predictions from intermediate layers via logit lens, computes BERTScore between each layer's decoding and the final greedy output, adds the raw token-level confidence as an extra feature, and trains either a logistic regressor or a random forest to predict whether the call is correct. The paper also introduces expected tool-calling utility (ETCU), a normalized decision-theoretic metric with a Bayes-optimal threshold, and evaluates on the simulated trial-and-error (STE) dataset with three Llama-3 models, comparing against raw confidence, histogram regression (HRE), and Nadaraya-Watson kernel regression (NWKR). The reported results show that MICE RF matches or improves smECE, yields significantly higher ETCU than HRE/NWKR at medium risk and often at high risk, remains competitive in zero-shot API generalization, and needs only a few hundred training examples.

Significance. The contribution is potentially useful: if confirmed, MICE gives a way to exploit model internals for safer call/no-call decisions, and ETCU is a sensible metric for comparing confidence estimators under asymmetric risk. The paper is transparent in several ways: code is released, statistical significance is assessed by permutation tests, feature ablations are reported, and zero-shot generalization is examined. These are real strengths. The main caveat is that the empirical support rests on a single synthetic dataset with labels defined by exact string match, and the baseline set omits the most standard recalibration methods, so the breadth of the claims in the title and abstract is not yet fully supported.

major comments (3)
  1. [§4.1] Section 4.1 states that 'we label a generated tool call as correct if and only if it exactly matches the one given by STE.' Tool calls generally admit many string-distinct but semantically equivalent serializations (argument order, quoting, whitespace, aliases), so this label is a surrogate for true correctness. Because the same labels are used both to train MICE and to compute the ETCU values in Table 1 and Figure 4, the reported utility improvements over HRE and NWKR could reflect skill at predicting the exact-match surrogate rather than at identifying calls that would succeed in the environment. The Limitations section does not discuss this. I ask for a label-validation analysis: at least a human-annotated or execution-based sample quantifying the mismatch rate, and a rerun of Table 1 under a semantically normalized correctness label, to show that the MICE advantage survives.
  2. [§4.4, Table 1] Section 4.4 compares MICE against raw confidence, HRE, and NWKR, but not against Platt scaling or isotonic regression, even though these are the standard recalibration baselines and are cited in §7. Since MICE LR is described as an extension of Platt scaling, the experiments do not currently isolate the contribution of the model-internal BERTScore features from the contribution of ordinary logistic recalibration. Adding Platt scaling and isotonic regression to Table 1 and Figure 4 would make the claim that MICE 'beats or matches strong baselines' directly supported.
  3. [§5, Table 1] The abstract and conclusion state that MICE 'significantly improves' ETCU in medium- and high-risk regimes, but the permutation tests summarized in the text show that MICE RF is not significant at high risk for Llama3.2 and that MICE LR is only significant at medium risk for Llama3.1. The paper's own summary is more cautious, but the abstract overstates uniformity; the claims should be aligned with the significance pattern.
minor comments (4)
  1. [§3.2] The phrase 'with loss of generality' before assuming tn=fn=0 is not accurate; this is a modeling simplification, and the following sentence's 'slightly less general' should be the only framing.
  2. [Table 1] The significance markers are placed immediately before negative numbers (e.g., '*-0.015∗†'), which is easy to misread as part of the value; the caption should clarify the marker placement.
  3. [§6] The phrase 'as suggested by reviewers' is an artifact of the review process and should be removed or replaced with a neutral motivation.
  4. [§7] Since Platt scaling and isotonic regression are mentioned in §7, a sentence explaining why only HRE and NWKR are used as main baselines would help readers understand the design choice.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MICE is an empirical estimator evaluated against external STE labels and an independently defined utility metric.

full rationale

The derivation chain is empirical and self-contained. MICE features are computed from model internals (logit lens decodes, BERTScore similarities to the final output, and raw token confidence), and a supervised classifier is trained on correctness labels derived from exact string match to the STE gold calls. The ETCU metric is defined independently from first principles in Eqs. (2) and (3) as expected utility under a Bayes-optimal threshold, with the four utility outcomes parameterized externally; it is not used to define the training objective or the correctness labels. Experiments compare against recalibration baselines HRE and NWKR on a held-out test set, so the reported gains are measured rather than forced by construction. The only overlap with prior work is that the STE dataset and some interpretability citations share authors, but no load-bearing claim is justified solely by those citations: the dataset is an external artifact used for evaluation, and the interpretability citations only motivate a hypothesis that is then tested empirically. The exact-match label validity is a legitimate measurement concern, but it is not a circularity because the prediction target is externally anchored to the STE benchmark rather than defined in terms of MICE's own outputs.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method is a supervised classifier, so its parameters are learned from training data. The entries above list the hand-chosen hyperparameters, the utility values that define the evaluation scenarios, and the domain assumptions about the dataset and feature representations that the claim depends on. No new physical or conceptual entity is postulated.

free parameters (5)
  • MICE LR L2 regularization strength = 2
    Chosen by hand in Section 4.4; it determines the logistic regression classifier's behavior and affects calibration.
  • MICE RF tree count = 1000
    Set in Section 4.4; random forest performance is sensitive to ensemble size.
  • MICE RF max depth = 20
    Set in Section 4.4; controls model capacity of the random forest.
  • MICE RF max features per split = 10
    Set in Section 4.4; controls feature subsampling in the random forest.
  • ETCU risk-level utility fp = -9, -1, -1/9
    Chosen in Section 3.2 to define high, medium, and low risk settings for the utility metric; these thresholds are not learned from data.
assumptions (6)
  • domain assumption A generated tool call is correct iff it exactly matches the STE gold tool call.
    Stated in Section 4.1; this binary label is the training target for MICE and the basis for all calibration and utility measurements. If multiple valid tool-call phrasings exist, labels are noisy.
  • domain assumption The STE dataset's synthetic labels, produced by a GPT-3.5-turbo simulation, are reliable enough to train and evaluate confidence estimators.
    Section 4.1 describes the simulation as presumptively correct; the paper does not independently verify the gold calls.
  • domain assumption BERTScore computed with DeBERTa-xlarge-mnli is a meaningful similarity measure between intermediate-layer decodes and the final output.
    Sections 2 and 4.4; the entire feature set depends on this semantic-similarity proxy. The paper notes the alignments were not always trivial.
  • domain assumption Intermediate layer hidden states can be projected through the unembedding matrix to produce meaningful per-token decodes (logit lens assumption).
    Section 2, Equation 1 uses h(i) W_out for i < ell; this is the logit lens technique, valid for residual-stream transformers with tied output embeddings but not guaranteed for all architectures.
  • standard math The Bayes decision rule with the stated utilities gives the optimal call/no-call policy.
    Section 3.2 derives Equation 2 from standard minimum Bayes risk; the assumptions on tp, fp, tn, fn are explicit.
  • standard math smECE with the reflected Gaussian kernel is a consistent calibration metric.
    Section 3.1 relies on Blasiok and Nakkiran (2024) for the consistency of kernel-smoothed ECE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools." pith.science (2026). https://pith.science/paper/LYBEBT63

@misc{pith2026250420168,
  author       = {Pith},
  title        = {Pith review of: MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYBEBT63}},
  note         = {Machine review of arXiv:2504.20168}
}
read the original abstract

Tool-using agents that act in the world need to be both useful and safe. Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions, but prior work shows that many models are poorly calibrated. Inspired by interpretability literature exploring the internals of models, we propose a novel class of model-internal confidence estimators (MICE) to better assess confidence when calling tools. MICE first decodes from each intermediate layer of the language model using logitLens and then computes similarity scores between each layer's generation and the final output. These features are fed into a learned probabilistic classifier to assess confidence in the decoded output. On the simulated trial and error (STE) tool-calling dataset using Llama3 models, we find that MICE beats or matches the baselines on smoothed expected calibration error. Using MICE confidences to determine whether to call a tool significantly improves over strong baselines on a new metric, expected tool-calling utility. Further experiments show that MICE is sample-efficient, can generalize zero-shot to unseen APIs, and results in higher tool-calling utility in scenarios with varying risk levels. Our code is open source, available at https://github.com/microsoft/mice_for_cats.

Figures

Figures reproduced from arXiv: 2504.20168 by the authors.

Figure 1
Figure 1. The MICE architecture. tures by decoding from the intermediate layers of a transformer-based large language model (LLM) and computes the similarities of those generations to the output of the final layer. Based on these features and the LLM’s raw confidence, it learns a model that outputs a confidence score. MICE excels on ETCU, increasingly outperforming two strong baselines as the cost of incorrect tool calls incr… view at source ↗
Figure 3
Figure 3. BERTScore similarities between the gener [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Expected tool-calling utility on the test set at varying risk levels. We include four trivial policies for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Feature importance for BERTScore features [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Sample complexity: AUC of MICE models and HRE baselines as the size of the training set varies on the Llama3-8B-Instruct model. Error bars are one standard deviation. half of the layers’ BERTScores plus confidence; second half of the layers’ BERTScores plus confi￾dence…
Figure 8
Figure 8. Figure 8: A version of Figure [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 24 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ayer, Hugh D

    Miriam C. Ayer, Hugh D. Brunk, George M. Ewing, W. T. Reid, and Edward Silverman. 1955. https://api.semanticscholar.org/CorpusID:121836360 An empirical distribution function for sampling with incomplete information . Annals of Mathematical Statistics, 26:641--647

    work page 1955

  4. [4]

    Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=rJVjQSQ8ye Linguistic calibration of long-form generations . In Forty-first International Conference on Machine Learning

    work page 2024

  5. [5]

    Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor V. Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://api.semanticscholar.org/CorpusID:257504984 Eliciting latent predictions from transformers with the tuned lens . ArXiv, abs/2303.08112

    arXiv 2023

  6. [6]

    Peter Bickel and Kjell Doksum. 1977. https://doi.org/10.2307/2286373 Mathematical Statistics: Basic Ideas and Selected Topics. , volume 56. Holden-Day Inc

    work page doi:10.2307/2286373 1977

  7. [7]

    Jaros aw B asiok and Preetum Nakkiran. 2024. https://openreview.net/forum?id=XwiA1nDahv Smooth ECE : Principled reliability diagrams via kernel smoothing . In The Twelfth International Conference on Learning Representations

    work page 2024

  8. [8]

    John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. https://aclanthology.org/C04-1046 Confidence estimation for machine translation . In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics , pages 315--321, Geneva, Switzerland. COLING

    work page 2004

Show all 60 references
  1. [9]

    Björn Böken. 2021. https://doi.org/10.1016/j.is.2020.101641 On the appropriateness of P latt scaling in classifier calibration . Information Systems, 95:101641

    2021

  2. [10]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30

    2017

  3. [11]

    Philip Dawid

    A. Philip Dawid. 1982. https://api.semanticscholar.org/CorpusID:121781338 The well-calibrated B ayesian . Journal of the American Statistical Association, 77:605--610

    1982

  4. [12]

    Shrey Desai and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.21 Calibration of pre-trained transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295--302, Online. Association for Computational...

    2020 doi

  5. [13]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The Llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  6. [14]

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710

    2024 arXiv

  7. [15]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. https://doi.org/10.1038/s41586-024-07421-0 Detecting hallucinations in large language models using semantic entropy . Nature, 630(8017):625--630

    2024 doi

  8. [16]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural L...

    2022 doi

  9. [17]

    Adam Gleave, Michael D Dennis, Shane Legg, Stuart Russell, and Jan Leike. 2021. https://openreview.net/forum?id=LwEQnp6CYev Quantifying differences in reward functions . In International Conference on Learning Representations

    2021

  10. [18]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. https://arxiv.org/abs/1706.04599 [cs] On calibration of modern neural networks . Preprint, arxiv:1706.04599 [cs]

    2017 arXiv

  11. [19]

    Helia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme, and Chris Kedzie. 2024. https://aclanthology.org/2024.acl-long.745/ LLM-Rubric : A multidimensional, calibrated approach to automated evaluation of natural language texts . In Proceedings of the 62nd Annual Meeting...

    2024

  12. [20]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations

    2021

  13. [21]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00407 How can we know when language models know? O n the calibration of language models for question answering . Transactions of the Association for Computational Linguistics, 9:962--977

    2021 doi

  14. [22]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 doi arXiv

  15. [23]

    Aviral Kumar and Sunita Sarawagi. 2019. https://api.semanticscholar.org/CorpusID:67855916 Calibration of encoder decoder models for neural machine translation . ArXiv, abs/1903.00802

    2019 arXiv

  16. [24]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    J.I. Marcum. 1960. A statistical theory of target detection by pulsed radar. IRE Transactions on Information Theory, 6(2):59--267

    1960

  18. [26]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. https://doi.org/10.18653/v1/2024.naacl-long.281 Language models implement simple W ord2 V ec-style vector arithmetic . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024 doi

  19. [27]

    Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

    Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. https://doi.org/10.1162/tacl_a_00494 Reducing conversational agents ' overconfidence through linguistic calibration . Transactions of the Association for Computational Linguistics, 10:857--872

    2022 doi

  20. [28]

    Elizbar A Nadaraya. 1964. On estimating regression. Theory of Probability & Its Applications, 9(1):141--142

    1964

  21. [29]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2014. https://api.semanticscholar.org/CorpusID:7982059 Binary classifier calibration: Non-parametric approach . ArXiv, abs/1401.3390

    2014 arXiv

  22. [30]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. 2015. https://api.semanticscholar.org/CorpusID:6292807 Obtaining well calibrated probabilities using bayesian binning . Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artific...

    2015

  23. [31]

    Alexandru Niculescu-Mizil and Rich Caruana. 2005. https://www.cs.cornell.edu/ alexn/papers/calibration.icml05.crc.rev3.pdf Predicting good probabilities with supervised learning . In Proceedings of the 22nd International Conference on Machine Learning (ICML)

    2005

  24. [32]

    Andrew Nobel. 1996. https://doi.org/10.1214/aos/1032526958 Histogram regression estimation using data-dependent partitions . The Annals of Statistics, 24(3):1084 -- 1105

    1996

  25. [33]

    nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Interpreting GPT : The logit lens . Blogpost

    2020

  26. [34]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

    2011

  27. [35]

    John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61--74. MIT Press

    1999

  28. [36]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  29. [37]

    Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. 2024. Bench CLAMP : A benchmark for evaluating language models on syntactic and semantic parsing. In Proceedings of the 37th International Conference on Neural Information Pr...

    2024

  30. [38]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. https://proceedings.neurips.cc/paper_files/paper/2023/file/d842425e4bf79ba039352da0f658a906-Paper-Conference.pdf Toolformer...

    2024

  31. [39]

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456--17472

    2022

  32. [40]

    Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. 2024. https://arxiv.org/abs/2405.21028 LACIE : Listener-aware finetuning for confidence calibration in large language models . Preprint, arXiv:2405.21028

    2024 arXiv

  33. [41]

    Elias Stengel-Eskin and Benjamin Van Durme. 2023 a . https://doi.org/10.1162/tacl_a_00598 Calibrated interpretation: Confidence estimation in semantic parsing . Transactions of the Association for Computational Linguistics, 11:1213--1231

    2023 doi

  34. [42]

    Elias Stengel-Eskin and Benjamin Van Durme. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.159 Did you mean ? C onfidence-based trade-offs in semantic parsing . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2621--2629, Singapore

    2023 doi

  35. [43]

    Nishant Subramani, Samuel Bowman, and Kyunghyun Cho. 2019. Can unconditional language models recover arbitrary sentences? Advances in Neural Information Processing Systems, 32

    2019

  36. [44]

    Nishant Subramani and Nivedita Suresh. 2020. https://arxiv.org/abs/2008.09049 Discovering useful sentence representations from large pretrained language models . arXiv preprint arXiv:2008.09049

    2020 arXiv

  37. [45]

    Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. https://doi.org/10.18653/v1/2022.findings-acl.48 Extracting latent steering vectors from pretrained language models . In Findings of the Association for Computational Linguistics: ACL 2022, pages 566--581, Dublin, I...

    2022 doi

  38. [46]

    Surat Teerapittayanon, Bradley McDanel , and H. T. Kung. 2017. https://doi.org/10.48550/arXiv.1709.01686 BranchyNet : Fast inference via early exiting from deep neural networks . Preprint, arxiv:1709.01686 [cs]. Version: 1

    2017 doi arXiv

  39. [47]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. https://doi.org/10.18653/v1/P19-1452 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Italy. Association for ...

    2019 doi

  40. [48]

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248

    2023 arXiv

  41. [49]

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. 2024. https://doi.org/10.18653/v1/2024.acl-long.570 LLM s in the imaginarium: Tool learning through simulated trial and error . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024 doi

  42. [50]

    Cheng Wang. 2024. https://arxiv.org/abs/2308.01222 Calibration in deep learning: A survey of the state-of-the-art . Preprint, arXiv:2308.01222

    2024 arXiv

  43. [51]

    Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020. https://doi.org/10.18653/v1/2020.acl-main.278 On the inference calibration of neural machine translation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3070--3079, Onl...

    2020 doi

  44. [52]

    Michael L. Waskom. 2021. https://doi.org/10.21105/joss.03021 seaborn: statistical data visualization . Journal of Open Source Software, 6(60):3021

    2021 doi

  45. [53]

    Geoffrey S Watson. 1964. Smooth regression analysis. Sankhy \=a : The Indian Journal of Statistics, Series A , pages 359--372

    1964

  46. [54]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

    2024

  47. [55]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.551 Do large language models know what they don't know? In Findings of the Association for Computational Linguistics : ACL 2023 , pages 8653--86...

    2023 doi

  48. [56]

    Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2024. https://aclanthology.org/2024.lrec-main.840 Jump to conclusions: Short-cutting transformers with linear transformations . In Proceedings of the 2024 Joint International Conference on Computational Linguistic...

    2024

  49. [57]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr BERTScore : Evaluating text generation with BERT . In Proceedings of the International Conference on Learning Representations

    2020

  50. [58]

    Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh

    Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. https://doi.org/10.48550/arXiv.2102.09690 Calibrate before use: Improving few-shot performance of language models . Technical Report arXiv:2102.09690, arXiv. ArXiv:2102.09690 [cs] type: article

    2021 doi arXiv

  51. [59]

    Ruiqi Zhong, Charlie Snell, Dan Klein, and Jason Eisner. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.312 Non-programmers can label programs indirectly via active examples: A case study with text-to- SQL . In Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023 doi

  52. [60]

    Jiawei Zhou, Jason Eisner, Michael Newman, Emmanouil Antonios Platanios, and Sam Thomson. 2022. https://doi.org/10.18653/v1/2022.acl-long.110 Online semantic parsing for latency reduction in task-oriented dialogue . In Proceedings of the 60th Annual Meeting of the Association ...

    2022 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.