Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Instruction tuning consistently raises model confidence while leaving accuracy nearly unchanged, and it uniformly reduces cross-rationale lexical diversity in generated answer rationales.

desk verdict A well-executed empirical study of confidence and rationale diversity in base vs instruct-tuned checkpoints, with a causal label that is slightly bigger than the experimental design supports. read the letter →

arxiv 2608.13430 v1 pith:YIFOXS37 submitted 2026-08-13 cs.CL cs.AI

Irina Proskurina , Mayank Kumar , Oyindolapo O. Komolafe This is my paper · ORCID
classification cs.CLcs.AI
keywords instructiontuningmodelconfidenceverbalizedlexicaldiversitySelf-BLEUcalibrationquestionansweringchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that instruction tuning, the post-training step that teaches a base language model to follow natural-language instructions, has a consistent and separable effect on how 7B-8B models express uncertainty in multiple-choice question answering. Across three model families (Qwen, Mistral, Llama) and three benchmarks (ARC-Easy, MMLU, CommonsenseQA), the tuned variants assign higher probability mass to one answer, report higher verbalized confidence, and generate rationales that look more alike from one sample to the next, while accuracy improves only slightly or not at all. The paper also shows that a surface-level diversity measure (unique bigrams) moves in different directions on different benchmarks, so the constant effect is not generic 'output becoming more uniform' but specifically a collapse in cross-rationale variability. These patterns survive when the comparison is restricted to questions where both variants choose the same answer and rationales are matched for length, which the authors take as evidence that confidence and rationale diversity are distinct signatures of instruction tuning.

What carries the argument

The argument is carried by a matched-pair comparison design and three quantitative instruments. Choice entropy $H_{\text{choice}}(x)=-\sum_j p_j \log p_j/\log M$ over the candidate answers converts likelihoods into a confidence score; a two-stage verbalized-confidence prompt extracts a numerical probability after fixing the likelihood-chosen answer; and lexical diversity is scored by Unique-2 (proportion of distinct bigrams) and 1-SelfBLEU (mean dissimilarity of each rationale to the other four sampled rationales for the same question). The controlled analysis—restricting to same-answer questions and pairing rationales by length before truncation—is what lets the paper attribute the persistence of the diversity drop to the instruction-tuned generation style rather than to different answers being chosen or to longer outputs.

What would settle it

The central claim would be falsified by a single matched base/instruct pair in which instruction tuning lowered verbalized confidence or raised 1-SelfBLEU, or by a controlled re-run where the same instruction-tuning recipe on the same data produced no increase in verbalized confidence. Concretely, one could take one of the three base models, apply a purely supervised instruction-tuning step on identical data to the official checkpoint, and check whether choice entropy and verbalized confidence move in the same direction as reported in Table 1; if they do not, the attribution of the effect to instruction tuning as such fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is a paired empirical contrast: moving from a base checkpoint to its instruction-tuned counterpart raises model confidence across every model–benchmark combination—choice entropy over candidate answers falls (e.g., Qwen on MMLU from 0.430 to 0.131) and verbalized confidence rises (e.g., Llama on ARC-Easy from 49.2% to 90.4%)—while accuracy changes incompletely and even stays flat (Llama on ARC-Easy at 82.2%). At the same time, cross-rationale diversity, measured as 1 minus Self-BLEU across five chain-of-thought rationales per question, falls in every setting (e.g., Mistral on ARC-Easy from 0.813 to 0.626). The instruction-tuned models are therefore not simply more fluent or narrower in one global sense: the decline is specific to how much repeated rationales resemble each other, not to how many distinct bigrams they contain, and the divergence between the two diversity measures persists under same-answer, length-matched comparisons.

Load-bearing premise

The load-bearing premise is that each 'Instruct' checkpoint is a faithful version of its base model with instruction tuning as the only meaningful change, so that any confidence and diversity difference can be credited to instruction tuning; in reality the tuned checkpoints also differ in data mixture and possibly in preference-based or other post-training steps.

Editorial extensions

If this is right

  • Users of instruction-tuned QA models should expect self-reported confidence to overstate accuracy gains; a confidence jump of tens of points with flat accuracy can be a feature of post-training, not evidence of competence.
  • Repeated sampling from an instruction-tuned model yields rationales that are more similar to each other, so agreement across generations is a weaker signal of reliability than it would be for a base model.
  • Likelihood-based and verbalized confidence diverge after instruction tuning, so calibration claims based on a single confidence proxy are incomplete.
  • Rationale diversity and confidence are not interchangeable: controlling for the chosen answer and rationale length leaves the diversity drop intact, meaning the two capture different effects of tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same-answer, length-matched control suggests, but does not prove, a mechanism: instruction tuning may be compressing the space of acceptable reasoning paths, which would predict that the effect is stronger on tasks with a single canonical solution than on open-ended ones—a testable extension the paper does not run.
  • If overconfidence and rationale homogenization share a cause in training (for example, a reward for concise, decisive outputs), then mitigation could target the training objective; the paper's paired design offers a template for such experiments.
  • Because the three model families differ in their post-training choices, the uniform direction of the confidence increase hints that the effect is robust to those differences, but extending the conclusion to larger or differently trained models goes beyond the 7B-8B evidence presented here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares three base language models (Qwen2.5-7B, Mistral-7B-v0.3, Llama-3.1-8B) with their official instruction-tuned checkpoints across three multiple-choice QA benchmarks (ARC-Easy, MMLU, CommonsenseQA). It measures answer confidence via normalized choice entropy and a two-stage verbalized-confidence protocol, and it measures rationale lexical diversity via Unique-2 and 1-SelfBLEU computed over five sampled chain-of-thought rationales per question. The central empirical claims are that instruction tuning consistently increases confidence (lower choice entropy, higher verbalized confidence) with limited accuracy change, that cross-rationale diversity (1-SelfBLEU) decreases across all nine model-benchmark pairs, that surface-level diversity (Unique-2) changes heterogeneously, and that the diversity shifts persist under a same-answer, length-matched control on CommonsenseQA.

Significance. If the descriptive findings are taken as comparisons between base and official instruct checkpoints, the paper provides a useful and clearly presented empirical contribution: it documents a robust decrease in cross-rationale lexical diversity and a robust increase in confidence across three model families, and it includes a thoughtful matched-answer, matched-length control. The use of standard metrics, public benchmarks, and per-example significance tests is a strength, and the authors are appropriately cautious about downstream risk in the ethical statement. However, the title, abstract, and Section 4 cast the results as causal effects of instruction tuning, and that attribution is not supported by the experimental design because the official Instruct checkpoints differ from their base versions in data mixture, preference optimization, and inference template. The paper is likely to be valuable after the causal claims are either supported by controlled training experiments or reframed as descriptive differences between shipped checkpoints.

major comments (3)
  1. [Experimental Settings (§3, Table 5)] The central causal claim that 'instruction tuning' increases confidence and decreases cross-rationale diversity is not supported by the model pairs used. Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-8B-Instruct are vendors' final post-trained checkpoints that also differ from their base versions in supervised fine-tuning data, preference optimization (RLHF or DPO), and likely other post-training steps; the paper itself distinguishes 'instruction tuning and preference-based post-training' in §2 but never controls for the latter. The word 'matched' in the abstract and §3 is therefore misleading. The Limitations section lists only benchmark and diversity-measure scope, omitting this training-recipe confound. To support the causal framing, the authors would need either to compare checkpoints that differ only in instruction tuning (e.g., the same base with and without the SFT stage, holding preference optimization fixed) or to explicitly reframe the contribution as documenting differences between shipped base and instruct models. The descriptive finding remains interesting, but the current causal language in the title, abstract, and §4 outruns the design.
  2. [§3, Model Confidence Evaluation (verbalized confidence)] The verbalized-confidence protocol fixes the answer to the likelihood-selected candidate and then prompts the model to output a single number between 0 and 1. Instruction-tuned models are trained to comply with such format restrictions, so part of the large verbalized-confidence increase (e.g., Llama from 49.2% to 90.4% on ARC-Easy) may reflect prompt-compliance behavior rather than a genuine change in calibrated confidence. The paper does not report the distribution of elicited probabilities, nor does it test an alternative elicitation that does not condition on a fixed answer. Because the choice-entropy result independently supports the confidence-increase pattern, this issue does not invalidate the paper, but it should be addressed with at least a robustness check or a discussion of this artifact before the verbalized-confidence measure is used as a load-bearing quantity.
  3. [Experimental Settings and Appendix B (inference protocol)] The paper does not specify whether the base and instruct models were prompted with the same raw text or whether the instruct models received their native chat template (and the base models a plain prompt). If the chat template is applied only to Instruct models, the comparison conflates template effects with training effects, which is especially relevant for verbalized confidence and for the lexical diversity of generated rationales. The generation settings in Appendix B (temperature 0.7, nucleus p=1.0, max 100 tokens) are stated as fixed, but template format is part of the generation setting. The authors should report the exact prompts, including any template wrappers, for both variants and, ideally, include a control that applies the instruct template to the base model.
minor comments (5)
  1. [Abstract and §4] The phrase 'likelihood-based calibration' in the abstract is imprecise; the paper primarily reports ECE for likelihood-based and verbalized confidence, which are calibration errors rather than a single 'likelihood-based calibration' construct.
  2. [Table 1 and Figure 1] The color coding in Table 1 is not accessible in grayscale; consider adding explicit arrows or boldface for the Instruct rows instead of relying on color alone. The labels in Figure 1 are very compact and hard to read at page width.
  3. [§4, 'Instruction Tuning Induces Heterogeneous Changes...'] The sentence 'the largest increases in Unique-2 are observed for Mistral on ARC-Easy and MMLU, while the largest increase occurs on CSQA' does not name the model for the CSQA increase (it is Mistral per Table 1); this should be clarified.
  4. [Appendix B] The verbalized-confidence prompt example does not show the exact formatting for the candidate answers (e.g., whether option labels are included as 'A. [choice A]' with the bracket placeholders literally present); please show a fully concrete example.
  5. [References] The reference to Huang et al. (2026) lists page numbers 1–59 for a short-paper track, which looks like an error; please verify the pagination.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical measurements with standard metrics on public checkpoints; no fitted parameter is relabeled as a prediction and no load-bearing self-citation chain is used.

full rationale

The paper's central claims are empirical observations: after comparing official base and instruction-tuned checkpoints, it reports lower choice entropy, higher verbalized confidence, and mostly lower cross-rationale diversity. All quantities are defined through standard, independently established formulas: choice entropy from normalized answer likelihoods (Eq. 2), verbalized confidence via a two-stage elicitation prompt, Unique-2 and Self-BLEU from generated rationales, and ECE from binned accuracy-confidence comparisons. None of these definitions presuppose the paper's conclusions, and no parameter is fitted to the benchmark data and then renamed as a finding. The paper does not derive a quantity from a self-citation; prior work is cited only for methodological provenance, such as the LM Evaluation Harness, SacreBLEU, and confidence elicitation protocols. The main threat to the paper's causal framing is that the Instruct checkpoints also differ from their Base counterparts in preference optimization, data mixture, and chat-template inference, so attributing the observed differences specifically to 'instruction tuning' is not fully identified. That is a validity limitation, not circularity: the descriptive Base-versus-Instruct differences remain well-defined measurements even if the causal label is contested. No equation reduces to its own input, no result is forced by a self-citation chain, and the study is self-contained against external benchmarks and metrics. Score 0 is therefore appropriate.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No parameters were fitted to the reported measurements; generation hyperparameters are controlled and shared between paired models. The main unstated burden is the matched-pair causal assumption.

free parameters (2)
  • Sampling temperature T = 0.7
    Set in Appendix B for rationale generation; lexical diversity values depend on T, though the paired comparison holds T fixed across models.
  • Number of sampled rationales K = 5
    Chosen for estimating cross-rationale diversity; larger K would reduce variance in Self-BLEU estimates.
assumptions (4)
  • domain assumption The model's prediction is the candidate with the highest conditional likelihood pLM(y|x), as in Eq. 1.
    Standard multiple-choice QA convention from Jiang et al. (2021); all confidence and accuracy measures build on this choice.
  • domain assumption Verbalized confidence elicited by a numeric prompt reflects the model's confidence.
    Section 3; instruction-tuned models may comply with the 'give a number' prompt differently, confounding the measure.
  • domain assumption Unique-2 and 1-SelfBLEU capture meaningful rationale diversity.
    Section 3; these are surface n-gram measures and may miss semantic diversity, which the authors leave to future work.
  • domain assumption Base and Instruct variants are matched for isolating instruction tuning.
    Experimental Settings and Table 5; the Instruct checkpoints include additional post-training, so the causal attribution is not clean.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity." pith.science (2026). https://pith.science/paper/YIFOXS37

@misc{pith2026260813430,
  author       = {Pith},
  title        = {Pith review of: Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YIFOXS37}},
  note         = {Machine review of arXiv:2608.13430}
}
read the original abstract

Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence. In question answering, verbalized model overconfidence may be associated with the consistency of the generated supporting rationales. In this paper, we study whether corresponding changes in the lexical diversity of generated answer rationales accompany changes in model confidence induced by instruction tuning. We evaluate three matched base and instruction-tuned models across question-answering benchmarks and find that instruction tuning consistently alters answer confidence, despite limited changes in predictive accuracy and decreases in likelihood-based calibration. Secondly, we observe a non-uniform effect of instruction tuning on rationale diversity: cross-rationale diversity consistently decreases, whereas surface-level lexical diversity varies in both direction and magnitude across models and benchmarks. Finally, we find that these differences persist after controlling for answer selection and rationale length, confirming that confidence and rationale diversity capture distinct effects of instruction tuning.

Figures

Figures reproduced from arXiv: 2608.13430 by the authors.

Figure 1
Figure 1. Effect of instruction tuning on answer uncer [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Average change in rationale length from Base to Instruct models across benchmarks. Positive values [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 6 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  2. [2]

    Publications Manual , year = "1983", publisher =

    1983

  3. [3]

    Nature , volume=

    Large language models encode clinical knowledge , author=. Nature , volume=. 2023 , publisher=

    2023

  4. [4]

    Nature medicine , volume=

    Evaluation and mitigation of the limitations of large language models in clinical decision-making , author=. Nature medicine , volume=. 2024 , publisher=

    2024

  5. [5]

    arXiv preprint arXiv:2303.17564 , year=

    Bloomberggpt: A large language model for finance , author=. arXiv preprint arXiv:2303.17564 , year=

    arXiv

  6. [6]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Lawbench: Benchmarking legal knowledge of large language models , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  7. [7]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  8. [8]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

Show all 44 references
  1. [9]

    arXiv preprint arXiv:2109.01652 , year=

    Finetuned language models are zero-shot learners , author=. arXiv preprint arXiv:2109.01652 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2207.05221 , year=

    Language models (mostly) know what they know , author=. arXiv preprint arXiv:2207.05221 , year=

    arXiv

  3. [11]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  4. [12]

    Calibrating the Confidence of Large Language Models by Eliciting Fidelity

    Zhang, Mozhi and Huang, Mianqiu and Shi, Rundong and Guo, Linsen and Peng, Chong and Yan, Peng and Zhou, Yaqian and Qiu, Xipeng. Calibrating the Confidence of Large Language Models by Eliciting Fidelity. Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  5. [13]

    On the Calibration of Large Language Models and Alignment

    Zhu, Chiwei and Xu, Benfeng and Wang, Quan and Zhang, Yongdong and Mao, Zhendong. On the Calibration of Large Language Models and Alignment. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023.findings-emnlp.654

    2023 doi

  6. [14]

    How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

    Jiang, Zhengbao and Araki, Jun and Ding, Haibo and Neubig, Graham. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics. 2021. doi:10.1162/tacl_a_00407

    2021 doi

  7. [15]

    A Survey of Confidence Estimation and Calibration in Large Language Models

    Geng, Jiahui and Cai, Fengyu and Wang, Yuxia and Koeppl, Heinz and Nakov, Preslav and Gurevych, Iryna. A Survey of Confidence Estimation and Calibration in Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024 doi

  8. [16]

    Avoidance Decoding for Diverse Multi-Branch Story Generation

    Park, Kyeongman and Yang, Nakyeong and Jung, Kyomin. Avoidance Decoding for Diverse Multi-Branch Story Generation. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.381

    2025 doi

  9. [17]

    A Character-Centric Creative Story Generation via Imagination

    Park, Kyeongman and Kim, Minbeom and Jung, Kyomin. A Character-Centric Creative Story Generation via Imagination. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.82

    2025 doi

  10. [18]

    and Rumshisky, Anna

    Deshpande, Vijeta and Ghose, Debasmita and Patterson, John D and Beaty, Roger E. and Rumshisky, Anna. Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models. Proceedings of the 2025 Conference on Empirical Methods in...

    2025 doi

  11. [19]

    Are Large Language Models Capable of Generating Human-Level Narratives?

    Tian, Yufei and Huang, Tenghao and Liu, Miri and Jiang, Derek and Spangher, Alexander and Chen, Muhao and May, Jonathan and Peng, Nanyun. Are Large Language Models Capable of Generating Human-Level Narratives?. Proceedings of the 2024 Conference on Empirical Methods in Natural...

    2024 doi

  12. [20]

    Collective Critics for Creative Story Generation

    Bae, Minwook and Kim, Hyounghun. Collective Critics for Creative Story Generation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.1046

    2024 doi

  13. [21]

    The Price of Format: Diversity Collapse in LLM s

    Yun, Longfei and An, Chenyang and Wang, Zilong and Peng, Letian and Shang, Jingbo. The Price of Format: Diversity Collapse in LLM s. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.836

    2025 doi

  14. [22]

    C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge

    Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan. C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hu...

    2019 doi

  15. [23]

    Jointly Measuring Diversity and Quality in Text Generation Models

    Alihosseini, Danial and Montahaei, Ehsan and Soleymani Baghshah, Mahdieh. Jointly Measuring Diversity and Quality in Text Generation Models. Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation. 2019. doi:10.18653/v1/W19-2311

    2019 doi

  16. [24]

    Investigating the Multilingual Calibration Effects of Language Model Instruction Tuning

    Huang, Jerry and Lu, Peng and Zeng, Qiuhao and Iwasawa, Yusuke and Matsuo, Yutaka and Chandar, Sarath and Marrese-Taylor, Edison and Li, Irene. Investigating the Multilingual Calibration Effects of Language Model Instruction Tuning. Proceedings of the 19th Conference of the E ...

    2026 doi

  17. [25]

    Nature , volume=

    Detecting hallucinations in large language models using semantic entropy , author=. Nature , volume=. 2024 , publisher=

    2024

  18. [26]

    Calibration-Tuning: Teaching Large Language Models to Know What They Don ' t Know

    Kapoor, Sanyam and Gruver, Nate and Roberts, Manley and Pal, Arka and Dooley, Samuel and Goldblum, Micah and Wilson, Andrew. Calibration-Tuning: Teaching Large Language Models to Know What They Don ' t Know. Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP...

    2024 doi

  19. [27]

    A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation

    Nadeem, Moin and He, Tianxing and Cho, Kyunghyun and Glass, James. A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10...

    2020 doi

  20. [28]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Calibrating language models with adaptive temperature scaling , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  21. [29]

    On the Efficacy of Sampling Adapters

    Meister, Clara and Pimentel, Tiago and Malagutti, Luca and Wilcox, Ethan and Cotterell, Ryan. On the Efficacy of Sampling Adapters. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.80

    2023 doi

  22. [30]

    arXiv preprint arXiv:2405.14782 , year=

    Lessons from the trenches on reproducible evaluation of language models , author=. arXiv preprint arXiv:2405.14782 , year=

    arXiv

  23. [31]

    A Call for Clarity in Reporting BLEU Scores

    Post, Matt. A Call for Clarity in Reporting BLEU Scores. Proceedings of the Third Conference on Machine Translation: Research Papers. 2018. doi:10.18653/v1/W18-6319

    2018 doi

  24. [32]

    Advances in neural information processing systems , volume=

    Large language models are zero-shot reasoners , author=. Advances in neural information processing systems , volume=

  25. [33]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Conceptnet 5.5: An open multilingual graph of general knowledge , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  26. [34]

    arXiv preprint arXiv:2002.07650 , year=

    Uncertainty estimation in autoregressive structured prediction , author=. arXiv preprint arXiv:2002.07650 , year=

    2002 arXiv

  27. [35]

    Shannon, C. E. , journal=. A mathematical theory of communication , year=

  28. [36]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    2009 arXiv

  29. [37]

    arXiv preprint arXiv:1803.05457 , year=

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    arXiv

  30. [38]

    Transactions of the Association for Computational Linguistics , volume=

    Benchmarking linguistic diversity of large language models , author=. Transactions of the Association for Computational Linguistics , volume=. 2025 , publisher=

    2025

  31. [39]

    International conference on machine learning , pages=

    On calibration of modern neural networks , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  32. [40]

    International Conference on Learning Representations , volume=

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms , author=. International Conference on Learning Representations , volume=

  33. [41]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  34. [42]

    Dan Gusfield , title =. 1997

    1997

  35. [43]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  36. [44]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.