Pith. sign in

REVIEW 3 major objections 5 minor 51 references

From Evidence to Belief: A Bayesian Epistemology Approach to Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language models update confidence only when evidence agrees with them, not when it contradicts or is irrelevant.

desk verdict A useful evidence taxonomy and a solid confirmation result, but the negative Bayesian claims rest on n=6 t-tests and a Section 5 that undercuts the irrelevance conclusion. read the letter →

arxiv 2504.19622 v1 pith:HKNVB2HW submitted 2025-04-28 cs.AI

Minsu Kim , Sangryul Kim , James Thorne This is my paper · ORCID
classification cs.AI
keywords Bayesianepistemologylanguagemodelconfidencebeliefupdatingevidencetypesirrelevantcontextverbalizedcalibration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models update their confidence the way Bayesian epistemology says a rational agent should: raising belief when evidence confirms a hypothesis, lowering it when evidence disconfirms, and leaving it unchanged when evidence is irrelevant. Probing GPT-3.5 and GPT-4o with seven kinds of evidence built from three question-answering datasets, the authors find that the models raise confidence and accuracy when given correct golden evidence, matching the confirmation assumption. But when evidence conflicts with the correct answer, confidence does not reliably drop, and when evidence is irrelevant, confidence and accuracy fall instead of staying flat. The paper concludes that language models partially align with Bayesian epistemology, following confirmation assumptions but failing to adhere to disconfirmation and irrelevance assumptions, and that their bias toward golden evidence helps explain the deviation. This matters because if model confidence is not a Bayesian posterior over evidence, then high-confidence answers cannot be read as justified beliefs.

What carries the argument

The load-bearing mechanism is a comparison between the model's probability of a hypothesis without evidence, $P(H|\theta)$, and its probability given evidence, $P(H|E,\theta)$, across a designed taxonomy of evidence: golden, conflicting, incomplete, contradictory, coincidental, and irrelevant evidence, together with reliability variations such as source credibility, specificity, timeliness, and experimental versus observational evidence. For each comparison the authors measure confidence three ways, verbalized confidence, token probability, and sampling, plus accuracy and expected calibration error. The taxonomy turns the abstract Bayesian confirmation, disconfirmation, and irrelevance assumptions into concrete, testable predictions: confirmation requires $P(H|E,\theta) > P(H|\theta)$, disconfirmation requires $P(H|\theta) > P(H|E,\theta)$, and irrelevance requires equality. The paper's evidence-generation prompts, which rewrite correct explanations into negated, coincidental, or out-of-context versions, are what make those predictions operational.

What would settle it

Re-run the confirmation experiments with coincidental evidence prompts stripped of any token that can be extracted as the answer, for example by paraphrasing or blanking concrete answer words like hemoglobin, and check whether models still reach high accuracy. If accuracy collapses or confidence drops once answer leakage is removed, the paper's attribution of the deviation to a failure to handle unreliable evidence is called into question.

Watch

Extended reading notes

Core claim

The central claim is that language models do not possess justified belief in the Bayesian sense: they behave like Bayesian agents only when evidence supports what they already know. Presented with golden evidence, both models show significantly higher confidence and accuracy than with no evidence, satisfying the confirmation assumption. Presented with conflicting evidence, however, they do not consistently lower confidence, with only GPT-4o using verbalized confidence showing a significant drop, while accuracy falls, so the disconfirmation assumption fails. Contradictory evidence containing both correct and wrong sentences is treated almost like golden evidence, and irrelevant evidence decreases accuracy rather than leaving belief unchanged, so the irrelevance assumption also fails. The paper also shows that stronger evidence raises verbalized and sampling confidence but does not always improve accuracy or calibration, meaning high confidence is not a guarantee of correct belief.

Load-bearing premise

The conclusions stand on the assumption that the automatically generated evidence prompts genuinely instantiate the intended epistemic categories; if coincidental evidence secretly contains the answer in a retrievable form, or if conflicting evidence is not truly contradictory, then the measured deviations from Bayesian assumptions are partly artifacts of data construction.

Editorial extensions

If this is right

  • Model confidence cannot be treated as a calibrated probability of being right: golden evidence inflates it, conflicting evidence leaves it nearly unchanged, and stronger evidence can make confidence rise while accuracy stays flat.
  • When contradictory context is present, a model's parametric prior can outweigh contradictory sentences, so adding a wrong statement to correct evidence need not reduce trustworthiness.
  • Irrelevant context is more damaging than conflicting context in these experiments, since it lowers accuracy while confidence drops only partially; context selection matters for safe deployment.
  • Under the paper's Bayesian standard, language models lack justified belief, so their high-confidence outputs should be treated as fluent guesses rather than epistemic commitments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence of the golden-evidence bias is that the same prompting setup should show smaller deviations when questions concern facts absent from the model's training data, because weaker parametric priors would let evidence dominate; this can be tested with a held-out corpus of novel facts.
  • The irrelevance gradient suggests a practical extension: if the degree of distraction tracks semantic or field distance, then retrieval systems could rank context not only by relevance but by predicted distraction, and the paper's within-field versus cross-field comparison provides a template for measuring that.
  • Bayesian updating is usually about sequential evidence, while this paper tests one-shot evidence; a sequential-evidence version, for example first irrelevant then confirming, could reveal whether model confidence obeys order effects and base-rate neglect, extending the framework without new assumptions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes evaluating whether large language models update their beliefs in accordance with three Bayesian epistemology assumptions (confirmation, disconfirmation, irrelevance) plus an evidence-power assumption. To do so, the authors construct a multi-type evidence dataset from SciQ, TriviaQA, and GSM8K, and measure verbalized confidence, token probability, and sampling-based confidence for GPT-3.5-turbo, GPT-4o, and Gemini-1.5-Flash. They report that golden evidence increases confidence and accuracy (supporting confirmation), that conflicting evidence does not reliably reduce confidence (interpreted as failing disconfirmation), that irrelevant evidence reduces accuracy but not always confidence (interpreted as failing irrelevance), and that stronger evidence raises verbalized and sampling confidence but not token-probability confidence in some conditions. They attribute the deviations to a bias toward golden evidence and to the degree of irrelevance.

Significance. If the findings are robust, the paper offers a philosophically grounded and empirically falsifiable test of whether LLM confidence behaves like Bayesian confirmation, and it ships code and data with explicit prompts, which is a strength. The strongest contribution is the confirmation result, which appears consistent across models, datasets, and confidence elicitation methods, and the systematic comparison of three confidence measures. However, the main negative conclusions about disconfirmation and irrelevance are based on statistical tests with severe power constraints and on evidence-category validity assumptions that are not fully met, so the significance is conditional and will depend on the adequacy of a revised analysis.

major comments (3)
  1. [Appendix H.2 and Section 4.1, Tables 4-6] The negative conclusions about disconfirmation are inferred from non-significant paired t-tests on only six dataset-level observations (three datasets by two models), as described in Appendix H.2; for example, the verbal-confidence comparison of conflicting evidence against no evidence yields p=0.99 (Table 4), and the token and sampling versions yield p=0.445 and p=0.392 (Tables 5 and 6). With n=6, these tests have very low power, so a non-significant p-value cannot support the paper's claim that conflicting evidence does not have a significant effect on confidence and consequently that LLMs fail the disconfirmation assumption (Section 4.1). A proper analysis using sample-level data, equivalence tests, or at least effect sizes and confidence intervals is needed before the central failure-to-disconfirm claim can be made.
  2. [Section 5 vs. Section 7] Section 5's cross-domain irrelevant-evidence ablation reports that confidence, accuracy, and ECE approached closer to the baseline no-evidence case and states that there is a possibility that the LLMs satisfy the irrelevant assumption, which directly contradicts the Section 7 conclusion that LLMs fail to adhere to irrelevance assumptions. If the irrelevance failure applies only to same-domain irrelevant evidence, the conclusion in Section 7 is overgeneralized; if it applies globally, the ablation's finding must be reconciled. The manuscript currently leaves the reader with an internal inconsistency that needs to be resolved, either by a more nuanced conclusion or by additional evidence that the same-domain result is the relevant one.
  3. [Appendix I.2 and Section 3.2] The prompts for generating coincidental evidence (Appendix I.2) embed the correct answer directly in the explanation text (e.g., 'hemoglobin', 'coagulation', 'golden eagles'), so the model can produce the correct answer by extraction even when the reasoning is supposed to be unreasonable or based on epistemic luck. This breaks the intended construct: Section 3.2 defines coincidental evidence as not containing the correct answer, yet the generated examples do. Consequently, the Section 4.1 claim that LLMs cannot handle coincidental evidence well is not a clean measure of whether models capture evidence unreliability; high accuracy and high confidence may simply reflect answer leakage. The dataset construction or the interpretation must be revised.
minor comments (5)
  1. [Appendix B, Tables 4-6] The significance thresholds (p<=0.05 significant, 0.05<p<=0.1 marginal) are used without any correction for the 18 comparisons made in the confirmation task; a multiple-comparison correction or an explicit justification for omitting one would strengthen the statistical claims.
  2. [Section 4.1] The statement that only GPT-4o with the verbal method exhibited behavior aligned with Bayesian disconfirmation is based on a single model-method combination; given the small number of tests and the low power, this should be hedged as exploratory rather than presented as a settled finding.
  3. [Table captions (Appendix A)] The captions of Tables 2 and 3 are identical; they should clearly distinguish that Table 2 reports token probability and Table 3 reports the sampling method.
  4. [Section 3.2] The definition of conflicting evidence states that all sentences are replaced with conflicting counterparts, but some of the negation examples in Appendix I.2 may still contain the original answer or close variants; a systematic check that the golden answer is absent from generated conflicting and coincidental evidence would improve construct validity.
  5. [Section 8 (Limitations)] The Limitations section notes that conflicting evidence is heterogeneous in whether it is entirely incompatible or merely obstructive; this heterogeneity weakens the disconfirmation analysis and should be analyzed separately or used to refine the evidence taxonomy, as the authors themselves suggest.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical measurement study with no fitted parameters or definitional reduction that would make its conclusions equivalent to its inputs.

full rationale

The paper is an empirical study; there is no fitted parameter, learned model, or closed-form derivation whose output is an input in disguise. The Bayesian confirmation, disconfirmation, and irrelevance conditions are normative definitions, and the paper tests them by measuring verbalized confidence, token probability, and sampling frequency under evidence perturbations. Equating these measurements with P(H|E,theta) is an operationalization, not an identity: the numbers come from model outputs, and the assumptions could in principle fail, which is exactly what the paper reports for disconfirmation and irrelevance. The coincidental-evidence examples in Appendix I.2 embed the answer in the explanation, and the n=6 paired t-tests in Appendix H.2 are low-power; these are construct-validity and statistical-power concerns about whether the evidence types and significance tests support the conclusions, but they are not circularity, because the conclusions are not forced by the construction of the Bayesian definitions. The only self-citations (Lee et al. 2023 for sampling confidence; Thorne et al. 2018 for FEVER) are methodological and external, and they are not load-bearing for the central claim. The internal tension between Section 5's cross-domain ablation ('there is a possibility that the LLMs satisfy the irrelevant assumption') and Section 7's conclusion that LLMs fail the irrelevance assumption is an inconsistency, not a circular reduction. No self-citation chain or uniqueness theorem is invoked to make the choice of framework forced. Therefore the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No parameters are fitted to data in this empirical study. The central tests rest on the assumption that confidence outputs are valid probability readings and that the auto-generated evidence types are valid.

free parameters (2)
  • Evidence perturbation degree = 0.5 (incomplete/contradictory); 0.3/0.5/0.8/1.0 (ablation)
    Hand-selected thresholds; results depend on these choices. For example, incomplete evidence at 50% deletion still acts as a hint, a finding that may change at other deletion levels.
  • Rouge-L correctness threshold = 0.3
    Hand-chosen threshold for counting an answer correct in SciQ and TriviaQA; different thresholds could alter accuracy and ECE values.
assumptions (4)
  • domain assumption Conditional probability is treated as a primitive notion (Hájek 2003), allowing Bayesian assumptions to be applied even when P(E)=0.
    This philosophical choice is invoked in Section 1 to justify applying Bayesian assumptions to language model evidence.
  • domain assumption Average verbalized, token, and sampling confidence scores are valid measurements of P(H|E,theta).
    Section 3 states 'we used an average confidence across all samples' to measure the probability norm. If confidence is not probability, the tests do not test Bayesian updating.
  • domain assumption The evidence generated by GPT-4/GPT-4o correctly realizes the intended types (conflicting, coincidental, etc.).
    Section 3.2 and Appendix I.2 define the types, but no human validation is reported. The coincidental evidence prompts leak the answer.
  • domain assumption A paired t-test on dataset-level aggregates (six numbers) is an appropriate significance test.
    Appendix H.2 explains p-values come from comparing six dataset-level means; this has very low power and can mislabel trends as failures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Evidence to Belief: A Bayesian Epistemology Approach to Language Models." pith.science (2026). https://pith.science/paper/HKNVB2HW

@misc{pith2026250419622,
  author       = {Pith},
  title        = {Pith review of: From Evidence to Belief: A Bayesian Epistemology Approach to Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HKNVB2HW}},
  note         = {Machine review of arXiv:2504.19622}
}
read the original abstract

This paper investigates the knowledge of language models from the perspective of Bayesian epistemology. We explore how language models adjust their confidence and responses when presented with evidence with varying levels of informativeness and reliability. To study these properties, we create a dataset with various types of evidence and analyze language models' responses and confidence using verbalized confidence, token probability, and sampling. We observed that language models do not consistently follow Bayesian epistemology: language models follow the Bayesian confirmation assumption well with true evidence but fail to adhere to other Bayesian assumptions when encountering different evidence types. Also, we demonstrated that language models can exhibit high confidence when given strong evidence, but this does not always guarantee high accuracy. Our analysis also reveals that language models are biased toward golden evidence and show varying performance depending on the degree of irrelevance, helping explain why they deviate from Bayesian assumptions.

Figures

Figures reproduced from arXiv: 2504.19622 by the authors.

Figure 1
Figure 1. The overall experimental method and simple examples of the evidence dataset for Confirmation Task. As [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The results of the Strength of Evidence task on the SciQ dataset with verbal confidence method. The [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The results for the degree of variations in evidence for the SciQ dataset with verbalized method. We [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The results of the Strength of Evidence task on the SciQ dataset with token probability method. The blue [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: The results of the Strength of Evidence task on the SciQ dataset with sampling method. The blue bar [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: The results of ablation for irrelevant evidence. The blue bar represents the result of no evidence [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 8 canonical work pages

  1. [1]

    Robert Audi. 1997. Epistemology: A Contemporary Introduction to the Theory of Knowledge. Routledge, New York

    work page 1997

  2. [2]

    Bender and Alexander Koller

    Emily M. Bender and Alexander Koller. 2020. https://doi.org/10.18653/v1/2020.acl-main.463 Climbing towards NLU : On meaning, form, and understanding in the age of data . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185--5198, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.463 2020

  3. [3]

    Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.703 Experience grounds language . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    doi:10.18653/v1/2020.emnlp-main.703 2020

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. http://arxiv.org/abs/1812.01193 e-snli: Natural language inference with natural language explanations

    arXiv 2018

  6. [6]

    Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. http://arxiv.org/abs/2303.08119 How many demonstrations do you need for in-context learning?

    arXiv 2023

  7. [7]

    Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. 2023. http://arxiv.org/abs/2311.09277 Contrastive chain-of-thought prompting

    arXiv 2023

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems

    arXiv 2021

Show all 51 references
  1. [9]

    Yu Feng, Ben Zhou, Haoyu Wang, Helen Jin, and Dan Roth. 2023. http://arxiv.org/abs/2212.10467 Generic temporal reasoning with differential analysis and explanation

    2023 arXiv

  2. [10]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. http://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling

    2020 arXiv

  3. [11]

    Gemini Team , Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Li...

    2024 arXiv

  4. [12]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. http://arxiv.org/abs/1706.04599 On calibration of modern neural networks

    2017 arXiv

  5. [13]

    Alan H\' a jek. 2003. https://doi.org/10.1023/b:synt.0000004904.91112.16 What conditional probability could not be . Synthese, 137(3):273--323

    2003

  6. [14]

    Alan H\' a jek and Stephan Hartmann. 2010. Bayesian epistemology. In DancyJ, editor, A Companion to Epistemology. Blackwell

    2010

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. http://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding

    2021 arXiv

  8. [16]

    Paul Horwich. 1982. Probability and Evidence. Cambridge University Press, Cambridge

    1982

  9. [17]

    Colin Howson. 2000. Hume's Problem: Induction and the Justification of Belief. Oxford University Press, New York

    2000

  10. [18]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. https://api.semanticscholar.org/CorpusID:265067168 A survey on hallucination in large language models: Principles, taxonomy...

    2023 arXiv

  11. [19]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  12. [20]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav For...

    2022 arXiv

  13. [21]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. http://arxiv.org/abs/2302.09664 Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    2023 arXiv

  14. [22]

    Bowman, and Ethan Perez

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish...

    2023 arXiv

  15. [23]

    Noah Lee, Na Min An, and James Thorne. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.278 Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569--4585, Singapore. Assoc...

    2023 doi

  16. [24]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  17. [25]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. http://arxiv.org/abs/2205.14334 Teaching models to express their uncertainty in words

    2022 arXiv

  18. [26]

    Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

    Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. https://doi.org/10.1162/tacl_a_00494 Reducing conversational agents ' overconfidence through linguistic calibration . Transactions of the Association for Computational Linguistics, 10:857--872

    2022 doi

  19. [27]

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Bel...

    2023 arXiv

  20. [28]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. http://arxiv.org/abs/2302.00093 Large language models can be easily distracted by irrelevant context

    2023 arXiv

  21. [29]

    Andrew Schwartz, Jo \ a o Sedoc, and Niranjan Balasubramanian

    Nikita Soni, H. Andrew Schwartz, Jo \ a o Sedoc, and Niranjan Balasubramanian. 2024. https://doi.org/10.18653/v1/2024.naacl-long.477 Large human language models: A need and the challenges . In Proceedings of the 2024 Conference of the North American Chapter of the Association ...

    2024 doi

  22. [30]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali...

    2023 arXiv

  23. [31]

    Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong H...

    2024 arXiv

  24. [32]

    William Talbott. 2006. Bayesian epistemology. In Edward Zalta, editor, Stanford Encyclopedia of Philosophy

    2006

  25. [33]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  26. [34]

    Manning, and Chelsea Finn

    Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. 2023 a . http://arxiv.org/abs/2311.08401 Fine-tuning language models for factuality

    2023 arXiv

  27. [35]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023 b . http://arxiv.org/abs/2305.14975 Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-t...

    2023 arXiv

  28. [36]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. http://arxiv.org/abs/2305.04388 Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting

    2023 arXiv

  29. [37]

    Olav Benjamin Vassend. 2023. https://doi.org/10.1007/s44204-023-00104-z What hinge epistemology and bayesian epistemology can learn from each other . Asian Journal of Philosophy, 2(2):1--21

    2023 doi

  30. [38]

    Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023 a . http://arxiv.org/abs/2212.10001 Towards understanding chain-of-thought prompting: An empirical study of what matters

    2023 arXiv

  31. [39]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 b . http://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models

    2023 arXiv

  32. [40]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. http://arxiv.org/abs/2109.01652 Finetuned language models are zero-shot learners

    2022 arXiv

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  34. [42]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. https://doi.org/10.18653/v1/W17-4413 Crowdsourcing multiple choice science questions . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106, Copenhagen, Denmark. Association for Computational Linguistics

    2017 doi

  35. [43]

    Jon Williamson. 2010. In Defence of Objective Bayesianism. Oxford University Press

    2010

  36. [44]

    Yijun Xiao and William Yang Wang. 2021. https://api.semanticscholar.org/CorpusID:232404053 On hallucination and predictive uncertainty in conditional language generation . ArXiv, abs/2103.15025

    2021 arXiv

  37. [45]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. http://arxiv.org/abs/2306.13063 Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

    2024 arXiv

  38. [46]

    Pengfei Yu and Heng Ji. 2024. http://arxiv.org/abs/2305.18582 Information association for language model updating by mitigating lm-logical discrepancy

    2024 arXiv

  39. [47]

    Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Cheng Zhou, Xinbing Wang, and Luoyi Fu. 2023. https://api.semanticscholar.org/CorpusID:265351856 Enhancing uncertainty-based hallucination detection with stronger focus . ArXiv, abs/2311.13230

    2023 arXiv

  40. [48]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023. http://arxiv.org/abs/2305.10425 Slic-hf: Sequence likelihood calibration with human feedback

    2023 arXiv

  41. [49]

    Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.335 Navigating the grey area: How expressions of uncertainty and overconfidence affect language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  42. [50]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.