Pith. sign in

REVIEW 5 major objections 5 minor 35 references

Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sampling test sorts LLM knowledge into six states, from well-known to confidently wrong.

desk verdict A useful six-category evaluation framework, but the abstract's layer-wise and RLHF claims are unsupported as written. read the letter →

arxiv 2501.01332 v1 pith:EPUD46XT submitted 2025-01-02 cs.CL

Yanbo Fang , Ruixiang Tang This is my paper · ORCID
classification cs.CL
keywords knowledgecategorizationconfidenceestimationchain-of-thoughtpromptinginstructiontuninglayer-wiseanalysissamplingconsistencyHaluEvalrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces K-(CSA)^2, a framework that sorts what a large language model knows into six categories by asking two questions at once: is the answer right, and is the model sure? Correctness is checked by exact match against ground truth; confidence is measured by prompting the same question several times and seeing how often the same answer comes back. The categories run from Highly Known (correct and consistent) to Confident Unknown (wrong and consistent). The paper then uses the categories to show that chain-of-thought prompting and instruction tuning reshape knowledge structures in different ways for internal versus context-provided knowledge, and that high-confidence knowledge of either sign concentrates in upper model layers. The point of the framework is that accuracy alone cannot distinguish true knowledge from confidently held misconceptions, so a metric like Category Score, which weights the six categories, gives a different and richer picture of model comprehension.

What carries the argument

The load-bearing object is the six-category mapping K-(CSA)^2 plus the confidence proxy that feeds it. Given a question, the model is queried seven times: once with greedy decoding ($T=0$) and six times with sampling. Correctness is exact match with the ground truth; confidence among wrong answers is $P_{\text{Confidence}} = \max_i (f_i/n)$, the largest fraction of identical responses. The two dimensions combine into the six categories of Table 3, and the categories in turn feed a Category Score (sum of six weights $6..1$ times category ratios) and transition ratios (upgrade, downgrade, stable) that track how a model's knowledge structure changes under training or prompting.

What would settle it

Run the same 3,000 HaluEval questions through the protocol with 6 versus 60 samples and with temperatures 0.2 versus 1.0, and count how often a knowledge point changes category; a large reassignment rate would show the confidence axis is an artifact of sampling settings rather than a stable property of the model's knowledge.

Watch

Extended reading notes

Core claim

The central claim is that a model's knowledge of a fact can be read off a small matrix of sampled responses: one greedy output plus six temperature-sampled outputs. If at least one output is correct, the knowledge point is 'known' and falls into Highly Known, Maybe Known, or Weakly Known depending on whether the greedy answer is correct and how much the sampled answers agree; if none is correct, it is 'unknown' and falls into Unconfident Unknown, May Confident Unknown, or Confident Unknown depending on how often the same wrong answer repeats. Confidence for wrong answers is the maximum frequency of any single answer among the samples. Applying this scheme across the HaluEval knowledge QA dataset, the paper reports that stronger models are more assertive in both correct and incorrect directions, that chain-of-thought prompting improves category structure especially on instruction-tuned models, that instruction tuning alone can initially hurt internal-knowledge scores, and that in layer-wise probes high-confidence categories (HK and CU) are encoded most strongly in upper layers while low-confidence categories (UU and MU) peak in middle-to-lower layers.

Load-bearing premise

The whole framework treats how often a model repeats the same answer across six temperature-sampled responses to an identical prompt as a measure of how confident the model is in that answer; if sampling variability reflects decoding randomness or question wording rather than knowledge confidence, every category and downstream finding built on it loses its footing.

Editorial extensions

If this is right

  • Category Score provides an evaluation number that penalizes confidently wrong answers more than uncertain wrong ones, which accuracy cannot do.
  • The finding that CoT and instruction tuning are complementary suggests model developers should evaluate and tune internal knowledge and context understanding separately.
  • Layer-wise patterns imply that interventions to correct confident misconceptions may need to act on upper layers, while low-confidence knowledge can be worked on in middle layers.
  • Training-stage curves show accuracy rising while Category Score dips, so monitoring only accuracy can miss temporary degradations in knowledge structure.
  • The six-category scheme gives a direct way to compare base, instruction-tuned, and chain-of-thought versions of the same model on the same knowledge points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One direct extension the authors do not spell out: the same sampling consistency measure could be turned into a hallucination detector for open-book QA, since a knowledge point that falls in CU (confidently wrong) despite attached context is exactly a case of overriding evidence.
  • The framework's confidence axis is behavior-based only; pairing PConfidence with the model's own token log-probabilities or with semantic-entropy estimates on the same questions would test whether 'confidence' tracks anything beyond repetition.
  • Because the paper shows CU knowledge concentrates in upper layers, a testable next step is to ablate or steer upper-layer representations and ask whether CU knowledge is more easily shifted than HK knowledge, which would give a causal handle on misconceptions.
  • The category transition ratios could be used as a fine-tuning early-stopping signal: stop when downgrade ratio starts to rise, rather than waiting for accuracy to saturate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces K-(CSA)², a framework that classifies LLM responses to knowledge-based QA into six categories (Highly Known, Maybe Known, Weakly Known, Unconfident Unknown, May Confident Unknown, Confident Unknown) using one greedy decode and six temperature-sampled decodes per question. Correctness is measured by exact match against ground truth; confidence is measured by the maximum frequency of an incorrect sampled response. The framework is applied to evaluate internal (parametric) and external (context-grounded) knowledge across seven LLMs, to compare base, instruction-tuned, and chain-of-thought variants, and to track category distributions over Pythia training checkpoints. The paper reports that stronger models are more assertive, that CoT produces synergistic gains with instruction tuning, and that layer-wise ground-truth probabilities are higher in upper layers for high-confidence knowledge and peak in middle-to-lower layers for low-confidence knowledge.

Significance. If the framework's measurement assumptions were validated, the six-category taxonomy would offer a practical way to evaluate LLM knowledge beyond binary accuracy, and the reported CoT/IT patterns would be informative for model alignment and evaluation. The paper has strengths: it uses a reproducible public benchmark (HaluEval), spans a range of open- and closed-source models, and explicitly separates correctness from confidence. However, the central layer-wise claim is not reproducible as written, the abstract's RLHF claim is not tested by the experiments, and the category definitions lack specified thresholds, so the framework's outputs and all findings built on them rest on ambiguous and under-validated measurements.

major comments (5)
  1. [Section 4.1, Figure 8] The layer-wise analysis is critically underspecified. The text states that "Ground truth probabilities are calculated by measuring the frequency of correct responses at each layer when the model was queried multiple times with the same input," but it does not explain what it means to query a layer of a transformer: no early-exit strategy, probing head, logit-lens readout, or other mechanism is described. It also does not state how the category labels, which are assigned from full-model final responses, are aligned to layer-level probabilities, nor how many queries per layer were used. Without this information, the abstract's headline finding that "higher layers in LLMs encode more high-confidence knowledge" cannot be checked or reproduced, and the heatmap values in Figure 8 have no clear interpretation.
  2. [Abstract and Section 1 Introduction] The abstract claims that "chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures" of LLMs, but the experiments in Section 3 do not isolate or test RLHF. The comparisons are among base, instruction-tuned (IT), and CoT variants; instruction tuning is not equivalent to RLHF, and no RLHF-specific checkpoint or fine-tuning run is used anywhere in the reported evaluations. This claim should either be removed from the abstract and introduction or supported by an actual RLHF comparison.
  3. [Section 2.1, Appendix A.3, Table 3] The boundaries among the six categories are not fully defined. For the unknown categories (UU, MU, CU), Appendix A.3 gives example values of PConfidence (1/5, 2/5, 5/5) but never states the threshold values that separate the categories. For the known categories, Table 3 uses informal quantifiers such as "few correct sampling answers" and "not all" without numerical criteria. Because every finding in Sections 3 and 4 depends on these category assignments, the missing thresholds make the entire framework non-reproducible and also prevent a sensitivity analysis. The Limitation section itself concedes that sample count and temperature affect categorization results, which further highlights the need for an explicit decision procedure and robustness checks.
  4. [Section 3, Figures 3, 7, 9, 11] Category ratios and their differences are reported without error bars, confidence intervals, or significance tests. With 3000 questions and only six sampled responses per question, many categories will have small counts, and the reported patterns (e.g., "the ratio of 6.CU rises" in Section 3.1, or "4.UU+5.MU decreases" in Section 4.2) may not be statistically stable. The paper should provide bootstrap confidence intervals or per-model variance estimates for the category ratios and for the transition ratios in Figures 4 and 5.
  5. [Section 2.2, Eq. (1)] The Category Score assigns weights w_i = 7 - i to the six categories, from 6 for Highly Known to 1 for Confident Unknown, but the paper provides no justification for this linear weighting scheme. Since the Category Score is used as a headline evaluation metric (e.g., in Figure 2), the choice of weights directly affects the conclusions drawn from score differences. A sensitivity analysis over alternative weightings, or a principled justification, is needed before the score can support the paper's comparative claims.
minor comments (5)
  1. [Section 2.1] There is a typo: "For close-source models, T = 1" should likely be "For open-source models, T = 1," and the temperature settings for each model type should be stated more clearly, since the text currently says both "T = 0.5" for closed-source models and "T = 1" for close-source models in the same paragraph.
  2. [Figure 3 caption] The caption says models are "sorted left to right by increasing combined accuracy (ratios of top-2 layers 1.HK + 2.MK)"; this should be "top-2 categories" rather than "top-2 layers."
  3. [Appendix A.4] There is a typo: "model's responses concentrated in unkonwn categories" should be "unknown categories."
  4. [Section 4.2, Finding 8] The phrase "accuracy is closely instructed with category scores" appears to be a typo; it should likely read "accuracy is closely correlated with category scores."
  5. [General] Several figures (especially Figures 8, 13, and 14) have very small text and dense numerical labels that are difficult to read; larger fonts and a clearer color scale would improve interpretability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction found; the framework is an empirical measurement taxonomy, and its findings rest on external benchmarks and observed response patterns rather than on inputs that already contain the conclusions.

full rationale

The paper's K-(CSA)^2 framework does not derive its conclusions from fitted parameters or from its own target results. The six knowledge categories are defined directly as a partition of observed greedy and sampled responses: HK, MK, and WK are determined by whether greedy and sampled answers are correct, while UU, MU, and CU are determined by the maximum frequency of repeated incorrect sampled responses (Section 2.1, Table 3, Appendix A.3, Eq. 2). The Category Score (Eq. 1) is a hand-chosen weighted sum of the observed category ratios; using this score to compare model variants is an evaluation design choice, not a fitted-input-then-prediction loop. The layer-wise claim in Section 4.1 is based on a stated empirical procedure: 'Ground truth probabilities are calculated by measuring the frequency of correct responses at each layer when the model was queried multiple times with the same input.' This procedure is severely under-specified, which is a reproducibility and validity problem, but it is not a case of the conclusion being equivalent to the input by construction. The acknowledged limitation that sample count and temperature can affect categorization is a measurement-fragility caveat, not circularity. No load-bearing self-citation is present; the definitions of the known categories are explicitly attributed to Gekhman et al. (2024), and the paper's empirical claims are tested on external open models and the HaluEval benchmark. Therefore no derivation step reduces by definition or self-citation to the claim it is used to support.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework relies on ad hoc Category Score weights, an arbitrary sample count, and a sampling-consistency definition of confidence. These are not derived from theory, so the empirical conclusions inherit their arbitrariness.

free parameters (4)
  • Category Score weights w_i = w_i = 7-i for i=1..6 (6,5,4,3,2,1)
    Ad hoc linear weights with no justification; this is the central comparison metric in Figures 2, 6, 10, and 12.
  • Sampling temperature T = T=0.5 for closed-source models, T=1 for open-source models (footnote 1)
    Confidence categories depend on temperature; values are taken from prior work (Gekhman et al.) but are not validated for this framework.
  • Number of samples per question = n=7 (1 greedy, 6 sampled)
    PConfidence = max_i(f_i/n), so category assignments and ratios depend directly on n; no sensitivity analysis is provided.
  • UU/MU/CU boundary thresholds = Implied: PConfidence=1/5 -> UU, 2/5 -> MU, 5/5 -> CU in the n=5 appendix example
    The appendix example does not state the threshold separating UU from MU and MU from CU for the actual n=7 setting.
assumptions (4)
  • domain assumption A knowledge point is known if at least one of the n responses is correct
    Section 2.1 uses 'at least once across multiple sampling attempts' as the known/unknown boundary, making a single lucky correct sample sufficient; this is a modeling choice, not a standard theorem.
  • domain assumption Confidence can be read from response consistency of identical prompts sampled at T>0
    Appendix A.3 defines PConfidence as the max response frequency, although the introduction promised 'variations of the same question'; no evidence links this consistency measure to calibrated model confidence.
  • domain assumption Layer-wise responses can be elicited and their correctness probability measured
    Section 4.1 reports per-layer ground truth probabilities for Llama models without specifying the probing or early-exit procedure, so the existence of a valid per-layer response is assumed.
  • domain assumption HaluEval ground truth answers are correct for scoring
    The paper uses exact match against HaluEval labels; this is a standard benchmark assumption, but no manual audit is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension." pith.science (2026). https://pith.science/paper/EPUD46XT

@misc{pith2026250101332,
  author       = {Pith},
  title        = {Pith review of: Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPUD46XT}},
  note         = {Machine review of arXiv:2501.01332}
}
read the original abstract

Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes LLM knowledge along two dimensions: correctness and confidence. The framework defines six categories of knowledge, ranging from highly confident correctness to confidently held misconceptions, enabling a nuanced evaluation of model comprehension beyond binary accuracy. Using this framework, we demonstrate how techniques like chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures of internal (pre-trained) and external (context-dependent) knowledge in LLMs. CoT particularly enhances base model performance and shows synergistic benefits when applied to aligned LLMs. Moreover, our layer-wise analysis reveals that higher layers in LLMs encode more high-confidence knowledge, while low-confidence knowledge tends to emerge in middle-to-lower layers.

Figures

Figures reproduced from arXiv: 2501.01332 by the authors.

Figure 1
Figure 1. Illustration of our framework, K-(CSA)2 , for categorizing knowledge comprehension in LLMs. The framework separates model responses into six categories: three for known knowledge (Highly Known (HK), Maybe Known (MK), Weakly Known (WK)) and three for unknown knowledge (Unconfident Unknown (UU), Mayconfident Unknown (MU), Confident Unknown (CU)). Greedy decoding represents deterministic answers, while random sampling … view at source ↗
Figure 2
Figure 2. Comparative analysis of model internal knowledge performance across different variations. Each subplot demonstrates performance differences between model variants, measuring changes in both accuracy (x-axis) and category score (y-axis). From left to right: (1) IT versus base models, showing a general decrease in performance; (2) CoT versus base models, indicating moderate improvements; (3) IT+CoT versus base models,… view at source ↗
Figure 3
Figure 3. Internal knowledge categories structure across models, sorted left to right by increasing com￾bined accuracy (ratios of top-2 layers 1.HK + 2.MK). The values within each section represent the ratio of knowledge points belonging to each category at differ￾ent training steps. There are six knowledge categories, each represented by a different color. multiple generations. For the full list of LLMs and their capacity we… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Average transition patterns across all evaluated models, comparing how CoT and IT+CoT affect category [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Mean category transition patterns for indi [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Comparative analysis of model external knowledge performance across different variations. Each subplot shows performance differences between variants in terms of accuracy (x-axis) and category score (y-axis). From left to right: (1) IT versus base models, showing varie…
Figure 7
Figure 7. Figure 7: Breakdown of external knowledge category ratios across different language models, showing the relative proportions of each category from HK to CU. Finding 3: Instruction-tuned models show su￾perior context understanding, with IT+CoT pro￾viding more benefits for both in…
Figure 8
Figure 8. Figure 8: Heatmap visualization of knowledge distribution across model layers for (a) Llama-2-7b and (b) Llama [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Changes in internal knowledge categories across different training steps for the Pythia-2.8b (Left figure) and Pythia-1.4b (Right figure) model 1000 35000 70000 105000 140000 Steps 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 Accuracy Accuracy Across Training Steps Pythia-1…
Figure 12
Figure 12. Figure 12: Accuracy and Category Score for external [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 13
Figure 13. Figure 13: Additional results for knowledge category structure transition [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: The detailed demonstrations of ratios for models across categories. [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 4 canonical work pages

  1. [1]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2024. https://openreview.net/forum?id=5x788rqbcj Physics of language models: Part 3.1, knowledge storage and extraction . In Forty-first International Conference on Machine Learning

    2024

  2. [2]

    Amos Azaria and Tom Mitchell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.68 The internal state of an LLM knows when it ' s lying . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967--976, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  3. [3]

    Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dar...

    arXiv 2022

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

  5. [5]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2024. https://arxiv.org/abs/2212.03827 Discovering latent knowledge in language models without supervision . Preprint, arXiv:2212.03827

    arXiv 2024

  6. [6]

    Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig

    Zorik Gekhman, G. Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. https://api.semanticscholar.org/CorpusID:269635770 Does fine-tuning llms on new knowledge encourage hallucinations? ArXiv, abs/2405.05904

    arXiv 2024

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  8. [8]

    Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023. https://arxiv.org/abs/2307.10236 Look before you leap: An exploratory study of uncertainty measurement for large language models . Preprint, arXiv:2307.10236

    arXiv 2023

Show all 35 references
  1. [9]

    Jiang, Alexandre Sablayrolles, and Arthur Mensch et al

    Albert Q. Jiang, Alexandre Sablayrolles, and Arthur Mensch et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . Preprint, arXiv:2310.06825

    2023 arXiv

  2. [10]

    Adam Tauman Kalai and Santosh S. Vempala. 2024 a . https://doi.org/10.1145/3618260.3649777 Calibrated language models must hallucinate . In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, STOC 2024, page 160–171, New York, NY, USA. Association for Computin...

    2024

  3. [11]

    Adam Tauman Kalai and Santosh S. Vempala. 2024 b . https://arxiv.org/abs/2311.14648 Calibrated language models must hallucinate . Preprint, arXiv:2311.14648

    2024 arXiv

  4. [12]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  5. [13]

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.397 H alu E val: A large-scale hallucination evaluation benchmark for large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natur...

    2023 doi

  6. [14]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024. https://arxiv.org/abs/2403.09972 Think twice before trusting: Self-detection for large language models through comprehensive answer reflection . Preprint, arXiv:2403.09972

    2024 arXiv

  7. [15]

    Justin Lovelace and Carolyn Ros \'e . 2022. https://doi.org/10.18653/v1/2022.emnlp-main.398 A framework for adapting pre-trained language models to knowledge graph completion . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 593...

    2022 doi

  8. [16]

    Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, et al. 2024. https://api.semanticscholar.org/CorpusID:268253470 Large language models surpass human experts in predicting neuroscience results . Nature human behaviour

    2024

  9. [17]

    OpenAI. 2024 a . https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  10. [18]

    OpenAI. 2024 b . https://openai.com/index/learning-to-reason-with-llms/ Learning to reason with llms

    2024

  11. [19]

    Long Ouyang, Jeffrey Wu, Xu Jiang, et al. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf Training language models to follow instructions with human feedback . In Advances in Neural Information Processing S...

    2022

  12. [20]

    Singhal, Shekoofeh Azizi, Tao Tu, et al

    K. Singhal, Shekoofeh Azizi, Tao Tu, et al. 2022. https://api.semanticscholar.org/CorpusID:255124952 Large language models encode clinical knowledge . Nature, 620:172 -- 180

    2022

  13. [21]

    Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm. arXiv preprint arXiv:2408.12076

    2024 arXiv

  14. [22]

    Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.18 Head-to-tail: How knowledgeable are large language models ( LLM s)? A . K . A . will LLM s replace knowledge graphs? In Proceedings of the 2024 Conference of the Nor...

    2024 doi

  15. [23]

    Gemma Team. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118

    2024 arXiv

  16. [24]

    Hugo Touvron, Louis Martin, and Kevin Stone et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288

    2023 arXiv

  17. [25]

    Trinh, Yuhuai Wu, Quoc V

    Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. https://api.semanticscholar.org/CorpusID:267032902 Solving olympiad geometry without human demonstrations . Nature, 625:476 -- 482

    2024

  18. [26]

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. https://arxiv.org/abs/2411.04368 Measuring short-form factuality in large language models . Preprint, arXiv:2411.04368

    2024 arXiv

  19. [27]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  20. [28]

    Kevin Wu, Eric Wu, and James Zou. 2024. https://arxiv.org/abs/2404.10198 Clasheval: Quantifying the tug-of-war between an llm's internal prior and external evidence . Preprint, arXiv:2404.10198

    2024 arXiv

  21. [29]

    An Yang, Baosong Yang, and Binyuan Hui et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    2024 arXiv

  22. [30]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.551 Do large language models know what they don ' t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--86...

    2023 doi

  23. [31]

    Haeun Yu, Pepa Atanasova, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.acl-long.444 Revealing the parametric knowledge of language models: A unified framework for attribution methods . In Proceedings of the 62nd Annual Meeting of the Association for Computat...

    2024 doi

  24. [32]

    Jiawei Zhang, Chejian Xu, Yu Gai, Freddy Lecue, Dawn Song, and Bo Li. 2024. https://arxiv.org/abs/2404.02935 Knowhalu: Hallucination detection via multi-form knowledge based factual checking . Preprint, arXiv:2404.02935

    2024 arXiv

  25. [33]

    Daniel Zheng, Richard Bai, Yizhe Zhang, Yi (Siri) Su, Xiaochuan Niu, and Navdeep Jaitly. 2024. https://arxiv.org/abs/2312.11539 Kglens: Towards efficient and effective knowledge probing of large language models with knowledge graphs . In ACL

    2024 arXiv

  26. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.