Pith. sign in

REVIEW 4 major objections 4 minor 38 references

Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models encode a linear signal in their internal state that predicts whether an upcoming factual recall will be correct, before the words are spoken.

desk verdict Linear probes separate self-labeled known/forgotten facts, but the self-referential labels make the 'self-awareness' claim circular; still a solid extension worth reviewing. read the letter →

arxiv 2505.21399 v1 pith:4F72O4EY submitted 2025-05-27 cs.CL cs.AIcs.LG

Hovhannes Tamoyan , Subhabrata Dutta , Iryna Gurevych This is my paper
classification cs.CLcs.AIcs.LG
keywords factualself-awarenesshallucinationdetectionlinearprobingresidualstreamknown-forgottenclassificationemergenceintraininglanguagemodelscalinginterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models carry a generation-time 'compass' for factual recall: before the model emits an attribute for a given entity and relation, its internal state already distinguishes upcoming correct recalls from upcoming wrong ones. The evidence is a linear probe on the Transformer's residual stream at the final prompt token, which separates 'known' from 'forgotten' facts at up to 82% test accuracy on Gemma 2 2B, with sparse-autoencoder latents showing the same layer-wise pattern. The signal survives quotation marks and unrelated prefix sentences, appears within the first portion of training, and needs a minimum model scale without strengthening further as models grow. A readable-at-generation signal matters because it could stop a hallucinated fact before it is uttered, something post-hoc fact-checking cannot do.

What carries the argument

The load-bearing object is the linear probe on the residual stream: for each prompt, the hidden state at the final token of the template is read out at every layer and classified by a single learned weight vector. The other half of the machinery is the model-dependent annotation scheme, which defines 'known' and 'forgotten' by where the gold attribute's tokens sit in the model's own output distribution (top $k=500$ vs. bottom $l=0.3$), and the paper also computes latent separation scores over sparse-autoencoder activations to show the feature is visible without probe supervision. This pair of tools carries the argument by demonstrating that the same linear readout succeeds across layers, model families, and perturbed prompts.

What would settle it

Re-derive the known/forgotten labels from an external criterion that does not consult the model's own logits — for example, whether the entity-relation-attribute triplet actually appears in the model's pretraining corpus — then retrain the linear probes on that labeling and compare accuracy; if the separation collapses toward chance, the measured signal is output confidence rather than factual self-awareness.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that language models encode meta-knowledge about fine-grained factual associations — not merely about whole entities — as linear directions in the residual stream, and this direction is active at generation time, before the attribute tokens are produced. For each (entity type, entity name, relation) prompt, the model's own output distribution labels the fact 'known' when the gold attribute tokens land in the top 500 predictions ($k=500$) and 'forgotten' when they fall in the bottom 30% of the vocabulary ($l=0.3$), and ambiguous cases are discarded from the dataset. Linear probes on the final-token residual stream then classify the two classes, reaching 0.820 test accuracy — a 0.311 gain over the random baseline — on Gemma 2 2B, with the same probe generalizing across the four entity categories, and the Pythia family showing the same effect at smaller magnitude. The authors read this as evidence that the model knows, at recall time, whether it will be able to recall the correct attribute, in contrast to the post-hoc truth-direction literature that checks the model's output after generation.

Load-bearing premise

The load-bearing premise is the label definition: a fact is called 'known' or 'forgotten' depending on where its gold attribute falls in the model's own output distribution (top $k=500$ tokens vs. bottom $l=0.3$), and the in-between cases are dropped, so the probe may be reading the model's output confidence rather than a distinct factual self-awareness.

Editorial extensions

If this is right

  • A pre-generation hallucination alarm is feasible: a linear readout can flag an upcoming wrong factual recall before the first attribute token is emitted, which post-hoc self-checking cannot do.
  • Factual self-awareness is a thresholded scaling property: Pythia 70M shows no decodable signal, while larger Pythia models improve monotonically, so a minimum model capacity is required for the representation to form.
  • The signal emerges early in training and saturates quickly, meaning additional next-token-prediction training does not keep sharpening this form of self-awareness.
  • The direction tracks entity-relation content rather than surface form: quotation marks and unrelated prefixed sentences barely weaken probe accuracy, while converting prompts into questions and repeating the same entity in few-shot context degrade it measurably.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the known/forgotten labels are defined by the same model's logits, part of the probe's accuracy plausibly reflects output confidence rather than an independent knowledge register; relabeling facts by an external oracle (for instance, presence of the triplet in the pretraining data) would separate the two readings.
  • If the linear direction is causal rather than merely correlational, steering the residual stream along the 'forgotten' direction before decoding could induce the model to abstain from answering — a controllable hallucination-mitigation handle the paper does not test.
  • The labeling scheme discards exactly the partially-known entities the introduction identifies as the hard case (most of the 1,865 entities that are neither fully known nor fully forgotten), so a graded three-way probe would target the region where real hallucination risk concentrates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a factual-recall dataset built from Wikidata for four entity types (player, movie, city, song), labels each (entity, relation) sample as 'known' or 'forgotten' depending on whether the gold attribute tokens fall in the model's own top-500 or bottom-30% next-token predictions, and trains linear probes on the final-prompt-token residual stream to distinguish these classes. The authors report test accuracy up to 0.82 (AUC around 0.90) on Gemma 2 2B, similar results on Gemma 2 9B, smaller positive gains on Pythia models, robustness to quotation, question, few-shot, and random-sentence perturbations, and scaling/checkpoint analyses suggesting an early and size-dependent emergence. The central claim is that LMs encode a linear 'factual self-awareness' signal that predicts whether they will correctly recall an attribute at generation time.

Significance. If the central claim were fully supported, the paper would offer a practically relevant and interpretable signal for detecting hallucination-prone recalls before decoding, complementing post-hoc truth-probe approaches. The use of open models, a Wikidata-derived dataset, multiple model families, and a code link are strengths for reproducibility, and the SAE comparison is a useful sanity check that the linear-probe finding is not purely an artifact of probe training. However, the significance is currently limited by the self-referential label definition: the labels are thresholded functions of the very logits produced from the probed representation, so the high probe accuracy may largely reflect output confidence rather than an independent knowledge-awareness state.

major comments (4)
  1. [Section 3 (Dataset) and central claim] The known/forgotten label for a sample is a deterministic function of the probed model's own output distribution: gold attribute tokens in the top-500 versus bottom-30% of the next-token distribution. Because the residual stream at the final prompt token is the input to the unembedding layer that produces those logits, a linear probe trained on this residual stream can achieve high accuracy simply by recovering the model's confidence in the gold token. This does not establish the headline claim of an 'internal compass that dictate the correctness of factual recall'; the label is a proxy for the model's output confidence, not for externally grounded recall success. I suggest two concrete controls: (i) train the same linear probe to classify samples with labels defined by whether greedy decoding matches the Wikidata attribute, and (ii) compare probe accuracy against a linear classifier applied directly to the pre-softmax logits of the gold token or to the full logit vector. If the residual-stream probe does not exceed these baselines, the 'self-awareness' interpretation should be substantially weakened. In addition, the paper discards all samples with intermediate confidence (between top-500 and bottom-30% of the vocabulary), which removes the hardest cases and makes the binary task artificially easy; reporting three-way or full-confidence classification would test the feature more fairly.
  2. [Section 4.2 and Appendix A (template selection)] The main experiments use template2_balanced, which was explicitly selected after experimenting with several templates and, according to Appendix A, 'only template2 consistently captures the self-awareness signal without interference from confounding factors.' This selection is made on the same data and task that is then used for the headline robustness and scaling results, so the reported robustness applies only to a template that was chosen for good behavior. The perturbation study (quotation, question, few-shot, random sentence) modifies this one template; it does not establish that the self-awareness signal is robust across template families. Please report probe results for all templates in Table 3 and for perturbations applied to at least one non-selected template, or clearly frame the contribution as template-specific.
  3. [Section 5 and Figure 8 (training dynamics)] The manuscript does not state whether the known/forgotten labels are recomputed at each checkpoint of Pythia 1.4B or are fixed using the final model. If labels are recomputed, the probe task changes over training because the set of 'known' samples changes, so the accuracy trajectory is not a single learning curve. If labels are fixed from the final model, early checkpoints are tested against labels that do not reflect their own behavior, which trivially yields near-random accuracy at initialization. The 'emergence early in training' conclusion therefore needs a precise statement of the labeling procedure, and preferably both analyses: fixed labels and per-checkpoint labels.
  4. [Table 1 and Figure 7 (cross-model scaling)] The claim that 'larger models develop more robust and generalizable representations' is based on a comparison across two different model families (Gemma and Pythia) and within Pythia on accuracy gains over random baseline that are small (0.028–0.120) and accompanied by baseline accuracies that vary with class balance (from 0.674 to 0.822). Because the labels are recomputed per model from that model's own output distribution, the 'same' samples receive different labels in different models, so the scaling comparison mixes changes in the label definition with changes in the representation. Please report the proportion of label agreement across models and, if possible, evaluate on a fixed external label set.
minor comments (4)
  1. [Section 4.1] The definition of MaxMin in the text ('maxj mint sknown,t l,j') does not match the figure captions ('MaxMin denotes the difference between max and min class means'); please align the notation and the captions.
  2. [Table 2] The header 'Train (shared across all) / Test (varies by modification)' is unclear; please specify whether the test labels are recomputed under each perturbation, and whether the same underlying samples and splits are used across all modification types.
  3. [Figure 1 and template examples] Several template examples contain garbled spacing and word order (for example 'The the is . director of movie Inception ...'), which makes the prompting scheme hard to follow; please clean the figures and the example sentences.
  4. [Section 3 (Dataset)] The choice of top-k = 500 and bottom-l = 0.3 is motivated by class balance, but the paper does not report how sensitive the main results are to intermediate (k, l) values; the heatmaps in Appendix D are a useful start, but they focus on extreme configurations and should be connected more explicitly to the headline accuracy.

Circularity Check

1 steps flagged · score 6.0 of 10

Partially circular: known/forgotten labels are thresholded functions of the probed model's own logits, and the probe reads that same token's residual stream, so the 0.82 accuracy partly re-derives the label from the representation that generated it; the paper's own 'proxy'/'model-dependent' concessions support this, while layer profiles, perturbation transfer, and scaling curves retain…

  1. self definitional [Section 3, Dataset and Linear Probe (Eq. 1); Contributions; Limitations]
    "We use a proxy definition for the same, using the logit distribution of the LM itself: if a model is able to signal that it can (or cannot) recall a certain attribute of an entity correctly (i.e., assign a high logit value to the respective token), we diagnose the behavior as factually self-aware. ... A sample is labeled as known if more of its gold label tokens appear among the top-k predictions in the logit space. Conversely, if more of the gold label tokens fall below the l-th percentile of the logit distribution, the sample is classified as forgotten."

    The probe target y in Eq. (1) is a deterministic thresholded function of the probed model's own next-token logits at the final prompt token, while the probe input x_{l,T_i} is that same token's residual stream. Transformer logits are an affine map from the final residual state, so a linear probe at the output layer can approximate the gold-token confidence that generated the label. The 0.82 accuracy and its transfer thus partly re-describe the model's own confidence rather than independently evidencing a distinct 'compass' for correct recall. The paper calls the annotation 'model-dependent' and a 'proxy'; its Limitations state the causal components are unknown. Robustness and scaling reuse the same labels, so the confound persists.

full rationale

The central measurement loop is: (i) known/forgotten labels are defined by thresholding the probe-target model's own next-token logits (top-500 vs. bottom-30% of the vocabulary, with intermediate ranks excluded from the retained samples); (ii) the linear probe is trained on the residual stream of the same final prompt token whose next-token distribution defines the label; and (iii) the resulting 0.82 accuracy is reported as evidence of an internal 'compass' that dictates correct recall. The reduction is strongest at the output layer, where logits are an affine function of the probed residual stream, so the probe can partially re-derive the label from the representation itself; for middle layers the link is not by-construction affine, and the intermediate-layer peak, the held-out generalization with a single fixed probe direction, the SAE agreement, and the transfer across prompt perturbations are genuine empirical content that keeps the derivation from being fully tautological. However, every robustness and scaling analysis reuses the same self-derived labels, and no external correctness signal (e.g., Wikidata validation of the answered attribute or another model's judgment) is ever introduced, so the construct validity of the headline claim remains partially defined by the model's own confidence. Two weaker concerns, flagged for completeness: (a) the input template template2_balanced was selected after trying four phrasings, with the appendix noting 'only template2 consistently captures the self-awareness signal,' a post hoc choice on the dependent variable for the headline number; and (b) the sole self-citation (Bajpai et al. 2024, co-authored by Dutta) appears only in Related Work and is not load-bearing. Because the paper is transparent about the 'model-dependent' and 'proxy' nature of its annotation and its Limitations admit the causal components are unknown, the result is partially rather than fully circular; score 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced; the 'self-awareness signal' is an interpretation of learned probe directions, not a postulated entity. The main free parameters are the annotation thresholds and the chosen template.

free parameters (3)
  • top-k threshold = 500
    Samples with the gold attribute in the top 500 output tokens are labeled 'known'. The paper analyzes k from 5 to 5000 and chooses 500 as a balance point.
  • bottom-l fraction = 0.3
    Samples with gold attribute in the bottom 30% of the output distribution are labeled 'forgotten'. Chosen to balance classes across models.
  • template2_balanced = N/A
    Selected after other templates were found to have spurious correlations; this is a post-hoc design choice that affects all reported results.
assumptions (3)
  • domain assumption Linear separability in the residual stream implies the model encodes self-awareness as a feature.
    The paper interprets linear probe accuracy as evidence of an internal 'compass'. This is standard in interpretability but is an interpretive leap.
  • ad hoc to paper The top-k/bottom-l logit proxy faithfully demarcates known from forgotten factual associations.
    Used to label the dataset (Section 3). The proxy is model-dependent and excludes intermediate confidence cases.
  • domain assumption The final token residual stream of the prompt contains the relevant information for the upcoming recall.
    Probes are trained only on the last token residual per layer, following prior work (Meng et al. 2022, Geva et al. 2023).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling." pith.science (2026). https://pith.science/paper/4F72O4EY

@misc{pith2026250521399,
  author       = {Pith},
  title        = {Pith review of: Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4F72O4EY}},
  note         = {Machine review of arXiv:2505.21399}
}
read the original abstract

Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs). Prior findings suggest LLMs can (sometimes) detect factual incorrectness in their generated content (i.e., fact-checking post-generation). In this work, we provide evidence supporting the presence of LLMs' internal compass that dictate the correctness of factual recall at the time of generation. We demonstrate that for a given subject entity and a relation, LLMs internally encode linear features in the Transformer's residual stream that dictate whether it will be able to recall the correct attribute (that forms a valid entity-relation-attribute triplet). This self-awareness signal is robust to minor formatting variations. We investigate the effects of context perturbation via different example selection strategies. Scaling experiments across model sizes and training dynamics highlight that self-awareness emerges rapidly during training and peaks in intermediate layers. These findings uncover intrinsic self-monitoring capabilities within LLMs, contributing to their interpretability and reliability.

Figures

Figures reproduced from arXiv: 2505.21399 by the authors.

Figure 1
Figure 1. Given an input comprising entity type, entity name, and relation, we obtain the model’s [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Top-five latent separation scores across transformer layers using SAE activations from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Latent separation scores across layers using Linear Probe activations from Gemma 2 2B. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Layer-wise linear probe accuracy for Gemma 2 2B and Pythia 12B. Orange/blue: train/test; [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Known-Forgotten sample ratio for each (k, l) configuration, aggregated across all models. Lower values (darker) indicate more balanced retention, helping identify the globally optimal (k, l) setting that generalizes across models. Class Balance Ratio Pythia 70M Pythia …
Figure 7
Figure 7. Figure 7: Accuracy gain over random base [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 9
Figure 9. Figure 9: Latent separation scores using SAE activations on Gemma 2 9B. Left: [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Latent separation scores using Linear Probe activations on Gemma 2 9B. Left: [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Accuracy gain over a random baseline from linear probes on: (a) Gemma 2 2B, and (b) [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: k-l parameters dependence on number of class balance (known and forgotten samples ratio) heatmaps for Gemma 2 models. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: k-l parameters dependence on number of class balance (known and forgotten samples ratio) heatmaps for Pythia models. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages

  1. [1]

    Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu, Andre Freitas, and Mustafa A. Mustafa. A survey of safety and trustworthiness of large language models through the lens of verification and validation. Artificial Intelligence...

    doi:10.1007/s10462-024-10824-0 2024

  2. [2]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43 0 (2), January 2025. ISSN 1046-8188. doi:10.1145/3703155. URL https://doi.org...

    doi:10.1145/3703155 2025

  3. [3]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield - Dodds, Nova DasSarma, Eli Tran - Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kra...

    arXiv doi:10.48550/arxiv.2207.05221 2022

  4. [4]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi\' e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 41451--41530. Curran Associates, Inc., 2023...

    work page 2023

  5. [5]

    Mitchell

    Amos Azaria and Tom M. Mitchell. The internal state of an LLM knows when it's lying. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pages 967--976. Association for Computational Linguistics, 2023. doi:10.18653/V1/2023.FINDINGS-EMNLP.68. URL http...

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  6. [6]

    Discovering latent knowledge in language models without supervision

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=ETKGuby0hcs

    2023

  7. [7]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, K...

    work page 2023

  8. [8]

    Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. CoRR, abs/2308.03188, 2023. doi:10.48550/ARXIV.2308.03188. URL https://doi.org/10.48550/arXiv.2308.03188

    arXiv doi:10.48550/arxiv.2308.03188 2023

Show all 38 references
  1. [9]

    On the self-verification limitations of large language models on reasoning and planning tasks

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenRevie...

    2025

  2. [10]

    Do i know this entity? knowledge awareness and hallucinations in language models

    Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257, 2024

    2024 arXiv

  3. [11]

    Gemma 2: Improving open language models at a practical size, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv. org/abs/2...

    2024 arXiv

  4. [12]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Sid Black, Eric Hallahan, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023

    2023 arXiv

  5. [13]

    Do large language models know what they don't know? In Anna Rogers, Jordan L

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know? In Anna Rogers, Jordan L. Boyd - Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, Toron...

    2023 doi

  6. [14]

    Can llms replace neil degrasse tyson? evaluating the reliability of llms as science communicators

    Prasoon Bajpai, Niladri Chatterjee, Subhabrata Dutta, and Tanmoy Chakraborty. Can llms replace neil degrasse tyson? evaluating the reliability of llms as science communicators. In Yaser Al - Onaizan, Mohit Bansal, and Yun - Nung Chen, editors, Proceedings of the 2024 Conferenc...

    2024

  7. [15]

    Tell me about yourself: Llms are aware of their learned behaviors

    Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: Llms are aware of their learned behaviors. arXiv preprint arXiv:2501.11120, 2025

    2025 arXiv

  8. [16]

    Large language models must be taught to know what they don't know

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Katie Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don't know. In Amir Globersons, Lester Mackey, Danielle Belgrave, An...

    2024

  9. [17]

    Self-contrast: Better reflection through inconsistent solving perspectives

    Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  10. [18]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali,...

    2023

  11. [19]

    INSIDE: llms' internal states retain the power of hallucination detection

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. INSIDE: llms' internal states retain the power of hallucination detection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Op...

    2024

  12. [20]

    LLM internal states reveal hallucination risk faced with a query

    Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. LLM internal states reveal hallucination risk faced with a query. CoRR, abs/2407.03282, 2024. doi:10.48550/ARXIV.2407.03282. URL https://doi.org/10.48550/arXiv.2407.03282

    2024 doi arXiv

  13. [21]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

    2023

  14. [22]

    Sparse autoencoders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  15. [23]

    The linear representation hypothesis and the geometry of large language models

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023

    2023 arXiv

  16. [24]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26, 2013

    2013

  17. [25]

    Inference-time intervention: Eliciting truthful answers from a language model

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36: 0 41451--41530, 2023 b

    2023

  18. [26]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023

    arXiv

  19. [27]

    Saes are highly dataset dependent: A case study on the refusal direction

    Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. Saes are highly dataset dependent: A case study on the refusal direction. In Alignment Forum, 2024

    2024

  20. [28]

    Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky - Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adri \` a Garriga - Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, N...

    2025 doi arXiv

  21. [29]

    Learning multi-level features with matryoshka sparse autoencoders

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. CoRR, abs/2503.17547, 2025. doi:10.48550/ARXIV.2503.17547. URL https://doi.org/10.48550/arXiv.2503.17547

    2025 doi arXiv

  22. [30]

    Wikidata

    Denny Vrandečić and Markus Krötzsch. Wikidata. https://www.wikidata.org, 2023

    2023

  23. [31]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35: 0 17359--17372, 2022

    2022

  24. [32]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  25. [33]

    Fact finding: Attempting to reverse-engineer factual recall on the neuron level

    Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall, 2023. AI Alig...

    2023

  26. [34]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J \'a nos Kram \'a r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024

    2024 arXiv

  27. [35]

    Demystifying prompts in language models via perplexity estimation

    Hila Gonen, Yonatan Belinkov, Ido Dagan, and Yoav Goldberg. Demystifying prompts in language models via perplexity estimation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023

    2023

  28. [36]

    Quantifying lms’ sensitivity to spurious prompt formatting

    Noam Sclar, Ehud Guriel, and Omer Levy. Quantifying lms’ sensitivity to spurious prompt formatting. In Proceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  29. [37]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  30. [38]

    State of what art? a call for multi-prompt llm evaluation

    Itay Mizrahi, Nimrod Sznajder, Libby Barak, and Yoav Goldberg. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics (TACL), 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.