Pith. sign in

REVIEW 5 major objections 5 minor 45 references

REAL: Reading Out Transformer Activations for Precise Localization in Language Model Steering

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read REAL is a framework that localizes behavior-relevant attention heads and layers by reading out vector-quantized activation codes, claiming a 20% average relative improvement (up to 81.5%) over ITI on truthfulness steering.

desk verdict Useful module-selection recipe, but the headline 20% gain over ITI is confounded by per-head weighting; needs equal-strength control. read the letter →

arxiv 2506.08359 v3 pith:QO2DPIO2 submitted 2025-06-10 cs.CL

Li-Ming Zhan , Bo Liu , Chengqiang Xie , Jiannong Cao , Xiao-Ming Wu This is my paper
classification cs.CL
keywords activationsteeringmoduleselectionvector-quantizedautoencoderattentionheadstruthfulnessknowledgeconflictsinference-timeinterventionrepresentationengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces REAL, a method for identifying which attention heads or layers in a Transformer are most responsible for a target behavior such as truthfulness, knowledge selection, or general alignment. Instead of linear probes, REAL trains a vector-quantized autoencoder on each module's activations, with a shared learnable codebook and a supervised contrastive loss, so that discrete codes separate behavior-relevant from behavior-irrelevant features. An autoregressive prior over these codes yields a likelihood for each response, and the AUC of those likelihoods ranks modules and sets steering strength. The central claim is that steering with REAL-selected modules outperforms linear-probe-based selection like ITI by 20% on average, up to 81.5%, and that the selected heads transfer across tasks without retraining.

What carries the argument

The central object is a vector-quantized autoencoder (VQ-AE) per module, whose latent embedding is split into semantic units, each quantized through a shared learnable codebook. A supervised contrastive loss pushes the quantized representations of positive and negative behavior examples apart, and a lightweight autoregressive prior (a single-layer GRU) models the discrete code sequences. The AUC-ROC of this prior over a development set is the behavior-relevance score, which both selects the top modules and weights the steering strength applied to each.

What would settle it

A direct test: for a fixed model and behavior, compute REAL's per-module AUC rankings and separately measure the actual improvement in the target metric when steering each module individually. If the rank correlation between AUC and per-module improvement is low or negative, or if the ranking changes substantially across random retraining seeds of the VQ-AE and prior, the selection framework's core assumption fails.

Watch

Extended reading notes

Core claim

REAL claims that behavior-relevant modules in an LLM can be located by quantizing each module's hidden activations into discrete semantic units, training an autoregressive model on the codes of positive examples, and ranking modules by how well the resulting likelihoods discriminate behavior-aligned from behavior-violating responses. The method is evaluated across eight Llama and Qwen models on nine datasets, covering truthfulness, open-domain QA under knowledge conflicts, and alignment tasks. When REAL-selected heads are plugged into the ITI protocol, MC1 and MC2 scores on TruthfulQA improve substantially, and when REAL selects layers for SAE-based steering on NQSwap and MacNoise, exact-match accuracy exceeds the SPARE baseline. The same truthfulness-selected heads also transfer to MQuAKE and CLUTRR knowledge-seeking tasks, supporting the claim of zero-shot cross-domain generalization.

Load-bearing premise

The load-bearing premise is that the AUC of the autoregressive prior, computed on a small dev set (82 TruthfulQA examples), is a stable and reliable predictor of how much mean-difference steering on that head or layer will improve generation.

Editorial extensions

If this is right

  • Steering vectors applied to REAL-selected heads improve truthfulness metrics over ITI-selected heads across multiple Llama and Qwen models.
  • The same heads selected on TruthfulQA generalize to knowledge-seeking tasks, improving exact-match accuracy on MQuAKE and CLUTRR without retraining.
  • REAL identifies a broader set of layers than SPARE for knowledge-conflict steering, including layers 5, 11, 12, and 13–16, yielding higher contextual and parametric exact match on NQSwap and MacNoise.
  • On general alignment behaviors, intervening on top-ranked layers with positive steering multipliers increases behavior-aligned token probabilities, while negative multipliers decrease them, and low-ranked layers show weaker or inconsistent control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because REAL only needs paired behavior-contrastive responses, it could serve as a general-purpose behavior-localization tool for any binary behavior elicitable in generation, including safety-relevant attributes beyond truthfulness.
  • The autoregressive prior's role may be mostly nonlinear separation in a quantized space; if a simpler classifier on the discrete codes matches the AUC, the GRU's sequential modeling could be unnecessary overhead.
  • A testable extension is replacing the small-dev-set AUC estimate with cross-validation across dev splits or with a pooled multi-module estimator, which would reveal whether the ranking stability is the real bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces REAL, a method for selecting behavior-relevant attention heads and layers for activation steering. For each candidate module, a vector-quantized autoencoder maps last-token activations into a codebook-quantized latent space, a supervised contrastive loss separates positive and negative behavior examples, and an autoregressive prior over the discrete codes is scored by AUC on a development set. The resulting score ranks heads and layers and also sets per-head steering strength in Eq. (6). REAL is evaluated on TruthfulQA, MQuAKE/CLUTRR, NQSwap/MacNoise, and four AI-risk behaviors across Llama and Qwen models, reporting substantial gains over ITI and LoFiT on truthfulness steering, gains over SPARE on knowledge-conflict selection, and cross-domain transfer.

Significance. If the central claim were established, REAL would be a useful contribution: it offers a nonlinear, module-selection mechanism that plugs into existing steering-vector methods, and the evaluation breadth across several model families and nine datasets is a clear strength. The paper also reports training costs and includes a qualitative head-overlap analysis. However, the central localization claim is not yet supported by the experimental design: the headline improvement over ITI conflates module selection with per-head weighting, hyperparameters are selected on test-set metrics, the VQ-AE is not compared with a plain nonlinear probe, and the general-alignment results lack a baseline. These issues are fixable with additional controlled comparisons, which is why I recommend major revision rather than rejection.

major comments (5)
  1. [§3.2, Eq. (6); Table 2] The reported 'REALITI vs ITI' gains change two variables simultaneously: the selected head set and the per-head steering strength, because Eq. (6) scales each selected head by s^(l,i)/s_max while ITI uses a single constant epsilon. The abstract and Section 4.1 attribute the gains to better localization, but the design cannot separate localization from the weighting schedule. Please add an equal-strength REAL baseline that applies a constant epsilon to the REAL-selected heads, and a score-weighted ITI baseline; the same confound applies to the cross-domain transfer numbers in Table 3 if the score-weighted intervention is used there.
  2. [§4.4, Fig. 6] The hyperparameters K, U, and alpha are selected by maximizing MC1/MC2 on the TruthfulQA test set; for example, the chosen K=32, U=8, and alpha=1e-3 are read off Fig. 6, which reports test-set scores. This is test-set selection, so the reported absolute numbers are optimistic and the comparison with baselines is not on equal footing. Please use a validation split for hyperparameter selection and report the test numbers for the single final configuration.
  3. [§3.2; Table 1] The module relevance score is an AUC computed on a development set of 82 TruthfulQA examples (Table 1), and this ranking drives both head selection and steering strength. The entire framework assumes this small-sample AUC is a stable and predictive proxy for steering success, but the paper provides no reliability analysis. I would like to see bootstrap confidence intervals on the AUC ranking, repeated dev-set splits, and a correlation between AUC rank and intervention gain on held-out data.
  4. [§3.1–3.2; Table 2] The motivation for REAL is that linear probes fail on entangled features, but no experiment separates the contribution of the VQ-AE from that of a plain nonlinear probe, such as a single-layer MLP classifier on the same activations, or a non-quantized autoencoder. Without such an ablation, the gains in Table 2 could be due to nonlinearity or to the score-weighting in Eq. (6) rather than to the codebook-based disentanglement that the paper emphasizes.
  5. [§4.3; Table 5] Table 5 reports average token probabilities for top-5 and last-5 layers with no baseline and no random-layer control, and the paper explicitly states it reports 'without baseline comparisons'. As a result, the claim of broad applicability for general alignment is not yet supported. Please add a random-layer-selection baseline and, ideally, a linear-probe layer-ranking baseline using the same steering vectors.
minor comments (5)
  1. [Abstract vs. Conclusion] The abstract says the method is evaluated across eight LLMs, while the conclusion says seven; please reconcile the count and the model list, especially since Appendix A mentions Vicuna-7B but Table 2 does not include it.
  2. [Eq. (6) vs. Table 5] Equation (6) uses a real-valued score ratio as a multiplier, while Table 5 reports discrete multipliers -1, 0, and 1; please clarify how the layer-level multipliers relate to the score-based weighting.
  3. [§4.1] The paper states that each head requires approximately 50 seconds and 758 MB of GPU memory; please clarify whether these figures include both the VQ-AE training and the autoregressive prior training, or only the VQ-AE.
  4. [Appendix B] In the noisy-OR layer scoring, the global threshold tau is stated as 5, but the percentile p used to compute tau is not defined; please specify the relationship between tau and p.
  5. [Reproducibility] No code or checkpoints are released, and although many results are averaged over three runs, standard deviations are not reported for Tables 3, 4, and 5; a reproducibility statement would help reviewers verify the experimental claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claims are evaluated on held-out and cross-domain data rather than being equivalent to the method's fitted inputs.

full rationale

The derivation chain is self-contained and not circular. REAL computes a per-module relevance score by training a VQ-AE with a supervised contrastive loss on behavior labels, fitting an autoregressive prior over the resulting discrete codes, and measuring AUC on a held-out dev split; this score is then used to select heads/layers and to set per-head steering strength in Eq. (6). Although the VQ-AE objective and the mean-difference steering vectors share the same behavior labels, this is supervised feature selection rather than circularity: the paper's reported outcomes, including TruthfulQA MC1/MC2 on held-out test examples, exact-match accuracy on MQUAKE/CLUTRR, NQSWAP/MACNOISE, and AI-risk behavior probabilities, are measured on held-out test sets or on different tasks and domains, so the claims are externally falsifiable. No load-bearing step reduces to its own input by construction: the AUC ranking could fail to predict steering gains, and the Eq. (6) weighting could in principle hurt, so the reported improvements are not forced. The paper cites prior work such as ITI, LoFiT, SPARE, and CAA, but it does not rely on any self-citation, imported uniqueness theorem, or ansatz smuggled in via citation to justify its central claim. The main validity caveat is a confound in the REALITI versus ITI comparison, since Eq. (6) changes both the head set and the per-head weighting; however, that is an experimental-design issue concerning attribution of the gain to localization, not a circular derivation.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The method rests on several domain assumptions about activation geometry and scoring. No new physical or external entities are introduced, so the invented-entities list is empty. The main free parameters are the VQ-AE hyperparameters and the selection thresholds, some of which are tuned on benchmark metrics.

free parameters (8)
  • Codebook size K = 32
    Chosen via ablation in Fig. 6b on Qwen2.5-7B and Llama3.1-8B-Instruct.
  • Number of semantic units U = 8
    Chosen via ablation in Fig. 6a; U=1 collapses the codebook for Llama3.1-8B-Instruct.
  • Contrastive loss weight alpha = 1e-3
    Chosen via ablation in Fig. 6c on Llama2-7B.
  • VQ-AE commitment coefficient beta = 0.25
    Fixed as a standard VQ hyperparameter in Appendix A.
  • Number of selected heads S = 48 (ITI protocol), 32 (LoFiT protocol)
    Set to match the corresponding baseline protocols in Section 4.1, not independently tuned.
  • Layer AUC threshold = 88%
    Used to select layers for knowledge-selection steering in Section 4.2.
  • Noisy-OR head threshold tau = 5th percentile
    Set as a global percentile for layer scoring in Appendix B.
  • Steering strength epsilon = Not reported
    The multiplier in eq. (6) is not given a value; presumably tuned per task, which is not disclosed.
assumptions (4)
  • domain assumption Last-token activations summarize a head's behavior-relevant content.
    Invoked in Section 3.1: 'we use the last token activations of attention heads, h_T, to distill behavior indicative information.'
  • ad hoc to paper AUC of codeword likelihoods on a small dev set is a stable predictor of steering effectiveness.
    The whole selection criterion in Section 3.2 relies on this proxy; the dev set for TruthfulQA has 82 examples.
  • domain assumption Supervised contrastive loss yields disentanglement between behavior-relevant and irrelevant features.
    Stated as a key insight in Section 3.1, not proven; the ablations give indirect support.
  • domain assumption Noisy-OR aggregation of head scores is a valid layer relevance score.
    Introduced in Appendix B for layer-level steering in general alignment tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REAL: Reading Out Transformer Activations for Precise Localization in Language Model Steering." pith.science (2026). https://pith.science/paper/QO2DPIO2

@misc{pith2026250608359,
  author       = {Pith},
  title        = {Pith review of: REAL: Reading Out Transformer Activations for Precise Localization in Language Model Steering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QO2DPIO2}},
  note         = {Machine review of arXiv:2506.08359}
}
read the original abstract

Inference-time steering aims to alter a large language model's (LLM's) responses without changing its parameters, but a central challenge is identifying the internal modules that most strongly govern the target behavior. Existing approaches often rely on simplistic cues or ad hoc heuristics, leading to suboptimal or unintended effects. We introduce REAL, a framework for identifying behavior-relevant modules (attention heads or layers) in Transformer models. For each module, REAL trains a vector-quantized autoencoder (VQ-AE) on its hidden activations and uses a shared, learnable codebook to partition the latent space into behavior-relevant and behavior-irrelevant subspaces. REAL quantifies a module's behavioral relevance by how well its VQ-AE encodings discriminate behavior-aligned from behavior-violating responses via a binary classification metric; this score guides both module selection and steering strength. We evaluate REAL across eight LLMs from the Llama and Qwen families and nine datasets spanning truthfulness enhancement, open-domain QA under knowledge conflicts, and general alignment tasks. REAL enables more effective inference-time interventions, achieving an average relative improvement of 20% (up to 81.5%) over the ITI method on truthfulness steering. In addition, the modules selected by REAL exhibit strong zero-shot generalization in cross-domain truthfulness-steering scenarios.

Figures

Figures reproduced from arXiv: 2506.08359 by the authors.

Figure 2
Figure 2. The top 48 attention heads in Llama2-7B-Chat identified by ITI and REAL, based on TRUTHFULQA. evidence from training, it may lack robustness and generalizability. Other approaches [3, 8, 7] involve computationally intensive cross-validation and manual selection of layers for intervention. These limitations highlight the need for a more principled, effective, and efficient method for module selection. Step 2: Steerin… view at source ↗
Figure 3
Figure 3. Overview of the proposed REAL framework. We use activations from each attention head to train a VQ-AE, aiming to learn a disentangled, quantized latent space. The VQ-AE is trained using a latent contrastive loss in conjunction with the standard VQ loss. The discrete encodings produced by the VQ-AE are then used to train a scoring function that outputs the probability of a given encoding corresponding to the target b… view at source ↗
Figure 4
Figure 4. t-SNE visualization comparing the highest-performing head (11th layer, 22nd head; top row) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Heatmaps of behavior-relevance scores for each attention head on [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on semantic unit count, codebook size, and contrastive loss weight. MC1 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 28 canonical work pages

  1. [1]

    Viégas, Hanspeter Pfister, and Martin Wattenberg

    Kenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information...

    work page 2023

  2. [2]

    Steering gpt-2-xl by adding an activation vector

    Alex Turner, Monte MacDiarmid, David Udell, Lisa Thiergart, and Ulisse Mini. Steering gpt-2-xl by adding an activation vector. InAI Alignment Forum, 2023

    work page 2023

  3. [3]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand, Aug...

    work page 2024

  4. [4]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  5. [5]

    Locating and editing factual associations in gpt.Advances in neural information processing systems, 35:17359–17372, 2022

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt.Advances in neural information processing systems, 35:17359–17372, 2022

    2022

  6. [6]

    Programming refusal with conditional activation steering.arXiv preprint arXiv:2409.05907, 2024

    Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering.arXiv preprint arXiv:2409.05907, 2024

    arXiv 2024

  7. [7]

    TruthX: Alleviating hallucinations by editing large language models in truthful space

    Shaolei Zhang, Tian Yu, and Yang Feng. TruthX: Alleviating hallucinations by editing large language models in truthful space. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8908–8949, Bangkok, Thailand, August 2024. Association...

    work page 2024

  8. [8]

    Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717, 2024. 10

    arXiv 2024

Show all 45 references
  1. [9]

    Truthflow: Truthful LLM generation via representation flow correction

    Hanyu Wang, Bochuan Cao, Yuanpu Cao, and Jinghui Chen. Truthflow: Truthful LLM generation via representation flow correction. InForty-second International Conference on Machine Learning, 2025

    2025

  2. [10]

    Farima Fatahi Bayat, Xin Liu, H. V . Jagadish, and Lu Wang. Enhanced language model truthfulness with learnable intervention and uncertainty expression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics, ACL 202...

    2024

  3. [11]

    Steering knowledge selection behaviours in LLMs via SAE-based representation engineering

    Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. Steering knowledge selection behaviours in LLMs via SAE-based representation engineering. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Pro...

    2025

  4. [12]

    Lofit: Localized fine-tuning on LLM representations

    Fangcong Yin, Xi Ye, and Greg Durrett. Lofit: Localized fine-tuning on LLM representations. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors,Advances in Neural Information Processing Systems 38: Annual ...

    2024

  5. [13]

    Induction heads as an essential mechanism for pattern matching in in-context learning

    Joy Crosbie and Ekaterina Shutova. Induction heads as an essential mechanism for pattern matching in in-context learning. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, Ap...

    2025

  6. [14]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  7. [15]

    Retrieval head mechanistically explains long-context factuality.arXiv preprint arXiv:2404.15574, 2024

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality.arXiv preprint arXiv:2404.15574, 2024

    2024 arXiv

  8. [16]

    A glitch in the matrix? locating and detecting language model grounding with fakepedia

    Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kiciman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia. In Lun-Wei Ku, Andre Martins, and Vivek Srikuma...

    2024

  9. [17]

    A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208, 2024

    Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. A primer on the inner workings of transformer-based language models.arXiv preprint arXiv:2405.00208, 2024

    2024 arXiv

  10. [19]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022

    2022

  11. [20]

    MQuAKE: Assessing knowledge editing in language models via multi-hop questions

    Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in ...

    2023

  12. [21]

    Hamilton

    Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natur...

    2019

  13. [22]

    Entity-based knowledge conflicts in question answering

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Em...

    2021

  14. [23]

    Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise

    Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Jiyoung Whang. Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors,Findings of the Association fo...

    2024

  15. [24]

    Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022

    Ethan Perez, Sam Ringer, Kamil ˙e Lukoši¯ut˙e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022

    2022 arXiv

  16. [25]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61s...

    2023

  17. [26]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 20...

    2024

  18. [27]

    In- context sharpness as alerts: An inner representation perspective for hallucination mitigation

    Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In- context sharpness as alerts: An inner representation perspective for hallucination mitigation. InForty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria,...

    2024

  19. [28]

    In-context learning creates task vectors

    Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 9318–9333. Association for Co...

    2023

  20. [29]

    Activation addition: Steering language models without optimization.CoRR, abs/2308.10248, 2023

    Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization.CoRR, abs/2308.10248, 2023

    2023 arXiv

  21. [30]

    Ponti, and Shay B

    Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay B. Cohen. Spectral editing of activations for large language model alignment.CoRR, abs/2405.09719, 2024

    2024 arXiv

  22. [31]

    Trusting your evidence: Hallucinate less with context-aware decoding

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North Ameri...

    2024

  23. [32]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  24. [33]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conferenc...

    2023

  25. [34]

    Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm.arXiv preprint arXiv:2408.12076, 2024

    Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm.arXiv preprint arXiv:2408.12076, 2024

    2024 arXiv

  26. [35]

    Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 202...

    2024

  27. [36]

    Combating misinformation in the age of llms: Opportunities and challenges.AI Magazine, 2023

    Canyu Chen and Kai Shu. Combating misinformation in the age of llms: Opportunities and challenges.AI Magazine, 2023

    2023

  28. [37]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors,Proceedin...

    2023

  29. [38]

    Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019

    2019

  30. [39]

    Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019

    T Wolf. Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  31. [40]

    Evaluating the ripple effects of knowledge editing in language models.Trans

    Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models.Trans. Assoc. Comput. Linguistics, 12:283–298, 2024

    2024

  32. [41]

    Advanced AI Risk

    Judea Pearl.Probabilistic reasoning in intelligent systems: networks of plausible inference. Elsevier, 2014. 13 A Datasets and Training Details Implementation DetailsExperiments were run on a single NVIDIA H100 80G GPU with Py- Torch [38] v2.3.1 and HuggingFace Transformers [3...

    2014

  33. [44]

    Average Confidence per Layer:for each layerl, we compute the average confidence score: ¯sl = 1 |Hl| X h∈Hl sl,h, whereH l is the set of heads in layerl

  34. [45]

    Typically, τ is computed as the (100−p) -th percentile over the collection of all head scores, we useτ= 5for all experiments: τ= Percentile {sl,h}l,h,100−p

    Fraction of Heads Above Global Threshold:we define a global threshold τ so that heads with scores higher than τ represent the top p% of all scores. Typically, τ is computed as the (100−p) -th percentile over the collection of all head scores, we useτ= 5for all experiments: τ= ...

  35. [46]

    linear” and “SqExp

    Noisy-OR Style Combination:We define the composite score using a noisy-OR [ 41] inspired function: Sor l = ¯sl +f l −¯sl fl. When both ¯sl and fl are viewed as normalized scores or probabilities, the Noisy-OR combination represents the probability that at least one of the cond...

    2003

  36. [2021]

    Association for Computational Linguistics. 11

  37. [4901]

    Association for Computational Linguistics, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.