Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM errors under irrelevant context are structured: the model derives an abstract class from the query, selects a context or query feature, and combines them to produce an answer.

desk verdict A genuinely new mechanistic story about how irrelevant context shifts LLM predictions, but the behavioral evidence is partly circular and needs null baselines before the headline claim is established. read the letter →

arxiv 2505.22630 v2 pith:BES2UD2M submitted 2025-05-28 cs.CL

Ziling Cheng , Meng Cao , Marc-Antoine Rondeau , Jackie Chi Kit Cheung This is my paper
classification cs.CL
keywords class-basedgeneralizationirrelevantcontexthallucinationmechanisticinterpretabilitylogitattributionactivationpatchingattentionknockoutstochasticparrotsfactualrecall
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the errors large language models make are noise or structure. In a controlled setting, the authors give a model a query together with an irrelevant context—a sentence about Honda, say, before a question about the original language of a film—and show that predictions shift roughly 38–57% of the time across the models tested. They propose that these shifts are the visible trace of class-based (mis)generalization: the model first derives an abstract class from the query (e.g., 'languages'), then selects a feature from the context or the query (e.g., Japan), and combines the class with the feature to produce an answer (e.g., 'Japanese'). Human annotation of 500 shifted answers finds 71% combine an identifiable context feature with the correct abstract class. If the claim holds, it challenges the simple 'stochastic parrot' picture: LLMs are not just stitching together memorized surface forms; they generalize over abstract categories in a systematic but unreliable way, a behavior the authors call stochastic chameleons.

What carries the argument

The central object is the class-based (mis)generalization hypothesis, made testable through the definition of query-based vs. context-based candidate sets: an answer is context-based if it enters the top three under C+Q without being in the top ten under Q alone. The hypothesis says the model first derives an abstract class from the query (e.g., 'languages'), then selects a feature from the context or query (e.g., Japan), and combines class with feature to produce the answer (e.g., 'Japanese'). The load-bearing evidence is the two-circuit competition revealed by activation patching: a query circuit transferring query-subject information from around layer 8 and a context circuit transferring context-subject and context-object information from around layer 17, with the final prediction decided by their relative strength in layers 17–24. Logit attribution supplies the class-to-instance trace: abstract class tokens are promoted in lower layers and concrete candidate tokens in higher layers, so the mechanism is visible in the residual stream.

What would settle it

Construct a null distribution for the PMI between contexts and their context-based candidates by randomly permuting the context–candidate pairs within each query class; if the observed mean PMI is not clearly above that null, the claimed context–candidate association is an artifact of selection. Alternatively, hold the context fixed and vary only the abstract class demanded by the query; if the model keeps producing context-derived answers even when the class and the context features are mismatched, the class-selection step is not doing the causal work.

Watch

Extended reading notes

Core claim

The central claim is that irrelevant context hallucinations are not arbitrary errors but the output of a structured two-step mechanism. Given a context–query pair, the model constructs an abstract class representation appropriate to the query ('language', 'place', 'company') in the lower transformer layers, then selects a feature—either from the query (France) or from the irrelevant context (Japan)—and combines class and feature into a candidate answer (French or Japanese). The paper defines query-based candidates as top-three answers under context+query that also appear in the top ten under the query alone, and context-based candidates as top-three answers that do not; predictions are query-dominant or context-dominant depending on which candidate wins. Logit attribution shows the abstract class tokens rising first and concrete candidate tokens rising later, with context- and query-candidate logits forming two competing groups that resolve around layers 17–24. Activation patching locates separate context and query circuits whose relative strength tracks which candidate wins, and attention knockout at the two critical layers flips a large share of predictions in both directions. The authors conclude that the machinery behind these errors is a genuine, if unreliable, form of generalization over abstract classes.

Load-bearing premise

The statistical validation assumes that an answer appearing only after context is added is the product of a class-based feature-selection mechanism, but such answers are selected precisely because they are context-sensitive, so the PMI test partly re-detects the criterion used to define the candidate set.

Editorial extensions

If this is right

  • A large share of wrong answers under irrelevant context is predictable: given the query's abstract class and the features available in the context, one can anticipate what the model will say.
  • Scaling up does not fix the problem: Pythia 12B and Llama-3 70B show class-based shifts at similar rates to the 7–8B models.
  • Rewording the query with alternative ParaRel templates leaves the phenomenon essentially unchanged.
  • The competition between query and context circuits is causally active: knocking out attention in layers 17–24 flips hundreds of predictions per thousand in both dominant directions.
  • Abstract class construction happens even without added context (Q-only), so class-to-instance refinement is a general feature of factual recall, not an artifact of the irrelevant context.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the two-circuit account is correct, targeted interventions at layers 17–24 should reduce irrelevant-context hallucinations, which the paper leaves as future work; one could test whether such interventions generalize beyond the ParaRel relations.
  • The class-plus-feature mechanism suggests a measurable prediction for prompt design: queries with an ambiguous required class should show larger answer-change rates under irrelevant context than queries with a clear class, a test the paper does not report.
  • The same class-plus-feature combination may underlie some few-shot demonstration effects: demonstrations provide context features that the class-selection step then incorporates, extending the mechanism beyond hallucination to in-context learning.
  • Automating the annotation of class and feature combination would let the 71% estimate be checked at scale and across relation types outside the 39 ParaRel relations used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes that LLM errors induced by irrelevant context follow a structured mechanism called class-based (mis)generalization: the model first derives an abstract class from the query (e.g., 'language'), then selects a feature from the context or query (e.g., 'Japan'), and combines them to produce an answer (e.g., 'Japanese'). The authors define context-based and query-based candidates in Eqs. (3)–(4), report behavioral results on 106M controlled context+query pairs, annotate 500 context-based candidates (claiming 71% combine context features with the correct class), and provide a PMI-based statistical test. They then present mechanistic evidence from logit attribution, activation patching, and attention knockout on Llama-3, Mistral, and Pythia, claiming hierarchical class-to-instance construction and competing context/query circuits. They also test larger models and prompt-template variations, finding the phenomenon persists.

Significance. If the claims hold, the paper would provide a nuanced counterpoint to the 'stochastic parrot' view: LLM errors are not unstructured surface stitching but show systematic class-level generalization, albeit unreliable. The paper's strengths include a large controlled dataset, multiple model families, a public code release, and a triangulation of behavioral and mechanistic methods. However, the central behavioral evidence currently rests on a candidate-selection procedure that may make the headline statistics partly self-fulfilling, and the mechanistic analysis requires stronger validation against null interventions. The idea is timely and the framework is potentially useful, but the evidence as presented is not yet sufficient to establish the mechanism.

major comments (4)
  1. [§5.3, Eqs. (3)–(6)] The PMI test is partially circular. Ccand is defined in Eq. (3) as a candidate that appears in top-3 under C+Q but not in top-10 under Q, so C and Ccand are selected specifically for their dependence. Any context-sensitive mechanism, including simple lexical priming, would yield positive PMI. The paper needs a null model—for example, permuted context–candidate pairs, candidates sampled from Q-only predictions, or a comparison with query-based candidates—to show that the observed PMI exceeds what the selection rule alone produces.
  2. [§5.2, Appendix D] The 71% annotation figure lacks inter-annotator agreement and a baseline. The candidates are preselected to be context-dependent, so labeling them 'context-influenced' may simply restate the selection criterion. The authors should report IAA and annotate a control set, such as query-based candidates or random predictions from Q-only, to show that the class+feature combination rate is specific to context-changed answers rather than a general property of top predictions.
  3. [§5.3, Eqs. (5)–(6)] The PMI estimation is underspecified. The text says each context Ci is paired with 'its corresponding candidate Ccand,i regardless of the 100 queries,' but Eq. (3) defines Ccand as a function of both C and Q. It is unclear how one candidate per context is obtained when the same context is paired with many different queries. Additionally, the one-sample t-test against E[PMI]=0 is not a natural null for dependence; a permutation or bootstrap test over context–candidate pairings would be more appropriate.
  4. [§6.2–6.3, Tables 5, 11–13] The critical layers (e.g., 17 and 24 for Llama) are identified from activation-patching maps and then used for attention knockout on the same 1000 datapoints, without a held-out split. The random-layer controls in Appendix H also show nontrivial probability changes in some conditions (e.g., '2 Low' reduces query probability by 7.5 in Llama query-dominant cases, close to the 8.4 reduction for layers 17+24). The mechanistic claim would be stronger with a clearly held-out validation, a statistical comparison of flip rates against random-layer and random-head baselines, and reporting of variance across runs.
minor comments (4)
  1. [Appendix E, Table 10] The text states 'A mean PMI of around 70' but Table 10 reports mean PMI values of 3.9, 3.7, and 3.8; this should be corrected to 'around 4' or similar.
  2. [Table 4 caption] The caption says 'The first and second row correspond to R1T,l, and the second row is R2T,l' which is confusingly phrased and appears to contain a typo; the intended distinction between the two residual-stream rows should be stated clearly.
  3. [Eq. (11)] The notation h0_{i*,l*} ∪ h1_{-i*,-l*} is not formally defined; the union of a restored activation with all corrupted activations should be described with explicit indexing.
  4. [§5.2] The abstract says '70%' while the body reports 71.0%; these should be reconciled for consistency.

Circularity Check

1 steps flagged · score 6.0 of 10

Sec. 5.3's PMI validation is circular: Ccand. is defined as context-changed by Eq. 3, so PMI(C, Ccand.) ≈ 4 re-detects the selection rule rather than independently confirming class-based generalization.

  1. self definitional [Section 4 (Eq. 3) and Section 5.3 (Eqs. 5-6)]
    "we define a context-based candidate x ∈ Ccand. to be a candidate among the top three predictions under C+Q but not among the top ten predictions under Q. ... we compute the pointwise mutual information (PMI) between them. ... This yields 100 pairs of (Ci, Ccand.,i) per class ... Across all models and expected classes, the mean PMI is approximately 4, suggesting a strong association between contexts and their corresponding candidates."

    Eq. (3) defines Ccand. as top-3(C+Q) \ top-10(Q), so every context-based candidate is selected precisely because it appears when C is added and not under Q alone. The PMI test then measures PMI(C_i, Ccand.,i) with P(C_i)=1/100 and the pair chosen by that same selection rule. Under the sampling design (100 contexts, 100 queries each), any context-consistent candidate—even an arbitrary one—yields PMI ≈ log(100) ≈ 4.6; the observed ≈4 is therefore near the value forced by the design rather than evidence for class+feature combination. No null baseline over shuffled or Q-only candidates is given, so the t-test against E[PMI]=0 is circular: it rejects independence that the selection rule already violated.

full rationale

The derivation chain's main circularity is confined to the statistical validation. Context-based candidates are defined by Eq. (3) as top-3 answers under C+Q that are absent from top-10 under Q; this set is, by construction, the set of answers whose appearance is tied to C. Computing PMI between C and this same set, with 100 contexts and P(C_i)=1/100, is expected to yield values near log(100)≈4.6 under any context-consistent candidate assignment, so the reported PMI≈4 and the rejection of E[PMI]=0 do not discriminate the class-based generalization hypothesis from weaker contextual priming. The 71% annotation and the mechanistic experiments (logit attribution, activation patching, attention knockout) are not circular by definition: they could have failed and contain independent observations, although the class-token lists in Appendix F.1 are manually predefined and the knockout flips only a minority of cases. There are no load-bearing self-citations; Cheng et al. (2025) and Yu et al. (2024) are cited for methods/related work, not to justify the central claim. Overall score 6: one load-bearing behavioral validation reduces by construction, while the core hypothesis retains some independent (if incomplete) evidence.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The paper's empirical claims depend on several hand-chosen thresholds and predefined token lists. The free parameters are the top-3/top-10 candidate definitions, the manually curated class token lists, and the data-driven selection of critical layers for intervention. The axioms are standard assumptions about eval decoding, interpretability methods, and dataset faithfulness. The invented entities are conceptual circuits inferred from interventions, with no independent evidence outside the paper.

free parameters (4)
  • top-3 threshold for context-based candidates = 3
    Atop3_C+Q threshold in Eq. (1) defines which candidates are considered strongly influenced by context; results are not tested for sensitivity to this threshold.
  • top-10 threshold for query-based candidates = 10
    Atop10_Q threshold in Eq. (2) excludes candidates that already appear under Q alone; the choice affects the size and composition of Ccand.
  • predefined class token lists = lists of tokens per relation type (Appendix F.1)
    Logit attribution for class relies on manually curated token sets (e.g., 'languages', 'language', 'tongue'). These are chosen by the authors and may not match the model's actual internal class representations.
  • critical layers for knockout = 17 and 24 (Llama); 18/24 (Mistral); 19/24 (Pythia)
    Attention knockout targets layers selected from the activation patching maps ('first blue spike' and 'highest blue spike'); these are data-driven selections, though random-layer controls partly validate them.
assumptions (4)
  • domain assumption Zero-temperature greedy decoding is representative of model predictions
    Section A states temperature=0 for evaluation; the analysis assumes top-1 outputs reflect the model's stable behavior.
  • domain assumption Vocabulary projection (logit lens) is a valid method to interpret internal representations
    Section 6.1 uses logit attribution; this assumes linear projections onto unembedding space are meaningful for understanding computation.
  • domain assumption Gaussian noise patching is a valid causal intervention that isolates circuit components
    Section 6.2 follows Meng et al. (2022) and assumes noise corruption and restoration identify causal contributions of tokens.
  • domain assumption The ParaRel relation templates faithfully instantiate the intended answer classes
    Section 4 uses ParaRel subdatasets; the classification into five knowledge types (language, place, etc.) is based on the authors' reading of the relations.
invented entities (2)
  • context circuit
    purpose: A hypothesized internal pathway that propagates features from the irrelevant context to the final answer position, competing with query-based reasoning (Section 6.2).
    Inferred from activation patching and attention knockout; not falsifiable outside the paper's controlled setting, and no independent test is provided beyond the same intervention data.
  • query circuit
    purpose: A hypothesized internal pathway that relies on the query alone to produce the answer, competing with the context circuit (Section 6.2).
    Same as context circuit; inferred from observation of patching maps without an external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs." pith.science (2026). https://pith.science/paper/BES2UD2M

@misc{pith2026250522630,
  author       = {Pith},
  title        = {Pith review of: Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BES2UD2M}},
  note         = {Machine review of arXiv:2505.22630}
}
read the original abstract

The widespread success of large language models (LLMs) on NLP benchmarks has been accompanied by concerns that LLMs function primarily as stochastic parrots that reproduce texts similar to what they saw during pre-training, often erroneously. But what is the nature of their errors, and do these errors exhibit any regularities? In this work, we examine irrelevant context hallucinations, in which models integrate misleading contextual cues into their predictions. Through behavioral analysis, we show that these errors result from a structured yet flawed mechanism that we term class-based (mis)generalization, in which models combine abstract class cues with features extracted from the query or context to derive answers. Furthermore, mechanistic interpretability experiments on Llama-3, Mistral, and Pythia across 39 factual recall relation types reveal that this behavior is reflected in the model's internal computations: (i) abstract class representations are constructed in lower layers before being refined into specific answers in higher layers, (ii) feature selection is governed by two competing circuits -- one prioritizing direct query-based reasoning, the other incorporating contextual cues -- whose relative influences determine the final output. Our findings provide a more nuanced perspective on the stochastic parrot argument: through form-based training, LLMs can exhibit generalization leveraging abstractions, albeit in unreliable ways based on contextual cues -- what we term stochastic chameleons.

Figures

Figures reproduced from arXiv: 2505.22630 by the authors.

Figure 1
Figure 1. Examples demonstrating class-based (mis)generalization with Llama-3 (8B). occurrences in pre-training data (Bender et al., 2021). This view is supported by evidence that LLMs can reproduce training artifacts, exploit spu￾rious correlations, and fail when faced with dis￾tribution shifts, among other issues (Carlini et al., 2021; Zhou et al., 2024; Dziri et al., 2023; Wu et al., 2024c; Mirzadeh et al., 2024). In this … view at source ↗
Figure 2
Figure 2. Class-based generalization framework: feature selection and combination. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Logit attribution (C+Q condition) along residual stream ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Left-hand plots demonstrate the context circuit, which extracts features from context and computes context-based candidates, while right-hand plots illustrate the query circuit. These circuits are the same in both context- and query-dominant cases; the difference lies …
Figure 5
Figure 5. Figure 5: Logit lens on Llama-3 shows how model’s top-1 predictions shift along the residual stream from abstract [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Additional logit attribution results for [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Logit Attribution Results For Mistral 7B. H.2 Additional Results Results for Llama and Mistral are presented in Ta￾ble 12 and [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Logit Attribution Results For Pythia 6.9B. Orig. L17+L24 2 Low 2 High Prob. Prob. ∆ Prob. ∆ Prob. ∆ Context-Dominant Ctx 25.5 13.1 -12.4 20.9 -4.6 21.1 -4.4 Query 8.6 14.8 +6.2 8.9 +0.3 12.6 +4.0 Query-Dominant Query 35.2 26.8 -8.4 25.7 -9.5 33.4 -1.8 Ctx 6.6 11.3 +4.7…
Figure 9
Figure 9. Figure 9: Activation patching under Q-Only condition reveals that query circuit is the same before and after adding [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Activation patching under C+Q condition for [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Activation patching under Q-Only condition reveals that query circuit is the same before and after adding [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Activation patching under Q-Only condition reveals that query circuit is the same before and after adding [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Activation patching under C+Q condition for [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic Computation

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A disentangled evaluation of math word problems finds arithmetic computation, not abstract formulation, is the main bottleneck for LLMs without chain-of-thought, and chain-of-thought mainly helps computation.

Reference graph

Works this paper leans on

57 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. https://doi.org/10.1162/tacl_a_00667 Evaluating correctness and faithfulness of instruction-following models for question answering . Transactions of the Association for Computational Linguistics, 12:681--699

    doi:10.1162/tacl_a_00667 2024

  2. [2]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  3. [3]

    Simone Balloccu, Patr \'i cia Schmidtov \'a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5/ Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLM s . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2024

  4. [4]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112

    arXiv 2023

  5. [5]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623

    2021

  6. [6]

    Emily M Bender and Alexander Koller. 2020. Climbing towards NLU : On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185--5198

    work page 2020

  7. [7]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

  8. [8]

    Meng Cao, Yue Dong, and Jackie Cheung. 2022 a . https://doi.org/10.18653/v1/2022.acl-long.236 Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340--3354, Dublin, Ireland. Association...

    doi:10.18653/v1/2022.acl-long.236 2022

Show all 57 references
  1. [9]

    Meng Cao, Yue Dong, Jingyi He, and Jackie Chi Kit Cheung. 2022 b . https://doi.org/10.18653/v1/2022.emnlp-main.663 Learning with rejection for abstractive text summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9768...

    2022 doi

  2. [10]

    Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.506 Factual error correction for abstractive summarization models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p...

    2020 doi

  3. [11]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21),...

    2021

  4. [12]

    Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.146 Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence . In Proceedings of the 2022 Conference on Empirical Methods in ...

    2022 doi

  5. [13]

    Ziling Cheng, Meng Cao, Leila Pishdad, Yanshuai Cao, and Jackie Chi Kit Cheung. 2025. https://arxiv.org/abs/2505.23701 Can llms reason abstractly over math word problems without cot? disentangling abstract formulation from arithmetic computation . Preprint, arXiv:2505.23701

    2025 arXiv

  6. [14]

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.84 Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps . In Proceedings...

    2024 doi

  7. [15]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. https://doi.org/10.1145/3626772.3657834 The power of noise: Redefining retrieval for rag systems . In Proceedings of the ...

    2024

  8. [16]

    Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. https://openreview.net/forum?i...

    2023

  9. [17]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. https://doi.org/10.1162/tacl_a_00410 Measuring and improving consistency in pretrained language models . Transactions of the Association for Computati...

    2021 doi

  10. [18]

    Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. https://doi.org/10.18653/v1/2021.acl-long.144 Causal analysis of syntactic agreement mechanisms in neural language models . In Proceedings of the 59th Annual Meeting o...

    2021 doi

  11. [19]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural L...

    2022 doi

  12. [20]

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. https://openreview.net/forum?id=5uwBzcn885 Patchscopes: A unifying framework for inspecting hidden representations of language models . In Forty-first International Conference on Machine Learning

    2024

  13. [21]

    Yihuai Hong, Dian Zhou, Meng Cao, Lei Yu, and Zhijing Jin. 2025. The reasoning-memorization interplay in language models is mediated by a single direction. arXiv preprint arXiv:2503.23084

    2025 arXiv

  14. [22]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  15. [23]

    Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. 2024. Do llms dream of elephants (when told not to)? latent concept association and associative memory in transformers. arXiv preprint arXiv:2406.18400

    2024 arXiv

  16. [24]

    Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.70 Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in ...

    2024 doi

  17. [25]

    Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. 2024. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612

    2024 arXiv

  18. [26]

    Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation

    2018

  19. [27]

    Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023. https://doi.org/10.18653/v1/2023.findings-acl.112 Large language models with controllable working memory . In Findings of the Association for Computational L...

    2023 doi

  20. [28]

    Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.565 Entity-based knowledge conflicts in question answering . In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    2021 doi

  21. [29]

    Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. 2024. Interpreting key mechanisms of factual recall in transformer-based language models. arXiv preprint arXiv:2403.19521

    2024 arXiv

  22. [30]

    Sara Vera Marjanovic, Haeun Yu, Pepa Atanasova, Maria Maistro, Christina Lioma, and Isabelle Augenstein. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.838 DYNAMICQA : Tracing internal knowledge conflicts in language models . In Findings of the Association for Computati...

    2024 doi

  23. [31]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  24. [32]

    Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023. https://openreview.net/forum?id=rJhk7Fpnvh Sources of hallucination by large language models on inference tasks . In The 2023 Conference on Empirical Methods in Natural Langua...

    2023

  25. [33]

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. https://openreview.net/forum?id=-h6WAS6eE4 Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems

    2022

  26. [34]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024. https://doi.org/10.18653/v1/2024.naacl-long.281 Language models implement simple W ord2 V ec-style vector arithmetic . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024 doi

  27. [35]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.759 Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on ...

    2022 doi

  28. [36]

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229

    2024 arXiv

  29. [37]

    Neel Nanda. 2023. https://www.neelnanda.io/mechanistic-interpretability/quickstart Mechanistic interpretability quickstart guide

    2023

  30. [38]

    Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. https://doi.org/10.18653/v1/2023.acl-long.559 D isent QA : Disentangling parametric and contextual knowledge with counterfactual question answering . In Proceedings of the 61st Annual ...

    2023 doi

  31. [39]

    nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens interpreting gpt: the logit lens

    2020

  32. [40]

    Chris Olah. 2022. https://www.transformer-circuits.pub/2022/mech-interp-essay Mechanistic interpretability, variables, and the importance of interpretable bases

    2022

  33. [41]

    Miller, and Sebastian Riedel

    Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rockt \"a schel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. https://openreview.net/forum?id=025X0zPfn How context affects language models' factual predictions . In Automated Knowledge Base Construction

    2020

  34. [42]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210--31227. PMLR

    2023

  35. [43]

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.435 A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis . In Proceedings of the 2023 Conference on Empirical Methods ...

    2023 doi

  36. [44]

    Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. https://openreview.net/forum?id=wjHVmgBDzc \ texttt\ ConflictBank\ \ : A benchmark for evaluating the influence of knowledge conflicts in LLM s . In The Thirty-eight ...

    2024

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  38. [46]

    Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024 a . https://openreview.net/forum?id=S7NVVfuRv8 How easily do irrelevant inputs skew the responses of large language models? In First Conference on Language Modeling

    2024

  39. [47]

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024 b . Retrieval head mechanistically explains long-context factuality. arXiv preprint arXiv:2404.15574

    2024 arXiv

  40. [48]

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Aky \"u rek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024 c . https://doi.org/10.18653/v1/2024.naacl-long.102 Reasoning or reciting? exploring the capabilities and limitations of language models through coun...

    2024 doi

  41. [49]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.486 Knowledge conflicts for LLM s: A survey . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p...

    2024 doi

  42. [50]

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024 b . https://arxiv.org/abs/2404.18824 Benchmarking benchmark leakage in large language models . arXiv preprint arXiv:2404.18824

    2024 arXiv

  43. [51]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. https://openreview.net/forum?id=ZS4m74kZpH Making retrieval-augmented language models robust to irrelevant context . In The Twelfth International Conference on Learning Representations

    2024

  44. [52]

    Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.466 Mechanistic understanding and mitigation of language model non-factual hallucinations . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7...

    2024 doi

  45. [53]

    Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.615 Characterizing mechanisms for factual recall in language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924--9959, Singa...

    2023 doi

  46. [54]

    Xiaowei Yuan, Zhao Yang, Yequan Wang, Shengping Liu, Jun Zhao, and Kang Liu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.234 Discerning and resolving knowledge conflicts through adaptive decoding with contextual information-entropy constraint . In Findings of the Assoc...

    2024 doi

  47. [55]

    Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.28 Explore spurious correlations at the concept level in language models for text classification . In Proceedings of the 62nd Annual Meeting of the Associati...

    2024 doi

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.