Pith. sign in

REVIEW 4 major objections 5 minor 72 references

Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Knowledge overshadowing — when a model's dominant fact masks a rarer correct fact and the model hallucinates — is driven by how attention inside a sparse knowledge circuit is allocated, and pruning that circuit recovers the hidden answer.

desk verdict A controlled synthetic study with a genuinely interesting training-dynamics result, but the circuit-based mechanism and recovery claims are not yet supported by the evidence. read the letter →

arxiv 2505.14406 v4 pith:JQZW6KLW submitted 2025-05-20 cs.CL

Haoming Huang , Yibo Yan , Jiahao Huo , Xin Zou , Xinfeng Li , Kun Wang , Xuming Hu This is my paper
classification cs.CL
keywords knowledgeovershadowinghallucinationcircuitsmechanisticinterpretabilityattentionheadscircuitpruningtrainingdynamicsEdgeAttributionPatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge overshadowing is a hallucination mode in which a model's strong learned association between background knowledge and a frequent fact masks a rarer fact that is also relevant, so the model answers with the frequent fact even after training on clean data. This paper argues that the phenomenon has a mechanistic, circuit-level origin: attention heads inside a sparse knowledge circuit allocate attention to the difference between dominant and subordinate inputs, and that allocation determines how strongly overshadowing occurs. To show this, the authors construct knowledge circuits with edge attribution patching, trace how overshadowing rises and falls across training as a function of knowledge popularity, model size, dataset size, and the loss share of subordinate knowledge, and then prune circuits to their optimal edge count to recover the suppressed correct answer. If the account holds, overshadowing is not an irreducible training artifact but a misallocation of attention inside a small subgraph — one that can be located and corrected.

What carries the argument

The central object is the knowledge circuit $\mathcal{C}$: a sparse directed subgraph of the transformer's computational graph, with nodes for attention heads, MLP layers, embeddings, and logits and edges for information flow, selected to be critical for distinguishing the dominant input $P_{\rm dom}$ from the subordinate input $P_{\rm sub}$. It is constructed by Edge Attribution Patching with Integrated Gradients: each edge is scored by the product of the parent activation difference (clean $P_{\rm sub}$ minus corrupt $P_{\rm dom}$) and the gradient of a metric $M$ defined as the logit difference between $Y_{\rm sub}$ and $Y_{\rm dom}$, edges with scores below a threshold $\tau$ are pruned, and $\tau$ is optimized by sweeping edge counts at uniform intervals and then applying golden-section search. The machinery does two jobs: it localizes overshadowing to specific attention heads and MLP nodes — the paper identifies a5.h1, m5, and m4 as the conduit of the rank reversal in the examined epoch — and the edge-pruning itself is the recovery mechanism. The quantitative backbone is the relative overshadowing rate $\mathrm{RO} = p(Y_{\rm dom}\mid P_{\rm sub})/p(Y_{\rm dom}\mid P_{\rm dom})$ and the subordinate-loss proportion, whose co-evolution with $\mathrm{RO}$ organizes the training-dynamics story.

What would settle it

Build an optimized circuit on one overshadowing prompt pair and feed it a held-out subordinate prompt that shares the same background knowledge: if the circuit does not recover that prompt's correct answer while the full model still hallucinates, the claim that the identified circuit is the mechanism of overshadowing on that background fails. A sharper check targets the method itself: replace the integrated-gradient edge scores with direct activation patching on the same edges; if the causal edges found by actual patching disagree with the attribution scores used here, the faithfulness premise that carries the entire mechanism story collapses.

Watch

Extended reading notes

Core claim

Stated on its own terms, the central claim is that 'the knowledge circuit's attentional allocation to differences between dominant and overshadowed knowledge inputs influences the extent of knowledge overshadowing.' The relative overshadowing rate $\mathrm{RO} = p(Y_{\rm dom}\mid P_{\rm sub})/p(Y_{\rm dom}\mid P_{\rm dom})$ typically spikes toward 100% early in training and later falls to 0%, and the trajectory is shaped by three factors: higher knowledge popularity $P$ and larger model size $M$ bring earlier onset, shorter duration, and quicker recovery, while larger dataset size $D$ brings earlier onset but slower recovery. Recovery begins when the loss proportion of subordinate knowledge rises far enough that optimization effort shifts onto subordinate errors. Inside the knowledge circuit, high-attention heads focused on $\{X_{\rm dom}, X_{\rm sub}\}$ appear as $\mathrm{RO}$ falls and disappear as it rises; in the representative epoch the paper dissects, head a5.h1 relays information to MLP layer m5 with layer-4 MLP mediation, and this is the layer where the rank of $Y_{\rm sub}$ overtakes $Y_{\rm dom}$. Finally, optimizing the circuit by pruning edges, via a two-stage sweep with golden-section search, eliminates the overshadowing effect on the tested prompts and flips the prediction from $Y_{\rm dom}$ to $Y_{\rm sub}$.

Load-bearing premise

The load-bearing premise is that the Edge Attribution Patching scores used to build each knowledge circuit faithfully identify which attention heads and MLP connections actually cause the overshadowed prediction; the paper's own check of this premise is a small ablation on fine-tuned models that does not test whether a circuit built from one set of prompts still works on held-out prompts.

Editorial extensions

If this is right

  • Overshadowing onset, duration, and recovery are predictable during training: higher knowledge popularity and larger model size compress the episode, while larger datasets make recovery slower.
  • Recovery from overshadowing begins when the loss proportion of subordinate knowledge rises enough to shift optimization effort onto subordinate errors, so tracking that proportion alongside the overshadowing rate gives an early signal of when a training run will escape the effect.
  • Persistent overshadowing in large pretrained models can be explained as insufficient optimization of subordinate knowledge at scale, which is why the hallucination survives high-quality training data.
  • Pruning a knowledge circuit to its optimal edge count recovers the masked subordinate answer without any weight update, making circuit optimization a candidate mitigation strategy for this type of hallucination.
  • High-attention heads focused on the dominant/subordinate difference are the mechanistic correlate of recovery: they are present in the circuit when the overshadowing rate falls and absent when it rises.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The recovery procedure is built per prompt, so the paper leaves implicit whether an optimized circuit transfers to a different subordinate instance sharing the same background knowledge; the paper itself notes its results rest on the Pythia suite and synthetic or fine-tuning data, so a cross-architecture, held-out transfer test would be the decisive follow-up.
  • Because the corrupt input replaces the subordinate token with a generic placeholder like 'something', the attribution may partly capture the model's reaction to a structural anomaly rather than to the dominant knowledge; building the corrupt input by substituting a random in-vocabulary token would separate these two effects.
  • The training-dynamics result points to a data-side intervention the authors do not implement: reweighting or resampling subordinate knowledge during the early-training window when its loss proportion is low could preempt the overshadowing spike before circuit-level recovery is needed.
  • If overshadowing is a general failure of circuits to attend to distinguishing tokens, the same construction and pruning tools could be used to diagnose other interference hallucinations, such as multimodal models defaulting to a dominant modality's prior — a connection the paper does not draw.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PhantomCircuit, a framework for analyzing knowledge overshadowing in LLMs through knowledge circuit analysis. It defines quantitative indicators (AO, RO, LP), studies training dynamics across knowledge popularity, model size, and dataset size on synthetic and fine-tuning datasets, constructs EAP-IG-style knowledge circuits, identifies attention-head and MLP components inside the circuit that correlate with overshadowing onset/recovery, and proposes a circuit-pruning-based recovery method. The main claims are that overshadowing dynamics are driven by loss allocation (LP), that circuit attentional allocation to {Xdom, Xsub} influences overshadowing extent, and that optimizing the circuit by pruning edges can recover the subordinate knowledge output.

Significance. If the central claims hold, the paper would extend knowledge overshadowing research from inference-time observation to a mechanistic, training-time account, and would offer a concrete, interpretability-motivated mitigation. The framework's focus on circuit evolution across training and the use of Pythia models from scratch is valuable in principle. However, the load-bearing mechanistic and recovery claims depend on the faithfulness of the constructed circuits and on evaluation procedures that presently contain circular or selected elements. The paper ships a repository link, though the manuscript text is inconsistent about code availability, and the experimental reporting of variance is incomplete. The significance is therefore conditional: the methodological lens is promising but the evidence in this version is not yet sufficient to support the strong mechanistic conclusions.

major comments (4)
  1. [Section 3.2.2, Eq. (6), Appendix B.1] The edge score in Eq. (6) is written as an endpoint gradient-times-activation-difference product, S(e) = E[ΔA(vp) · ∂M(Ysub|Psub)/∂A(vp)], which is exact only for a linear model. True EAP-IG averages gradients along the corrupt-to-clean path, and the faithfulness guarantees cited from Hanna et al. (2024) apply to that path-integral formulation. Appendix B.1 says IG is 'often used to refine' the estimate, making it ambiguous whether the path integral was actually computed. Since all Section 4.2 mechanistic conclusions about attention heads inherit the faithfulness of these edge scores, the paper must either state precisely which scoring procedure was implemented or provide task-specific faithfulness evidence that transfers to the endpoint approximation.
  2. [Section 4.3, Table 1] The faithfulness ablation removes the top-scoring attention heads from circuits built on the same instances used for evaluation and measures the drop in M on those same instances. This cannot distinguish a faithful causal circuit from an artifact of optimizing the same metric used at construction time. The study lacks held-out Psub instances, ablations of low-scoring heads, and random-ablation baselines. As written, Table 1 does not support the claim that the circuit, or its attention-head interpretation, is faithful for knowledge overshadowing.
  3. [Section 3.2.4, Eq. (7), Table 2] The recovery evaluation is circular. Eq. (7) selects τopt by maximizing M(Copt(τ), Psub, Ysub), and the same Psub and Ysub are subsequently used to report recovery in Table 2 and Figure 6. The reported successes are therefore selected on the evaluation set. Moreover, Ysub and Ydom are identified by the R-PMI heuristic described in Section 3.2.4, so the apparent recovery reflects, at least in part, the heuristic's own labeling. A valid evaluation requires a held-out split (or at least a separate selection criterion) for both the threshold optimization and the identification of Xsub, Ysub, Ydom.
  4. [Section 4.1.4, Figures 3 and 4] All dynamic-training claims (onset, duration, recovery as functions of P, M, D) are based on single training runs with no reported variance. Figures 3 and 4 show RO and LP curves without error bars or multiple seeds. Given that the models are trained from scratch with randomly sampled tokens, the qualitative differences attributed to P, M, and D could be within-run noise. The paper should report at least 3–5 seeds with error bars, or explicitly state that the plots are representative and provide replication evidence.
minor comments (5)
  1. [Abstract and Section 3.2.2] The abstract states the code is at a GitHub repository, while Section 3.2.2 says 'Our code will be available upon acceptance.' Please reconcile these statements and ensure the repository contains the exact circuit-construction and optimization scripts used for the reported results.
  2. [Section 3.2.2, Eq. (6)] The notation 'Exp' in Eq. (6) is ambiguous: it could mean expectation over Psub samples or exponential. Please use explicit expectation notation (e.g., E_{Psub}[...]) and define the sampling distribution.
  3. [Figure 5] Figure 5(c) is described as showing the circuit structure 'with 400 edges totally,' but the text does not specify the model, epoch, and P/D configuration used for this figure, making it difficult to connect the a5.h1/m4/m5 analysis to a particular experimental setting. Please add this information to the caption.
  4. [Section 4.3] The sentence 'removed the top 10%, 20%, and 50% of attention heads by score' should clarify whether 'score' refers to the circuit edge score S(e) or attention score; the two are different quantities and the ablation result depends on this choice.
  5. [Limitations] Limitation 1 acknowledges that findings rely primarily on Pythia and synthetic/fine-tuning data. This is appropriate, but the natural-language case in Appendix E is a single pair (politician/singer) and a multi-hop anecdote; consider presenting a small but systematic set of natural cases if generalization is claimed.

Circularity Check

2 steps flagged · score 6.0 of 10

The recovery result is a train-set fit: Eq. (7) selects the circuit threshold by maximizing the same M on the same Psub/Ysub used to report recovery, and the faithfulness ablation verifies only the definition of the edge scores.

  1. fitted input called prediction [Section 3.2.4, Eq. (7); Section 4.2, Figure 6]
    "τopt = arg max τ M(Copt(τ), Psub, Ysub) ... The performance of the optimized circuit Copt is then evaluated by feeding it the clean input Psub, while Pdom serves as the baseline for contrast. Finally, the circuit successfully produces Ysub, demonstrating the elimination of the overshadowing effect."

    The optimized threshold τopt is chosen by maximizing M on the very same Psub and Ysub that are then used to demonstrate recovery. M is the logit difference between Ysub and Ydom, and both Ysub and Ydom are themselves identified by the R-PMI heuristic (Eq. 8) operating on Psub. Therefore the reported 'recovery' is the objective function of the optimization, not an independent prediction on held-out data. The circuit's success in producing Ysub is by construction the value that Eq. (7) was designed to produce.

  2. self definitional [Section 4.3, Table 1]
    "We first construct optimal circuits for pythia-410m and gpt2-medium models trained on the fine-tuning dataset. Subsequently, we removed the top 10%, 20%, and 50% of attention heads by score and observed the impact on performance metric M and attention scores, with results shown in Table 1. ... The results show that ablating just 10% of the high-attention heads leads to a significant drop in both the attention scores on Xdom, Xsub and the circuit performance metric M."

    The ablated heads are selected by the edge scores S(e) defined in Eq. (6) as the gradient of M, so removing the highest-scoring heads and measuring the drop in M verifies the scoring definition rather than an independent causal mechanism. The circuit construction and the faithfulness test use the same fine-tuning dataset, the same Psub/Ysub, and the same metric M, so the ablation cannot distinguish a faithful circuit from an artifact of optimizing M. The 'faithfulness' conclusion is thus partly guaranteed by the way the scores and the ablation are defined.

full rationale

The training-dynamics analyses in Sections 4.1 and 4.2 (Figures 3 and 4) are self-contained empirical measurements of RO under varying P, M, and D, and they are not circular. The circuit-construction method itself follows the standard EAP-IG paradigm and is not circular per se. However, the recovery evaluation in Section 3.2.4 and Section 4.2 is circular: Eq. (7) chooses τ by maximizing M(Copt(τ), Psub, Ysub), and then the same Psub/Ysub, with Ysub identified by the R-PMI heuristic in Eq. (8), is used to claim that the circuit 'successfully produces Ysub'. The recovery is therefore a fit to the evaluation instances, not a prediction. In addition, the faithfulness ablation in Section 4.3 removes heads whose scores are the gradient of M and measures the drop in M, which restates the scoring definition rather than providing independent held-out validation. Because the central mitigation claim and the mechanistic faithfulness claim inherit these construction choices, while the dynamics findings remain independent, the overall circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on four domain assumptions: synthetic data models real overshadowing, EAP-IG is faithful, R-PMI labels are correct, and attention scores reflect causal use. The tuned numerical parameters are the circuit pruning threshold, optimized per prompt, and the hand-set attention threshold 0.2; top-k is unstated.

free parameters (3)
  • circuit pruning threshold tau (or edge count n) = not reported; optimized per prompt via golden-section search
    tau/n determines which edges survive in the knowledge circuit. In Section 3.2.4 it is tuned to maximize M on the same Psub later used for evaluation, so the recovery results depend on this per-instance fit.
  • high-attention threshold = 0.2
    Set by hand in Section 4.2, justified by Xsub being one fifth of the prompt length in the synthetic dataset. Used to define high-attention heads whose emergence and recovery are linked to RO.
  • top-k for R-PMI candidate sets = not specified
    The R-PMI component identification in Section 3.2.4 and Appendix B.2 requires a top-k next-token set; k is never stated, and the identification of Ysub and Ydom depends on it.
assumptions (4)
  • domain assumption Synthetic datasets of randomly sampled token IDs with shared Xbg, repeated Xdom, and a single Xsub reproduce the knowledge overshadowing phenomenon.
    The dynamics and mechanistic conclusions in Section 4.2 are drawn from these synthetic groups, which lack semantic relationships. Generalization to real language is assumed in Section 4.1.1 and Appendix C.1.
  • domain assumption EAP-IG edge attribution scores faithfully identify the causal subgraph responsible for a model behavior.
    Section 3.2.2 constructs the knowledge circuit by pruning edges using Eq. (6) and relies on prior faithfulness results. The paper does not verify faithfulness for this task beyond a small ablation in Section 4.3.
  • domain assumption R-PMI based deletion identifies the true overshadowed component Xsub and the intended outputs Ysub and Ydom.
    Section 3.2.4 and Appendix B.2 use SR-PMI and rank elevation to label the instances that define the recovery metric. Incorrect labels would make the recovery demonstration circular or meaningless.
  • domain assumption Attention scores of heads in the pruned circuit measure the model's causal use of {Xdom, Xsub}.
    Section 3.2.3 and Figure 5 equate high attention with functional contribution to overshadowing. No causal verification is provided beyond the small ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis." pith.science (2026). https://pith.science/paper/JQZW6KLW

@misc{pith2026250514406,
  author       = {Pith},
  title        = {Pith review of: Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQZW6KLW}},
  note         = {Machine review of arXiv:2505.14406}
}
read the original abstract

Large Language Models (LLMs), despite their remarkable capabilities, are hampered by hallucinations. A particularly challenging variant, knowledge overshadowing, occurs when one piece of activated knowledge inadvertently masks another relevant piece, leading to erroneous outputs even with high-quality training data. Current understanding of overshadowing is largely confined to inference-time observations, lacking deep insights into its origins and internal mechanisms during model training. Therefore, we introduce PhantomCircuit, a novel framework designed to comprehensively analyze and detect knowledge overshadowing. By innovatively employing knowledge circuit analysis, PhantomCircuit dissects the function of key components in the circuit and how the attention pattern dynamics contribute to the overshadowing phenomenon and its evolution throughout the training process. Extensive experiments demonstrate PhantomCircuit's effectiveness in identifying such instances, offering novel insights into this elusive hallucination and providing the research community with a new methodological lens for its potential mitigation.

Figures

Figures reproduced from arXiv: 2505.14406 by the authors.

Figure 1
Figure 1. Illustrative comparison of previous research [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A group of imbalanced natural language data of knowledge overshadowing. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) ∼ (c) show the dynamic variation of RO relating to P, M and D in model training phase. Higher Knowledge Popularity (P) and Model Size (M) tend to result in an earlier onset, shorter duration, and quicker recovery from knowledge overshadowing. In contrast, a larger Dataset Size (D) also leads to an earlier onset but is associated with a slower recovery phase. (d) ∼ (f) show the duration of onset stage, high RO (>… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The co-evolution between RO & LP in different M. Early high overall loss and low LP leads to intensive optimization of Pdom and RO rises up to near 100%. As training progresses, subordinate knowledge loss proportion (LP) rises, shifting optimization focus to Psub error…
Figure 5
Figure 5. Figure 5: The circuit-based analysis of proposed PHANTOMCIRCUIT. (a) shows the average attention scores allocated to {Xdom, Xsub} across epoch. (b) focuses on the onset and recovery phases and shows that the appear and disappear of high-attention heads (attention score greater t…
Figure 6
Figure 6. Figure 6: Overshadowing recovery via optimized circuit. In the fine-tuned model, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The dynamics analysis of knowledge overshadowing in fine-tuning dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The relationship between circuit performance, high-attention heads proportion, average attention score on [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 11 canonical work pages

  1. [1]

    Elham Asgari, Nina Monta \ n a-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung, and Dominic Pimenta. 2025. A framework to assess clinical safety and hallucination rates of llms for medical text summarisation. npj Digital Medicine, 8(1):1--15

    work page 2025

  2. [2]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, and 81 others. 2024. https://arxiv.org/abs/2403.17297 Internlm2 technical report . Preprint, arXiv:2403.17297

    arXiv 2024

  5. [5]

    Neeloy Chakraborty, Melkior Ornik, and Katherine Driggs-Campbell. 2025. Hallucination detection in foundation models for decision-making: A flexible definition and review of the state of the art. ACM Computing Surveys

    work page 2025

  6. [6]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1--45

    2024

  7. [7]

    I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, and 1 others. 2023. Factool: Factuality detection in generative ai--a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13528

    arXiv 2023

  8. [8]

    Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, and 1 others. 2025. Llm agents for education: Advances and applications. arXiv preprint arXiv:2503.11733

    arXiv 2025

Show all 72 references
  1. [9]

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318--16352

    2023

  2. [10]

    Yunkai Dang, Mengxi Gao, Yibo Yan, Xin Zou, Yanggan Gu, Aiwei Liu, and Xuming Hu. 2024. Exploring response uncertainty in mllms: An empirical evaluation under misleading scenarios. arXiv preprint arXiv:2411.02708

    2024 arXiv

  3. [11]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495

    2023 arXiv

  4. [12]

    Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Ji-Rong Wen, and Zhicheng Dou. 2025. Understand what llm needs: Dual preference alignment for retrieval-augmented generation. In Proceedings of the ACM on Web Conference 2025, pages 4206--4225

    2025

  5. [13]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, and 1 others. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    2022 arXiv

  6. [14]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12

    2021

  7. [15]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  9. [17]

    Ahmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, and Hinrich Schütze. 2025. https://arxiv.org/abs/2506.03434 Time course mechinterp: Analyzing the evolution of components and knowledge in large language models . Preprint, arXiv:2506.03434

    2025 arXiv

  10. [18]

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17806

    2024 arXiv

  11. [19]

    Kaichen Huang, Jiahao Huo, Yibo Yan, Kun Wang, Yutao Yue, and Xuming Hu. 2024 a . Miner: Mining the underlying pattern of modality-specific neurons in multimodal large language models. arXiv preprint arXiv:2410.04819

    2024 arXiv

  12. [20]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions ...

    2025

  13. [21]

    Xinting Huang, Madhur Panwar, Navin Goyal, and Michael Hahn. 2024 b . Inversionview: A general-purpose method for reading information from neural activations. arXiv preprint arXiv:2405.17653

    2024 arXiv

  14. [22]

    Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. 2024. Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model. arXiv preprint arXiv:2406.11193

    2024 arXiv

  15. [23]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  16. [24]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  17. [25]

    Callie Y Kim, Christine P Lee, and Bilge Mutlu. 2024. Understanding large-language model (llm)-powered human-robot interaction. In Proceedings of the 2024 ACM/IEEE international conference on human-robot interaction, pages 371--380

    2024

  18. [26]

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, and 1 others. 2025. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419

    2025 arXiv

  19. [27]

    Junyu Luo, Cao Xiao, and Fenglong Ma. 2023. Zero-resource hallucination prevention for large language models. arXiv preprint arXiv:2309.02654

    2023 arXiv

  20. [28]

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647

    2024 arXiv

  21. [29]

    Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. 2024. The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey. arXiv preprint arXiv:2404.11584

    2024 arXiv

  22. [30]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251

    2023 arXiv

  23. [31]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1--13

    2024

  24. [32]

    nostalgebraist. 2020. Interpreting GPT : the logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  25. [33]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill, 5(3):e00024--001

    2020

  26. [34]

    Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, and Huajun Chen. 2025. How do llms acquire new knowledge? a knowledge circuits perspective on continual pre-training. arXiv preprint arXiv:2502.11196

    2025 arXiv

  27. [35]

    Zhixuan Pan, Shaowen Wang, and Jian Li. 2025. Understanding llm behaviors via compression: Data generation, knowledge acquisition and scaling laws. arXiv preprint arXiv:2504.09597

    2025

  28. [36]

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. 2024. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199

    2024 arXiv

  29. [37]

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646

    2024

  30. [38]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922

    2023 arXiv

  31. [39]

    Jiamin Su, Yibo Yan, Fangteng Fu, Han Zhang, Jingheng Ye, Xiang Liu, Jiahao Huo, Huiyu Zhou, and Xuming Hu. 2025. Essayjudge: A multi-granular benchmark for assessing automated essay scoring capabilities of multimodal large language models. arXiv preprint arXiv:2502.11916

    2025 arXiv

  32. [40]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261

    2022 arXiv

  33. [41]

    Aaquib Syed, Can Rager, and Arthur Conmy. 2023. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348

    2023 arXiv

  34. [42]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. arXiv preprint arXiv:2402.16438

    2024 arXiv

  35. [43]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  36. [44]

    Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7601--7614

    2024

  37. [45]

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987

    2023 arXiv

  38. [46]

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593

    2022 arXiv

  39. [47]

    Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, and 1 others. 2025. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585

    2025 arXiv

  40. [48]

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160

    2023 arXiv

  41. [49]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    2023

  42. [50]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2024

  43. [51]

    Shuhang Xun, Sicheng Tao, Jungang Li, Yibo Shi, Zhixin Lin, Zhanhui Zhu, Yibo Yan, Hanqian Li, Linghao Zhang, Shikang Wang, and 1 others. 2025. Rtv-bench: Benchmarking mllm continuous perception, understanding and reasoning through real-time video. arXiv preprint arXiv:2505.02064

    2025

  44. [52]

    Yibo Yan and Joey Lee. 2024. Georeasoner: Reasoning on geospatially grounded context for natural language understanding. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4163--4167

    2024

  45. [53]

    Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. 2024 a . A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. arXiv preprint arXiv:2412.11936

    2024 arXiv

  46. [54]

    Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, and 1 others. 2024 b . Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04509

    2024 arXiv

  47. [55]

    Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S Yu, Carla Gomes, Bart Selman, and Qingsong Wen. 2025 a . Position: Multimodal large language models can significantly advance scientific reasoning. arXiv preprint arXiv:2502.02871

    2025 arXiv

  48. [56]

    Yibo Yan, Shen Wang, Jiahao Huo, Philip S Yu, Xuming Hu, and Qingsong Wen. 2025 b . Mathagent: Leveraging a mixture-of-math-agent framework for real-world multimodal mathematical error detection. arXiv preprint arXiv:2503.18132

    2025 arXiv

  49. [57]

    Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. 2024 c . Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. In Proceedings of the ACM Web Conference 2024...

    2024

  50. [58]

    Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469

    2023 arXiv

  51. [59]

    Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. Knowledge circuits in pretrained transformers. arXiv preprint arXiv:2405.17969

    2024 arXiv

  52. [60]

    Murong Yue. 2025. A survey of large language model agents for question answering. arXiv preprint arXiv:2503.19213

    2025 arXiv

  53. [61]

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024 a . Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation. arXiv preprint arXiv:2402.09267

    2024 arXiv

  54. [62]

    Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. 2024 b . Knowledge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039

    2024 arXiv

  55. [63]

    Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu, Chi Han, Yi R Fung, Kathleen McKeown, Chengxiang Zhai, Manling Li, and 1 others. 2025 a . The law of knowledge overshadowing: Towards understanding, predicting, and preventing llm hallucination. arXiv preprint arXiv:2502.16143

    2025 arXiv

  56. [64]

    Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, and 1 others. 2025 b . Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents. ACM Computing Survey...

    2025

  57. [65]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2):1--38

    2024

  58. [66]

    Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. 2024. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprint arXiv:2408.09429

    2024 arXiv

  59. [67]

    Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2024. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality. arXiv preprint arXiv:2410.04780

    2024 arXiv

  60. [68]

    Junyi Zhu, Shuochen Liu, Yu Yu, Bo Tang, Yibo Yan, Zhiyu Li, Feiyu Xiong, Tong Xu, and Matthew B Blaschko. 2024. Fastmem: fast memorization of prompt improves context awareness of large language models. arXiv preprint arXiv:2406.16069

    2024 arXiv

  61. [69]

    Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. 2024. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2410.03577

    2024 arXiv

  62. [70]

    Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De. 2025. https://arxiv.org/abs/2503.21676 How do language models learn facts? dynamics, curricula and hallucinations . Preprint, arXiv:2503.21676

    2025 arXiv

  63. [71]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  64. [72]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.