{"id":"f5f9df43-ebd1-4f8b-8f35-3d6b5ab12f95","arxiv_id":"2505.14406","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PhantomCircuit traces knowledge overshadowing to attention circuits during training and prunes circuit edges to recover the overshadowed answer.","lead":"This paper studies a type of AI hallucination called knowledge overshadowing, where a model's strong memory of common knowledge hides a rarer relevant fact. It uses circuit analysis to trace where this happens inside the model and proposes a pruning method to recover the correct answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central mechanism claim depends on an EAP-IG circuit score that Eq. (6) states without the integrated-gradients path integral, and the only faithfulness check ablates top-scored heads on the same instances used to build the circuits; neither the attention-head story nor the recovery result is…","rationale":"I read the paper as attempting to provide a mechanistic, training-time account of knowledge overshadowing: dynamic observations (Section 4.2) plus circuit-level attention analysis plus a pruning-based recovery. The dynamic observations are plausible and the synthetic setup is well controlled, but they are descriptive; the load-bearing step is the circuit construction. The reader's weakest assumption (EAP-IG faithfulness) is the right target. My specific reading of Eq. (6) makes the problem sharper: the formula as written is not the IG path integral, so unless the implementation differs from the text, the paper's circuits are not EAP-IG circuits. Table 1's ablation is inadequate as a faithfulness check because it only removes high-scoring heads on in-distribution instances and does not compare against a control ablation. A held-out recovery evaluation would also separate the mitigation claim from the metric used to select the circuit. These concerns do not refute the descriptive training-dynamics findings, and they could be resolved by a precise re-implementation and a held-out faithfulness test, so I would keep the CONDITIONAL verdict rather than escalate. No formal verification or released code exists to independently confirm the method, but the controlled synthetic data and natural-language case study are genuine evidence for the descriptive part.","tokens_in":20690,"tokens_out":6445,"duration_ms":65014,"concrete_test":"Run both variants on the same Pythia-410M fine-tuned checkpoint and 100 Psub prompts: (i) the score in Eq. (6) and (ii) the exact EAP-IG path integral from Hanna et al. (2024). Compare the top-200 edge sets (e.g., Jaccard overlap) and perform direct activation patching of those edges to measure M. If overlap is low or patching the Eq. (6) circuit does not reproduce the full-model M, the circuit is not the claimed EAP-IG circuit and the mechanistic claims are unsupported. Additionally, rebuild Copt on a train subset and evaluate recovery on held-out Psub instances; if recovery collapses on held-out prompts, the mitigation claim is overfit to the optimized instances.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanistic claim (Section 4.2) is that attentional allocation inside the knowledge circuit determines the extent of overshadowing. All attention-head conclusions inherit the correctness of the circuit built in Section 3.2.2 and Appendix B.1, which is presented as EAP-IG. However, Eq. (6) defines S(e) as E[ΔA(vp)·∂M(Ysub|Psub)/∂A(vp)], an endpoint gradient-times-difference. True EAP-IG (Hanna et al., 2024) averages gradients along the full corrupt-to-clean path; Eq. (6) is only exact for a linear model. If the implementation follows the equation, the cited faithfulness results for EAP-IG do not transfer. The paper is internally ambiguous: Appendix B.1 mentions IG only as a possible refinement, and the abstract points to a repository while Section 3.2.2 states the code will be available upon acceptance, so the actual score cannot be inspected. The single faithfulness study (Table 1, Section 4.3) removes the highest-scoring heads from circuits built for those same instances and observes a drop in M; it does not ablate low-scoring heads or evaluate held-out Psub, so it cannot distinguish a faithful mechanism from an artifact of optimizing the same metric used for circuit construction. If the circuits are not faithful, the a5.h1/m4/m5 interpretation and the pruning-based recovery claim do not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PhantomCircuit, a framework for analyzing knowledge overshadowing in LLMs through knowledge circuit analysis. It defines quantitative indicators (AO, RO, LP), studies training dynamics across knowledge popularity, model size, and dataset size on synthetic and fine-tuning datasets, constructs EAP-IG-style knowledge circuits, identifies attention-head and MLP components inside the circuit that correlate with overshadowing onset/recovery, and proposes a circuit-pruning-based recovery method. The main claims are that overshadowing dynamics are driven by loss allocation (LP), that circuit attentional allocation to {Xdom, Xsub} influences overshadowing extent, and that optimizing the circuit by pruning edges can recover the subordinate knowledge output.","tokens_in":20989,"tokens_out":2050,"duration_ms":21806,"significance":"If the central claims hold, the paper would extend knowledge overshadowing research from inference-time observation to a mechanistic, training-time account, and would offer a concrete, interpretability-motivated mitigation. The framework's focus on circuit evolution across training and the use of Pythia models from scratch is valuable in principle. However, the load-bearing mechanistic and recovery claims depend on the faithfulness of the constructed circuits and on evaluation procedures that presently contain circular or selected elements. The paper ships a repository link, though the manuscript text is inconsistent about code availability, and the experimental reporting of variance is incomplete. The significance is therefore conditional: the methodological lens is promising but the evidence in this version is not yet sufficient to support the strong mechanistic conclusions.","major_comments":[{"comment":"The edge score in Eq. (6) is written as an endpoint gradient-times-activation-difference product, S(e) = E[ΔA(vp) · ∂M(Ysub|Psub)/∂A(vp)], which is exact only for a linear model. True EAP-IG averages gradients along the corrupt-to-clean path, and the faithfulness guarantees cited from Hanna et al. (2024) apply to that path-integral formulation. Appendix B.1 says IG is 'often used to refine' the estimate, making it ambiguous whether the path integral was actually computed. Since all Section 4.2 mechanistic conclusions about attention heads inherit the faithfulness of these edge scores, the paper must either state precisely which scoring procedure was implemented or provide task-specific faithfulness evidence that transfers to the endpoint approximation.","section":"Section 3.2.2, Eq. (6), Appendix B.1"},{"comment":"The faithfulness ablation removes the top-scoring attention heads from circuits built on the same instances used for evaluation and measures the drop in M on those same instances. This cannot distinguish a faithful causal circuit from an artifact of optimizing the same metric used at construction time. The study lacks held-out Psub instances, ablations of low-scoring heads, and random-ablation baselines. As written, Table 1 does not support the claim that the circuit, or its attention-head interpretation, is faithful for knowledge overshadowing.","section":"Section 4.3, Table 1"},{"comment":"The recovery evaluation is circular. Eq. (7) selects τopt by maximizing M(Copt(τ), Psub, Ysub), and the same Psub and Ysub are subsequently used to report recovery in Table 2 and Figure 6. The reported successes are therefore selected on the evaluation set. Moreover, Ysub and Ydom are identified by the R-PMI heuristic described in Section 3.2.4, so the apparent recovery reflects, at least in part, the heuristic's own labeling. A valid evaluation requires a held-out split (or at least a separate selection criterion) for both the threshold optimization and the identification of Xsub, Ysub, Ydom.","section":"Section 3.2.4, Eq. (7), Table 2"},{"comment":"All dynamic-training claims (onset, duration, recovery as functions of P, M, D) are based on single training runs with no reported variance. Figures 3 and 4 show RO and LP curves without error bars or multiple seeds. Given that the models are trained from scratch with randomly sampled tokens, the qualitative differences attributed to P, M, and D could be within-run noise. The paper should report at least 3–5 seeds with error bars, or explicitly state that the plots are representative and provide replication evidence.","section":"Section 4.1.4, Figures 3 and 4"}],"minor_comments":[{"comment":"The abstract states the code is at a GitHub repository, while Section 3.2.2 says 'Our code will be available upon acceptance.' Please reconcile these statements and ensure the repository contains the exact circuit-construction and optimization scripts used for the reported results.","section":"Abstract and Section 3.2.2"},{"comment":"The notation 'Exp' in Eq. (6) is ambiguous: it could mean expectation over Psub samples or exponential. Please use explicit expectation notation (e.g., E_{Psub}[...]) and define the sampling distribution.","section":"Section 3.2.2, Eq. (6)"},{"comment":"Figure 5(c) is described as showing the circuit structure 'with 400 edges totally,' but the text does not specify the model, epoch, and P/D configuration used for this figure, making it difficult to connect the a5.h1/m4/m5 analysis to a particular experimental setting. Please add this information to the caption.","section":"Figure 5"},{"comment":"The sentence 'removed the top 10%, 20%, and 50% of attention heads by score' should clarify whether 'score' refers to the circuit edge score S(e) or attention score; the two are different quantities and the ablation result depends on this choice.","section":"Section 4.3"},{"comment":"Limitation 1 acknowledges that findings rely primarily on Pythia and synthetic/fine-tuning data. This is appropriate, but the natural-language case in Appendix E is a single pair (politician/singer) and a multi-hop anecdote; consider presenting a small but systematic set of natural cases if generalization is claimed.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting phenomenon, and the circuit-analysis lens is a reasonable direction. However, the central mechanistic and recovery claims presently rest on (i) a circuit scoring equation that may not be EAP-IG, (ii) a faithfulness check that is not discriminative, and (iii) a recovery evaluation that selects on the evaluation set. These are not merely presentational issues; they determine whether the reported results support the stated conclusions. I recommend major revision with the expectation that the authors either provide held-out and multi-seed evidence or substantially soften the mechanistic claims. I also note the inconsistency between the abstract's repository link and the main text's 'available upon acceptance' statement, which should be resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth reading for the training-dynamics experiment, and the circuit mechanism story is plausible but not yet proven. The paper's core new result is the controlled synthetic setup showing that with higher P and M, RO peaks early and recovers fast, and that larger D delays recovery. The LP co-evolution observation—overshadowing starts when subordinate loss is a negligible fraction of total loss, and recovery begins when LP rises—is the most interesting thing here. That's a clean, falsifiable descriptive claim about how this particular failure mode emerges during training.\n\nWhat the paper does well: the synthetic construction is careful (controlled token lengths, disjoint entities, varying P/M/D), and the authors are honest about the limitations of the setup. The circuit analysis is a reasonable first probe: looking at which attention heads concentrate on the dominant-vs-subordinate distinction, and using logit lens for the rank reversal at layer 5, gives a testable hypothesis. The Table 2 case flips are concrete evidence that pruning a circuit can change the model's top prediction in some cases.\n\nWhere it gets soft: Eq. 6 is not integrated gradients as Hanna et al. define it. It's an endpoint gradient times an activation difference—a first-order approximation that equals true IG only for linear models. The paper says \"similar to EAP-IG\" and Appendix B.1 hedges by saying \"methods like IG are often used to refine,\" but there's no path integral anywhere. So you can't inherit EAP-IG's faithfulness results without either implementing actual IG or validating the approximation on this task. The faithfulness ablation (Table 1) doesn't fill that gap: it removes top-scored heads from circuits built on the same instances, so it cannot distinguish causal importance from fitting the metric. The recovery evaluation is circular: Eq. 7 picks tau by maximizing M on the same Psub instances used for the demonstration, and Table 2 shows selected successes with no aggregate rate or held-out evaluation. Also, Figures 3 and 4 are single-run curves with no error bars or seed variance. And the abstract points to a GitHub URL but Section 3.2.2 says code will be released on acceptance; as it stands the actual S(e) computation is not inspectable.\n\nNet: the descriptive dynamics result is probably real under the stated synthetic conditions and could be cited for that. But the causal claim—that attentional allocation in circuits causes overshadowing and that pruning recovers the subordinate knowledge—is not yet supported. The paper deserves peer review, because the question and the controlled paradigm are worthwhile, but it needs held-out recovery evaluation, multiple seeds, a faithful circuit score (or at least a stated one), and code. If the authors deliver those, the mechanism story could become solid.","headline":"A controlled synthetic study with a genuinely interesting training-dynamics result, but the circuit-based mechanism and recovery claims are not yet supported by the evidence.","tokens_in":21552,"tokens_out":2572,"would_cite":true,"duration_ms":24940,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge overshadowing — when a model's dominant fact masks a rarer correct fact and the model hallucinates — is driven by how attention inside a sparse knowledge circuit is allocated, and pruning that circuit recovers the hidden answer.","keywords":["knowledge overshadowing","hallucination","knowledge circuits","mechanistic interpretability","attention heads","circuit pruning","training dynamics","Edge Attribution Patching"],"falsifier":"Build an optimized circuit on one overshadowing prompt pair and feed it a held-out subordinate prompt that shares the same background knowledge: if the circuit does not recover that prompt's correct answer while the full model still hallucinates, the claim that the identified circuit is the mechanism of overshadowing on that background fails. A sharper check targets the method itself: replace the integrated-gradient edge scores with direct activation patching on the same edges; if the causal edges found by actual patching disagree with the attribution scores used here, the faithfulness premise that carries the entire mechanism story collapses.","tokens_in":20469,"feed_emoji":"🧠","tokens_out":13580,"duration_ms":113380,"temperature":0.7,"pith_summary":"Knowledge overshadowing is a hallucination mode in which a model's strong learned association between background knowledge and a frequent fact masks a rarer fact that is also relevant, so the model answers with the frequent fact even after training on clean data. This paper argues that the phenomenon has a mechanistic, circuit-level origin: attention heads inside a sparse knowledge circuit allocate attention to the difference between dominant and subordinate inputs, and that allocation determines how strongly overshadowing occurs. To show this, the authors construct knowledge circuits with edge attribution patching, trace how overshadowing rises and falls across training as a function of knowledge popularity, model size, dataset size, and the loss share of subordinate knowledge, and then prune circuits to their optimal edge count to recover the suppressed correct answer. If the account holds, overshadowing is not an irreducible training artifact but a misallocation of attention inside a small subgraph — one that can be located and corrected.","feed_headline":"Pruning attention circuits unmasks overshadowed knowledge","feed_subtitle":"Rare correct facts resurface when the model's circuit is pruned to focus on what differs.","key_machinery":"The central object is the knowledge circuit $\\mathcal{C}$: a sparse directed subgraph of the transformer's computational graph, with nodes for attention heads, MLP layers, embeddings, and logits and edges for information flow, selected to be critical for distinguishing the dominant input $P_{\\rm dom}$ from the subordinate input $P_{\\rm sub}$. It is constructed by Edge Attribution Patching with Integrated Gradients: each edge is scored by the product of the parent activation difference (clean $P_{\\rm sub}$ minus corrupt $P_{\\rm dom}$) and the gradient of a metric $M$ defined as the logit difference between $Y_{\\rm sub}$ and $Y_{\\rm dom}$, edges with scores below a threshold $\\tau$ are pruned, and $\\tau$ is optimized by sweeping edge counts at uniform intervals and then applying golden-section search. The machinery does two jobs: it localizes overshadowing to specific attention heads and MLP nodes — the paper identifies a5.h1, m5, and m4 as the conduit of the rank reversal in the examined epoch — and the edge-pruning itself is the recovery mechanism. The quantitative backbone is the relative overshadowing rate $\\mathrm{RO} = p(Y_{\\rm dom}\\mid P_{\\rm sub})/p(Y_{\\rm dom}\\mid P_{\\rm dom})$ and the subordinate-loss proportion, whose co-evolution with $\\mathrm{RO}$ organizes the training-dynamics story.","core_discovery":"Stated on its own terms, the central claim is that 'the knowledge circuit's attentional allocation to differences between dominant and overshadowed knowledge inputs influences the extent of knowledge overshadowing.' The relative overshadowing rate $\\mathrm{RO} = p(Y_{\\rm dom}\\mid P_{\\rm sub})/p(Y_{\\rm dom}\\mid P_{\\rm dom})$ typically spikes toward 100% early in training and later falls to 0%, and the trajectory is shaped by three factors: higher knowledge popularity $P$ and larger model size $M$ bring earlier onset, shorter duration, and quicker recovery, while larger dataset size $D$ brings earlier onset but slower recovery. Recovery begins when the loss proportion of subordinate knowledge rises far enough that optimization effort shifts onto subordinate errors. Inside the knowledge circuit, high-attention heads focused on $\\{X_{\\rm dom}, X_{\\rm sub}\\}$ appear as $\\mathrm{RO}$ falls and disappear as it rises; in the representative epoch the paper dissects, head a5.h1 relays information to MLP layer m5 with layer-4 MLP mediation, and this is the layer where the rank of $Y_{\\rm sub}$ overtakes $Y_{\\rm dom}$. Finally, optimizing the circuit by pruning edges, via a two-stage sweep with golden-section search, eliminates the overshadowing effect on the tested prompts and flips the prediction from $Y_{\\rm dom}$ to $Y_{\\rm sub}$.","pith_inferences":["The recovery procedure is built per prompt, so the paper leaves implicit whether an optimized circuit transfers to a different subordinate instance sharing the same background knowledge; the paper itself notes its results rest on the Pythia suite and synthetic or fine-tuning data, so a cross-architecture, held-out transfer test would be the decisive follow-up.","Because the corrupt input replaces the subordinate token with a generic placeholder like 'something', the attribution may partly capture the model's reaction to a structural anomaly rather than to the dominant knowledge; building the corrupt input by substituting a random in-vocabulary token would separate these two effects.","The training-dynamics result points to a data-side intervention the authors do not implement: reweighting or resampling subordinate knowledge during the early-training window when its loss proportion is low could preempt the overshadowing spike before circuit-level recovery is needed.","If overshadowing is a general failure of circuits to attend to distinguishing tokens, the same construction and pruning tools could be used to diagnose other interference hallucinations, such as multimodal models defaulting to a dominant modality's prior — a connection the paper does not draw."],"forward_implications":["Overshadowing onset, duration, and recovery are predictable during training: higher knowledge popularity and larger model size compress the episode, while larger datasets make recovery slower.","Recovery from overshadowing begins when the loss proportion of subordinate knowledge rises enough to shift optimization effort onto subordinate errors, so tracking that proportion alongside the overshadowing rate gives an early signal of when a training run will escape the effect.","Persistent overshadowing in large pretrained models can be explained as insufficient optimization of subordinate knowledge at scale, which is why the hallucination survives high-quality training data.","Pruning a knowledge circuit to its optimal edge count recovers the masked subordinate answer without any weight update, making circuit optimization a candidate mitigation strategy for this type of hallucination.","High-attention heads focused on the dominant/subordinate difference are the mechanistic correlate of recovery: they are present in the circuit when the overshadowing rate falls and absent when it rises."],"supporting_citations":[{"why":"Defines knowledge overshadowing and its quantitative RO/P framing, and supplies the R-PMI procedure that PhantomCircuit's automated recovery simplifies.","marker":"(Zhang et al., 2025a)"},{"why":"Earlier formulation of overshadowing as amalgamated hallucination and the source of the R-PMI-based component identification.","marker":"(Zhang et al., 2024b)"},{"why":"EAP-IG, the edge attribution patching with integrated gradients method the circuit construction adapts, plus the faithfulness evidence the paper relies on.","marker":"(Hanna et al., 2024)"},{"why":"Introduces knowledge circuits and the optimized construction procedure that Appendix B.1 explicitly adapts.","marker":"(Yao et al., 2024)"},{"why":"The Pythia suite supplies the 70M to 2.8B models used for all training-dynamics and circuit experiments.","marker":"(Biderman et al., 2023)"},{"why":"Cited alongside Hanna et al. as the basis for the claim that EAP-IG circuits are built with high faithfulness and efficiency.","marker":"(Marks et al., 2024)"},{"why":"Attribution patching as the foundation of automated circuit discovery underpinning the edge-scoring approach.","marker":"(Syed et al., 2023)"},{"why":"The logit lens used for the layer-wise logit evolution analysis in the circuit study.","marker":"(nostalgebraist, 2020)"}],"fun_headline_variants":["Pruning attention circuits unmasks overshadowed knowledge","Knowledge circuit surgery reveals hidden LLM facts","Attention masks knowledge: circuit analysis explains why","How to spot and fix overshadowed knowledge in LLMs","Circuit pruning flips erroneous predictions to correct ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Edge Attribution Patching scores used to build each knowledge circuit faithfully identify which attention heads and MLP connections actually cause the overshadowed prediction; the paper's own check of this premise is a small ablation on fine-tuned models that does not test whether a circuit built from one set of prompts still works on held-out prompts.","fun_headline_variants_meta":{"raw":{"variants":["Pruning attention circuits unmasks overshadowed knowledge","Knowledge circuit surgery reveals hidden LLM facts","Attention masks knowledge: circuit analysis explains why","How to spot and fix overshadowed knowledge in LLMs","Circuit pruning flips erroneous predictions to correct ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1650,"prompt_tokens":981,"completion_tokens":669,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":597}},"tokens_in":597,"tokens_out":669,"duration_ms":7229,"temperature":1.0,"reasoning_tokens":597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:34:18.997506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an optimized circuit on one overshadowing prompt pair and feed it a held-out subordinate prompt that shares the same background knowledge: if the circuit does not recover that prompt's correct answer while the full model still hallucinates, the claim that the identified circuit is the mechanism of overshadowing on that background fails. A sharper check targets the method itself: replace the integrated-gradient edge scores with direct activation patching on the same edges; if the causal edges found by actual patching disagree with the attribution scores used here, the faithfulness premise that carries the entire mechanism story collapses.","supporting_citations":[],"review_version":1}