{"id":"15b7d5de-c4de-4940-be2d-2216ca6e51e4","arxiv_id":"2606.27510","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Re-derivation of activation patching NIE reveals it captures interaction effects in addition to direct causal effects, demonstrated via GPT-2 IOI circuit where INT explains component ranking issues and faithfulness instability.","lead":"The paper re-derives the activation patching estimand from causal mediation analysis and shows that the natural indirect effect includes interaction effects measuring how one component's causal impact depends on others. This explains observed instabilities in faithfulness scores and positions interaction effects as a diagnostic rather than a flaw to remove.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Whether standard NIE decomposition from mediation analysis applies without distortion to activation patching interventions on transformer computation graphs","rationale":"The reader's weakest assumption directly identifies the load-bearing step. The abstract presents the re-derivation as the foundation; any mismatch between the patching operation and the mediation assumptions would make the subsequent claims about INT scaling, local affinity, and faithfulness instability rest on an unverified translation rather than a proven equivalence.","tokens_in":1733,"tokens_out":305,"duration_ms":14288,"concrete_test":"Take the paper's definition of the patching intervention (clean run vs. patched run on a single component) and re-derive the difference in output expectation step-by-step from the transformer forward pass; check whether it equals the standard NIE expression in §2 without invoking extra assumptions on the functional form of the remaining components.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on re-deriving the activation patching estimand as NIE + INT. This requires that the intervention (replacing one component's activation while leaving others at their observed values) corresponds exactly to the do-operator on a single mediator in the causal graph assumed by the mediation analysis, with no additional distortion from residual connections, attention mixing, or the fact that all components are computed jointly from the same input. If the patching operation implicitly affects multiple mediators or violates the no-interference assumption used in the decomposition, then the extracted INT term does not cleanly measure the claimed interaction effects.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that activation patching, which estimates the natural indirect effect (NIE) of a component on model behavior, actually captures both the component's direct causal effect and hidden interaction effects (INT) with other components. Re-deriving the estimand from causal mediation analysis, the authors prove that INT scales with the distance between clean and patched activations, is negligible when the model is locally affine, and combinatorially decomposes into pairwise and higher-order group interactions. They demonstrate in the GPT-2 IOI circuit that INT causes components to appear invisible or inflated under NIE-based analysis and explains instability in faithfulness scores, arguing that INT serves as a diagnostic for prompt-dependent causal conclusions rather than a quantity to eliminate.","tokens_in":1850,"tokens_out":645,"duration_ms":39263,"significance":"If the central derivation and proofs hold, this is a significant contribution to mechanistic interpretability. Activation patching is the dominant causal attribution tool, and identifying that its NIE estimand systematically includes interaction effects explains documented instabilities and warns against greedy component ranking. Credit is due for the formal proofs of INT scaling and combinatorial decomposition (which yield falsifiable predictions) as well as the empirical demonstration on the established GPT-2 IOI circuit. The work reframes INT as a useful diagnostic rather than a nuisance, with potential to improve the reliability of causal claims in the field.","major_comments":[{"comment":"The re-derivation of the activation patching estimand as NIE + INT (main text, causal mediation section) rests on the assumption that replacing one component's activation while holding others at observed values corresponds exactly to the do-operator on a single mediator with no interference. In transformer computation graphs, residual connections and attention mixing mean all components are computed jointly from the same input; this may violate the no-interference assumption underlying the standard NIE decomposition, potentially distorting the extracted INT term. A concrete test (e.g., invariance of INT under patching order or residual-stream ablations) is needed to confirm the mapping holds without distortion.","section":"Re-derivation of the activation patching estimand"},{"comment":"§ on proof of INT scaling and local affinity: the claim that INT is negligible when the model is locally affine is load-bearing for the recommendation to treat INT as diagnostic rather than eliminable, but the proof sketch does not explicitly state the functional-form assumptions on the response surface or how the affine approximation is verified empirically in the GPT-2 experiments.","section":"Proof that INT scales with distance and is negligible when locally affine"}],"minor_comments":[{"comment":"Abstract: the acronym INT is introduced in the second sentence without an immediate parenthetical definition, which reduces readability for readers unfamiliar with mediation analysis.","section":"Abstract"},{"comment":"The combinatorial decomposition into pairwise and higher-order interactions is stated clearly but would benefit from an explicit small example (e.g., three-component case) to illustrate the group-level terms before the GPT-2 results.","section":"Combinatorial decomposition"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which help strengthen the causal foundations and proof details of our work. We respond to each major comment below and will revise the manuscript accordingly where appropriate.","responses":[{"response":"We model the transformer as a structural causal model in which component activations are the mediators, and the activation-patching intervention is defined directly on a single mediator while others take their observed (natural) values. This matches the standard NIE definition from causal mediation analysis; residual streams and attention are part of the joint structural equations but do not alter the interventional semantics at the mediator level. We agree a concrete check is valuable and will add a subsection with an empirical invariance test of INT estimates under varied patching orders on the GPT-2 IOI circuit, plus a brief discussion of the no-interference assumption in residual architectures.","revision_made":"yes","referee_comment":"[Re-derivation of the activation patching estimand] The re-derivation assumes replacing one component's activation while holding others corresponds to the do-operator on a single mediator with no interference. In transformers, residual connections and attention mixing may violate this, distorting INT. A concrete test (e.g., invariance under patching order or residual-stream ablations) is needed."},{"response":"The local-affinity claim rests on a first-order Taylor expansion of the response surface (model output as a function of the mediators) around the clean activation point; under this approximation all second- and higher-order terms, including INT, vanish. We will expand the proof section to state the required differentiability and neighborhood-size assumptions explicitly. We will also add empirical verification in the GPT-2 experiments by reporting second-order finite-difference measures of nonlinearity to confirm that INT remains small precisely where the local affine condition holds.","revision_made":"yes","referee_comment":"[Proof that INT scales with distance and is negligible when locally affine] The claim that INT is negligible when locally affine is load-bearing, but the proof sketch does not explicitly state the functional-form assumptions on the response surface or how the affine approximation is verified empirically in the GPT-2 experiments."}],"tokens_in":1552,"tokens_out":459,"duration_ms":26579,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core point is that standard activation patching estimates the natural indirect effect through a component, but that NIE also picks up interaction terms measuring how the component's effect depends on the state of other parts of the model. The authors re-derive the patching estimand from causal mediation analysis, split it into direct and interaction pieces, prove that the interaction term scales with the distance between clean and patched activations, vanishes under local affinity, and decomposes into pairwise and higher-order terms. They then run the decomposition on the GPT-2 IOI circuit and show that components whose importance is conditional on other components either drop out or get inflated, and that INT variance tracks the known instability in faithfulness scores.\n\nThe work is clearest when it treats the interaction term as information rather than noise to be removed. The suggested remedies for eliminating INT each have the failure modes the authors flag, and the empirical results line up with the math on the IOI task. The citation pattern is light and focused on the relevant mediation and interpretability literature.\n\nThe main soft spot is whether the patching intervention maps exactly onto the single-mediator do-operator assumed in the decomposition. Residual streams and attention mixing mean that changing one activation can have downstream effects that are not purely the intended mediator intervention. The paper's proofs and GPT-2 results suggest the decomposition still holds usefully in practice, but the mapping is an assumption that could be probed more directly with controlled synthetic graphs.\n\nThis is for mechanistic interpretability groups that rely on activation patching for circuit ranking. It is worth sending to peer review because the formal split is checkable, the empirical demonstration is on a known circuit, and the diagnostic framing is actionable.","headline":"The paper shows activation patching NIE mixes in interaction effects from other components, formalizes the split via mediation analysis, and treats INT as a useful diagnostic rather than a bug.","tokens_in":2314,"tokens_out":418,"would_cite":true,"duration_ms":14703,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Activation patching's natural indirect effect includes hidden interaction effects between model components.","keywords":["activation patching","natural indirect effect","interaction effects","causal mediation analysis","mechanistic interpretability","transformer circuits","IOI task","faithfulness scores"],"falsifier":"Compute the isolated direct effect of a component while holding all other components at their clean values and compare the result to the NIE obtained by standard patching; a systematic difference matching the predicted INT term would confirm the decomposition.","tokens_in":2639,"feed_emoji":"🔍","tokens_out":560,"duration_ms":32231,"temperature":0.7,"pith_summary":"The paper re-derives the quantity estimated by activation patching using causal mediation analysis and shows that the natural indirect effect attributed to one component also contains interaction effects measuring how that component's causal influence depends on the states of other components. These interactions cause standard patching results to miss or inflate the apparent importance of components whose effects are conditional, as demonstrated in the GPT-2 IOI circuit where they produce invisible or artificially strong components and explain unstable faithfulness scores. The authors prove that interaction magnitude scales with the distance between clean and patched activations, becomes negligible in locally affine models, and decomposes into pairwise and higher-order group terms. They argue that interaction effects should be retained as a diagnostic rather than eliminated, because their size and sign indicate when causal attributions are prompt-dependent and when greedy single-component ranking will overlook mechanisms that require joint search.","feed_headline":"Activation patching mixes direct effects with interactions","feed_subtitle":"Re-derivation shows the natural indirect effect also captures how one component's impact depends on others, producing misattributions and un","key_machinery":"The decomposition of the natural indirect effect (NIE) into direct causal effect plus interaction effects (INT) obtained by applying causal mediation analysis to activation patching.","core_discovery":"Re-deriving the activation patching estimand from causal mediation analysis reveals that the natural indirect effect (NIE) decomposes into the component's isolated causal effect plus interaction effects (INT) that quantify how much the component's effect itself depends on the states of other components. In the GPT-2 IOI circuit this produces components whose causal importance is conditional and therefore invisible or inflated under standard patching, while INT variance accounts for previously observed instability in faithfulness scores. INT scales directly with activation distance between clean and patched runs, vanishes when the model is locally affine, and factors combinatorially into pair","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Hidden interactions bias activation patching","Patching mixes effects with component interactions","Interaction effects confound mediator attributions","Activation patching entangled by hidden interactions"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The causal mediation analysis decomposition of NIE into direct and interaction components applies without distortion to the specific intervention used in activation patching on transformer models.","fun_headline_variants_meta":{"raw":{"variants":["Hidden interactions bias activation patching","Patching mixes effects with component interactions","Interaction effects confound mediator attributions","Activation patching entangled by hidden interactions"]},"model":"grok-4.3","cost_usd":0.005615,"raw_usage":{"total_tokens":2717,"prompt_tokens":727,"num_sources_used":0,"completion_tokens":44,"cost_in_usd_ticks":56149500,"prompt_tokens_details":{"text_tokens":727,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1946,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":727,"tokens_out":44,"duration_ms":22793,"temperature":1.0,"reasoning_tokens":1946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T01:44:58.820882+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Compute the isolated direct effect of a component while holding all other components at their clean values and compare the result to the NIE obtained by standard patching; a systematic difference matching the predicted INT term would confirm the decomposition.","supporting_citations":[],"review_version":1}