{"id":"5aa1bff5-02c5-4c86-a70a-9f8d77d1ab8e","arxiv_id":"2505.24731","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Circuit stability, measured as rank correlation between soft circuits across subtasks, is proposed as a predictor of language model generalization.","lead":"This paper defines circuit stability: how consistently a language model uses the same internal computation paths across different inputs. The authors report case studies where stability patterns align with performance drops, but the predictive claim is not actually tested out of sample.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central predictive claim is not tested out-of-sample: the alpha=0.6 clusters and accuracy comparisons are fit on the same 64 arithmetic subtasks, so the reported alignment is retrospective rather than predictive.","rationale":"The reader's REJECT verdict is well supported. I considered whether the most load-bearing concern should be the faithfulness of EAP-IG soft circuits, which is the reader's stated weakest assumption. That is a genuine risk, and exact activation patching on selected subtasks would be worth running. However, even a perfectly faithful circuit estimator would not save the paper's central claim, because no experiment evaluates prediction out of sample: the same subtasks are used to select alpha, form clusters, and measure accuracy, so the observed alignment is compatible with post hoc selection. The formal Definition 4 stability quantity is also never computed, making it hard to connect the formal framework to the empirical clusters. I do not see evidence of bad faith; the analysis is exploratory and clearly described in places, and the code is released. But the headline claim is stronger than the evidence. A held-out split with a fixed threshold and a trivial digit-count baseline is a small, decisive addition. Until then, rejecting the central claim as stated is the appropriate outcome.","tokens_in":21598,"tokens_out":6418,"duration_ms":82053,"concrete_test":"Pre-register alpha=0.6. Randomly split the 64 arithmetic subtasks into 48 fitting cells and 16 held-out cells, or hold out all subtasks with operand digit count greater than 6. Using only the fitting cells, fit a predictor of subtask accuracy from pairwise Spearman distances to a reference subtask, alpha-equivalence cluster membership, and within-cluster average stability. Fix the predictor before looking at held-out accuracies. Evaluate on held-out cells by RMSE or rank correlation against a baseline predictor that uses only operand digit counts. If the circuit-based predictor does not beat that baseline on held-out cells, the claim that circuit stability predicts length or compositional generalization is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that circuit stability can 'characterize and predict' generalization. That predictive claim is never actually tested. In Section 4, soft circuits are computed for all 64 subtasks, then alpha=0.6 is selected after inspecting Figure 2, in which 80% of circuit families emerge between alpha=0.58 and 0.79. The five named clusters are then compared with accuracies measured on the same 64 subtasks in Figure 4. This is an in-sample, post hoc description: the threshold and the clusters are chosen because they separate the subtasks that also differ in accuracy. The t-SNE visual confirmation in Figure 1 uses the same circuits and a perplexity of 3, which can produce separated-looking blobs even from noise; it does not independently validate the threshold. Section 5 likewise compares circuit behavior and model performance on the same constructed partitions. Section 6 is the only intervention, but it reports stability and accuracy before and after chain-of-thought without using the former to forecast the latter. In addition, Definition 4 is never computed: no infimum over a collection of partitions and no explicit expectation over subtask pairs is reported; the experiments use alpha-equivalence connected components instead. Thus, even if EAP-IG soft circuits are perfectly faithful, the paper's evidence does not establish the central predictive claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes circuit stability and circuit equivalence for transformer language models, defining soft circuits via edge attribution patching and using Spearman rank correlation as a similarity measure. It reports three case studies: two-operand addition on gemma-2-2b, Boolean expression evaluation on phi-1.5, and sports understanding with chain-of-thought prompting on Llama-3.1-8b and Gemma-2-9b. The central claim, stated in the abstract, is that circuit stability and instability can characterize and predict different aspects of generalization, offering a way to assess models without constructing new benchmarks. The paper also contributes formal definitions, an open-source codebase, and a discussion of circuit discovery methods.","tokens_in":21876,"tokens_out":4855,"duration_ms":54639,"significance":"If the central claim were established, circuit stability would be a valuable and low-cost evaluation signal, connecting mechanistic interpretability to generalization prediction. The formalization of soft circuits and equivalence is a useful step, and the release of code plus detailed experimental settings supports reproducibility. However, the empirical evidence is confined to retrospective, in-sample analyses: the threshold that defines circuit families is chosen after inspecting the data, and the accuracy comparisons use the same subtasks. The predictive claim is therefore not actually tested, and several load-bearing assumptions, such as the faithfulness of EAP-IG soft circuits, are left unvalidated. The idea is promising, but the current evidence does not support the advertised prediction capability.","major_comments":[{"comment":"The central predictive claim is not tested out-of-sample. The threshold alpha=0.6 is selected after inspecting Figure 2, and the resulting clusters are compared with accuracies measured on the same 64 subtasks in Figure 4. This is a retrospective, in-sample description: neither the threshold nor the clusters are used to forecast accuracy on held-out subtasks or held-out tasks. The t-SNE in Figure 1 is computed from the same circuits and does not provide independent validation because low perplexity t-SNE can create separated blobs even from noise. To support the abstract's 'predict' claim, the authors would need to fix the partition and alpha a priori, or select them on a training set and evaluate predictive accuracy on a held-out set of subtasks or tasks.","section":"Section 4.2, Figures 1–2, and Figure 4"},{"comment":"Definition 4 (Eq. 2) is never instantiated in the experiments. The definition requires an infimum over a collection of partitions and an expectation over subtask pairs drawn from the partition distribution. Instead, the arithmetic case study computes pairwise Spearman correlations between all 64 soft circuits and thresholds them at alpha=0.6 to form connected components. This measures a different quantity: the empirical clustering of pairwise alpha-equivalence, not circuit stability as formally defined. At minimum, the paper should either compute the defined quantity or explicitly state that the experiments operationalize a weaker, pairwise notion and justify its relationship to Definition 4.","section":"Section 3, Definition 4, and Section 4.2"},{"comment":"The faithfulness of EAP-IG soft circuits is assumed rather than established. Section 2 cites Miller et al. (2024) showing that circuit discovery is sensitive to the performance metric and threshold, but no robustness analysis is provided for the soft circuits used here. If the soft-circuit rankings are not faithful to the model's actual causal structure, then Spearman correlation between them does not measure consistency of reasoning. A concrete test would be to compare EAP-IG scores against activation patching on a subset of edges, or to verify that the top-ranked edges recover previously identified arithmetic circuits (e.g., for addition). Without such validation, the connection between 'circuit stability' and generalization is not established.","section":"Appendix B, Definition 3, and Section 2"},{"comment":"The statement 'not is associative' is mathematically incorrect. The operator 'not' is unary, so associativity is not defined for it. The intended claim is likely that redundant parentheses do not change the semantics of an expression containing only 'not' and literals, which is true but requires a different justification. This error weakens the interpretation of the Not subtask results: the observed 40% performance drop after adding parentheses is still evidence of a problem, but the asserted reason (failure to understand associativity) is unsound.","section":"Section 5.2, Not subtask"},{"comment":"The claim that performance differences between subtask clusters are 'not merely an artifact of statistical noise' is not supported by uncertainty quantification. Figure 4 reports exact-match accuracies with n=1000 per subtask but no error bars, confidence intervals, or significance tests. Given that several adjacent cells differ by only a few percent, and the cluster boundary differences are sometimes around 10–20%, the lack of error bars makes it impossible to verify the claim. At minimum, the authors should report bootstrap intervals or pairwise significance tests for the cluster boundary differences.","section":"Section 4.3, Figure 4"}],"minor_comments":[{"comment":"In the sentence 'we see that chain-of-thought significantly circuit stability', a verb appears to be missing; it should likely read 'significantly improves circuit stability'.","section":"Section 6"},{"comment":"The word 'meaingful' should be 'meaningful'.","section":"Section 5, introductory paragraph"},{"comment":"The phrase 'well-studied in the the literature' contains a duplicated 'the'.","section":"Appendix A"},{"comment":"The multiple panels in Figure 3 are not labelled with the corresponding alpha values, which makes it difficult to follow the description of how circuit families emerge as alpha increases.","section":"Figure 3"},{"comment":"The abbreviation 'EA-IG' is used in one place; it should be 'EAP-IG' to match the edge attribution patching with integrated gradients terminology used elsewhere.","section":"Appendix B"}],"recommendation":"reject","confidential_remarks":"The core idea is worth pursuing, but the manuscript as written does not provide evidence for the predictive claim: the threshold and clusters are chosen post hoc, Definition 4 is not measured, and the soft-circuit faithfulness is unvalidated. The 'not is associative' error also needs correction before resubmission. I would be open to a revised version that includes a genuine out-of-sample prediction test and a faithfulness check, but that would be a substantial new experimental effort."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. The formal framework is genuinely new and worth engaging with: the soft-circuit definitions, epsilon-stability, and alpha-equivalence are clean and clearly stated, and the three case studies are a reasonable first pass at showing the idea has empirical traction. The paper also ships its code, and the Boolean-expression study uses independent expectations about when circuits should or should not be stable, which is the most convincing evidence in the paper. The chain-of-thought experiment is preliminary but suggestive.\n\nThat said, the central claim in the abstract—that circuit stability can \"predict\" generalization—is not supported by the experiments as they stand. The alpha = 0.6 threshold is selected after inspecting Figure 2, and the resulting clusters are then compared with accuracies measured on the same 64 arithmetic subtasks. That is an in-sample, post hoc description, not a prediction. Definition 4, the actual epsilon-stability quantity, is never computed; the experiments use alpha-equivalence connected components instead, which is a related but weaker notion. The t-SNE confirmation uses perplexity = 3, which can produce separated-looking blobs from noise, and Figure 4 has no error bars. These are real problems, especially because the paper explicitly claims prediction.\n\nThe arithmetic case study is therefore best read as a characterization, not a prediction. The Boolean study does better because the expected circuit differences are derived from the structure of the task (parentheses matter for Not+And but not for Not alone), and the observed stability patterns line up with those expectations. That gives me some confidence the framework can encode meaningful signal. The CoT study is too underpowered to carry weight: five random partitions, two models, and a significance test without correction.\n\nWho is this for? Researchers working on mechanistic interpretability and LLM evaluation, particularly those interested in circuit reuse and task decomposition. The limitations section is honest, but it does not fix the gap between what is claimed and what is shown. To make the predictive claim stick, the author would need to pre-register a threshold and partition scheme, or at least hold out a set of subtasks and show that circuit stability predicts their accuracy before looking at the accuracy numbers.\n\nThe paper deserves a serious referee, because the definitions are novel and the empirical approach points in a useful direction. I would recommend engaging with it, but with the expectation of major revisions on the prediction framing. If the author can turn the arithmetic study into a genuine out-of-sample test, this could become a solid contribution.","headline":"A genuinely new formal definition of circuit stability, but the headline 'predicts generalization' claim is not actually tested out-of-sample, so the paper is a promising start rather than a finished contribution.","tokens_in":22347,"tokens_out":1599,"would_cite":false,"duration_ms":21662,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model generalizes when its reasoning stays consistent across subtasks.","keywords":["circuit stability","soft circuits","mechanistic interpretability","generalization","language models","attribution patching","chain-of-thought prompting","benchmark saturation"],"falsifier":"Find a task where a model's accuracy stays flat across a subtask boundary but the EAP-IG soft circuits change sharply; if circuit stability is low while generalization is perfect, stability is not necessary for generalization. The reverse—high stability with a large accuracy cliff—would also falsify sufficiency. A concrete candidate is addition with a carry boundary, where the correct algorithm may change with operand length while performance remains high.","tokens_in":21390,"feed_emoji":"🧠","tokens_out":5591,"duration_ms":60800,"temperature":0.7,"pith_summary":"This paper argues that a language model generalizes when it applies the same reasoning process, its circuit, across related subtasks. The author formalizes \"soft circuits\" as edge-importance scores computed by ablating each edge of the model's computational graph, then defines circuit stability as the rank correlation between the soft circuits of different subtasks. Three case studies—arithmetic addition, Boolean expression evaluation, and sports understanding—show that high stability tracks successful length, structural, and compositional generalization, while instability flags where the model fails. If the claim holds, evaluating generalization would not require building ever-harder benchmarks; one could instead measure whether a model's internal reasoning is consistent. The paper offers this as a step toward tying interpretability directly to generalization.","feed_headline":"Stable circuits predict generalization without new benchmarks","feed_subtitle":"Measure whether a model uses the same reasoning on every subtask, and you can see where it will fail.","key_machinery":"The central object is the soft circuit, a continuous relaxation of the discrete circuit used in mechanistic interpretability: instead of marking each edge in the computational graph as in or out, every edge receives a real score $c(e) = \\mathbb{E}_{(x,y)\\sim\\mathcal{D}}[\\mathcal{L}(M_{\\{e\\}}(x),y) - \\mathcal{L}(M(x),y)]$, the expected change in a performance metric when that edge is ablated. Soft circuits are computed with edge attribution patching with integrated gradients (EAP-IG), a Taylor-approximation of activation patching that needs only a constant number of forward and backward passes. Similarity between soft circuits is measured by Spearman's $\\rho$, and this feeds the two formal definitions: $\\varepsilon$-circuit stability (the expected pairwise similarity across subtasks exceeds $\\varepsilon$) and $\\alpha$-equivalence (two subtasks' circuits correlate at least $\\alpha$). The paper's mechanism is to partition a task into subtasks, compute one soft circuit per subtask, and read generalization off the resulting cluster structure.","core_discovery":"The central claim is that circuit stability—a model's tendency to reuse the same computational subgraph when solving inputs drawn from different subtasks—is a predictor and characterizer of generalization. Stability is measured on soft circuits, where each edge of the transformer's computational graph receives a real-valued importance equal to the expected change in a performance metric when that edge is ablated. Soft circuits are compared with Spearman's rank correlation, and two subtasks are $\\alpha$-equivalent when their circuits correlate above a threshold. In gemma-2-2b on addition, subtasks cluster into circuit families that match performance: commutativity violations show up as different circuits for (o1,o2) versus (o2,o1), and within a family accuracy decays smoothly and predictably. In phi-1.5 on Boolean expressions, instability across parenthesization reveals whether the model has internalized operator precedence and associativity. On sports understanding, chain-of-thought prompting raises circuit stability while raising accuracy, suggesting stability is an actionable, not just descriptive, quantity.","pith_inferences":["A practical extension the paper does not pursue: circuit stability could be computed on a held-out set of subtasks before new benchmarks are built, turning it into a cheap screen for which capability gaps are worth probing.","The faithfulness assumption could be tested directly: if EAP-IG scores are replaced by exact activation patching on a small subset of edges, the stability rankings should survive; if they flip, stability is an artifact of the approximation.","The framework suggests a definition of a learned skill as an equivalence class of soft circuits, which connects to compositionality research where a skill is a reusable subroutine."],"forward_implications":["For arithmetic, circuit instability across digit-length subtasks predicts where gemma-2-2b's exact-match accuracy drops, including the 20-plus percentage-point gap between commutative partners.","For Boolean expressions, the presence or absence of circuit equivalence across parenthesized and non-parenthesized subtasks tells whether phi-1.5 has internalized associativity and operator precedence.","Chain-of-thought prompting increases circuit stability on sports understanding for Llama-3.1-8b and Gemma-2-9b, linking interpretability metrics to an intervention that improves accuracy.","Because soft circuits can be estimated efficiently, circuit stability offers a benchmark-free signal for tracking and comparing model generalization."],"supporting_citations":[{"why":"Supplies the transformer computational graph representation (nodes and edges) on which soft circuits are defined.","marker":"Elhage et al., 2021"},{"why":"Establishes automated circuit discovery and the noisy-to-clean patching baseline that the paper's method builds on.","marker":"Conmy et al., 2023"},{"why":"Introduces attribution patching, the efficient linear approximation that underlies the EAP-IG soft-circuit estimator.","marker":"Nanda, 2023"},{"why":"Provides edge attribution patching with integrated gradients (EAP-IG), the specific circuit discovery algorithm used in all experiments.","marker":"Hanna et al., 2024"},{"why":"Shows circuit discovery metrics are sensitive to performance metric and threshold, motivating the paper's soft-circuit relaxation.","marker":"Miller et al., 2024"},{"why":"Supplies the BIG-bench evaluation protocol, including the Boolean expression task and exact string match accuracy.","marker":"Srivastava et al., 2023"},{"why":"Provides the sports understanding task used in the chain-of-thought case study.","marker":"Suzgun et al., 2023"},{"why":"Introduces chain-of-thought prompting, the intervention the paper uses to induce circuit stability.","marker":"Wei et al., 2022"},{"why":"Supports the interpretation that arithmetic abilities are mediated by shared attention heads and MLPs, grounding the cluster analysis.","marker":"Nikankin et al., 2025"}],"fun_headline_variants":["Circuit stability predicts generalization without new benchmarks","Stable circuits expose generalization gaps","Circuit stability: a new predictor of model generalization","No new benchmarks: circuit stability forecasts model accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that soft circuits, as computed by edge attribution patching with integrated gradients, faithfully reflect the model's actual reasoning, so that a high rank correlation between two soft circuits really means the model is reasoning consistently.","fun_headline_variants_meta":{"raw":{"variants":["Circuit stability predicts generalization without new benchmarks","Stable circuits expose generalization gaps","Circuit stability: a new predictor of model generalization","No new benchmarks: circuit stability forecasts model accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1695,"prompt_tokens":849,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":792}},"tokens_in":465,"tokens_out":846,"duration_ms":10516,"temperature":1.0,"reasoning_tokens":792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:14:04.973095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a task where a model's accuracy stays flat across a subtask boundary but the EAP-IG soft circuits change sharply; if circuit stability is low while generalization is perfect, stability is not necessary for generalization. The reverse—high stability with a large accuracy cliff—would also falsify sufficiency. A concrete candidate is addition with a carry boundary, where the correct algorithm may change with operand length while performance remains high.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the transformer computational graph representation (nodes and edges) on which soft circuits are defined."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows circuit discovery metrics are sensitive to performance metric and threshold, motivating the paper's soft-circuit relaxation."},{"cited_title":"Brown, et al","cited_arxiv_id":null,"evidence_quote":"Supplies the BIG-bench evaluation protocol, including the Boolean expression task and exact string match accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting, the intervention the paper uses to induce circuit stability."}],"review_version":1}