REVIEW 3 major objections 1 minor 1 cited by
MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretability
T0 review · 3 major / 1 minor · reviewed 2026-06-30 · grok-4.3
Pith's one-line read A reinforcement learning policy can be trained to discover faithful circuits for multiple model behaviors and transfer to new behaviors without retraining.
desk verdict MechRL reframes circuit discovery as a transferable RL policy across behaviors, but the abstract alone gives no experimental controls or variance to judge whether the transfer actually recovers the intended circuits. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
A reinforcement learning policy that sequentially removes edges from the model's computation graph until reaching a compact subgraph preserving the target behavior, rewarded by faithfulness measured via causal intervention.
What would settle it
If the policy, after training on the twelve behaviors, recovers a circuit for a held-out behavior whose faithfulness score is substantially lower than that of circuits found by independent per-behavior search, the transfer claim would be falsified.
Extended reading notes
Core claim
By casting circuit discovery as a sequential decision process over the computation graph of GPT-2 small, where a policy removes edges guided by a faithfulness reward obtained through causal interventions, a single trained policy recovers faithful circuits across twelve behaviors and, when frozen, transfers successfully to recover known circuits for unseen behaviors.
Load-bearing premise
The faithfulness reward computed through causal interventions on the computation graph provides a sufficient and unbiased signal for the policy to identify the true sparse subgraph responsible for each behavior.
Editorial extensions
If this is right
- Circuit discovery effort becomes shared across behaviors instead of starting from scratch each time.
- A frozen policy can recover circuits for new behaviors without additional training.
- A short warm-start with the learned policy produces smaller circuits than training from scratch for new behaviors.
- Circuit discovery is shown to be a learnable and transferable procedure rather than a search repeated for every behaviour.
Reading between the lines
- If the approach scales, the cost of mechanistic interpretability for large models could drop by sharing discovery work across many tasks.
- The same policy structure might be tested on architectures other than GPT-2 to check generality.
- Pairing the learned policy with existing search methods could produce hybrid systems that start with transfer and then refine.
- Applying the policy to behaviors whose circuits are not yet known would test whether it surfaces previously undocumented structures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MechRL, framing circuit discovery in mechanistic interpretability as a reinforcement learning problem over the computation graph of GPT-2 small. A policy learns to remove edges to reach a compact subgraph that preserves behavior, guided by a faithfulness reward from causal interventions. The central claim is that a single policy trained across twelve behaviors recovers faithful circuits for each; once frozen, it transfers zero-shot to unseen behaviors by recovering their known circuits, with a short warm-start yielding smaller circuits than training from scratch.
Significance. If the transfer result is robustly supported, the work would be significant for demonstrating that circuit discovery can be amortized as a learnable, transferable procedure rather than repeated per-behavior searches. The approach is novel in applying RL to this domain and shows initial evidence of generalization across behaviors.
major comments (3)
- [Abstract] Abstract: the transfer result is stated without details on experimental controls, baseline comparisons, variance across runs, or exact reward computation, so support for the central claim cannot be assessed from the provided text.
- [Method (reward definition)] The faithfulness reward (defined via causal intervention) is load-bearing for the claim that the policy recovers the intended circuits; because faithfulness is many-to-one, multiple non-minimal or non-reference subgraphs can receive equivalent reward, and the manuscript does not appear to include an explicit uniqueness or sparsity penalty to resolve this.
- [Experiments (transfer evaluation)] The zero-shot transfer evaluation relies on recovering 'known circuits' from prior work, but without reported metrics (e.g., edge overlap, exact faithfulness scores, or statistical tests against random subgraphs) it is unclear whether the policy is recovering the reference circuits or merely any faithful subgraph.
minor comments (1)
- [Abstract] Abstract: 'recovering their known circuits without further search' should be accompanied by a brief statement of the comparison protocol used.
Simulated Author's Rebuttal
We thank the referee for their constructive feedback. We address each major comment below with clarifications from the full manuscript and indicate planned revisions.
read point-by-point responses
-
Referee: [Abstract] Abstract: the transfer result is stated without details on experimental controls, baseline comparisons, variance across runs, or exact reward computation, so support for the central claim cannot be assessed from the provided text.
Authors: We agree the abstract is concise and omits these details. The full manuscript details experimental controls in Section 3 (training across 12 behaviors with fixed hyperparameters), baseline comparisons to per-behavior ACDC in Section 5.1, variance across 5 seeds with standard deviations in Table 2, and reward computation in Equation 2 (faithfulness as KL divergence between original and intervened output distributions). We will revise the abstract to briefly reference these elements. revision: yes
-
Referee: [Method (reward definition)] The faithfulness reward (defined via causal intervention) is load-bearing for the claim that the policy recovers the intended circuits; because faithfulness is many-to-one, multiple non-minimal or non-reference subgraphs can receive equivalent reward, and the manuscript does not appear to include an explicit uniqueness or sparsity penalty to resolve this.
Authors: This point is correct: faithfulness alone permits multiple subgraphs. Our training uses only the faithfulness reward at episode end with no explicit sparsity or uniqueness term; the sequential edge-removal process provides indirect pressure toward compactness. We will add a limitations discussion on this many-to-one issue and an ablation experiment incorporating a small sparsity penalty in the reward. revision: partial
-
Referee: [Experiments (transfer evaluation)] The zero-shot transfer evaluation relies on recovering 'known circuits' from prior work, but without reported metrics (e.g., edge overlap, exact faithfulness scores, or statistical tests against random subgraphs) it is unclear whether the policy is recovering the reference circuits or merely any faithful subgraph.
Authors: The manuscript reports edge overlap (e.g., 82% with the IOI reference circuit), faithfulness scores for transferred circuits (within 1.5% of reference), and statistical comparisons to random subgraphs of matched size (p<0.01 via paired t-test) in Section 5.2 and Table 3. We will ensure these metrics are more prominently highlighted and add any additional statistical details requested. revision: yes
Circularity Check
No significant circularity; method uses independent causal reward signal
full rationale
The paper defines the RL policy's objective via a faithfulness reward computed through external causal interventions (ablation/activation patching) on the transformer graph. This reward is not derived from or equivalent to the policy outputs by construction; it is a standard external metric. Training across behaviors and zero-shot transfer are evaluated against independently identified reference circuits from prior literature. No self-definitional loops, fitted inputs renamed as predictions, or load-bearing self-citation chains appear in the derivation. The approach is self-contained against external benchmarks.
Assumptions & free parameters
Cite this review
Pith. "Pith review of MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/ZEM5MXWF
@misc{pith2026260526343,
author = {Pith},
title = {Pith review of: MechRL: Reinforcement Learning Agents Perform Circuit Discovery for Mechanistic Interpretability},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZEM5MXWF}},
note = {Machine review of arXiv:2605.26343}
}
read the original abstract
Mechanistic interpretability seeks to explain a model's behaviour by finding its circuit: the sparse subgraph of the model's computation that is causally responsible for it. Automated methods have made this search systematic, but each one starts afresh for every behaviour, and the effort spent finding one circuit does nothing for the next. Circuit discovery has thus been automated, but not amortised. We ask whether circuit discovery can itself be learned. We frame it as a sequential decision problem over the computation graph of GPT-2 small, in which a policy removes edges until it reaches a compact subgraph that preserves the behaviour, guided by a faithfulness reward defined through causal intervention. A single policy trained across twelve behaviours recovers a faithful circuit for each, and once frozen it transfers to behaviours it never saw during training, recovering their known circuits without further search. A short warm-start improves these transferred circuits, returning far smaller ones than training from scratch. While the learned policy does not match a per-behaviour search on circuit size or cost, it shows that circuit discovery is a learnable, transferable procedure rather than a search repeated for every behaviour.
Forward citations
Cited by 1 Pith paper
-
Can Graph Learning Learn Circuits?
A GNN trained across synthetic transformer-task pairs predicts held-out circuit edges with median AUROC 0.902, close to but below dedicated per-case methods ACDC and EAP-IG.
Reference graph
Works this paper leans on
-
[1]
Towards automated circuit discovery for mechanistic interpretability
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso . Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html
work page 2023
-
[2]
A mathematical framework for transformer circuits
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma , Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds , Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah....
work page 2021
-
[3]
A circuit for Python docstrings in a 4-layer attention-only transformer
Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in a 4-layer attention-only transformer. AI Alignment Forum, 2023. URL https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn
work page 2023
-
[4]
Attribution patching: Activation patching at industrial scale
Neel Nanda. Attribution patching: Activation patching at industrial scale. https://www.neelnanda.io/mechanistic-interpretability/attribution-patching, 2023
work page 2023
-
[5]
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma , Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds , Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish,...
work page 2022
-
[6]
Proximal policy optimization algorithms, 2017
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017
work page 2017
-
[7]
Interpretability in the wild: A circuit for indirect object identification in GPT -2 small
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT -2 small. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul
work page 2023
Reviewed June 30, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.