Pith. sign in

REVIEW 3 major objections 5 minor 8 references

Automated Attribution Graph Interpretation via Probe Prompting

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Probe prompting automatically groups transformer circuit features into interpretable supernodes from their activation patterns on concept-targeted probe prompts, and the paper reports descriptive evidence that these supernodes are behaviora

desk verdict The abstract promises 45,596 causal steering validations; the body delivers 5 prompts, 39 features, and explicitly defers causal tests to future work—fix that mismatch and this is a worthwhile proof-of-concept, not a bomb. read the letter →

arxiv 2511.07002 v2 pith:ZXVNZJUB submitted 2025-11-10 cs.CL

Giuseppe Birardi , Gonçalo Paulo This is my paper · ORCID
classification cs.CL
keywords probepromptingattributiongraphscircuitinterpretationcross-layertranscoderssupernodesbehavioralsignaturesmechanisticinterpretabilitylayerwisehierarchy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the thousands of sparse features in a transformer's attribution graph can be automatically grouped into a small number of interpretable supernodes by measuring each feature's behavior on a handful of concept-targeted probe prompts, rather than by clustering activation vectors geometrically. The authors argue that these behavioral groupings are more coherent—features in one group tend to fire on the same tokens across contexts—and that the resulting subgraphs preserve about 83% of explanatory coverage while shrinking from thousands of nodes to tens. They also report evidence for a layerwise division of labor: features in early layers transfer across entity substitutions, while late-layer features specialize in promoting the specific output token. The payoff would be cutting circuit interpretation from hours to minutes and giving researchers a shared vocabulary for feature roles.

What carries the argument

The central object is the Cross-Prompt Activation Signature (CPAS): for each feature, a behavioral profile computed from its activation on a set of probe prompts that preserve syntax but vary semantics. The profile records cosine similarity to the seed activation, robust z-scores, peak token identity and position, activation density, and sparsity. Aggregated across probes, these signatures feed transparent decision rules that assign each feature to one of three categories—Semantic (stable word or concept detector), Relationship (diffuse, sentence-spanning binder), or Say-X (output-promotion via functional tokens like 'is' or 'the')—and these categories become the supernodes. The work is carr

What would settle it

Run the published pipeline on a new entity-swap pair with identical syntax (e.g., 'The capital of France is' vs 'The capital of England is') and check whether transfer is again layer-dependent (early features ~64%, late Say-X ~36%). Additionally, ablate the labeled Say-X supernodes and measure the target logit: if removal does not lower the logit of the correct answer, the Say-X labels are not causally implicated in output promotion. A negative result on either test would undermine the central claims.

Watch

Extended reading notes

Core claim

The central claim is that a feature's role in a circuit is best read from its cross-prompt activation signature—how consistently and where it activates across systematically varied prompts—rather than from its raw activation vector or layer position. Using a rule-based classifier over these signatures, the paper labels features as semantic detectors, relationship binders, or output-promotion ('Say-X') nodes, and groups them into supernodes. On five factual-recall prompts in a 2-billion-parameter model equipped with a public cross-layer transcoder dictionary, these supernodes score higher on behavioral coherence than cosine or layer-adjacency baselines (2.3× peak-token consistency, 5.8× activ

Load-bearing premise

The load-bearing premise is that the attribution graph—computed under a replacement model that freezes attention patterns and layer norms—faithfully reflects the model's causal computation; if attention routing carries the effects the graph is supposed to capture, the supernode labels and the early-vs-late hierarchy could describe a distorted mechanism.

Editorial extensions

If this is right

  • If the method works as claimed, initial circuit analysis drops from roughly two hours per prompt to minutes, allowing broad surveys of circuit structure across many prompts and models.
  • A standardized Semantic/Relationship/Say-X vocabulary would make it possible to compare circuit motifs across tasks and to study whether the same relational features recur (e.g., 'capital-of' vs 'member-of').
  • The reported early-transfer/late-specialization split, if confirmed, implies that activation steering or weight edits aimed at transferring behavior should target early-layer backbone features, while late-layer output promoters are task-specific.
  • The subgraph compression (83% Completeness at ~54% Replacement) offers a legibility-vs-coverage trade-off that could be used as a default for human review, though the paper notes this applies to factual recall, not long reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The initial abstract's claim that 45,596 entity-swap interventions show predicted steering in every case is not supported by the body's reported experiments (five prompts, 39 features in the transfer study); the body itself labels its validation correlational. This discrepancy means the headline claim should be treated as a target, not as achieved evidence, until the large-scale data are released.
  • Because the attribution graphs come from replacement models that freeze attention patterns, the method may systematically miss features whose causal effect flows through attention changes; an attention-aware attribution variant could alter which features are labeled Say-X and could shift the layerwise hierarchy.
  • The rule thresholds (e.g., peak consistency ≥ 0.80, Say-X layer ≥ 7) were hand-tuned on the same circuits used for evaluation; a direct test would apply the pipeline to held-out prompt families and check whether default thresholds remain sensible without re-tuning.
  • A concrete way to stress-test the three-category taxonomy: generate probe prompts that include polysemous or multi-token entities and check whether the directionality rules ('is' → forward, 'of' → backward) still assign correct targets; the paper itself notes the 'of' directionality is currently implemented forward in code, a known issue.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces 'probe prompting,' a rule-based pipeline that groups features of an attribution graph into concept-aligned supernodes (Semantic, Relationship, Say-X) based on cross-prompt activation signatures (CPAS). The method is evaluated on five prompts in four factual domains using Gemma-2-2B with a public CLT dictionary. The body reports three descriptive findings: (C1) concept-aligned grouping yields higher behavioral coherence than cosine or layer-adjacency baselines on the Michael Jordan circuit; (C2) subgraphs preserve 83% Completeness while compressing graph size; (C3) an entity-swap study (Dallas→Oakland) suggests early-layer features transfer more robustly than late-layer Say-X features. The abstract, however, claims validation of 'predicted steering behavior' across 45,596 entity-swap interventions, which does not appear anywhere in the body. The manuscript is transparent about its small scale and correlational nature in Section 5.2 and Appendix A.4, but the abstract's causal claim is unsupported.

Significance. If the body's modest claims were confirmed with proper held-out evaluation, the method could meaningfully reduce the manual cost of circuit interpretation and provide a reusable, transparent taxonomy. The paper ships a public codebase, interactive demo, deterministic pipeline, and clearly documents many limitations. However, the abstract advertises a large-scale causal validation that the manuscript does not contain, and the evaluation of the main behavioral-coherence claim is circular because the same metrics used to define categories are used as success criteria. The likely contribution is a proof-of-concept tool, not the strong empirical result claimed in the abstract.

major comments (3)
  1. [Abstract vs. Section 4/5.2] The abstract states: 'Across four factual domains, on Gemma-2-2B with a public CLT dictionary and 45,596 entity-swap interventions, we find that the labeled supernodes have the predicted steering behavior in every one of them.' The body never reports this study. Section 4 lists five prompts, Table 3 reports 39 features and a single entity swap (Dallas→Oakland), and Section 5.2 says causal ablations and steering are 'planned for future work.' No arithmetic from the described setup yields 45,596. This is a direct internal contradiction between the advertised central claim and the reported evidence.
  2. [Section 3.2 and Table 2] The evaluation of C1 (behavioral coherence) is circular. The classification rules in Section 3.2 define Semantic (Dictionary) by peak consistency ≥0.80 and Say-X by functional dominance and functional consistency, while Table 2 compares concept-aligned grouping to baselines on 'Peak Token Consistency' and 'Sparsity Consistency' — the very quantities the rule-based classifier is designed to optimize. The paper itself concedes in Appendix A.4 that 'No held-out test set: All tuning was performed on circuits included in evaluation.' Without a held-out split or evaluation on metrics not used in the decision rules, the 2.3×/5.8× improvements in Table 2 are expected by construction, not evidence of superior grouping.
  3. [Table 3 and Section 4] The transfer hierarchy claim (C3) rests on a single entity-swap pair and an arbitrary threshold (cosine > 0.80) for defining transfer. The 10.1-layer difference is reported without significance tests or confidence intervals, and the authors acknowledge the small sample in the text. Given that the abstract elevates this to a claim about 'predicted steering behavior' across tens of thousands of interventions, the single-pair descriptive finding cannot support the advertised claim. Replication across multiple state-capital pairs and a proper test of the layer-difference hypothesis are needed before this can be called a 'backbone-and-specialization' result.
minor comments (5)
  1. [Appendix A.3] Typo: 'michael hordan plays' appears in the example-data directory listing; should be 'michael jordan plays.'
  2. [Appendix A.7] In the target-token mapping rules, 'Texas’scapital' is missing a space; also the possessive marker example uses a curly apostrophe inconsistently with the rest of the text.
  3. [Section 5.2] The sentence 'the current prototype remains unreliable in multilingual contexts without explicit cross-lingual adaptation' is repeated nearly verbatim in Section 3.2 and Appendix A.7; consider consolidating.
  4. [Table 2 caption] The caption says 'same pinned nodes' and then notes that Replacement/Completeness are near-invariant; it would be clearer to state explicitly that these metrics are not discriminative for the comparison, as the text does.
  5. [Section 4] The set of prompts is described as 'four factual domains' but the list includes five prompts; clarify whether the antonymy and anatomy pairs count as factual domains or as separate categories.

Circularity Check

3 steps flagged · score 6.0 of 10

Core behavioral-coherence and transfer findings are partly circular: group membership and transfer labels are defined by the same behavioral metrics used as evaluation, and thresholds were hand-tuned on the evaluated circuits.

  1. self definitional [§3.2 Decision Rules and Classification Logic; Table 2 (C1)]
    "… Semantic (Dictionary): … Peak consistency≥0.80 on a single token type; few distinct peaks (≤1). … Say-X: … Functional dominance (≥50%), high functional consistency (≥0.90), and layer≥7. … Concept-aligned grouping achieves 2.3× higher token consistency and 5.8× higher activation similarity than cosine clustering …"

    The category definitions are thresholds on the same signature statistics later used as the C1 evaluation: peak-token consistency, functional consistency, sparsity, and cross-prompt activation similarity. Table 2 then reports that concept-aligned supernodes have higher Peak Token Consistency and Activation Pattern Similarity than cosine/layer-adjacency baselines. Because the grouping rules were constructed to require high values of these exact quantities, the 2.3×/5.8× margins are partly guaranteed by construction rather than being an independent test of behavioral coherence.

  2. fitted input called prediction [§4, Table 3 (C3)]
    "Transfer measured by matching Dallas features to Oakland and computing activation overlap (threshold: cosine ¿ 0.80). Features with high overlap (¿0.80) are classified as “transferred”; low overlap (¡0.50) as “failed.” … Behavioral Validation: Activation Overlap (transferred) 1.000 (SD = 0.000); Peak Token Consistency 96%; Entity-Appropriate Activation 100% (Oakland features→‘Sacramento’)"

    A Dallas feature is classified as 'transferred' precisely when its activation overlap with Oakland exceeds 0.80, and the 'Behavioral Validation' row then reports Activation Overlap 1.000 (SD = 0.000) for the transferred set as evidence. Reporting a selection criterion's own quantity as validation is tautological. Since the overlap is computed from the same probe-prompt activations that generated the CPAS signatures used for grouping, the entity-swap result is not a held-out causal prediction; it restates the input used to assign the transfer label.

1 more flagged steps
  1. fitted input called prediction [§3.2; Appendix A.4 Known Limitations 1]
    "Thresholds were chosen via iterative refinement on held-out examples, balancing recall (capturing relevant features in each category) against precision (avoiding false groupings). … No held-out test set: All tuning was performed on circuits included in evaluation. Future work should use proper train/validation/test splits."

    The paper's own appendix corrects the §3.2 claim of held-out tuning: no held-out test set was used, and the decision thresholds were tuned on the same circuits that are later evaluated. Those thresholds determine the supernode groupings whose behavioral coherence and layer-hierarchy results are reported as findings. This is evaluation leakage of the fitted-input-called-prediction form: the fitted classification parameters are not independent of the data on which the paper's central descriptive claims are made.

full rationale

The two quantitative claims that would support the published abstract—45,596 entity-swap interventions with predicted steering behavior—are absent from the body: §4 reports five prompts, 39 transfer features, and a single Dallas→Oakland swap, while §2 and §5.2 say causal ablations and steering are future work. I treat that as an unsupported-reporting defect rather than circularity, because no derivation exists to compare with its inputs. The C1 and C3 evaluation, however, is partly circular by construction. The §3.2 decision rules classify features using thresholds on peak-token consistency, functional consistency, sparsity, and layer, and the CPAS signatures come from the same probe prompts used in Tables 2 and 3. Reporting higher peak-token consistency/activation similarity for groups formed with those thresholds is an internal consistency check, not independent confirmation. Table 3 labels a feature as transferred when activation overlap exceeds 0.80 and then reports an overlap of 1.000 for transferred features as validation, which restates the selection criterion. Appendix A.4 also admits that all thresholds were tuned on the evaluated circuits despite the §3.2 'held-out' wording. The self-citation to Prompt Rover is lineage rather than load-bearing, and the CLT replacement-model caveat is a validity limitation, not a circular step. The Neuronpedia Replacement/Completeness numbers and the qualitative feature activations are independent, but the central behavioral-coherence and transfer claims are substantially determined by the metrics used to define them, so a score of 6 is appropriate.

Assumptions & free parameters 11 free parameters · 6 assumptions · 4 invented entities

The central result rests on a chain of assumptions: CLT replacement-model faithfulness, the functional/semantic token dichotomy, the hand-tuned thresholds, and the adequacy of LLM-generated probes. Many thresholds are free parameters tuned on the evaluation circuits (Appendix A.4), and the novel entities (CPAS, supernodes, Say-X) lack external causal validation in the body despite the abstract's claim of 45,596 steering-validated interventions.

free parameters (11)
  • peak_consistency_threshold = >= 0.80
    Semantic (Dictionary) classification threshold; hand-tuned; Appendix A.4 says all tuning on evaluation circuits.
  • distinct_peaks_threshold = <= 1
    Limits Semantic Dictionary to single-token detectors; hand-tuned.
  • sayx_layer_threshold = >= 7
    Late-layer prior for output-promotion features; hand-tuned.
  • functional_vs_semantic_ratio = >= 50%
    Say-X requires functional-peak dominance; hand-tuned.
  • functional_confidence_threshold = >= 0.90
    High confidence required to map functional peak to target token; hand-tuned.
  • median_sparsity_threshold = < 0.45
    Relationship category requires diffuse activation; hand-tuned.
  • cross_prompt_stability_threshold = >= 60%
    Features must meet category criteria on at least 60% of probes; hand-tuned.
  • target_token_window = ±7 tokens
    Directionality mapping window for Say-X target identification; default configurable value.
  • cumulative_influence_tau = 0.60–0.80
    Node-selection threshold; chosen via UI to capture top 70-80% influence.
  • transfer_cosine_threshold = 0.80
    Entity-swap transfer classification threshold.
  • conflict_resolution_weights = 0.4/0.3/0.2/0.1
    Weights for peak consistency, category confidence, layer prior, sparsity in conflict resolution.
assumptions (6)
  • domain assumption CLT replacement models freezing attention patterns and layer norms faithfully linearize model computation and yield faithful attribution graphs.
    Invoked throughout; Section 4: 'Each circuit uses Neuronpedia's attribution graph generation with replacement model (freezing attention patterns, layer norms)'. If false, all supernode labels rest on a distorted graph.
  • ad hoc to paper Functional vs semantic token labeling and directionality rules correctly identify output-promotion (Say-X) target tokens.
    Section 3.1/Appendix A.7 define the vocabulary and directionality; the mapping is heuristic and even contains a known bug for 'of' directionality (A.7 implementation note).
  • ad hoc to paper Hand-tuned classification thresholds encode valid expert judgment and generalize beyond the tuning set.
    Section 3.2 says thresholds chosen via iterative refinement on held-out examples; Appendix A.4 admits no held-out test set and all tuning was on evaluation circuits.
  • domain assumption LLM-generated probe prompts (Claude-3.5-Sonnet, temperature 0.7) adequately elicit the target concepts.
    Section 4: probes generated via LLM and manually reviewed; quality of probes is assumed sufficient.
  • domain assumption Neuronpedia Replacement and Completeness metrics are valid measures of explanatory coverage.
    Section 3.1: metrics rely on attribution path-tracing in replacement models.
  • domain assumption CLT features are sufficiently monosemantic for semantic labels to be meaningful.
    Section 2: relies on SAE/CLT literature; the paper does not re-validate monosemy.
invented entities (4)
  • CPAS (Cross-Prompt Activation Signature)
    purpose: Summary of a feature's activation behavior across probe prompts, used to classify features into supernodes.
    Defined and evaluated within the same paper; no external benchmark validates CPAS as a feature descriptor.
  • Supernode
    purpose: Concept-aligned group of attribution-graph features.
    The central unit of the proposed method; its causal validity is asserted only by the paper's own descriptive metrics.
  • Say-X category
    purpose: Output-promotion features that peak on functional tokens and promote a specific target token.
    The abstract claims steering validation, but the body says steering/ablations are future work; no independent causal evidence is provided.
  • Semantic (Dictionary/Concept) and Relationship categories
    purpose: Feature taxonomy for semantic detectors and relation binders.
    Defined by threshold rules in Section 3.2; validated only via descriptive activation statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Attribution Graph Interpretation via Probe Prompting." pith.science (2026). https://pith.science/paper/ZXVNZJUB

@misc{pith2026251107002,
  author       = {Pith},
  title        = {Pith review of: Automated Attribution Graph Interpretation via Probe Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXVNZJUB}},
  note         = {Machine review of arXiv:2511.07002}
}
read the original abstract

Even though we know the precise computations that lead from a large language model (LLM) input to its output this computation remains very hard to interpret. One way to make it easier to understand this process is by creating a sparse computational graph that captures most of the model behavior with smallest number of computational nodes. Cross-layer transcoders (CLT) decompose the dense computations of the MLP but the resulting circuits still contain thousands of nodes even for short prompts. Existing automated interpretation methods label individual features from corpus activations, and it often happens that these labels are not validated by causal intervention. We introduce probe prompting, a transparent rule-based pipeline that groups the features of an attribution graph into concept-aligned supernodes from their responses on a small set of concept-targeted probe prompts, summarized as Cross-Prompt Activation Signatures (CPAS). Across four factual domains, on Gemma-2-2B with a public CLT dictionary and 45,596 entity-swap interventions, we find that the labeled supernodes have the predicted steering behavior in every one of them. Code, datasets, and an interactive demo are released anonymously as a reusable harness for calibrating supernode labels against causal interventions.

Figures

Figures reproduced from arXiv: 2511.07002 by the authors.

Figure 1
Figure 1. Probe prompting pipeline overview. Four-stage process: (1) Attribution Graph: Start with [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Activation characteristics distinguish feature categories across layers.(Left) Violin plot showing [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Rule-based feature typing for probe-prompted subgraphs. Decision rules used to assign each feature [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Stacked-prompts activation map and Cross-Prompt Activation Signature (CPAS) for Feature 20-clt [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Cross-prompt transfer validation (Dallas [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  2. [2]

    On the geometrical nature of insight, 07 2025

    Giuseppe Birardi. On the geometrical nature of insight, 07 2025. URL https://www.lesswrong. com/posts/nfGZtKzz8WzxF3MAs/on-the-geometrical-nature-of-insight . Con- ceptual essay introducing the Prompt Rover framing

    2025

  3. [3]

    Prompt rover

    Giuseppe Birardi. Prompt rover. https://github.com/peppinob-ol/prompt_rover, 2025. Software repository

    2025

  4. [4]

    Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and C...

    2023

  5. [5]

    The circuits research landscape: Results and perspectives, 2025

    Jack Lindsey, Emmanuel Ameisen, Neel Nanda, Stepan Shabalin, Mateusz Piotrowski, Tom McGrath, Michael Hanna, Owen Lewis, Curt Tigges, Jack Merullo, Connor Watts, Gonc ¸alo Paulo, Joshua Batson, Liv Gorton, Elana Simon, Max Loeffler, Callum McDougall, and Johnny Lin. The circuits research landscape: Results and perspectives, 2025. URL https://www.neuronped...

    2025

  6. [6]

    Neuronpedia: Open interpretability platform and apis, 2025

    Neuronpedia. Neuronpedia: Open interpretability platform and apis, 2025. URL https://www. neuronpedia.org/. Documentation: https://docs.neuronpedia.org/

    2025

  7. [7]

    Turner, Callum McDougall, Monte MacDiarmid, C

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosem...

    2024

  8. [8]

    Sparse crosscoders for cross-layer features and model understanding, 2024

    Transformer Circuits Team. Sparse crosscoders for cross-layer features and model understanding, 2024. URL https://transformer-circuits.pub/2024/crosscoders/. Transformer Circuits Note. 26

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.