Pith. sign in

REVIEW 3 major objections 4 minor 22 references

CLT-Forge is an open-source library that makes training and interpreting cross-layer transcoders — compact feature-sharing decompositions of transformer MLPs — scalable to about 1.5 million features on a moderate 8-GPU setup while matching

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:41 UTC pith:64NKPNZL

load-bearing objection Useful library, but the equations describe per-layer encoders — not a shared-feature CLT — so the parameter count and the central claim are both in doubt. the 3 major comments →

arxiv 2603.21014 v2 pith:64NKPNZL submitted 2026-03-22 cs.LG cs.CL

CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs

Florent Draye , Vedant Palit , Abir Harrasse , Tung-Yu Wu , Jiarui Liu , Punya Syon Pandey , Roderick Wu , Chih-Hao Hsu
show 3 more authors
Terry Jingchen Zhang Zhijing Jin Bernhard Sch\"olkopf
This is my paper
classification cs.LG cs.CL
keywords cross-layer transcodersmechanistic interpretabilityactivation cachingquantizationfeature shardingattribution graphsautomated interpretabilitydistributed training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces CLT-Forge, an open-source library that unifies the full workflow for cross-layer transcoders (CLTs): distributed training, activation caching, automated feature interpretation, attribution-graph computation, and visualization. The central claim is that by sharding features across GPUs and compressing cached activations with per-layer quantization plus entropy coding, a research group can train a CLT with roughly 1.5 million features on a 1-billion-parameter transformer using eight 80GB GPUs, then analyze it end-to-end. The authors report that on a small reference model the implementation matches prior work (explained variance around 0.8, replacement score around 0.8, graph completeness around 0.95) while cutting activation storage from about 20TB to about 4TB with only a stated 2–3% drop in reconstruction quality. If these figures hold, the main bottleneck for mechanistic interpretability shifts from infrastructure to research questions.

Core claim

The paper's central discovery is that feature-wise sharding combined with compressed activation caching makes cross-layer transcoder training feasible at scale in an open-source setting, and that the full pipeline — training, autointerp, attribution graphs, and interactive visualization — can be delivered as one integrated library. The reported evaluation shows parity with prior closed implementations on a small model and a concrete scaling configuration for a 1B-parameter model with 1.5M features across 8 GPUs. The activation store is compressed from roughly 20TB to 4TB using int8/int4 symmetric per-layer quantization with zstd coding, enabling storage and training of CLTs that would otherw

What carries the argument

The load-bearing machinery is the cross-layer transcoder (CLT) itself: features are shared across layers through layer-specific encoders and decoders, where a decoder at a target layer sums contributions from features extracted at that layer and all earlier layers, yielding compact attribution graphs. Training is made scalable by feature-wise sharding (each GPU owns a subset of the feature dimension) and by cached activations that are symmetrically quantized per layer to int8/int4 and then zstd-compressed, cutting storage 4–12× compared with float16. Attribution scores are computed as the product of a decoder vector, a frozen Jacobian of the model's forward pass, and an encoder vector, which

Load-bearing premise

The central claim depends on the assumption that int8/int4 quantization plus zstd compression preserves activation fidelity closely enough (stated as a 2–3% drop) that CLT training and downstream attribution are unaffected; the paper asserts this without showing the evaluation.

What would settle it

Run a head-to-head experiment on a fixed model and dataset, training one CLT on float16 cached activations and another on int4-compressed activations, and compare explained variance, replacement score, and graph completeness; if the gap exceeds the reported 2–3% or changes circuit-level conclusions, the scalability trade-off does not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Research groups can train CLTs with roughly 1.5 million features on 1-billion-parameter models using only eight 80GB GPUs, rather than proprietary infrastructure.
  • The integrated autointerp pipeline makes it practical to interpret millions of features in a single parallelized pass, storing results in a unified per-worker database.
  • Low-rank fine-tuning allows pretrained CLTs to be adapted to smaller datasets or new domains without full retraining, reducing compute and memory costs.
  • Attribution graphs, pruning, and interventions are computed within the same library, so end-to-end circuit analysis becomes reproducible and scriptable.
  • The activation-storage footprint for training a 1B model on 300M tokens drops from roughly 20TB to about 4TB, making large-scale CLT training feasible on modest cluster storage.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 2–3% fidelity impact of quantization is asserted but not demonstrated in the paper; the scalability claim rests on this trade-off holding for larger models and varied activation distributions.
  • Feature-wise sharding and compressed caching could transfer to sparse autoencoders and other dictionary-learning methods, broadening the library's reach beyond cross-layer transcoders.
  • If the fidelity loss grows with model scale or activation skew, the storage savings may come at a hidden cost to downstream attribution quality; a head-to-head float16 versus int4 ablation would settle this.
  • The visual interface's cluster-level interventions could enable a workflow for automatically discovering high-level circuits by clustering features and testing interventions at that coarser granularity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CLT-Forge, an open-source library for end-to-end training and analysis of Cross-Layer Transcoders (CLTs). The library combines feature-sharded distributed training, quantized and zstd-compressed activation caching, automated interpretability, attribution-graph computation via Circuit-Tracer, and a Dash-based visualization interface. The authors report that on GPT-2 their implementation matches prior work, with explained variance ~0.8, replacement score ~0.8, and graph completeness ~0.95, and they claim scalability to a LLaMA 1B CLT with ~1.5M features.

Significance. If the implementation is indeed correct, CLT-Forge would be a useful contribution to the mechanistic interpretability community by unifying several fragmented workflows into one open-source system. The MIT license, the integration of feature sharding, and the inclusion of autointerp and visualization are strengths, as is the honest discussion of limitations (frozen attention, unstable replacement-score optimization). However, the central architectural definition in Section 2.1 is internally inconsistent, and key empirical claims about quantization and scalability are not supported by any reported experiments. As written, the paper does not yet establish that the library implements the CLT architecture it claims.

major comments (3)
  1. [§2.1, Eq. (1) vs. Eq. (3) and Eq. (4)] The text states that W^ℓ_enc and b^ℓ_enc in Eq. (1) are 'layer-specific parameters.' This contradicts the defining property of a CLT, stated in the same section, that features are shared across layers while decoding is layer-specific. If each layer has its own encoder, the same feature index at different layers corresponds to different directions in activation space, so the compact attribution graphs and the match to prior CLT metrics are unsupported. The parameter count in Eq. (4) then also omits the L encoder matrices (approximately 3.2B parameters for the LLaMA 1B example) and the stated ≈27.4B is wrong. If a shared encoder is intended, Eq. (1) must drop the ℓ superscript and Eq. (4) should include the single shared encoder. The authors must clarify which architecture the released code implements and correct the formalism accordingly.
  2. [§3.1, quantization claim] The claim that int8/int4/int2 quantization plus zstd compression causes 'only a 2–3% drop in activation reconstruction quality and a similar impact on CLT reconstruction performance' is stated without any supporting data. This trade-off is load-bearing for the library's scalability story: if the actual degradation is larger, the reported EV and replacement scores in Section 4 cannot be replicated with the compressed cache. Provide a quantitative evaluation (e.g., activation reconstruction error and downstream CLT metrics as a function of quantization level, with the compression settings used) before making the scalability claim.
  3. [§4, evaluation section] The evaluation reports only point estimates (~0.8 EV, ~0.8 replacement, ~0.95 completeness) with no error bars, no comparison table against the cited prior works, and no quantitative results for the claimed advantage of feature-sharded training over DDP ('significantly improving memory efficiency' is not measured). For a systems/library paper, these claims need to be supported with concrete numbers, such as memory usage, throughput, and convergence curves for both sharding and DDP, and ideally a reproducibility statement about the number of runs. Without this, the central scalability claim remains unsubstantiated.
minor comments (4)
  1. [§2.1, Eq. (4)] The phrase 'with ed=d features' is confusing; it should read 'with d_features = e·d.' Also clarify in the text that the count L(L−1)/2 + L includes both diagonal (ℓ=ℓ′) and off-diagonal decoder pairs, consistent with Eq. (3).
  2. [§3.1] The storage numbers (20TB to 4TB) are given without specifying the quantization bits, compression settings, or whether both MLP inputs and outputs are stored. Please add these details so the claims are reproducible.
  3. [§3.5 and throughout] The library name is spelled inconsistently as 'Circuit Tracer' and 'Circuit-Tracer.' Use one form consistently.
  4. [Appendix B, Figure 3] The caption states 'final value 0.77 for L0=3 per layer,' while the main text reports ~0.8 EV at total L0=100. Clarify the relationship between per-layer L0 and total L0 and why the values differ.

Circularity Check

0 steps flagged

No circularity: evaluation is externally benchmarked; self-citations are not load-bearing.

full rationale

This is a systems/infrastructure paper. Its central claims (scalable training, feature sharding, compressed activation caching, autointerp, attribution graphs, visualization) are implementation contributions rather than predictions derived from fitted parameters. The only quantitative evaluation claims (EV ~0.8, replacement score ~0.8, graph completeness ~0.95) are explicitly benchmarked against external prior work (Hanna et al. 2025; Merullo et al. 2025; Lindsey et al. 2025a), so they are not forced by the library's own construction. The quantization fidelity assertion ('only a 2–3% drop') is unsupported by the shown evaluation, but it is an empirical claim, not a circular reduction. Self-citations (Draye et al. 2025, Harrasse et al. 2025) are companion studies referenced for detailed graph and multilingual analysis; they are not used to define or fit any quantity that the paper then re-predicts. The Eq. (1) 'layer-specific encoder' wording does conflict with the stated shared-feature CLT definition, and Eq. (4) omits encoder matrices in the parameter count; these are correctness/consistency concerns and could invalidate the architecture claim if the released code follows Eq. (1), but they are not a derivation that reduces to its own inputs. No circular step is exhibited.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claims rest on established but unproven interpretability assumptions (linear representation, CLT reconstruction, Jacobian-based attribution) plus an untested engineering assumption about quantization preserving activation fidelity. No new entities are invented; the free parameters in the config are standard hyperparameters, not fitted to make a scientific prediction.

axioms (4)
  • domain assumption Linear representation hypothesis: LLM semantic features correspond to directions in activation space (Park et al. 2023).
    Underlies the entire transcoder approach; cited but not proven.
  • domain assumption Cross-layer transcoder decoder form (Eq. 3) with layer-pair decoders can faithfully reconstruct MLP outputs.
    Adopted from Ameisen et al. 2025; the paper does not justify sufficiency.
  • domain assumption Attribution score (Eq. 5) with frozen/stop-gradient Jacobian approximates causal feature influence.
    Lifted from Circuit-Tracer/Anthropic; no proof of causal validity.
  • domain assumption Compressed activation caching (int8/int4 + zstd) preserves activation fidelity sufficient for CLT training.
    Section 3.1 claims 2-3% quality drop, but no experiment is shown.

pith-pipeline@v1.3.0-alltime-deepseek · 8946 in / 10285 out tokens · 96289 ms · 2026-08-02T17:41:44.584669+00:00 · methodology

0 comments
read the original abstract

Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge.

Figures

Figures reproduced from arXiv: 2603.21014 by Abir Harrasse, Bernhard Sch\"olkopf, Chih-Hao Hsu, Florent Draye, Jiarui Liu, Punya Syon Pandey, Roderick Wu, Terry Jingchen Zhang, Tung-yu Wu, Vedant Palit, Zhijing Jin.

Figure 1
Figure 1. Figure 1: Overview of the CLT-Forge framework. is similarly limited; while tools such as Neuronpe￾dia (Lin and Bloom) provide very useful interfaces, they remain difficult to extend and integrate into research workflows. In this work, we introduce the first unified library for CLTs that enables scalable training and post￾hoc interpretability analysis. Our contributions are: • Scalable training infrastructure: Sup￾po… view at source ↗
Figure 2
Figure 2. Figure 2: Example of a GPT-2 attribution-graph for the sentence ’The opposite of "large" is’ with replacement-score [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of training dynamics for a GPT-2 CLT. a) Sparsity per-layer level. b) L0 scheduler, linear [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 3 linked inside Pith

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others. 2025. https://transformer-circuits.pub/2025/attribution...

    2025

  2. [2]

    Anthropic. 2025. Circuits updates --- january 2025. https://transformer-circuits.pub/2025/january-update/index.html. Accessed: 2026-02-26

    2025

  3. [3]

    Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. 2024. Saelens. https://github.com/decoderesearch/SAELens. Accessed: 2026-02-27

    2024

  4. [4]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decomposing language models with d...

    2023

  5. [5]

    Florent Draye, Anson Lei, Ingmar Posner, and Bernhard Sch \"o lkopf. 2025. Sparse attention post-training for mechanistic interpretability. arXiv preprint arXiv:2512.05865

    Pith/arXiv arXiv 2025

  6. [6]

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. 2024. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375--24410

    2024

  7. [7]

    EleutherAI. 2024. Sparsify: Training sparse autoencoders and transcoders. https://github.com/EleutherAI/sparsify. Accessed: 2026-02-26

    2024

  8. [8]

    Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen. 2025. Circuit-tracer: A new library for finding feature circuits. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 239--249

    2025

  9. [9]

    Abir Harrasse, Florent Draye, Zhijing Jin, and Bernhard Sch \"o lkopf. 2025. Tracing multilingual representations in llms with cross-layer transcoders. arXiv preprint arXiv:2511.10840

    arXiv 2025

  10. [10]

    Harish Kamath, Emmanuel Ameisen, Isaac Kauvar, Rodrigo Luger, Wes Gurnee, Adam Pearce, Sam Zimmerman, Joshua Batson, Thomas Conerly, Chris Olah, and Jack Lindsey. 2025. https://transformer-circuits.pub/2025/attention-qk/index.html Tracing attention computation through feature interactions . Transformer Circuits Thread

    2025

  11. [11]

    Georg Lange. 2025. Cross-layer transcoder. https://github.com/Goreg12345/crosslayer-transcoder. GitHub repository

    2025

  12. [12]

    Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023

    Johnny Lin and Joseph Bloom. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL https://www. neuronpedia. org. Software available from neuronpedia. org

    2023

  13. [13]

    Jack Lindsey, Emmanuel Ameisen, Neel Nanda, Stepan Shabalin, Mateusz Piotrowski, Tom McGrath, Michael Hanna, Owen Lewis, Curt Tigges, Jack Merullo, Connor Watts, Gonçalo Paulo, Joshua Batson, Liv Gorton, Elana Simon, Max Loeffler, Callum McDougall, and Johnny Lin. 2025 a . https://neuronpedia.org/graph/info The circuits research landscape: Results and per...

    2025

  14. [14]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others. 2025 b . https://transformer-circuits.pub/2025/attribut...

    2025

  15. [15]

    Callum McDougall. 2024. SAE Visualizer . https://github.com/callummcdougall/sae_vis

    2024

  16. [16]

    Jack Merullo, Connor Watts, Max Loeffler, Liv Gorton, Elana Simon, Tom McGrath, and Owen Lewis. 2025. Replicating circuit tracing for a simple known mechanism. https://www.goodfire.ai/research/replicating-circuit-tracing-for-a-simple-mechanism

    2025

  17. [17]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658

    Pith/arXiv arXiv 2023

  18. [18]

    Gon c alo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. 2024. Automatically interpreting millions of features in large language models. arXiv preprint arXiv:2410.13928

    Pith/arXiv arXiv 2024

  19. [19]

    Curt Tigges. 2025. Cross-layer coding. h ttps://github.com/curt-tigges/crosslayer-coding

    2025

  20. [20]

    E. Tredal. 2025. Open cross-layer transcoder (openclt). https://github.com/etredal/openCLT. GitHub repository

    2025

  21. [21]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  22. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...