Pith. sign in

REVIEW 3 major objections 5 minor 22 references

A Procrustes rotation before joint sparse autoencoder training yields more cross-seed universal BERT features than post-hoc alignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 06:27 UTC pith:PS2NCA75

load-bearing objection Clean ablation shows pre-joint Procrustes beats post-hoc matching for cross-seed SAE features; useful engineering result, not a conceptual breakthrough. the 3 major comments →

arxiv 2607.08499 v1 pith:PS2NCA75 submitted 2026-07-09 cs.CL

Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoders

Bendeg\'uz V\'aradi , Zolt\'an Kmetty This is my paper
classification cs.CL
keywords sparse autoencoderscross-seed universalityProcrustes alignmentmechanistic interpretabilityBERTTop-K sparsityend-to-end SAE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Independently trained BERT models with different random seeds end up with misaligned internal feature spaces, so the same linguistic concept can sit in different latent dimensions and look unrelated. This paper claims that first computing a single orthogonal Procrustes rotation to line up the two seeds' activation spaces, then training one shared Top-K sparse autoencoder end-to-end with a light cross-seed penalty, recovers substantially more "universal" features (Pearson correlation at least 0.70 across seeds) than either independent autoencoders matched after the fact or joint training without the rotation. The result holds on five seed pairs and three classification datasets (sentiment, politeness, emotion). A short ablation study shows that high-universality features correspond to readable sociolinguistic patterns such as sentence-initial modal verbs in requests. If the claim holds, researchers can treat features that survive this pipeline as seed-independent rather than training artifacts.

Core claim

Conditioning joint end-to-end Top-K sparse autoencoder training on an orthogonal Procrustes alignment of the two seeds' activation spaces produces more cross-seed universal features (Pearson r ≥ 0.70) than post-hoc matching baselines on all three evaluated datasets, and the rotation helps only when it is applied before joint training rather than after separate training.

What carries the argument

Procrustes-conditioned joint SAE: an orthogonal rotation matrix W_align that minimises the Frobenius distance between the two seeds' hidden-state matrices is applied to one seed's activations before they enter a single shared Top-K sparse autoencoder trained with local reconstruction, downstream KL/MSE, cross-code, and auxiliary dead-feature losses.

Load-bearing premise

A single orthogonal rotation estimated from only 500 sequences is assumed to be a faithful enough map of the two seeds' native activation geometries that later high correlations reflect shared structure rather than the rotation itself.

What would settle it

On a held-out seed pair and dataset, replace the true Procrustes matrix with random orthogonal matrices or a mismatched (shuffled-token) matrix and check whether the count of r ≥ 0.70 features collapses to the unrotated baseline; if true Procrustes no longer dominates, the alignment claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder that first estimates an orthogonal Procrustes rotation between the activation spaces of two independently trained BERT seeds (Eq. 1) and then trains a single shared Top-K SAE under a multi-term objective combining KL/downstream consistency, local reconstruction, cross-seed sparse-code MSE, and an AuxK dead-feature revival loss (Eqs. 2–7). On five seed pairs and three classification datasets (SST-2, Stanford Politeness, TweetEval Emotion), the full pipeline is reported to yield higher top-10/top-100 Pearson correlations and more features with r ≥ 0.70 than index-matched independent SAEs, post-hoc rotation, and joint training without rotation (Table 1). A short counterfactual ablation on one politeness feature and permutation/NRE checks in the appendices are offered as supporting evidence of interpretability and alignment quality.

Significance. Cross-seed feature misalignment is a genuine obstacle for mechanistic interpretability of independently trained models. The paper’s clean ablation ladder (independent matched → post-hoc Procrustes → joint without rotation → rotation-only → full pipeline) and the five-pair, three-dataset design give a concrete empirical ranking that is useful even if the absolute universality numbers remain threshold-sensitive. The rotation-only ablation and Appendix D’s 30-rotation nulls already show that Procrustes conditioning of joint training, rather than L_cross alone, drives most of the gain. If the ranking holds under broader model families and more rigorous qualitative validation, the method would be a lightweight, practical addition to the SAE toolkit for cross-seed and model-diffing studies.

major comments (3)
  1. §3.2 / Eq. (6) and Table 1: L_cross = MSE(c_A, c_B) directly optimizes the same sparse codes whose Pearson correlation is later used as the success metric. The authors correctly flag the Goodhart risk and report a rotation-only ablation (λ_cross = 0) that still beats joint-no-rotation; however, the absolute universal counts remain sensitive to the arbitrary r ≥ 0.70 cutoff and to dead-feature fractions often exceeding 70 %. A sensitivity analysis over a range of thresholds (e.g. 0.5–0.9) and a clearer statement of how much of the headline gain survives when λ_cross = 0 on every dataset would make the central claim robust rather than partially co-defined by the training objective.
  2. §5 / Table 1 and §6: The evaluation is confined to bert-base-uncased layer 10 and three English classification datasets. On SST-2 the absolute number of universal features remains small (78 ± 54) and the dead rate is ~87 %. The limitations section acknowledges the single-family restriction, but the central claim of a “consistent method for extracting linguistic information from black-box models” currently rests on a narrow regime; at least one additional architecture or layer, or an explicit comparison against Feature-Aligned SAEs [8], is needed before the ranking can be treated as general.
  3. §5.1 / Table 2: The qualitative validation consists of a single feature (Feature 389) with five hand-crafted counterfactuals. While the ablation is suggestive, it does not yet establish that high-universality features systematically encode interpretable sociolinguistic patterns. Expanding the analysis to a small random sample of high-r features (or an automated LLM-assisted labeling protocol with inter-annotator checks) would better support the interpretability claim that accompanies the quantitative ranking.
minor comments (5)
  1. Figure 1 caption and surrounding text refer to “Figure 3” for the PCA projection; numbering should be made consistent throughout.
  2. Appendix C reports mean-pooling results on only two seed pairs while the main table uses five; the discrepancy should be stated explicitly so readers do not over-interpret the larger universal counts.
  3. Eq. (5) writes MSEM(ĥ_B, W_align h_B); a short note clarifying that the decoder output is subsequently unrotated before downstream injection would remove ambiguity.
  4. The phrase “industry standard hyper-parameters” in §3.2 is vague; pointing readers to Table A1 earlier would help.
  5. Typographical inconsistencies appear in author names (e.g., “Bendeg´ uz”, “E¨ otv¨ os”) and in the arXiv date line; these should be cleaned for the camera-ready version.

Circularity Check

1 steps flagged

L_cross = MSE(c_A, c_B) directly optimizes sparse-code similarity later scored as Pearson-r universality (r≥0.70), a partial Goodhart loop the authors flag and partially ablate via λ_cross=0.

specific steps
  1. other [§3.2 Eqs. (2),(6); Goodhart paragraph; §4 universality definition; Table 1 ablations]
    "Lcross = MSEM(cA, cB) (6) ... It is worth noting that L cross may seem to violate Goodhart’s law by optimizing for cross-seed SAE sparse codes while the feature correlations are measured as a metric of success. ... We define ”universality” if a feature active in both models has at least r≥0.70. ... Joint — rotation only ... Joint — full⋆"

    The explicit training term L_cross minimizes squared discrepancy between the very sparse codes whose Pearson correlation is later thresholded at 0.70 to count “universal” features. Including λ_cross=1 therefore tends, by construction, to raise the reported universality counts relative to λ_cross=0. The paper mitigates this by showing the rotation-only condition already outperforms baselines, so the circularity is partial rather than total.

full rationale

This is an empirical methods paper proposing a joint SAE pipeline and ranking it against post-hoc matched baselines on three datasets; it does not claim a first-principles derivation or uniqueness theorem. The sole circularity risk is that the training objective includes L_cross = MSE(c_A, c_B) while the headline success metric is the fraction of features with Pearson r(c_A, c_B) ≥ 0.70. The authors themselves note the Goodhart concern and neutralize most of it by reporting the rotation-only ablation (λ_cross=0), which still beats joint-no-rotation and the post-hoc baselines on every dataset, plus a 30-rotation permutation null (Appendix D) that isolates Procrustes. No self-citation is load-bearing, no uniqueness is imported from the authors, and no fitted parameter is renamed a prediction. Residual sensitivity of absolute universal counts to the arbitrary 0.70 cutoff and high dead-feature rates remains, but does not collapse the central ranking by construction. Hence a modest score of 3 rather than 0 or 6+.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 1 invented entities

The central claim rests on standard non-convexity of dictionary learning, the classical Orthogonal Procrustes solution, and a collection of previously published SAE training tricks, plus a handful of free hyper-parameters and an ad-hoc universality threshold. No new physical or mathematical entities beyond the named pipeline itself are introduced; independent evidence for the pipeline is purely internal empirical comparison.

free parameters (7)
  • λ_DS (downstream loss weight) = 1.5
    Set to 1.5 by the authors; controls relative strength of layer-wise MSE.
  • λ_cross (cross-seed sparse-code loss weight) = 1.0
    Selected by grid search over {0.1, 0.5, 1.0, 2.0}; directly influences the measured correlation metric.
  • Top-K sparsity level k = 32
    Hard sparsity constraint; chosen as 32 following prior Top-K SAE practice.
  • dictionary size n_dict = 6144
    Overcomplete dictionary dimension; set to 6144.
  • Procrustes alignment sample size = 500
    Number of sequences used to estimate W_align; validated post-hoc in Appendix B but still a free design choice.
  • universality threshold = 0.70
    Pearson r ≥ 0.70 used to count “universal” features; ad-hoc cutoff that defines the headline metric.
  • λ_aux (auxiliary dead-feature loss weight) = 1/32
    Taken from Gao et al. as 1/32; scales the residual-reconstruction term for dead units.
axioms (4)
  • domain assumption Dictionary learning is non-convex, so independently trained networks learn misaligned feature spaces even under identical architecture and data.
    Stated as the fundamental challenge in the abstract and Introduction; underpins the entire motivation for alignment.
  • standard math The Orthogonal Procrustes problem yields the unique optimal rotation minimizing the Frobenius distance between two centered activation matrices.
    Equation (1) and the SVD construction W_align = V U^T; classical linear-algebra result.
  • domain assumption Top-K activation structurally enforces sparsity and eliminates the shrinkage bias of L1 penalties.
    Adopted from Gao et al. [4]; used to justify omitting an explicit sparsity regularizer.
  • ad hoc to paper A feature active in both models with per-token Pearson r ≥ 0.70 counts as “universal.”
    Definition introduced in Evaluation §4 and used for all headline counts; no external justification for the precise cutoff.
invented entities (1)
  • Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder no independent evidence
    purpose: Single shared SAE trained on Procrustes-aligned activations of two seeds, combining Top-K, end-to-end, cross, and auxiliary losses to maximize cross-seed feature correlation.
    The named architecture is the paper’s central construct; all evidence for its superiority is internal to the reported experiments.

pith-pipeline@v1.1.0-grok45 · 17725 in / 3199 out tokens · 46328 ms · 2026-07-10T06:27:12.803625+00:00 · methodology

0 comments
read the original abstract

We present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) for extracting cross-seed universal features from independently trained BERT models. Cross-seed feature universality is a fundamental challenge in mechanistic interpretability: because dictionary learning is non-convex, independently trained networks learn misaligned feature spaces, so apparently identical features may differ by random initialization. We address this by computing an orthogonal Procrustes rotation between seeds' activation spaces before joint SAE training, combining Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss based on previous SAE literature. Evaluating on five independent seed pairs (ten BERT models) across three benchmark datasets (SST-2, Stanford Politeness, TweetEval Emotion), our full pipeline produces more universal features (Pearson r $\geq$ 0.70 across seeds) than post-hoc alignment baselines on all three datasets. A minimal qualitative analysis confirms that high-universality features encode interpretable sociolinguistic patterns.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 8 internal anchors

  1. [1]

    arXiv preprint arXiv:2503.05613 (2025)

    Shu, D., Wu, X., Zhao, H., Rai, D., Yao, Z., Liu, N., Du, M.: A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613 (2025)

    work page arXiv 2025

  2. [2]

    Sparse Autoencoders Find Highly Interpretable Features in Language Models

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., Sharkey, L.: Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Transformer Circuits Thread 2(5), 6 (2023)

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A.,et al.: Towards monosemanticity: Decom- posing language models with dictionary learning. Transformer Circuits Thread 2(5), 6 (2023)

    work page 2023

  4. [4]

    Scaling and evaluating sparse autoencoders

    Gao, L., Tour, T.D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., Wu, J.: Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Improving Dictionary Learning with Gated Sparse Autoencoders

    Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kram´ ar, J., Shah, R., Nanda, N.: Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Galichin, A.V., Dontsov, A., Druzhinina, P., Razzhigaev, A., Rogov, O., Tutubalina, E., Oseledets, I.: I have covered all the bases here: Interpreting rea- soning features in large language models via sparse autoencoders. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, pp. 30771–30779 (2026)

    work page 2026

  7. [7]

    Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning

    Braun, D., Taylor, J., Goldowsky-Dill, N., Sharkey, L.: Identifying function- ally important features with end-to-end sparse dictionary learning, 2024. URL https://arxiv.org/abs/2405.12241 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders

    Marks, L., Paren, A., Krueger, D., Barez, F.: Enhancing neural net- work interpretability with feature-aligned sparse autoencoders. arXiv preprint arXiv:2411.01220 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Anonymous: ORTHOGONAL SAE: FEATURE DISENTANGLEMENT THROUGH COMPETITION-AWARE ORTHOGONALITY CONSTRAINTS (2025)

    work page 2025

  10. [10]

    In: Forty-second International Conference on Machine Learning (2025)

    Thasarathan, H., Forsyth, J., Fel, T., Kowal, M., Derpanis, K.G.: Universal sparse autoencoders: Interpretable cross-model concept alignment. In: Forty-second International Conference on Machine Learning (2025)

    work page 2025

  11. [11]

    InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantly

    Enouen, J., Liu, Y.: Instashap: Interpretable additive models explain shapley values instantly. arXiv preprint arXiv:2502.14177 (2025) 16

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Transformer Circuits Thread, 3982–3992 (2024)

    Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., Olah, C.: Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 3982–3992 (2024)

    work page 2024

  13. [13]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Claflin, O.: Feature integration spaces: Joint training reveals dual encoding in neu- ral network representations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, pp. 35384–35392 (2026)

    work page 2026

  14. [14]

    Advances in Neural Information Processing Systems34, 5656–5668 (2021)

    Csisz´ arik, A., K˝ or¨ osi-Szab´ o, P., Matszangosz, A., Papp, G., Varga, D.: Similarity and matching of neural network representations. Advances in Neural Information Processing Systems34, 5656–5668 (2021)

    work page 2021

  15. [15]

    Lan, M., Torr, P., Meek, A., Krueger, D., Barez, F.: Sparse autoencoders reveal universal feature spaces across large language models (2024)

    work page 2024

  16. [16]

    Atlas-Alignment: Making Interpretability Transferable Across Language Models

    Puri, B., Berend, J., Lapuschkin, S., Samek, W.: Atlas-alignment: Making inter- pretability transferable across language models. arXiv preprint arXiv:2510.27413 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp

    Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., Potts, C.: Recursive deep models for semantic compositionality over a sentiment tree- bank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642. Association for Computational Linguistics, Seattle, Washington, USA (2013).https://www.a...

    work page 2013

  18. [18]

    In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Danescu-Niculescu-Mizil, C., Sudhof, M., Jurafsky, D., Leskovec, J., Potts, C.: A computational approach to politeness with application to social factors. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 250–259 (2013)

    work page 2013

  19. [19]

    In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp

    Mohammad, S., Bravo-Marquez, F., Salameh, M., Kiritchenko, S.: Semeval-2018 task 1: Affect in tweets. In: Proceedings of the 12th International Workshop on Semantic Evaluation, pp. 1–17 (2018)

    work page 2018

  20. [20]

    Discovering Latent Concepts Learned in BERT

    Dalvi, F., Khan, A.R., Alam, F., Durrani, N., Xu, J., Sajjad, H.: Discovering latent concepts learned in bert. arXiv preprint arXiv:2205.07237 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    R., Millman , K

    Harris, C.R., Millman, K.J., Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J., Kern, R., Picus, M., Hoyer, S., Kerkwijk, M.H., Brett, M., Haldane, A., R´ ıo, J.F., Wiebe, M., Peterson, P., G´ erard-Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., Oliphant, T.E.: Array program...

    work page doi:10.1038/s41586-020-2649-2 2020

  22. [22]

    Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017) 17

    work page 2017