Scaling Interpretable Transformers with Parity Bottleneck Layers

Andrew Mack; Kraig Yuheng Tou; Lauren Greenspan; Mark Henry; Zhengxun Wu

REVIEW 5 major objections 5 minor 15 references

Scaling Interpretable Transformers with Parity Bottleneck Layers

T0 review · 5 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-layer interpretable sparse bottlenecks can scale: a parity-hash dictionary yields native features that match or beat post-hoc SAEs at GPT-2 scale.

desk verdict A genuinely new architecture with a correct coherence theorem, but the interpretability-by-construction claim outruns the evidence; worth refereeing. read the letter →

arxiv 2607.20652 v1 pith:LNYHFI4V submitted 2026-07-22 cs.LG cs.AI

Andrew Mack , Kraig Yuheng Tou , Mark Henry , Zhengxun Wu , Lauren Greenspan This is my paper · ORCID

classification cs.LGcs.AI

keywords interpretabilitybyconstructionsparsebottleneckparityhashingbalancedbinarycodesincoherentdictionariesautoencodersfeaturesteeringcausalinterventions

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that 'interpretable by construction' is practical at GPT-2 scale. Its central move is to replace learned over-complete dictionaries—the expensive part of per-layer sparse bottlenecks—with a fixed, parameter-free parity-hash dictionary whose feature directions are computed on-chip from integer indices. A binary error-correcting code supplies a deterministic worst-case bound on pairwise feature overlap, so the dictionary is nearly orthogonal by design. Trained at 203M and 1.3B parameters, the resulting ParityTransformers match dense baselines on language modeling, match or exceed post-hoc sparse autoencoders on sparse probing, and outperform them on feature absorption, steering effectiveness, and fine-grained causal interventions. The disclosed cost is a 6.7–9.4x token-budget overhead and a 1.22–2.06x throughput overhead.

What carries the argument

The Deep Parity Bottleneck (DPB): a hierarchical sparse bottleneck whose decoder defines feature directions as parity-hashed sign vectors, computed on-chip via bitwise AND, popcount, and parity from a small seed matrix that fits in registers. The encoder is a multi-level beam search: at each level it scores only the children of active parents via a universal GEMM and keeps the top-K per level, making the search cost logarithmic in dictionary size. The seed matrix is chosen so the codebook is a binary linear code whose nonzero codewords are nearly balanced, which yields the deterministic coherence bound mu <= (1+2*sqrt(d))/d. The first d features are overridden to standard basis vectors so le

What would settle it

At small scale, replace the hierarchical beam-search encoder with exhaustive top-K scoring over the full dictionary and compare the selected features and causal-edit success; if they differ materially, the claim that the beam search faithfully approximates the optimal sparse code is false.

Watch

Extended reading notes

Core claim

The central claim is that a per-layer sparse bottleneck can be both memory-cheap and interpretable if the dictionary is algebraic rather than learned. Each feature direction is a ±1/sqrt(d) sign vector generated on the fly from its index via parity of selected index bits; the seed matrix is chosen so the dictionary is a binary linear code with balanced codewords, giving pairwise inner products at most (1+2*sqrt(d))/d—about 0.06 for d=1024 and 0.04 for d=2048. Because a hierarchical beam-search encoder selects only the top-K surviving features at each MLP input, any computation the MLP performs depends only on features in the sparse code. Empirically, ParityTransformers match dense baselines

Load-bearing premise

The argument rests on the untested hypothesis that any approximately orthogonal over-complete basis, when used as a sparse bottleneck, induces interpretable features—and on the related assumption that the hierarchical beam search approximates the optimal sparse code faithfully; the paper's own evidence is mixed, with worse auto-interpretability scores than post-hoc SAEs.

Editorial extensions

If this is right

Per-layer interpretable bottlenecks are feasible at GPT-2 scale at a disclosed cost: 6.7–9.4x more training tokens and 1.22–2.06x lower throughput than dense baselines.
Model features are native to the forward pass: the MLP sees only the top-K surviving dictionary features, so there is no hidden channel for unrepresented structure to influence MLP computation.
On the tested metrics, DPB features match post-hoc SAEs on sparse probing (top-1 0.802 vs 0.786) and outperform them on feature absorption, steering effectiveness, and fine-grained causal interventions.
The dictionary's coherence is guaranteed deterministically, not probabilistically: any pair of distinct features has inner product at most about 0.06 for d=1024 and 0.04 for d=2048.
Parity bottleneck features can support sparse retrieval indices directly, matching post-hoc SAE features for poisoned-document retrieval while avoiding the cost of training per-layer SAEs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the coherence guarantee, rather than the specific code, is what matters, the same on-chip hashing trick could be replayed with other algebraic dictionaries to trade coherence against index-arithmetic cost or to support wider bottlenecks than 2^r features.
The register-level dictionary suggests a path to activation-sparse inference: if feature generation never touches HBM, inference FLOPs could scale with the number of active features, potentially making very wide sparse layers cheaper than dense ones at deployment time.
The hierarchical beam search is the least-tested component; a natural next experiment is exhaustive top-K scoring at small scale to measure how much pruning changes the selected features and downstream interpretability.
If interpretability-by-design matures, the post-hoc SAE pipeline—training a separate dictionary per layer—could be replaced by reading out features the model already computes, which would change how model auditing and data attribution are done in practice.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this paper actually builds something new: a GPT-2-scale transformer with fixed, algebraically generated sparse bottlenecks (Kloosterman-code hashed directions) that are computed on-chip instead of stored, and it trains 200M and 1.3B models with honest disclosure of the cost. Second, the main interpretability claim—that the DPB features are naturally the ones the model uses—is true by construction but not the same as saying they are interpretable, and the paper's own AutoInterp scores undercut that. The central hypothesis in Section 2 is asserted, not established.

What is good: Theorem A.1 is a correct coherence bound for the dictionary, with the three label-pair cases handled cleanly and the Weil/Carlitz-Uchiyama/Lachaud-Wolfmann bound cited properly. The engineering is real: the hierarchical beam search with per-parent budgets makes this tractable, and the 6.7-9.4x token tax / 1.22-2.06x throughput overhead are stated without spin. The feature absorption result (0.01 vs 0.807) is striking if it holds up. The poisoned-data retrieval study is a legitimate application and the 5-run error bars there are welcome.

The soft spots are real. The stress-test issue is load-bearing: Theorem A.1 bounds the coherence of the full dictionary, but the features that actually reach the MLP are chosen by a greedy tree-constrained beam search. The paper never analyzes whether that search produces a faithful sparse code. If it prunes useful features, 'interpretable by construction' weakens to 'interpretable when the basis and the beam search cooperate.' The AutoInterp gap (0.796 vs 0.859-0.891) is consistent with the beam search discarding semantics.

The empirical comparisons are also confounded: SAEs are trained on a different base model, in-loop DPB edits are not the same protocol as reconstruction edits for SAEs, key figures have no error bars, and the abstract's '8-14x' mixes the 205M and 1.3B scales. The dense baseline token budget is inconsistently reported (3.0B vs 6.82B in Table 5) and was chosen by extrapolation to land at the target loss. No code or data is released, which matters for a paper whose headline is an empirical comparison. These are fixable with a controlled re-evaluation.

Who this is for: people working on interpretability-by-design, sparse bottlenecks, or faithfulness of SAEs. It deserves a real peer review—the construction and theorem are worth referee time even if the current claims need tightening. I'd send it to review, but the decision should be conditional on addressing the beam-search faithfulness gap and the confounded baselines.

Referee Report

5 major / 5 minor

Summary. The paper introduces the ParityTransformer, a GPT-2-scale architecture in which every MLP input passes through a Deep Parity Bottleneck (DPB). The DPB replaces a learned over-complete dictionary with a fixed, parameter-free algebraic dictionary: feature directions are generated on-chip from feature indices via a parity hash based on a Kloosterman code, with a deterministic coherence guarantee (Theorem A.1). A hierarchical beam-search encoder selects a small number of active features per layer. The authors train 200M and 1.3B models, report language-modeling capability and an 'interpretability tax' (6.7-9.4x training tokens, 1.22-2.06x throughput), and compare DPB features against post-hoc SAEs on causal interventions, steering, SAEBench, and poisoned-document retrieval. The central claim is that DPB features are native to the model's forward pass by construction, thereby addressing SAE-faithfulness questions.

Significance. If the full set of claims held, this would be a substantial engineering and interpretability contribution: per-layer wide sparse bottlenecks at GPT-2 scale, a deterministic incoherence guarantee, elimination of HBM dictionary storage, and a concrete attack on the SAE-faithfulness problem. The algebraic core is genuinely strong: Theorem A.1 is correctly proved, with all three label-pair cases handled, the Weil/Carlitz-Uchiyama/Lachaud-Wolfmann bound properly cited, and the basis-override pairs verified separately. The paper is also unusually transparent about training budgets, throughput overheads, and the limitations of individual metrics. However, the interpretability-by-construction program rests on an unproven hypothesis in Section 2 and on a heuristic hierarchical encoder whose faithfulness is not analyzed. Moreover, several headline empirical comparisons (causal interventions, steering, capability parity) are confounded by unmatched protocols or baselines. The result is a paper with a sound theoretical lemma but with the central interpretability and empirical claims only partially supported.

major comments (5)

[§2.2, 'Unfolded Features'] The paper states that 'for downstream interpretability tasks, we unfold each feature index into two features — one for positive and one for negative activations — effectively doubling the dictionary size.' If these unfolded features enter the same geometric dictionary used by the decoder, then every pair (φ_i, −φ_i) has inner product −1, so dictionary coherence becomes μ=1 and Theorem A.1 no longer applies. This is not cosmetic: the deterministic incoherence guarantee is a central advertised property, and the main evaluations appear to use the unfolded representation. Please clarify whether unfolding is only a labeling convention for analysis or a true dictionary expansion; if the latter, the guarantee and all downstream claims must be re-evaluated.
[§2.2 / Table 6] Theorem A.1 bounds pairwise coherence of the full Kloosterman dictionary, but the active features are chosen by the hierarchical beam search, which scores only children of active parents (e.g., 16×256 candidates at level 1 out of 32,768 features, Table 6) and keeps the top Kℓ. The theorem is silent on whether this search approximates the optimal sparse code, and pruned branches are unreachable by construction. The Section 2 hypothesis that any approximately orthogonal over-complete basis induces interpretable features is therefore confounded with the tree-structured encoder. The AutoInterp deficit in Table 4 (0.796 vs 0.859–0.891) is consistent with the search discarding useful features. A direct test — comparing beam-search codes with exhaustive sparse codes on the same dictionary, or with a random basis under the same search — would establish whether the advertised guarantees apply to
[Appendix C.2 / Figure 2] The causal-intervention comparison is not layer-matched. The protocol says the evaluation is restricted to the most influential MLP layer, but for IOI the DPB layer is selected by attention-patching effect (layer 11, attention e=0.42) even though DPB edits MLP features. For gendered pronouns the DPB evaluation layer has MLP effect e=0.15, while the dense baseline layer has e=0.94. The two methods are also patched differently (in-bottleneck edits with norm preservation vs SAE decode-and-patch into the residual stream). This makes it difficult to attribute the edit-success differences in Figure 2 to feature quality. Please report the MLP mediation effect for every selected layer and rerun with a common selection rule, or provide an ablation controlling for the patch target.
[§3.3 / Appendix D] The steering comparison uses single-layer steering for SAEs and multi-layer steering for DPB, with different α ranges (2–10 vs 20–75), chosen because different strategies 'provided better results' for each approach. The Pareto frontier in Figure 3a therefore does not isolate feature quality; it compares two different intervention designs. A control with the same steering mechanism (e.g., single-layer steering for both, or multi-layer SAE steering) is needed before claiming that DPB features give superior steering effectiveness.
[§3.1 / Tables 1–2] The 'matches or outperforms' capability claim is computed against dense baselines that were stopped as soon as they came within ~1% of the PT validation loss (GPT-Large at 6.82B tokens; GPT-Small at 2.13B) or trained to a fixed 3B budget (GPT-Large-cold). This is appropriate for measuring the token-budget overhead, but it does not support a capability-parity claim: a dense baseline trained on the full 20B tokens may do better. Please either train full-token-budget dense baselines or explicitly restrict the claim to 'at matched validation loss' and remove 'imposes no obvious ceiling in capabilities.'

minor comments (5)

[Appendix B.1] The text states a poisoning rate of 'approximately 0.078%', but 100/(100+12,800)=0.00775≈0.78%; Figure 4's label 'Poison frac. (0.0078)' is correct. Please fix the percentage in the text.
[§2.2] The heading 'Auxillary losses' contains a typo ('Auxiliary'), and equation (7) uses S^{(t)} without a nearby definition of the active set at token t.
[Table 5] The column 'wf' is not defined in the caption; the text later explains it is the warmdown fraction, but the table should be self-contained.
[§3.1, footnote] The footnote about the MLP 'ansatz' and the discarded DPB-before-attention experiments contains no quantitative results, even though it is cited to justify a major architectural choice. Add a short description of the failed attention experiments or remove the claim.
[Table 4] SCR and TPP are reported despite the text noting that SAEBench does not recommend these metrics below 2B parameters. The table should mark those rows as not recommended, or the text should explain why they are included beyond completeness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the mathematical guarantee is an external coding-theoretic bound and the empirical claims are benchmark comparisons, not fitted-then-predicted quantities.

full rationale

The paper's central mathematical result, Theorem A.1, is a standalone coding-theoretic coherence bound for the Kloosterman-code dictionary, proved in Appendix A using external results (Weil; Carlitz–Uchiyama; Lachaud–Wolfmann). It is not fitted from the model and does not depend on any downstream claim. The empirical sections compare ParityTransformers against independently trained dense baselines and post-hoc SAEs on SAEBench, steering, causal interventions, and poisoned-document retrieval; these are external benchmarks, not parameters fitted to a subset of the data and then re-reported as predictions. The phrase 'features are native to the model's forwards pass by construction' is an explicitly constructive property of the architecture (the DPB is the sole input to each MLP), not a derived empirical prediction, and the paper itself labels it 'by construction' and separates it from the empirical question of alignment with conceptual features. The unproven hypothesis about approximately orthogonal bases and the lack of analysis of the beam-search encoder are scientific gaps, not circular reductions. There are no load-bearing self-citations: the authors do not rely on their own prior work as justification for the central claims, and the cited coding-theory literature is external and checkable. Therefore no step in the derivation chain reduces, by definition or by fit, to its own inputs.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The new mathematical content (coherence theorem) is genuine and parameter-free, resting on external Weil/Carlitz-Uchiyama/Lachaud-Wolfmann results. The non-mathematical load-bearing premises are: the interpretability hypothesis (section 2), beam-search encoder fidelity (section 2.2), and the MLP-ansatz choice (section 3.1 footnote). Hand-chosen design quantities: K_l, Delta_l, auxiliary-loss weights, EMA details, n_min, the dense-baseline token budget, and steering alpha ranges. No new particles, forces, dimensions, or other entities are introduced; the 'computational features' taxonomy is a framing device, not a postulated entity.

free parameters (7)

per-level active feature counts K_0, K_1, K_2 = 16, 32, 64
Table 6; hand-chosen sparsity budget; controls the number of features the MLP ever sees and thus every downstream interpretability metric.
per-parent child budget Delta_l = 256
Table 6; beam width of the hierarchical encoder; hand-chosen; features in pruned branches can never be selected.
auxiliary loss weights = 0.05 (variance), 0.1 (reconstruction)
Appendix G; applied only to PT-Large-2L-Aux, the model used in the headline interpretability comparisons.
EMA score-standardization window / token subsample = not specified; 64-token subsamples
Section 2.2; the EMA machinery that eliminated the 90% dead-feature collapse is a free design choice whose smoothing window is unreported.
variance-loss activity threshold n_min = not specified
Section 2.2, Eq. 7; F is defined as features active on at least n_min tokens; the threshold sets which features receive variance pressure.
dense-baseline token budget (GPT-Large-cold) = 3.0B
Appendix G; chosen by extrapolation from LR sweeps to land at the target CE loss; drives the headline 6.7x data-efficiency ratio.
steering hyperparameters (alpha, top-k) = alpha in [2,10] PT; alpha in [20,75] SAE
Appendix D; different steering mechanisms and alpha-ranges per method make the Pareto comparison method-specific rather than matched.

assumptions (6)

ad hoc to paper Any approximately orthogonal over-complete basis, when used as a sparse bottleneck, will induce interpretable features.
Section 2 'Design principles': the load-bearing interpretability hypothesis. The paper's own evidence is mixed: sparse probing 0.802 beats SAEs, but AutoInterp 0.796 trails 0.859-0.891.
standard math Weil bound for binary Kloosterman sums: |K(a)| <= 2*sqrt(d).
Theorem A.3, cited to Weil 1948, Carlitz-Uchiyama 1957, Lachaud-Wolfmann 1987; external and non-circular.
standard math Trace-map properties (Lemma A.2): Tr(y) in {0,1}, linearity, non-vanishing.
Proved in Appendix A.2 from finite-field axioms; standard background.
ad hoc to paper The MLP ansatz: implicit emulation of a very wide, sparsely-activating, sparsely-connected computational graph.
Section 3.1 footnote; justifies inserting DPB only before MLPs; the paper reports attention-bottleneck variants reached similar loss but were dropped for worse interpretability.
domain assumption Beam-search encoder fidelity: greedy top-down top-K selection approximates the optimal sparse code.
Section 2.2; Theorem A.1 bounds dictionary pair coherence but not beam-search selection error; pruned branches are unreachable, constraining the effective dictionary.
domain assumption Residual-stream dimension must be a power of two (d=2^t).
Appendix A; required by the Kloosterman construction (r=2t columns); restricts applicability to transformer widths that are powers of two.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scaling Interpretable Transformers with Parity Bottleneck Layers." pith.science (2026). https://pith.science/paper/LNYHFI4V

@misc{pith2026260720652,
  author       = {Pith},
  title        = {Pith review of: Scaling Interpretable Transformers with Parity Bottleneck Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LNYHFI4V}},
  note         = {Machine review of arXiv:2607.20652}
}

read the original abstract

Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams. Sparse autoencoders (SAEs) are designed to recover such features post-hoc, but training models that are interpretable by construction has remained impractical, as a per-layer over-complete bottleneck is prohibitively expensive in both memory and compute. To overcome this issue, we introduce the ParityTransformer, a GPT-2-scale architecture whose intermediate representations are efficient and wide / sparse by design. At each layer, a Deep Parity Bottleneck (DPB) replaces a learned over-complete basis with a parameter-free algebraic dictionary, providing a deterministic incoherence guarantee and eliminating the memory requirements that have prevented per-layer interpretable bottlenecks at scale. A DPB is a hierarchically structured sparse bottleneck which efficiently enforces sparsity using a multi-level mixture-of-experts approach: a hardware-aware implementation that closes the cost gap between activation sparse and dense training to a manageable interpretability tax. Empirically, ParityTransformers perform at least as well as post-hoc SAEs on sparse probing tasks, while out-performing on measures of feature absorption, steering effectiveness, and fine-grained causal interventions. Because subsequent computation acts only on features that survive the sparse bottleneck, the ParityTransformer's features are native to the model's forwards pass by construction, addressing the question of whether SAEs probe features the model actually uses during computation. We see this as a step toward training models whose internal representations are interpretable by design rather than recovered post hoc.

Figures

Figures reproduced from arXiv: 2607.20652 by the authors.

**Figure 2.** Edit success rate (fraction of completions flipped to the target) for varying number of [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** b so that readers can validate the fluency / concept strength for themselves. Full methodology and LLM judge prompts are deferred to Appendix D [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Poisoned-document retrieval precision at [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: SAE explained variance across layers for the post-hoc SAEs trained on the fine-tuned [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

**Figure 6.** Figure 6: Poisoned-document retrieval precision for additional values of [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

**Figure 7.** Figure 7: Steering examples F Compute Requirements Training a single 1.3B ParityTransformer on 20B tokens takes around 160 B200 hours, for an approximate cost of $640, while a 200m ParityTransformer takes around 32 B200 hours, for an approximate cost of $128. We trained 2 varian…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 linked inside Pith

[1]

Compute the clean logit differenced clean =logit(a src)−logit(a tgt)on the source prompt
[2]

deploy" and

The ι-th bit ofM uis (M u)ι = D coord(ξι), u(1) E F2 + D coord(ξ −1 ι ), u(2) E F2 .(23) Comparing Equations (21) and (23), the index with u= (τ(a), τ(b)) generates exactly the codeword ca,b. Since a7→τ(a) is a bijection, ranging over all 2t-bit indices u is the same as ranging over all pairs (a, b)∈F 2 2t, in a different order: M is a generator matrix fo...
[3]

URL https://transformer-circuits.pub/2024/scaling-monosemanticity/ index.html. A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248, 2023. J. Vig, S. Gehrmann, Y . Belinkov, S. Qian, D. Nevo, Y . Singer, and S. Shieber. Investigating ...

arXiv 2024
[4]

Measure the patched logit differenced patched
[5]

A value of e= 0 indicates the patch had no effect; e= 1 indicates the logit difference fully shifted to match the target

Compute the normalized effect:e= (d clean −d patched)/(dclean −d tgt). A value of e= 0 indicates the patch had no effect; e= 1 indicates the logit difference fully shifted to match the target. We patch all positions simultaneously rather than only the final token, as MLP-mediated circuits may process task-relevant information at subject or object token po...

2024
[7]

Compute the target logit differenced tgt on the target prompt
[8]

Patch the component output at layer ℓ from the target prompt into the source prompt atall token positions
[10]

The same concept definitions, seed texts, LLM judge, and scoring rubric are used for both models

compared to single-feature steering (20–75) because perturbations are applied at every layer simultaneously. The same concept definitions, seed texts, LLM judge, and scoring rubric are used for both models. The scaling factor α is swept to trace the concept-fluency tradeoff, but differ between PT and baseline due to the aforementioned reasons. D.1 Steerin...

Show all 15 references

[12]

Collect positive texts (containing the target concept) and negative (contrastive) texts
[13]

Run the model, record feature activations on each set
[14]

Compute the difference in mean feature activations between positive and negative sets
[15]

religion,

Retain the top-kfeatures with the largest positive differential activation. Steering vector applicationFor post-hoc SAEs, steering vectors v are top k weighted sums of the the decoder column Wdec[i] normalized to unit norm. For Parity Transformer with DPB, L0 features are stan...

2026
[256]

When [IO] and [S] went to the [place], [S] gave a [object] to

These selected settings are used in the main retrieval experiments for the correponding method reported in the paper. 21 B.5 Additional Poisoned-Document Retrieval Results 0 1 2 3 4 5 6 7 8 9 10 11 Layer 0.00 0.05 0.10 0.15 0.20Positive precision Triggered, k = 1 0 1 2 3 4 5 6...
[2016]

URLhttps://arxiv.org/abs/1606.06031. S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry. TRAK: Attributing model behavior at scale. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 270...

2023 arXiv
[2024]

URLhttps://arxiv.org/abs/2408.05451. K. Jordan, Y . Jin, V . Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/ muon/. A. Karvonen, C. Rager, J. Lin, C. Tigges, J. Bl...

2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Compute the clean logit differenced clean =logit(a src)−logit(a tgt)on the source prompt

[2] [2]

deploy" and

The ι-th bit ofM uis (M u)ι = D coord(ξι), u(1) E F2 + D coord(ξ −1 ι ), u(2) E F2 .(23) Comparing Equations (21) and (23), the index with u= (τ(a), τ(b)) generates exactly the codeword ca,b. Since a7→τ(a) is a bijection, ranging over all 2t-bit indices u is the same as ranging over all pairs (a, b)∈F 2 2t, in a different order: M is a generator matrix fo...

[3] [3]

URL https://transformer-circuits.pub/2024/scaling-monosemanticity/ index.html. A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid. Steering language models with activation engineering.arXiv preprint arXiv:2308.10248, 2023. J. Vig, S. Gehrmann, Y . Belinkov, S. Qian, D. Nevo, Y . Singer, and S. Shieber. Investigating ...

arXiv 2024

[4] [4]

Measure the patched logit differenced patched

[5] [5]

A value of e= 0 indicates the patch had no effect; e= 1 indicates the logit difference fully shifted to match the target

Compute the normalized effect:e= (d clean −d patched)/(dclean −d tgt). A value of e= 0 indicates the patch had no effect; e= 1 indicates the logit difference fully shifted to match the target. We patch all positions simultaneously rather than only the final token, as MLP-mediated circuits may process task-relevant information at subject or object token po...

2024

[6] [7]

Compute the target logit differenced tgt on the target prompt

[7] [8]

Patch the component output at layer ℓ from the target prompt into the source prompt atall token positions

[8] [10]

The same concept definitions, seed texts, LLM judge, and scoring rubric are used for both models

compared to single-feature steering (20–75) because perturbations are applied at every layer simultaneously. The same concept definitions, seed texts, LLM judge, and scoring rubric are used for both models. The scaling factor α is swept to trace the concept-fluency tradeoff, but differ between PT and baseline due to the aforementioned reasons. D.1 Steerin...