Pith. sign in

REVIEW 3 major objections 5 minor 79 references

A variance floor fixes the crispness collapse that kills legible transformers, yielding units that are readable, surgically editable, and rewirable by a decorrelation dial at quality parity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 05:34 UTC pith:BZ6IEYTD

load-bearing objection Clean algebraic fix for crispness collapse plus a usable end-to-end legible model; the 78%/50% headline is a knife-edge single-seed snapshot, but the identity, ablations, and edit locality still make it worth reading. the 3 major comments →

arxiv 2607.08946 v1 pith:BZ6IEYTD submitted 2026-07-09 cs.LG cs.CL

Training, Reading, and Editing Legible Transformers

Mark Oskin This is my paper
classification cs.LG cs.CL
keywords legible transformerscrispness penaltyvariance floorfuzzy set operatorsmodel editingdecorrelation dialdetection versus namingfan-out fan-in circuits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

A transformer can be built from bounded, named operators that already look like fuzzy set operations, but training still has to keep those operators crisp and alive. The natural crispness penalty fails: it is secretly a variance-minimizer, so units collapse into dead constants that ignore their inputs and quality collapses with them. The paper shows that writing the target property itself as a loss — a per-channel variance floor — repairs the collapse, recovers both legibility and language-model quality, and, together with a learned per-unit gate, retires the old hand-set GELU reservation. The resulting model has the majority of its feed-forward and half of its attention value channels as crisp, contextual detectors. Because those detectors fire cleanly and sparsely, edits land far more locally than on ordinary neurons and can express explicit conjunctions; a further between-unit decorrelation pressure then trades circuit reuse for independence, turning shared concepts into single editable units and whole predictions into short lists of named operations, all without hurting quality.

Core claim

The crispness pressure E[v(1−v)] equals μ(1−μ)−var and therefore cannot distinguish a live detector from a dead constant; a per-channel variance floor that is exactly the selective metric written as a loss, paired with a learned per-unit gate, recovers both legibility and quality and produces an end-to-end model whose crisp sparse units admit far more local deep edits and a fan-out-to-fan-in circuit dial at parity with a conventional baseline.

What carries the argument

The identity E[v(1−v)] = μ(1−μ) − var, which shows the crispness penalty is a variance-minimizer, and the per-channel variance floor L_ctx that prices out the zero-variance endpoints so the only zero-loss state is crisp and contextual.

Load-bearing premise

That the fixed variance threshold used to call a unit selective is a stable definition of legibility rather than a knife-edge that small drifts can push large numbers of channels across.

What would settle it

Train the same architecture with the variance floor at several nearby thresholds and seeds; if selective fractions and edit-locality gains swing wildly or quality collapses while the continuous crisp fraction stays high, the floor is not a robust definition of the target state.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper shows that legible-by-construction transformers (bounded fuzzy-set FFN operators and attention values) require an explicit training objective to remain usable: a crispness penalty collapses units to dead constants because E[v(1-v)]=μ(1-μ)-var is a pure variance minimizer, and a per-channel variance floor L_ctx (the selective metric written as a hinge loss) repairs the collapse, recovering both selective fractions and quality. A learned per-unit gate α retires the hand-set reserved-GELU fraction, routing 87% of load-bearing computation through crisp operators and yielding an annealed model with 78% selective FFN operands and 50% selective attention channels at parity with a GELU baseline. In the tuned-lens frame the units separate clean detection from harder naming; crisp sparse units support 50–184× more local deep edits (including single-unit A∩B conjunctions); and a between-unit decorrelation term acts as a free dial that trades fan-out reuse for fan-in independence, making concepts surgically editable and predictions attributable to short lists of named operations.

Significance. If the results hold, the work supplies a practical, algebraically motivated objective that turns prior legible substrates into end-to-end models that can be read and edited without post-hoc dictionaries. The identity and variance-floor fix are clean contributions that connect specification gaming to anti-collapse regularizers; the edit-locality gains and free fan-out/fan-in dial are concrete payoffs for interpretability and model editing. Strengths include the exact identity, coherent ablations (floor placement, tax schedules, λ_d sweep), and the demonstration that quality metrics (perplexity, LAMBADA, BLiMP) recover independently of the selective count. The construction is parameter-neutral and the measurements are defined on ordinary residual-stream quantities, so the circuit findings are potentially transferable.

major comments (3)
  1. Section 3.4 and the footnote in Section 7 explicitly state that the selective population sits just above the knife-edge threshold τ=0.003 (typical variance only a small multiple of τ), so that “a small, transient drift in variance can carry many channels across the line, and the count can swing by tens of points between training checkpoints while the crispness underneath barely moves.” All headline numbers (78% FFN, 50% attention, per-head depth trajectory) and the edit-site selection that underpins the 50–184× locality claims are therefore single-seed, end-of-epoch snapshots of this fragile statistic. The continuous crisp fraction (~0.90) is acknowledged to be more stable but is not the quantity used for the claims. A modest multi-seed sweep or τ-perturbation table is required before the legibility and editability headlines can be treated as robust.
  2. All quantitative results (quality, selective fractions, edit KL, fan-in/fan-out, attribution sparsity) are reported from a single seed at 125 M parameters. While the algebraic identity and the qualitative direction of the ablations are convincing, the load-bearing numerical claims (especially the depth-dependent locality factors and the exact 78%/50% figures) cannot be assessed for variance. At minimum the paper should report the range of selective fractions and LAMBADA across 3–5 seeds for the annealed and baseline models, or clearly relegate the percentages to “illustrative of a single run.”
  3. Section 6.2 and Figure 12 compare Boolean versus GELU edit locality only on units that fire above an (unspecified) threshold; layers L1, L2 and L4 contain no editable Boolean output units at all. The reported 184× and 52× advantages therefore condition on the existence of firing sites that the objective itself produces. A fairer comparison would either (a) include non-firing units at matched efficacy or (b) report the fraction of units that are editable under identical criteria, so that the locality gain is not confounded with the mere presence of sparse detectors.
minor comments (5)
  1. Figure 4 and Table 1: the annealed schedule is described only as “10^{-2}→10^{-1}”; the exact ramp (linear, cosine, step) and the epoch at which the final tax is reached should be stated so the experiment is reproducible.
  2. Section 5.3 and Figure 9: “decodable” is defined by projection onto a “nameable content category,” yet the seven-way dictionary used for the curated sample in Figure 7 is never listed. A short appendix table of the category definitions would remove ambiguity.
  3. Equation (8): the hinge is written ReLU(1-var_c/τ); the text later refers to a “variance floor.” Clarifying that the loss is zero once var_c≥τ (i.e., a soft floor rather than a hard constraint) would prevent misreading.
  4. Appendix A notes a late-training instability above roughly half-Boolean; the main text should cross-reference this ceiling when claiming that the learned gate “retires the hand-set partition,” so readers understand the residual GELU blend is not optional.
  5. Typographical: “theselective” appears repeatedly without a space (e.g., Sections 3.4, 4.3); “lambada” is lower-cased inconsistently with LAMBADA.

Circularity Check

1 steps flagged

Variance floor is explicitly 'the selective metric written as a loss', so the headline selective % recovery is partly by construction; quality/edit metrics remain independent.

specific steps
  1. self definitional [Abstract + Section 4.3 Eq. (8) + Section 3.4]
    "a per-channel variance floor, the target legibility metric written as a loss, which recovers both legibility and quality. ... This is the selective metric of Section 3.4 turned into a loss: paired with the crispness term, the unique zero-loss state is crisp and contextual"

    Selective is defined as crisp AND var_c ≥ τ. L_ctx is exactly the hinge ReLU(1 - var_c/τ) that enforces the contextual half of that definition. Optimizing L_ctx therefore raises the selective count by construction; reporting '78% selective' after training with that loss is not an independent empirical discovery of legibility but the direct optimization target. Quality recovery and edit KL remain non-circular external checks.

full rationale

The algebraic identity E[v(1-v)]=μ(1-μ)-var is a genuine, parameter-free derivation that correctly diagnoses why pure crispness collapses variance; it is not circular. Writing the selective definition (crisp + var≥τ) directly as the hinge loss L_ctx and then reporting higher selective fractions is self-definitional for that particular count, as the paper itself states. However, the same objective also recovers independent external metrics (dev perplexity, LAMBADA, BLiMP) and yields measured edit-locality gains (KL_off) that are not forced by the loss definition. Self-citations to the author's prior operator papers supply the substrate but are not load-bearing for the new objective, reading, or editing claims; the paper recaps the constructions and evaluates them end-to-end against a conventional baseline. The selective threshold itself is acknowledged as knife-edge, but that is a robustness issue rather than circularity. Overall mild and localized; central quality and edit results stand independently.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

The central claims rest on a small set of hand-chosen loss coefficients and a threshold that defines the target metric, plus the architectural premise that sigmoid-bounded fuzzy-set operators are the right substrate. No new physical entities are invented; the free parameters are ordinary training hyperparameters whose values are reported but not derived.

free parameters (6)
  • variance floor threshold τ = 0.003
    Defines the contextual half of the selective metric and the hinge in L_ctx; set to 0.003 by hand.
  • crispness coefficient λ_c = 3e-3
    Weight on the crispness penalty that induces the collapse the floor repairs.
  • sparsity coefficient λ_s = 1e-3
    Weight on the L1 sparsity pressure on bounded operands.
  • variance-floor coefficient λ_ctx = 3e-3
    Weight on the per-channel ReLU variance hinge; chosen after a too-strong 1e-2 ablation.
  • GELU-tax schedule for learned gate α = 1e-2 → 1e-1
    Annealed output-norm-weighted tax that drives the population crisp; final load-bearing crisp fraction 0.87 depends on it.
  • decorrelation coefficient λ_d = 0 to 1
    Swept over two orders of magnitude to expose the fan-out/fan-in dial.
axioms (4)
  • domain assumption Sigmoid-bounded fuzzy intersection and set-difference on two operand banks are a parameter-neutral, legible replacement for GELU feed-forward and attention value projections.
    Taken from the author’s prior constructions (Oskin 2026a,b) and used as the substrate throughout Sections 3–7.
  • ad hoc to paper A channel is legible (selective) when >50% of its mass lies within 0.1 of {0,1} and its across-context variance ≥ τ=0.003.
    Definition introduced in Section 3.4; the paper notes the count is knife-edge fragile near the floor.
  • domain assumption Quality parity with a conventional GELU baseline on perplexity, LAMBADA and BLiMP is the relevant success criterion for the objective.
    Stated throughout; no claim of benchmark improvement is made.
  • standard math E[v(1−v)] = μ(1−μ) − var for v∈[0,1] exactly characterizes the crispness penalty as a variance minimizer.
    Elementary identity used in Section 4.2 to diagnose collapse.
invented entities (3)
  • per-channel variance floor L_ctx no independent evidence
    purpose: Prevents the crispness penalty from driving units to dead constants by forbidding zero-variance solutions.
    New loss term; shares hinge shape with VICReg but is applied to counteract an induced collapse rather than SSL invariance.
  • learned per-unit operator/GELU gate α=σ(θ) no independent evidence
    purpose: Retires the hand-set reserved-GELU partition so the model itself chooses the crisp fraction.
    New architectural scalar per unit; empirical outcome is empty pure-GELU bucket and 87% load-bearing crisp computation.
  • between-unit decorrelation dial (fan-out ↔ fan-in) no independent evidence
    purpose: Trades circuit reuse for independence, making concepts single editable units and predictions short named explanations.
    New pressure and measurement pair (fan-in atoms vs fan-out readers); quality flat across the sweep.

pith-pipeline@v1.1.0-grok45 · 37059 in / 3840 out tokens · 40535 ms · 2026-07-13T05:34:44.863309+00:00 · methodology

0 comments
read the original abstract

A transformer can be built from operators that are legible by construction -- bounded, named units that read as fuzzy set operations rather than dense activations -- but legibility must be pressed for during training, and the pressure has a failure mode. A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant. An identity, E[v(1-v)] = mu(1-mu) - var, shows why -- the penalty is a variance-minimizer blind to the difference between a live detector and a constant -- and names the fix: a per-channel variance floor, the target legibility metric written as a loss, which recovers both legibility and quality. A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work: given the choice the model keeps no unit as pure GELU and routes 87% of its load-bearing computation through crisp operators. The result is the most legible transformer we have built -- 78% of its feed-forward operands and 50% of its attention value channels are crisp-and-contextual detectors, and per-head legibility rises from 18% in shallow layers to 78% in deep ones. Read in the correct rotated per-layer frame, these units separate a clean detection (what a unit responds to) from a harder naming (what its output decodes to); and because the objective makes each unit crisp and sparse, edits to them are far more local -- 50-184x in the deep layers where the edit sites concentrate -- and can target explicit conjunctions a single neuron cannot express. Finally, a between-unit decorrelation pressure exposes a legibility dial: it trades a circuit's reuse for independence at no quality cost, turning concepts into single, surgically editable units and a prediction into a short explanation read off a handful of named operations. Quality holds at parity with a conventional baseline throughout.

Figures

Figures reproduced from arXiv: 2607.08946 by Mark Oskin.

Figure 1
Figure 1. Figure 1: The legible substrate, on both sublayers. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The three metric states, each a real feed-forward operand channel of the annealed model over [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Why the crispness pressure cannot sharpen an operator without risking its death. The penalty [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Repairing the crispness collapse (fully-Boolean [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Variance floor on each sublayer (12L). Left: selective fractions — each sublayer’s legibility recovers only when its own floor is applied (black outline marks the floored sublayer): a local effect. Right: LAMBADA accuracy recovers from either floor: a global effect. Quality is a coupled property of the residual stream; legibility is local to each sublayer’s operators. 4.5 Learning the operator fraction ins… view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of the learned gate α = σ(θ) across the four runs. The pure-gelu band (α < 0.1, bottom) is empty in every run, including the untaxed null; the crisp band (α > 0.9, top) fills as the tax rises. The model is never handed a reserved gelu partition and never forms one on its own. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Legibility of the annealed model on both sublayers. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The prompt “It was the best of times, it was the ___” (completion: worst), each layer’s attention and feed-forward writes decoded under the raw logit lens (left) and the tuned lens (Belrose et al., 2023) (right). The raw column reads at the final prediction position; the tuned column reads each write at its most on-task source position (tagged @k; untagged tuned cells and all raw cells are the final positi… view at source ↗
Figure 9
Figure 9. Figure 9: Per-layer legibility of the annealed model’s feed-forward operands: the fraction that are [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: How the fuzzy-conjunction block computes concepts, redundant early and distributed late. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Firing crispness predicts edit locality, within the annealed model. Each point is one Boolean [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Off-target collateral kloff per layer band at matched efficacy (eff∗ = 2.0 nats), Boolean units (anneal) against the conventional gelu baseline; the count of Boolean and gelu units (nB/nG) is printed under each band. The targetedness advantage emerges with depth: Boolean units are no better than gelu in the earliest layers (L0–L2, 1.4×), where their outputs barely fire, but 184× more local at L6–L8 and 52… view at source ↗
Figure 13
Figure 13. Figure 13: A conjunctive A ∩ B edit. Two difference-of-means concept directions — “king” and “France” — are installed into the two operand banks of one Boolean feed-forward unit (layer 6, j=17); the unit’s unchanged fuzzy-AND A⊙B then fires only for the conjunction. Across four context classes, A fires on king contexts, B on France, and the intersection (peak over positions) is nonzero only for “both” (0.35), exactl… view at source ↗
Figure 14
Figure 14. Figure 14: Fan-out versus fan-in, from real circuits (edge width [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Figure [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Reading a prediction, and what decorrelation does to it. (a) The decorrelated model generating from “The cat is much bigger ” token by token: each produced token is arced back to the prior word its top value head reads (a lighter dashed arc marks a strong secondary head — here the induction head L9.H3 that copies the earlier cat to predict cat); attention is legible at every step, and below, its direct-lo… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

79 extracted references · 24 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2606.31845 , year =

    Oskin, Mark , title =. arXiv preprint arXiv:2606.31845 , year =

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2607.04319 , year =

    Oskin, Mark , title =. arXiv preprint arXiv:2607.04319 , year =

    Pith/arXiv arXiv

  3. [3]

    2020 , howpublished =

    nostalgebraist , title =. 2020 , howpublished =

    2020

  4. [4]

    arXiv preprint arXiv:2303.08112 , year =

    Eliciting Latent Predictions from Transformers with the Tuned Lens , author =. arXiv preprint arXiv:2303.08112 , year =

    Pith/arXiv arXiv

  5. [5]

    Proceedings of EMNLP 2021 , pages =

    Transformer Feed-Forward Layers Are Key-Value Memories , author =. Proceedings of EMNLP 2021 , pages =. 2021 , url =

    2021

  6. [6]

    Proceedings of EMNLP 2022 , pages =

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space , author =. Proceedings of EMNLP 2022 , pages =. 2022 , url =

    2022

  7. [7]

    Transformer Circuits Thread, Anthropic , year =

    A Mathematical Framework for Transformer Circuits , author =. Transformer Circuits Thread, Anthropic , year =

  8. [8]

    Transformer Circuits Thread, Anthropic , year =

    Privileged Bases in the Transformer Residual Stream , author =. Transformer Circuits Thread, Anthropic , year =

  9. [9]

    International Conference on Learning Representations (ICLR) , year =

    Residual Connections Encourage Iterative Inference , author =. International Conference on Learning Representations (ICLR) , year =

  10. [10]

    International Conference on Learning Representations (ICLR) , year =

    Highway and Residual Networks learn Unrolled Iterative Estimation , author =. International Conference on Learning Representations (ICLR) , year =

  11. [11]

    arXiv preprint arXiv:2310.16270 , year =

    Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism , author =. arXiv preprint arXiv:2310.16270 , year =

    Pith/arXiv arXiv

  12. [12]

    Proceedings of ACL 2023 , year =

    Analyzing Transformers in Embedding Space , author =. Proceedings of ACL 2023 , year =

    2023

  13. [13]

    Proceedings of CoNLL 2023 , year =

    Future Lens: Anticipating Subsequent Tokens from a Single Hidden State , author =. Proceedings of CoNLL 2023 , year =

    2023

  14. [14]

    2022 , note=

    Bardes, Adrien and Ponce, Jean and LeCun, Yann , booktitle=. 2022 , note=

    2022

  15. [15]

    International Conference on Machine Learning (ICML) , series=

    Barlow Twins: Self-Supervised Learning via Redundancy Reduction , author=. International Conference on Machine Learning (ICML) , series=. 2021 , note=

    2021

  16. [16]

    International Conference on Machine Learning (ICML) , series=

    Whitening for Self-Supervised Representation Learning , author=. International Conference on Machine Learning (ICML) , series=. 2021 , note=

    2021

  17. [17]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  18. [18]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Exploring Simple Siamese Representation Learning , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

  19. [19]

    Lu, Lu and Shin, Yeonjong and Su, Yanhui and Karniadakis, George Em , journal=. Dying. 2020 , note=

    2020

  20. [20]

    International Conference on Machine Learning (ICML) , year=

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , author=. International Conference on Machine Learning (ICML) , year=

  21. [21]

    International Conference on Machine Learning (ICML) , series=

    Concept Bottleneck Models , author=. International Conference on Machine Learning (ICML) , series=. 2020 , note=

    2020

  22. [22]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Towards Robust Interpretability with Self-Explaining Neural Networks , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  23. [23]

    Transformer Circuits Thread , year=

    Towards Monosemanticity: Decomposing Language Models with Dictionary Learning , author=. Transformer Circuits Thread , year=

  24. [24]

    Transformer Circuits Thread , year=

    Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet , author=. Transformer Circuits Thread , year=

  25. [25]

    arXiv preprint arXiv:2405.12241 , year=

    Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning , author=. arXiv preprint arXiv:2405.12241 , year=

    Pith/arXiv arXiv

  26. [26]

    Higgins, Irina and Matthey, Loic and Pal, Arka and others , booktitle=. beta-

  27. [27]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  28. [28]

    arXiv preprint arXiv:1308.3432 , year=

    Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , author=. arXiv preprint arXiv:1308.3432 , year=

    Pith/arXiv arXiv

  29. [29]

    International Conference on Learning Representations (ICLR) , year=

    Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets , author=. International Conference on Learning Representations (ICLR) , year=

  30. [30]

    arXiv preprint arXiv:1803.04585 , year=

    Categorizing Variants of Goodhart's Law , author=. arXiv preprint arXiv:1803.04585 , year=

    Pith/arXiv arXiv

  31. [31]

    Towards Faithfully Interpretable

    Jacovi, Alon and Goldberg, Yoav , booktitle=. Towards Faithfully Interpretable. 2020 , note=

    2020

  32. [32]

    arXiv preprint arXiv:2604.10604 , year=

    NSFL: A Post-Training Neuro-Symbolic Fuzzy Logic Framework for Boolean Operators in Neural Embeddings , author=. arXiv preprint arXiv:2604.10604 , year=

    Pith/arXiv arXiv

  33. [33]

    International Conference on Machine Learning (ICML) , year=

    Codebook Features: Sparse and Discrete Interpretability for Neural Networks , author=. International Conference on Machine Learning (ICML) , year=

  34. [34]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Backpack Language Models , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  35. [35]

    Transformer Circuits Thread , year=

    Softmax Linear Units , author=. Transformer Circuits Thread , year=

  36. [36]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Large Memory Layers with Product Keys , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  37. [37]

    arXiv preprint arXiv:2407.04153 , year=

    Mixture of a Million Experts , author=. arXiv preprint arXiv:2407.04153 , year=

    Pith/arXiv arXiv

  38. [38]

    arXiv preprint arXiv:2002.05202 , year=

    GLU Variants Improve Transformer , author=. arXiv preprint arXiv:2002.05202 , year=

    Pith/arXiv arXiv 2002

  39. [39]

    International Conference on Learning Representations (ICLR) , year=

    Bilinear MLPs Enable Weight-Based Mechanistic Interpretability , author=. International Conference on Learning Representations (ICLR) , year=

  40. [40]

    International Conference on Learning Representations (ICLR) , year=

    Learning Language Representations with Logical Inductive Bias , author=. International Conference on Learning Representations (ICLR) , year=

  41. [41]

    arXiv preprint arXiv:2510.12269 , year=

    Tensor Logic: The Language of AI , author=. arXiv preprint arXiv:2510.12269 , year=

    arXiv

  42. [42]

    arXiv preprint arXiv:2006.13155 , year=

    Logical Neural Networks , author=. arXiv preprint arXiv:2006.13155 , year=

    Pith/arXiv arXiv 2006

  43. [43]

    Artificial Intelligence , volume=

    Analyzing Differentiable Fuzzy Logic Operators , author=. Artificial Intelligence , volume=. 2022 , note=

    2022

  44. [44]

    Artificial Intelligence , volume=

    Logic Tensor Networks , author=. Artificial Intelligence , volume=. 2022 , note=

    2022

  45. [45]

    arXiv preprint arXiv:2508.08172 , year=

    Neural Logic Networks for Interpretable Classification , author=. arXiv preprint arXiv:2508.08172 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2404.19756 , year=

    KAN: Kolmogorov--Arnold Networks , author=. arXiv preprint arXiv:2404.19756 , year=

    Pith/arXiv arXiv

  47. [47]

    Transformer Circuits Thread , year=

    Toy Models of Superposition , author=. Transformer Circuits Thread , year=

  48. [48]

    arXiv preprint arXiv:2211.12312 , year=

    Interpreting Neural Networks through the Polytope Lens , author=. arXiv preprint arXiv:2211.12312 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv preprint arXiv:2309.08600 , year=

    Sparse Autoencoders Find Highly Interpretable Features in Language Models , author=. arXiv preprint arXiv:2309.08600 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2406.04093 , year=

    Scaling and Evaluating Sparse Autoencoders , author=. arXiv preprint arXiv:2406.04093 , year=

    Pith/arXiv arXiv

  51. [51]

    Proceedings of BlackboxNLP , year=

    Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 , author=. Proceedings of BlackboxNLP , year=

  52. [52]

    International Conference on Machine Learning (ICML) , year=

    The Linear Representation Hypothesis and the Geometry of Large Language Models , author=. International Conference on Machine Learning (ICML) , year=

  53. [53]

    arXiv preprint arXiv:2604.02608 , year=

    Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens , author=. arXiv preprint arXiv:2604.02608 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2410.14670 , year=

    Decomposing the Dark Matter of Sparse Autoencoders , author=. arXiv preprint arXiv:2410.14670 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv preprint arXiv:2405.14860 , year=

    Not All Language Model Features Are One-Dimensionally Linear , author=. arXiv preprint arXiv:2405.14860 , year=

    Pith/arXiv arXiv

  56. [56]

    OpenAI Blog , year=

    Language Models Can Explain Neurons in Language Models , author=. OpenAI Blog , year=

  57. [57]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Compositional Explanations of Neurons , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  58. [58]

    arXiv preprint arXiv:2506.15538 , year=

    Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework , author=. arXiv preprint arXiv:2506.15538 , year=

    arXiv

  59. [59]

    arXiv preprint arXiv:2410.13928 , year=

    Automatically Interpreting Millions of Features in Large Language Models , author=. arXiv preprint arXiv:2410.13928 , year=

    Pith/arXiv arXiv

  60. [60]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Knowledge Neurons in Pretrained Transformers , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  61. [61]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Locating and Editing Factual Associations in GPT , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  62. [62]

    International Conference on Learning Representations (ICLR) , year=

    Mass-Editing Memory in a Transformer , author=. International Conference on Learning Representations (ICLR) , year=

  63. [63]

    arXiv preprint arXiv:2308.10248 , year=

    Activation Addition: Steering Language Models Without Optimization , author=. arXiv preprint arXiv:2308.10248 , year=

    Pith/arXiv arXiv

  64. [64]

    Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Steering Llama 2 via Contrastive Activation Addition , author=. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  65. [65]

    arXiv preprint arXiv:2310.01405 , year=

    Representation Engineering: A Top-Down Approach to AI Transparency , author=. arXiv preprint arXiv:2310.01405 , year=

    Pith/arXiv arXiv

  66. [66]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Refusal in Language Models Is Mediated by a Single Direction , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  67. [67]

    International Conference on Learning Representations (ICLR) , year=

    Editing Models with Task Arithmetic , author=. International Conference on Learning Representations (ICLR) , year=

  68. [68]

    International Conference on Learning Representations (ICLR) , year=

    Function Vectors in Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  69. [69]

    Findings of EMNLP , year=

    In-Context Learning Creates Task Vectors , author=. Findings of EMNLP , year=

  70. [70]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Investigating Gender Bias in Language Models Using Causal Mediation Analysis , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  71. [71]

    International Conference on Learning Representations (ICLR) , year=

    Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small , author=. International Conference on Learning Representations (ICLR) , year=

  72. [72]

    International Conference on Learning Representations (ICLR) , year=

    Towards Best Practices of Activation Patching in Language Models: Metrics and Methods , author=. International Conference on Learning Representations (ICLR) , year=

  73. [73]

    arXiv preprint arXiv:2510.03659 , year=

    Does Higher Interpretability Imply Better Utility? A Pairwise Analysis on Sparse Autoencoders , author=. arXiv preprint arXiv:2510.03659 , year=

    arXiv

  74. [74]

    arXiv preprint arXiv:2502.06809 , year=

    Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution , author=. arXiv preprint arXiv:2502.06809 , year=

    Pith/arXiv arXiv

  75. [75]

    Paperno, Denis and Kruszewski, Germ. The. Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  76. [76]

    , journal=

    Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R. , journal=

  77. [77]

    2023 , howpublished=

    A Framework for Few-Shot Language Model Evaluation , author=. 2023 , howpublished=

    2023

  78. [78]

    arXiv preprint arXiv:2408.05451 , year=

    Mathematical Models of Computation in Superposition , author=. arXiv preprint arXiv:2408.05451 , year=

    Pith/arXiv arXiv

  79. [79]

    International Conference on Learning Representations (ICLR) , year=

    Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models , author=. International Conference on Learning Representations (ICLR) , year=