Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Replacing self-attention with prototype routing makes a language model's concepts inspectable and editable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:56 UTC pith:HJQ65BA3

load-bearing objection ProtoT is a genuinely new mixer and the performance reporting is honest, but the 'interpretable by design' claim rests on intervention evidence that only works on the discovery sentence. the 4 major comments →

arxiv 2602.11852 v2 pith:HJQ65BA3 submitted 2026-02-12 cs.AI cs.CLcs.LG

Prototype Transformer: Towards Language Model Architectures Interpretable by Design

Yordan Yordanov , Matteo Forasassi , Bayar Menzat , Ruizhi Wang , Chang Qi , Markus Kaltenberger , Amine M'Charrak , Tommaso Salvatori
show 1 more author
Thomas Lukasiewicz
This is my paper
classification cs.AI cs.CLcs.LG
keywords prototype-based routinginterpretable language modelsself-attention replacementlinear attentionconcept learningmechanistic interpretabilitytargeted model editingprototype intervention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces ProtoT, an autoregressive language model in which the standard self-attention mixer is replaced by a set of learned prototype vectors that act as communication channels between past and future tokens. The authors claim that this simple architectural change causes the prototypes to self-organize into nameable, largely disentangled concepts during training, such as "woman" or "COVID," without any concept-level supervision. Because each concept is tied to a specific prototype, the model offers a direct route to interpretability: inspecting which prototypes fired, and editing them by re-initializing or masking a prototype, changes predictions in a targeted way. ProtoT also scales linearly with sequence length and keeps competitive perplexity and GLUE performance, though it trails the strongest Transformer baseline. The paper's central wager is that interpretability can be designed into the architecture rather than recovered post hoc.

Core claim

The central claim is that a prototype-based routing bottleneck—where R learned vectors filter the past through softmax write gates, aggregate it via time-discounted prefix means, and read it back through softmax read gates—creates a semantic routing structure that spontaneously learns nameable concepts. The authors show that prototypes in deeper layers align with concepts such as "female," "male," "New Zealand," "Mental Health," "COVID," school narratives, and function-word patterns; that LLM-assisted labeling scores for disentanglement improve on those of a matched LLaMA-style attention model; and that disrupting a prototype decreases or increases probabilities of related target words in a

What carries the argument

The prototype mixer: R learned parameter vectors (prototypes) act as channels. A write gate routes each past token into channels using softmax over prototypes; a discounted prefix mean maintains per-channel memory (an exponential moving average with learned decay, giving each prototype a half-life); and a read gate retrieves channel content for the current token. The recurrence allows caching, so per-token generation is O(1) and total cost is linear in sequence length, unlike quadratic self-attention. The semantic bottleneck—prototypes never interact directly—is what the paper argues pushes each channel to specialize into a nameable concept.

Load-bearing premise

The paper's interpretability claims rest on the assumption that the LLM-generated theme labels of top-activating sentences—plus a small set of hand-picked intervention contexts whose baseline probabilities are high enough—genuinely reflect what a prototype encodes; if the labels are biased or the selected prototypes are unrepresentative, the central contribution is unsupported.

What would settle it

Train a ProtoT model on a corpus stripped of gendered or COVID-related content; if the supposed "female" and "COVID" prototypes still emerge, or if re-initializing them changes target-word probabilities, the claimed concept-to-prototype correspondence would not be tracking content. More directly, a systematic sweep over all prototypes with a held-out concept benchmark would settle whether intervention effects generalize beyond the few hand-picked cases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If ProtoT's central claim holds, an autoregressive LM can expose its own concept inventory by listing which prototypes fire, replacing post-hoc interpretability with built-in inspection.
  • Targeted edits follow directly: masking or re-initializing a single prototype produces a measurable, concept-specific change in next-token probability, pointing toward a mechanism for correcting or suppressing specific model behaviors.
  • Because generation cost is O(1) per token, the architecture is a candidate for long-context and on-device settings where quadratic attention is prohibitive.
  • Robustness under meaning-preserving perturbations can be attributed to specific prototype channels, so a user can see where stability comes from rather than treating robustness as a black-box aggregate.
  • The gap to strong attention baselines is stated as real but shrinking with scale; if the trend continues, interpretability-by-design need not be paid for in accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if prototype hubs are as stable as the cross-seed COVID result suggests, one could pre-register a concept dictionary by training small models and then use prototype masks as a cheap alignment or safety interface on larger ones—an extension the paper does not pursue.
  • Editorial inference: the read-before-write temporal pattern suggests prototypes act as predictive memory slots; a natural testable extension is whether making the read gate explicitly anticipatory, or adding a separate "planned" channel, improves long-range coherence.
  • Editorial inference: because the authors identify polysemanticity but find it limited, ProtoT may offer a tractable setting for measuring superposition directly, and for testing whether prototype count R controls the concept vocabulary—R=64 showed diminishing perplexity returns, but concept coverage may still grow.
  • Editorial inference: the intervention results for less contrastive concepts like COVID are noisier than the gender contrast, which hints that editing works best when a concept has a semantic opposite; a useful extension would be to design edit protocols that use paired opposite concepts as anchors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Prototype Transformer (ProtoT), an autoregressive language model that replaces standard self-attention with R learnable prototype vectors. Prototypes route past context into R communication channels via a write gate and read back via a read gate, giving linear complexity in sequence length. The authors claim that this structure makes prototypes automatically capture nameable concepts, enables targeted behavioral edits, and provides competitive performance on perplexity, GLUE, and text generation compared with LLaMA, Mamba, and DeltaNet on a 250M-token FineWeb-Edu subset. The interpretability section uses LLM-aided labeling of top-activating sequences, interpretability metrics, prototype re-initialization/masking interventions, and robustness analyses.

Significance. If the claims are substantiated, ProtoT is a meaningful step toward intrinsically interpretable LMs: it is a clean alternative to self-attention with linear computational complexity, and the paper provides extensive ablations, transparent reporting of limitations (e.g., long-context bottleneck, variable cross-seed intervention effects), and a detailed appendix. The paper is also careful to compare against a matched LLaMA backbone and two linear-attention baselines. However, the central interpretability claims currently rest on post-hoc selected prototypes, LLM-judged labels, and intervention tests that are largely confined to the exact sentences used to discover the prototypes. The targeted-edit claim is not yet empirically distinguished from post-hoc cherry-picking, and long-context performance degrades at 2048 tokens. The architecture's practical value therefore depends on strengthening the causal evidence.

major comments (4)
  1. [§5.1 and A.3.1 (Table 6)] The intervention experiments do not establish targeted edits beyond the discovery context. Prototypes L9 P7 and L9 P18 are selected because they top-activate on 'women/girls' sentences from FineWeb, and the test set includes the exact seed sentence from that ranking. In Table 6, the only large effects occur on that seed context: for target 'women', random re-initialization of L9 P7 gives -17.8% on the seed, but the other six contexts give -3.00%, +1.37%, -0.67%, +1.81%, +0.11%, and -1.56%. For L9 P18, the claimed 'consistently increased' effect is +11.50% on the seed but -0.13%, +1.43%, -0.31%, and +0.12% on other contexts. This is evidence of a causal role only for the discovery example, not a general property. A pre-registered protocol with held-out contexts and random-prototype controls is needed to support the paper's targeted-edit claim.
  2. [A.3.2 (Table 7)] The cross-seed COVID experiment contradicts functional specificity. For the same context 'covid - 19 lambda variant...', write-gate masking changes the probability of 'COVID' by +0.5 percentage points (original model), -8.4 pp (seed 124), and +6.3 pp (seed 325); random re-initialization gives +0.3, -8.6, and +4.6 pp. Many other entries are within noise, and the paper concedes the effects 'appear more variable' (A.3.2). The robust emergence of a concept slot across seeds is not evidence that the slot has a stable causal role; a model can allocate a dedicated slot without using it consistently for prediction. The claim that targeted edits are possible for these concepts requires a consistent intervention signature across seeds and contexts, not merely the presence of a slot.
  3. [§5.1 and A.10] The LLM-aided interpretability scores are not validated against human judgments and are computed only on the ten most activating sequences per prototype. The prompt in A.10 provides multiple worked examples of identifying 'themes' and explicitly instructs the evaluator to find a main theme, which biases toward high disentanglement scores. No inter-annotator agreement, no random/null prototype baseline, and no comparison with a labeled concept dataset is provided. Consequently, the statement that 'the vast majority of prototypes' are disentangled is not quantitatively supported. This is a central methodological weakness because the concept-capture claim rests almost entirely on these scores and on visual inspection of selected examples.
  4. [Table 1] The long-context scalability claim is overstated. ProtoT's test perplexity degrades from 80.5 at context 1024 to 81.9 at context 2048, while LLaMA, Mamba, and DeltaNet all improve (e.g., LLaMA 65.3 -> 63.6). The paper acknowledges a bottleneck and attributes it to hidden size and the low-rank value projection, but the abstract and introduction state that ProtoT 'scales well' and 'maintain[s] competitive performance.' With the current evidence, linear-complexity comes with a performance degradation at longer contexts, and the large-scale result (29.5 vs 25.8 for LLaMA) still shows a substantial gap. This limitation should be stated directly, not presented as a success.
minor comments (6)
  1. [Table 3] The row labeled 'ProtoAttn' should be 'ProtoT' for consistency with the rest of the paper.
  2. [Eqs. (1)-(2)] The notation is confusing: 'PM' is used both as a function name and as the prefix-mean object, and the softmax subscripts are ambiguous. Please rewrite the equations with clearer indexing and separate notation for the write and read gates.
  3. [Table 1] Formatting issues: 'LLaMA78.7', 'ProtoT (h=512)', and 'ProtoT (L=12)' lack spaces; align the table for readability.
  4. [Reproducibility statement] The code and intervention benchmark are planned for release only upon acceptance; no URL or permanent artifact is provided. Since the intervention test sentences were hand-constructed and LLM-paraphrased, please release the exact prompts and random seeds used for generation to enable replication.
  5. [A.3.1 and A.10] The main text says the LLM-aided evaluator is GPT-5.1, while the test-corpus generation in A.3.1 uses Gemini 2.5 Pro. Please clarify which LLM is used for which task, and specify whether the same evaluator was used for the LLaMA comparison.
  6. [§4] The training regime is described as 10 epochs, but the large-scale run uses 339k documents; please clarify whether the large-scale model also trains for 10 epochs or for a fixed number of tokens.

Circularity Check

1 steps flagged

Prototype intervention evidence is partially circular: the test set is built from the same top-activating seed sentence used to select the prototype, so the 'female prototype' causal result is a biased re-test rather than an independent prediction.

specific steps
  1. self definitional [Appendix A.3.1 (and Section 5.1)]
    "To create a controlled and relevant test set, we began with a seed sentence identified from our initial visualization analysis. This sentence was the top-ranked example from the FineWeb test set that maximally activated the ‘write‘ gate of our primary target, prototype L9 P7. To expand our test set while maintaining semantic consistency, we then prompted a large language model (Gemini 2.5 Pro) to generate six additional sentences thematically similar to the seed sentence, each required to contain the keywords ‘women‘ and ‘girls‘."

    L9 P7 was labeled a 'female' prototype because it 'consistently activates on sentences containing words such as ‘women‘ and ‘girls‘' (A.3.1). The intervention test set then reuses the exact top-ranked seed sentence from that discovery ranking, plus paraphrases that are required to contain the same keywords. Evaluating the prototype's causal role on data that was used to identify it does not independently test the concept claim: the −17.80%/−10.62% effects appear on the seed-like context, while the other six contexts show small, sign-inconsistent changes (Table 6). This is a selection-dependent re-test, not a held-out prediction.

full rationale

Performance evaluations (GLUE, perplexity, throughput, text generation) are independent benchmarks and support the architecture's capability and scaling claims; they are not circular. The interpretability-by-design claim, however, rests partly on prototype intervention experiments that are partially circular: the prototype is selected by ranking write-gate activations, and the intervention corpus is built from the top-ranked seed sentence plus LLM paraphrases constrained to the same concept keywords. Thus the headline 'female prototype' result re-uses its discovery data. The Appendix's own cross-seed COVID experiment shows sign-inconsistent intervention effects (e.g., +0.5pp, −8.4pp, +6.3pp for the same context across seeds), and the paper concedes effects 'appear more variable', further weakening the targeted-edit generalization; that is an admitted limitation rather than a separate circular step. There is no load-bearing self-citation or imported uniqueness theorem. Because the architecture, scaling, and benchmark results are independent, the circularity is localized to the causal-interpretability validation: score 4.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 1 invented entities

The performance claims rest on standard training assumptions, dataset representativeness, and the chosen hyperparameters. The interpretability claims rest on the validity of LLM-as-judge labeling and on the assumption that prototype re-initialization isolates causal roles. No physically new entities are introduced; the prototype vectors are learned parameters with internal-only evidence.

free parameters (7)
  • Number of prototypes R = 32
    Selected by perplexity-vs-speed ablation (Table 19); central to architecture capacity and interpretability.
  • Local convolution kernel size (layers 0-1) = 5
    Selected by kernel-size ablation (Table 16); needed to stabilize layer-0 routing.
  • Layer-0 read temperature initialization = 3.0
    Selected as a routing regularizer (Table 15); affects layer-0 health.
  • Alpha-gate initialization = 1.0
    Selected by ablation (Table 17); differs from ReZero's 0.0 and is used as a debugging signal.
  • Low-rank value projection ratio = h/2
    Chosen to save compute at the value stream (Table 18); contributes to the context-length bottleneck.
  • Dropout probability = 0.1
    Selected by ablation (Table 20) for all models in the multi-epoch training regime.
  • Per-model peak learning rates = ProtoT 2.0e-3, LLaMA 1.6e-3, Mamba 3.8e-3, DeltaNet 6.8e-3 (adjusted to 2.3e-3 for large-scale Mamba)
    Found via Optuna/BoTorch search (Sec. 4); used for all reported comparisons.
axioms (5)
  • domain assumption Standard transformer training assumptions (AdamW, warmup, cosine schedule, dropout, RMSNorm) are valid and fairness-preserving across baselines.
    Invoked throughout Sec. 4; if optimizer or schedule choices favor one architecture, the comparative claims weaken.
  • domain assumption The FineWeb-Edu 250M-token subset and custom 16k BPE tokenizer are representative for evaluating language modeling and downstream transfer.
    Sec. 4; the paper uses a small, filtered subset and does not validate on standard public benchmarks like C4 or The Pile.
  • domain assumption LLM-as-a-judge Elo scores (Gemma-3-4B-IT) approximate human text-quality preferences.
    Appendix B.1; no human validation or correlation analysis is reported, yet conclusions about generation quality rely on it.
  • domain assumption LLM-aided theme labeling (GPT-5.1) scores disentanglement and coverage faithfully.
    Sec. 5.1 and Appendix A.10; the 'nameable concept' and 'disentanglement' claims rest entirely on this subjective evaluator.
  • domain assumption Re-initializing or masking a prototype channel is a valid causal intervention that isolates that prototype's functional role.
    Appendix A.3; the paper does not control for off-target effects on other prototypes or for interactions beyond the measured probability changes.
invented entities (1)
  • Prototype parameter vectors (P_k) no independent evidence
    purpose: Create R semantic communication channels that aggregate past context and are claimed to capture nameable concepts.
    Prototypes are novel architectural components, but their semantic assignments are demonstrated only through in-paper activation inspection and interventions. There is no external, falsifiable handle outside the paper (e.g., a pre-registered prediction of concept identity before looking at activations).

pith-pipeline@v1.3.0-alltime-deepseek · 32611 in / 12715 out tokens · 123042 ms · 2026-08-02T23:56:35.505023+00:00 · methodology

0 comments
read the original abstract

While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination. We introduce the Prototype Transformer (ProtoT), an autoregressive LM architecture that replaces the quadratic-cost self-attention module of the Transformer with a linear-cost module based on prototypes, which are learned parameter vectors. In ProtoT, prototypes create communication channels that aggregate contextual information at different time scales. We show that this structure leads prototypes to automatically capture nameable concepts, such as "woman", during training, offering a path toward interpreting model reasoning and making targeted edits to model behavior. Compared with baselines, ProtoT scales well with model and data size, is robust to input perturbations, and performs well on text generation and downstream tasks, including GLUE. These results suggest that ProtoT is a promising step toward autoregressive language models that are more interpretable by design.

Figures

Figures reproduced from arXiv: 2602.11852 by Amine M'Charrak, Bayar Menzat, Chang Qi, Markus Kaltenberger, Matteo Forasassi, Ruizhi Wang, Thomas Lukasiewicz, Tommaso Salvatori, Yordan Yordanov.

Figure 1
Figure 1. Figure 1: A single autoregressive step of the ProtoT mixer module. Prototypes [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: histograms of the numerical scores provided by LLM aided interpretability [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: Sequences most strongly activating prototype 11 at layer 0, which encodes the con￾cept of narrative in a scholastic context. Right: Write-gate, read-gate, and memory curves for a sequence that strongly activates prototype 23 at layer 4. Read-gate peaks precede write-gate activa￾tions, spiking on the token immediately before those that trigger write-gate routing. across the most strongly activating se… view at source ↗
Figure 4
Figure 4. Figure 4: Visualization for prototype L9 P7 Rank: #1 | Avg: 0.04695 | Sum: 1.45544 | Perplexity: 108.91024 | Tokens: 31 Full sentence: <bos> family rules and communication our family is getting torn apart by a little 10 - year old boy who has adhd. my biggest problem with him is his arguing Rank: #2 | Avg: 0.04290 | Sum: 1.33002 | Perplexity: 11.95463 | Tokens: 31 Full sentence: <bos> the ten commandments then god s… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization for prototype L9 P18 Rank: #1 | Avg: 0.07597 | Sum: 2.35518 | Perplexity: 17.38632 | Tokens: 31 Full sentence: <bos> to set the record straight, today isn t actually presidents day. it is still known as washington s birthday, according to the federal government and Rank: #2 | Avg: 0.07422 | Sum: 2.30096 | Perplexity: 78.61736 | Tokens: 31 Full sentence: <bos> save on america s first 12 ¢ stam… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization for the control prototype L9 P2 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Two example prototypes illustrating the relationship between half-life and repetition. The [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: ProtoT interpretability metrics across depth. [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: LLaMA interpretability metrics across depth. [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization for prototype L0 P18. Half-life = 12.8 LLM scoring and labeling We provide visualizations of some random selected examples and resulting statistics of the LLM scoring process. We also show ablation for different model configu￾rations, including R=16, R=64, two extra seeds for R=32 and R=32 without low rank projection. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Visualization for prototype L1 P14. Half-life = 13.2 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Visualization for prototype L7 P31. Half-life = 12.7 30 [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Visualization for prototype L8 P5. Half-life = 0.140 [PITH_FULL_IMAGE:figures/full_fig_p031_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Visualization for prototype L10 P8. Half-life = 0.510 31 [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: LLM aided interpretability results for prototype [PITH_FULL_IMAGE:figures/full_fig_p032_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: LLM aided interpretability results for prototype [PITH_FULL_IMAGE:figures/full_fig_p032_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: LLM aided interpretability results for prototype [PITH_FULL_IMAGE:figures/full_fig_p033_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: LLM aided interpretability results for prototype [PITH_FULL_IMAGE:figures/full_fig_p033_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: histograms for LLM aided interpretability for model configuration R=16 [PITH_FULL_IMAGE:figures/full_fig_p033_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: histograms for LLM aided interpretability for model configuration R=32 (S=124) [PITH_FULL_IMAGE:figures/full_fig_p034_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: histograms for LLM aided interpretability for model configuration R=32 (S=135) [PITH_FULL_IMAGE:figures/full_fig_p034_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: histograms for LLM aided interpretability for model configuration R=64 [PITH_FULL_IMAGE:figures/full_fig_p034_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: histograms for LLM aided interpretability for R=32 (S=124) and no low rank projection [PITH_FULL_IMAGE:figures/full_fig_p034_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: histograms for LLM aided interpretability for LLaMA model trained for 1) [PITH_FULL_IMAGE:figures/full_fig_p034_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Full prompt used for LLM aided evaluation and labeling experiment [PITH_FULL_IMAGE:figures/full_fig_p036_25.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Graph Memory Transformer (GMT)

    cs.LG 2026-04 unverdicted novelty 7.0

    Graph Memory Transformer replaces FFN sublayers with a graph memory cell using 128 centroids and transition matrices per block, yielding stable training at 82.2M parameters but higher validation loss than a 103M dense...

  2. Collapse-Free Prototype Readout Layer for Transformer Encoders

    cs.LG 2026-04 unverdicted novelty 7.0

    DDCL-Attention introduces a collapse-free prototype readout for transformers that decomposes the training loss exactly into reconstruction and diversity terms while providing stability guarantees via singular perturba...

  3. Graph Memory Transformer (GMT)

    cs.LG 2026-04 unverdicted novelty 5.0

    Graph Memory Transformer (GMT) swaps dense FFN sublayers for a graph of 128 centroids and a learned 128x128 transition matrix per block, yielding a 82M-parameter decoder-only LM that trains stably but trails a 103M de...

Reference graph

Works this paper leans on

34 extracted references · 25 linked inside Pith · cited by 2 Pith papers

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [5]

    in the”, “of the

    Table 16: Kernel size ablation of the local convolution in ProtoT. Reported best dev perplexity (lower is better), averaged over 3 seeds. Best values are in bold. Variant Kernel size Performance (dev perplexity)↓ ProtoT (k=4)4 98.3 ProtoT (k=5)597.1 ProtoT (k=6)696.9 ProtoT (k=7)7 97.5 Alpha-gate initialization:The results in Table 17 show that 0.8 and 1....

    2020

  3. [6]

    why does my dog eat poo p?

    was selected as a control, as it did not exhibit a clear, gender-coded semantic preference. Test Case Construction.To create a controlled and relevant test set, we began with a seed sen- tence identified from our initial visualization analysis. This sentence was the top-ranked example from the FineWeb test set that maximally activated the ‘write‘ gate of ...

    2016

  4. [7]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, S ¨oren Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models. a...

    Pith/arXiv arXiv

  5. [8]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Pith/arXiv arXiv

  6. [9]

    Sarthak Jain and Byron C. Wallace. Attention is not explanation. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics,

    2019

  7. [10]

    In- terpreting attention layer outputs with sparse autoencoders.arXiv preprint arXiv:2406.17759,

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. In- terpreting attention layer outputs with sparse autoencoders.arXiv preprint arXiv:2406.17759,

    Pith/arXiv arXiv

  8. [12]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    12 Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Pith/arXiv arXiv

  9. [13]

    Revisiting small batch training for deep neural networks.arXiv preprint arXiv:1804.07612,

    Dominic Masters and Carlo Luschi. Revisiting small batch training for deep neural networks.arXiv preprint arXiv:1804.07612,

    Pith/arXiv arXiv

  10. [14]

    Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models.arXiv preprint arXiv:1609.07843,

    Pith/arXiv arXiv

  11. [16]

    A practical review of mecha- nistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646,

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mecha- nistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646,

    arXiv

  12. [17]

    Sentence-BERT: Sentence embeddings using Siamese BERT- networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP),

    2019

  13. [18]

    Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909,

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units.arXiv preprint arXiv:1508.07909,

    Pith/arXiv arXiv

  14. [20]

    BERT rediscovers the classical NLP pipeline.arXiv preprint arXiv:1905.05950,

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline.arXiv preprint arXiv:1905.05950,

    Pith/arXiv arXiv 1905

  15. [21]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023a. Hugo Touvron, Louis Martin...

    Pith/arXiv arXiv

  16. [22]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461,

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding.arXiv preprint arXiv:1804.07461,

    Pith/arXiv arXiv

  17. [24]

    MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers.arXiv preprint arXiv:2002.10957,

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers.arXiv preprint arXiv:2002.10957,

    Pith/arXiv arXiv 2002

  18. [26]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv

  19. [27]

    Image classification at supercomputer scale

    Chris Ying, Sameer Kumar, Dehao Chen, Tao Wang, and Youlong Cheng. Image classification at supercomputer scale. InProceedings of the Systems for Machine Learning Workshop at NeurIPS 2018,

    2018

  20. [28]

    Towards best practices of activation patching in language models: Metrics and methods.arXiv preprint arXiv:2309.16042,

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods.arXiv preprint arXiv:2309.16042,

    Pith/arXiv arXiv

  21. [29]

    Deconstruct- ing what makes a good optimizer for language models.arXiv preprint arXiv:2407.07972,

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstruct- ing what makes a good optimizer for language models.arXiv preprint arXiv:2407.07972,

    Pith/arXiv arXiv

  22. [30]

    to help us produce the prototype interpretability html

    14 A APPENDIX A.1 AI USAGE We have used LLMs for proofreading the paper and to polish writing, for retrieval and discovery of related work, and for low-level coding help, e.g. to help us produce the prototype interpretability html. We have checked all AI output, and have verified that the resulting code is correct and works as expected. A.2 ADDITIONALDETA...

    2018

  23. [34]

    You are analyzing a single prototype (a neuron-like feature) from a neural language model.\n

    34 A.10 PROMPT FORLLM-AIDED INTERPRETABILITY EXPERIMENT "You are analyzing a single prototype (a neuron-like feature) from a neural language model.\n" "For this prototype you are given, for each of its top-ranked sentences, the full sentence and the subset " "of its most activating tokens. Each example is formatted as:\n\n" " Most activating tokens senten...

    1955

  24. [512]

    To avoid leakage, we fine-tune on the official training split, select hyperparameters and checkpoints on the official development split using early stopping, and export test predictions in the official TSV format for submission to the GLUE server. We follow the official GLUE metrics: accuracy for SST-2, QNLI, MNLI, QQP, RTE, and WNLI (or the primary metri...

    2016

  25. [1995]

    Automatically Interpreting Millions of Features in Large Language Models.arXiv preprint arXiv:2410.13928,

    Gonc ¸alo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically Interpreting Millions of Features in Large Language Models.arXiv preprint arXiv:2410.13928,

    Pith/arXiv arXiv

  26. [2015]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202,

    Pith/arXiv arXiv 2002

  27. [2017]

    Jamba: A hybrid transformer- mamba language model.arXiv preprint arXiv:2403.19887,

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model.arXiv preprint arXiv:2403.19887,

    Pith/arXiv arXiv

  28. [2018]

    Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small.arXiv preprint arXiv:2211.00593,

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small.arXiv preprint arXiv:2211.00593,

    Pith/arXiv arXiv

  29. [2019]

    Toy models of superposi- tion.arXiv preprint arXiv:2209.10652,

    11 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion.arXiv preprint arXiv:2209.10652,

    Pith/arXiv arXiv

  30. [2020]

    Cottrell, and Julian McAuley

    Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth.arXiv preprint arXiv:2003.04887,

    Pith/arXiv arXiv 2003

  31. [2022]

    Dissecting recall of factual associations in auto-regressive language models.arXiv preprint arXiv:2304.14767,

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models.arXiv preprint arXiv:2304.14767,

    Pith/arXiv arXiv

  32. [2023]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  33. [2024]

    What does BERT look at? An analysis of BERT’s attention.arXiv preprint arXiv:1906.04341,

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? An analysis of BERT’s attention.arXiv preprint arXiv:1906.04341,

    Pith/arXiv arXiv 1906

  34. [2025]

    A transformer and prototype-based interpretable model for contextual sarcasm detection.arXiv preprint arXiv:2503.11838,

    Ximing Wen and Rezvaneh Rezapour. A transformer and prototype-based interpretable model for contextual sarcasm detection.arXiv preprint arXiv:2503.11838,

    Pith/arXiv arXiv