Pith. sign in

REVIEW 4 major objections 6 minor 20 references

The paper claims that SQS, a signed and shrunk quadratic activation for gated linear units, preserves the bilinear weight spectrum that makes eigenvector interpretability possible while matching state-of-the-art performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SQS is a new GLU activation that reportedly preserves bilinear weight spectra for interpretability while achieving competitive performance on MNIST, FMNIST, and Tiny Stories.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection SQS is a new, plausibly competitive activation, but the paper never defines the matrix it eigen-decomposes, so the central interpretability claim is unsupported. the 4 major comments →

arxiv 2509.01874 v1 pith:57ANZWSS submitted 2025-09-02 cs.LG cs.AI

Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function

Jason Abohwo , Thomas Mosen This is my paper
classification cs.LG cs.AI
keywords Signed Quadratic ShrinkGated Linear Unitsbilinear MLPweight-based interpretabilityeigendecompositionmechanistic interpretabilityactivation functions
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Signed Quadratic Shrink (SQS), an activation function for gated linear units designed to give neural networks the weight-based interpretability of bilinear MLPs without the performance and data-efficiency costs that bilinear layers incur. It claims that SQS makes the quadratic-form structure of a bilinear layer—where each logit reads as x^T A x and can be decomposed into weighted eigen-directions—survive in a nonlinear, trainable gate. On MNIST, Fashion-MNIST, and TinyStories, SQS matches the loss and convergence of GELU and SwiGLU, beats ReLU and bilinear layers, and produces eigenvectors with high cosine similarity to bilinear eigenvectors. If right, it means interpretability need not be traded away when switching to competitive activations.

Core claim

On the paper's terms: SQS is defined as σ(x) = (x/|x|)·(|x| − c)/(1 + (λ|x|)^p)^(1/p), which for p = 1 becomes (x − c·sgn(x))/(1 + λ|x|). This replaces the identity gate of a bilinear GLU with a signed, shifted, shrinking quadratic that is quasi-linear for very small and very large inputs. The discovery claim is that the bilinear layer's interpretability mechanism—each output logit computed as x^T A x and decomposable into weighted eigen-directions—survives inside this nonlinear gate, so the trained model's weights can be read as class-specific eigenfeatures. The experimental evidence is that SQS-GLU eigenvectors are highly similar to those of a bilinear MLP (cosine similarity above 0.5, nea

What carries the argument

The central object is the signed, shifted, shrunk quadratic activation SQS. With p = 1 it reduces to σ(x) = (x − c·sgn(x))/(1 + λ|x|): quasi-linear for very small and very large inputs, approximately quadratic in between while retaining sign. Inside a GLU, (Wx)⊙σ(Vx), it keeps each logit close to a sum of squared projections along eigenvectors, so an interaction matrix A extracted from the trained weights retains the spectral structure of a bilinear layer. The shrink term bounds large activations and the signed shift avoids the flat gradients of a pure quadratic; hyperparameters c and λ control the shift and shrinkage, and the paper sets λ = 0.5, c = 0.01, p = 1 for its experiments.

Load-bearing premise

The load-bearing premise is that a usable, meaningful interaction matrix can be extracted from a trained SQS-GLU by treating it like a bilinear layer, even though the SQS gate is nonlinear and input-dependent; the paper does not specify how this extraction is done.

What would settle it

Run an SQS-GLU and a bilinear MLP on the same data, extract each logit's interaction matrix, then rank each logit by how much dropping its least-important eigenvectors changes classification decisions; if SQS-GLU's top eigenvectors do not carry the class information, the weight-spectra interpretability claim is refuted. Alternatively, if SQS-to-bilinear eigenvector cosine similarity drops consistently below about 0.5 across many seeds, the claim that SQS preserves bilinear spectra fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Weight-based mechanistic interpretability becomes available in a performant GLU, without the activation-space machinery of sparse-autoencoder analyses.
  • SQS-GLU reaches competitive loss and perplexity on language modeling with TinyStories, offering a drop-in transformer activation that keeps training speed comparable to ReLU, GELU, and SwiGLU.
  • On vision benchmarks, SQS converges faster than ReLU-GLU and bilinear MLPs, being first to reach 80%, 85%, and 90% accuracy.
  • The eigenvectors of an SQS-GLU can be used to explain individual classification decisions, inheriting the bilinear layer's decomposition into ranked eigen-directions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A formal, published definition of how the interaction matrix is extracted from a trained SQS-GLU would let others verify the eigenvector-similarity result without reimplementing the paper's undocumented procedure in Section 4.1.
  • Because SQS's shrink term bounds large activations, it likely acts as an implicit regularizer; one testable extension is measuring weight spectra and generalization under lower weight decay to see whether λ can substitute for part of explicit regularization.
  • The p parameter, kept at 1 for speed, could interpolate between a smooth quadratic and a near-linear gate, potentially exposing a dataset-specific interpretability–performance trade-off curve.
  • A testable extension is applying SQS to transformer MLP blocks and checking whether the eigenvectors of the combined weight matrix correspond to interpretable features such as syntactic roles in generated stories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Signed Quadratic Shrink (SQS), an activation function intended for use inside Gated Linear Units (GLUs). The central claim is that SQS-GLUs achieve performance competitive with standard activation functions such as SwiGLU and GELU while preserving the weight-based interpretability properties of bilinear MLPs, whose logits can be written as x^T A_a x and analyzed via eigendecomposition. Experiments on MNIST, Fashion-MNIST, and Tiny Stories compare SQS-GLU with ReLU-GLU, GELU-GLU, SwiGLU, and Bilinear GLU. The paper reports qualitative eigenvector visualizations for SQS-GLU, cosine similarities between SQS-GLU and bilinear MLP eigenvectors, and loss/accuracy/perplexity numbers. The main contribution is the SQS activation itself and the claim that it reconciles performance with interpretability.

Significance. If the central claim were established, the paper would be a meaningful contribution: designing a nonlinear activation that preserves the bilinear spectral structure would allow weight-based mechanistic interpretability in high-performing GLU architectures, addressing a known limitation of bilinear MLPs. The idea of modifying a quadratic form with a shrink factor to obtain a practical activation is interesting, and the paper provides a concrete, cheaply computable activation. However, the paper's core evidence for interpretability is not currently valid: no procedure is given to map a trained SQS-GLU to an interaction matrix, the hyperparameters underlying the interpretability figures were selected using the very same interpretability criterion, and the derivation of the activation is internally inconsistent. As such, the significance is currently speculative rather than demonstrated.

major comments (4)
  1. [Section 4.1] The paper's central interpretability claim is unsupported because the interaction matrix for an SQS-GLU is never defined. For a bilinear MLP, g_a(x)=(W_a^T x)(V_a^T x)=x^T A_a x with A_a symmetric, so eigendecomposition is immediate. For SQS-GLU, g_a(x)=(W_a^T x)σ(V_a^T x), where σ(v) is nonlinear and input-dependent (Algorithm 1). No fixed symmetric matrix A_a satisfies g_a(x)=x^T A_a x for all x. The text says 'we follow the same procedure' as Pearce et al., but for bilinear layers that procedure is unambiguous, whereas for SQS-GLU it is not. Without a concrete definition of how A_a is extracted from the trained weights, the eigenvectors in Figure 2 and Appendix B, and the cosine similarity comparisons, have no clear meaning. This is the load-bearing gap for the paper's main claim.
  2. [Section 3.1 and Algorithm 1] The activation is not derived consistently with the stated motivation. The paper begins with a quadratic activation g(x)=((|x|+c)^2-c^2)/(1+(λ|x|)^p)^{1/p}, then states that factoring out |x| and adding directional information yields σ(x)=sign(x)(|x|-c)/(1+(λ|x|)^p)^{1/p}. Factoring |x| from the numerator would produce |x|(|x|+2c), not |x|-c. Algorithm 1 implements y=(x-c*s)/(1+λ*x*s), which is the p=1 case of the latter, not of the derived quadratic. Moreover, the claim in the text that σ is 'quasi-linear for |x|<<1 and |x|>10' is false for the implemented function: for large |x|, σ(x) tends to sign(x)/λ (a constant), and at x=0 there is a discontinuity (σ(0)=-c if sign(0)=1). This undermines the theoretical motivation for why SQS should preserve bilinear structure.
  3. [Section 4] The hyperparameter selection is circular with respect to the interpretability claim. The paper states that λ=0.5, c=0.01 were chosen because they 'yield the most interpretable eigenvectors whilst maintaining performance,' and the same runs are then used to report the eigenvector interpretability and cosine similarity with bilinear MLPs. Since interpretability is assessed visually and by similarity to bilinear eigenvectors—criteria that were used to pick hyperparameters—the results in Figure 2 and Section 4.1 are selected on the outcome variable. A valid evaluation would require a separate validation set for hyperparameter selection and a pre-specified interpretability metric, or at least a demonstration that results are robust across a range of hyperparameters.
  4. [Table 1 and Figure 3] The performance claims are weaker than the text suggests. On MNIST and FMNIST, SQS final loss is higher than GELU and SwiGLU (e.g., MNIST 0.0834 vs 0.0680 for GELU, 0.0728 for SwiGLU), and SQS only clearly outperforms ReLU and Bilinear MLPs. On Tiny Stories, SQS's advantages are tiny (e.g., final loss 1.9020 vs 1.9021 for ReLU) and no error bars, number of seeds, or significance tests are reported for Tiny Stories, unlike MNIST/FMNIST where 5-run averages are stated. The claim that SQS is 'first to reach 80%, 85%, and 90% accuracy' is only supported by Figure 3, not by a quantitative table. For a new activation function, the performance evidence is insufficient to support the stated competitiveness.
minor comments (6)
  1. [Section 3.1, equation for GLU] The formula (Wx)⊙σ(Vx) = x/||x||· Wx ⊙ (|Vx|−c(Wx))/(...) appears to be a typographical error: it multiplies Wx by x/||x|| and subtracts c(Wx) in the numerator, which does not match the scalar definition of σ. It should be (Wx)⊙( sign(Vx)⊙(|Vx|-c)/(1+λ|Vx|) ) for p=1.
  2. [Algorithm 1] The comment 'parameter p is defined but not used in the forward pass' indicates that Algorithm 1 only implements p=1. If p is intended to be a general hyperparameter, an algorithm for p>1 should be provided, and the derivative for backpropagation should be stated. Also, using sign(0)=1 creates a discontinuity at 0; the implications for training and for the claim of quasi-linearity should be discussed.
  3. [Section 4.1] There is a malformed citation 'Bilinear MLP [2? ]' in the text. Several figure references are duplicated or misnumbered: 'Figures 4 5 7 7 8' in Appendix B, and Figure 9 has only the caption 'Caption' with no actual caption text.
  4. [Appendix A] Table 2 does not list the model dimension or embedding size for the Section 4.1 experiments, despite these being essential for reproducing the eigenvector visualizations. Table 3 lists model dim only for Section 4.2; please clarify whether Section 4.1 uses the same architecture.
  5. [Figure 2] Cosine similarity values are reported as averages over 5 runs but no error bars or standard deviations are shown, nor is it explained how eigenvectors are aligned across runs or across architectures. Without this, the claim that similarity is 'never below .5' is not well supported.
  6. [Abstract and Conclusion] The paper claims 'weight-based interpretability' as a general property, but interpretability is only evaluated on MNIST and Fashion-MNIST; no interpretability analysis is presented for the Tiny Stories language model, so the broader claim is overreaching.

Circularity Check

2 steps flagged

Interpretability claim is partly circular: λ/c are tuned to maximize eigenvector interpretability, and the interaction-matrix extraction is undefined (or bilinear by default), so 'preserving bilinear weight spectra' is either tautological or unverified.

specific steps
  1. fitted input called prediction [Section 4, Experiments (first paragraph) and Section 4.1]
    "For all uses of SQS, we setλ =.5, c =.01 (as we find that this yields the most interpretable eigenvectors whilst maintaining performance on par with current activation functions), andp = 1 (for computational efficiency."

    The hyperparameters λ and c are explicitly chosen to maximize the interpretability of the eigenvectors. The paper then uses models trained with these selected hyperparameters to report the central claim that 'SQS-GLU facilitates the learning of interpretable features as revealed through eigenvector decomposition'. The interpretability result is therefore the selection criterion restated as a finding, not an independent prediction; it is statistically forced by the hyperparameter choice.

  2. other [Section 4.1, Figure 2 caption; Appendix B.0.1]
    "we follow the same procedure elucidated by Pearce et al. [2]. More specifically, we analyze a shallow Feed-Forward network consisting of an embedding projection, an SQS-GLU, and an output projection ... Figure 2 displays the top eigenvectors for five classes in both datasets. ... Following the methodology of Pearce et al. [2], we compute the eigen-spectrum of the interaction matrices extracted from the SQS-GLU layer."

    For bilinear MLPs, the procedure defines an interaction matrix A_a = sym(W_a V_a^T) because the logit is exactly x^T A_a x. For SQS-GLU the logit is (W_a^T x) σ(V_a^T x) with nonlinear σ, so no exact interaction matrix exists. The paper never defines how the interaction matrix is extracted from the SQS-GLU layer. If the same bilinear formula is used, then the 'SQS-GLU eigenvectors' are by construction the bilinear eigenvectors of the weight outer product, making the claimed preservation of bilinear weight spectra tautological; if a different formula is used, the claim is unverifiable. In either case the central claim does not rest on a demonstrated property of the SQS function.

full rationale

The performance comparison (Section 4.2, Table 1) is self-contained against external baselines and is not circular. However, the paper's central novel claim—that SQS enables weight-based interpretability by preserving bilinear weight spectra—is substantially circular or underdefined. First, the hyperparameters λ=.5, c=.01 are selected specifically to yield 'the most interpretable eigenvectors'; the same runs are then presented as evidence that SQS-GLU learns interpretable eigenfeatures. Thus the interpretability result is an artifact of the selection process, not an independent empirical finding. Second, the paper never defines the interaction matrix whose eigenvectors are plotted for SQS-GLU. The bilinear MLP has a unique, well-defined interaction matrix A_a = sym(W_a V_a^T) because its logit is a quadratic form; SQS-GLU's logit contains a nonlinear gate σ, so no such matrix exists. If the authors reuse the bilinear extraction, then the eigenvectors are by construction those of a bilinear approximation, making the 'preservation' claim true by definition rather than by the network's behavior; if they use a different extraction, the procedure is omitted. The references to Pearce et al. and Sharkey are external, not self-citations, and do not themselves create circularity. The performance results and speed measurements remain independent, but the interpretability claim—the paper's raison d'être—reduces to a fitted selection and an undefined extraction, warranting a score of 6.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The central claim rests on the unstated possibility of extracting a meaningful interaction matrix from a nonlinear gate, on a hyperparameter choice made after seeing the results, and on a qualitative description of the activation that the paper's own algorithm contradicts.

free parameters (3)
  • λ = 0.5
    Chosen to maximize interpretability and performance on MNIST/FMNIST; no held-out validation reported (Section 4).
  • c = 0.01
    Same as λ: selected by hand to yield interpretable eigenvectors while keeping performance (Section 4).
  • p = 1
    Set to 1 for computational efficiency; the forward pass does not use p (Algorithm 1 note).
axioms (3)
  • domain assumption Bilinear interaction matrices' eigendecomposition yields interpretable features.
    Inherited from Pearce et al. [2]; assumed valid for the comparison baseline.
  • ad hoc to paper An interaction matrix can be assigned to an SQS-GLU.
    The gate σ(Vx) depends on x, so (Wx)⊙σ(Vx) is not a quadratic form; the paper gives no construction of a single matrix.
  • ad hoc to paper SQS is quasi-linear for small and large |x|.
    Stated in Section 3.1 but contradicted by Algorithm 1, where the function jumps at 0 and saturates at ±1/λ.
invented entities (1)
  • Signed Quadratic Shrink (SQS) activation function no independent evidence
    purpose: Enable GLUs to learn interpretable weight-based features while matching performance of standard activations.
    New function proposed in this paper; no external evidence that it preserves bilinear spectra; its specification is ambiguous due to malformed equations.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function." pith.science (2026). https://pith.science/paper/57ANZWSS

@misc{pith2026250901874,
  author       = {Pith},
  title        = {Pith review of: Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/57ANZWSS}},
  note         = {Machine review of arXiv:2509.01874}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding the inner workings of machine learning models is critical for ensuring their reliability and robustness. Whilst many techniques in mechanistic interpretability focus on activation driven analyses, being able to derive meaningful features directly from the weights of a neural network would provide greater guarantees and more computational efficiency. Existing techniques for analyzing model features through weights suffer from drawbacks such as reduced performance and data inefficiency. In this paper, we introduce Signed Quadratic Shrink (SQS), an activation function designed to allow Gated Linear Units (GLUs) to learn interpretable features without these drawbacks. Our experimental results show that SQS achieves performance competitive with state-of-the-art activation functions whilst enabling weight-based interpretability

Figures

Figures reproduced from arXiv: 2509.01874 by Jason Abohwo, Thomas Mosen.

Figure 1
Figure 1. Figure 1: ReLU [5], GeLU[4], Swish [10] SQS (p = 1, λ = .5, c = .5) functions. We include x · SQS to provide intuition for the use of SQS within a GLU[3]. As noted by Pearce et al.[2], a given output logit of a bilinear layer can be formulated as g(x)a = (x T Aax) = Pd i λi(v T i x) 2 . With this, regular MLPs can be seen as a special case in which λ and v are shared between output logits. This then reduces the bili… view at source ↗
Figure 2
Figure 2. Figure 2: A) On the Top Eigenvectors after Eigendecomposition on each output dimension of the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MNIST Test loss and accuracy over time. Note that log is applied to both graphs, to test [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Eigen-spectrum for the 5 logit of the Signed Quadratic Shrink-GLU trained on MNIST [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Eigen-spectrum for the 6 logit of the Signed Quadratic Shrink-GLU trained on MNIST [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Eigen-spectrum for the 7 logit of the Signed Quadratic Shrink-GLU trained on MNIST [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Eigen-spectrum for the 8 logit of the Signed Quadratic Shrink-GLU trained on MNIST [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Eigen-spectrum for the 9 logit of the Signed Quadratic Shrink-GLU trained on MNIST [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Caption 11 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    A technical note on bilinear layers for interpretability

    Lee Sharkey. A technical note on bilinear layers for interpretability. 2023. arXiv:2305.03452 [cs.LG]. https://arxiv.org/abs/2305.03452. 4 Figure 3: MNIST Test loss and accuracy over time. Note that log is applied to both graphs, to test loss on the left and to iteration on the right, and that "None" is the Bilinear MLP Table 1: Performance of various act...

    Pith/arXiv arXiv 2023

  2. [2]

    Pearce, Thomas Dooms, Alice Rigg, Jose M

    Michael T. Pearce, Thomas Dooms, Alice Rigg, Jose M. Oramas, and Lee Sharkey. Bilinear MLPs enable weight-based mechanistic interpretability. 2024. arXiv:2410.08417 [cs.LG].https: //arxiv.org/abs/2410.08417

    Pith/arXiv arXiv 2024

  3. [3]

    Dauphin, Angela Fan, Michael Auli, and David Grangier

    Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolutional Networks. 2017. arXiv:1612.08083 [cs.CL].https://arxiv.org/abs/ 1612.08083

    Pith/arXiv arXiv 2017

  4. [4]

    Gaussian Error Linear Units (GELUs)

    Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). 2023. arXiv:1606.08415 [cs.LG]. https://arxiv.org/abs/1606.08415

    Pith/arXiv arXiv 2023

  5. [5]

    Deep Learning using Rectified Linear Units (ReLU)

    Abien Fred Agarap. Deep Learning using Rectified Linear Units (ReLU). 2019. arXiv:1803.08375 [cs.NE]. https://arxiv.org/abs/1803.08375

    Pith/arXiv arXiv 2019

  6. [6]

    The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

    Li Deng. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]. IEEE Signal Processing Magazine , 29(6):141–142, 2012. doi:10.1109/MSP.2012.2211477

    work page arXiv 2012

  7. [7]

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. 2017. arXiv:1708.07747 [cs.LG]. https: //arxiv.org/abs/1708.07747

    Pith/arXiv arXiv 2017

  8. [8]

    TinyStories: How Small Can Language Models Be and Still Speak Coherent English? 2023

    Ronen Eldan and Yuanzhi Li. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? 2023. arXiv:2305.07759 [cs.CL].https://arxiv.org/abs/2305.07759

    Pith/arXiv arXiv 2023

  9. [9]

    Sparse Autoencoders Find Highly Interpretable Features in Language Models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. 2023. arXiv:2309.08600 [cs.LG]. https://arxiv.org/abs/2309.08600

    Pith/arXiv arXiv 2023

  10. [10]

    Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions. 2017. arXiv:1710.05941 [cs.NE]. https://arxiv.org/abs/1710.05941

    Pith/arXiv arXiv 2017

  11. [11]

    GLU Variants Improve Transformer

    Noam Shazeer. GLU Variants Improve Transformer. 2020. arXiv:2002.05202 [cs.LG].https: //arxiv.org/abs/2002.05202

    Pith/arXiv arXiv 2020

  12. [12]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. 2024. arXiv:2406.04093 [cs.LG]. https://arxiv.org/abs/2406.04093

    Pith/arXiv arXiv 2024

  13. [13]

    Interpreting Attention Layer Outputs with Sparse Autoencoders

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting Attention Layer Outputs with Sparse Autoencoders. 2024. arXiv:2406.17759 [cs.LG]. https://arxiv.org/abs/2406.17759

    Pith/arXiv arXiv 2024

  14. [14]

    Modular Networks: Learning to Decompose Neural Computation

    Louis Kirsch, Julius Kunze, and David Barber. Modular Networks: Learning to Decompose Neural Computation. 2018. arXiv:1811.05249 [cs.LG].https://arxiv.org/abs/1811.05249

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  15. [15]

    Gradient Routing: Masking Gradients to Localize Computation in Neural Networks

    Alex Cloud, Jacob Goldman-Wetzler, Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient Routing: Masking Gradients to Localize Computation in Neural Networks

    work page

  16. [16]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. 2023. arXiv:2307.09288 [cs.CL]. https://arxiv.org/abs/2307.09288. 6

    Pith/arXiv arXiv 2023

  17. [17]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. 2019. arXiv:1711.05101 [cs.LG]. https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  18. [18]

    SGDR: Stochastic Gradient Descent with Warm Restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts

    work page

  19. [2017]

    what is that man doing here?

    arXiv:1608.03983 [cs.LG].https://arxiv.org/abs/1608.03983. A Experimental Details This section details the experimental configurations used in our evaluation of Signed Quadratic Shrink (SQS). A.1 Datasets • MNIST: Dataset of 70,000 grayscale images (60,000 train, 10,000 test) of handwritten digits (0-9) • Fashion MNIST: Dataset of 70,000 grayscale images ...

    Pith/arXiv arXiv 2048

  20. [2024]

    arXiv:2410.04332 [cs.LG].https://arxiv.org/abs/2410.04332

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.