{"id":"ecc79908-c784-4366-989f-576a3452f4a7","arxiv_id":"2509.01874","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"SQS is a new GLU activation that reportedly preserves bilinear weight spectra for interpretability while achieving competitive performance on MNIST, FMNIST, and Tiny Stories.","lead":"A new activation function called Signed Quadratic Shrink (SQS) is proposed for Gated Linear Units. It claims to make neural network weights interpretable while matching standard performance, but the supporting evidence is incomplete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SQS-GLU's nonlinear gate breaks the bilinear form; Section 4.1 never defines the interaction matrix whose eigenvectors are plotted, so the weight-spectrum interpretability claim is unsupported.","rationale":"The paper's central claim is that SQS-GLU preserves the weight-spectrum interpretability of bilinear MLPs while matching state-of-the-art performance. The performance part is plausible but not the decisive issue. The decisive issue is the interpretability mechanism: for a bilinear layer, the output logit is a quadratic form whose symmetric matrix has a canonical eigen-decomposition; for SQS-GLU, the nonlinear gate means the logit is not a quadratic form for any fixed matrix. Section 4.1 reports eigenvectors of 'interaction matrices' from SQS-GLU without defining how those matrices are obtained. This is not a minor missing detail: without it, Figure 2 and the cosine-similarity comparison to bilinear MLPs have no clear mathematical object, and the paper's central claim is unverifiable. The proposed R^2 test directly checks whether a candidate A_a actually reproduces the SQS-GLU function; if the authors can specify A_a and pass such a test, the claim could be rehabilitated, but as presented the manuscript does not provide the necessary definition. Supporting inconsistencies (e.g., Algorithm 1 notes p is unused, and some figure/table references are incomplete) reinforce the impression of an unfinished manuscript, but the missing extraction procedure is the load-bearing concern. I therefore agree with the reader's REJECT verdict: the central claim is not currently supported, not merely under-optimized.","tokens_in":7138,"tokens_out":5024,"duration_ms":55764,"concrete_test":"Request the exact construction of A_a used for Figure 2, then evaluate reconstruction fidelity on a trained SQS-GLU: define A_a as specified and compute R^2 = 1 - sum_x (g_a(x) - x^T A_a x)^2 / sum_x (g_a(x) - mean(g_a))^2 over held-out inputs. If R^2 is close to 1, the bilinear approximation is validated; if R^2 is appreciably below 1, or if no A_a is specified, the eigenvectors in Figure 2 do not characterize the SQS-GLU function and the central claim fails. A useful baseline is A_a = (W_a V_a^T + V_a W_a^T)/2, the matrix the bilinear procedure would produce.","verdict_should_be":"REJECT","load_bearing_attack":"Bilinear MLP logits are exact quadratic forms: g_a(x) = (W_a^T x)(V_a^T x) = x^T A_a x with symmetric A_a, so eigen-decomposing A_a is meaningful. For SQS-GLU, g_a(x) = (W_a^T x) σ(V_a^T x), where Algorithm 1 defines σ(v) = sign(v)(|v| - c) / (1 + λ|v|). Because σ is nonlinear and not even approximately affine over the operating range (e.g., at v=1, σ ≈ 0.66; at v=10, σ ≈ 1.67), no fixed matrix A_a satisfies g_a(x) = x^T A_a x for all inputs. The eigen-decomposition reported in Figure 2 and Section 4.1 therefore requires an extraction procedure that maps trained SQS-GLU weights to some A_a, but no such procedure is given. The text only says it 'follows the same procedure' as Pearce et al.; for bilinear layers that procedure is immediate, but for SQS-GLU it is not. Without a definition of A_a, the claim that SQS 'preserves bilinear weight spectra' and enables weight-based interpretability is not a demonstrated result. The additional statement in Section 4 that λ=.5, c=.01 yields 'the most interpretable eigenvectors' is circular absent a formal interpretability metric, but the missing matrix definition is the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Signed Quadratic Shrink (SQS), an activation function intended for use inside Gated Linear Units (GLUs). The central claim is that SQS-GLUs achieve performance competitive with standard activation functions such as SwiGLU and GELU while preserving the weight-based interpretability properties of bilinear MLPs, whose logits can be written as x^T A_a x and analyzed via eigendecomposition. Experiments on MNIST, Fashion-MNIST, and Tiny Stories compare SQS-GLU with ReLU-GLU, GELU-GLU, SwiGLU, and Bilinear GLU. The paper reports qualitative eigenvector visualizations for SQS-GLU, cosine similarities between SQS-GLU and bilinear MLP eigenvectors, and loss/accuracy/perplexity numbers. The main contribution is the SQS activation itself and the claim that it reconciles performance with interpretability.","tokens_in":7544,"tokens_out":7053,"duration_ms":73170,"significance":"If the central claim were established, the paper would be a meaningful contribution: designing a nonlinear activation that preserves the bilinear spectral structure would allow weight-based mechanistic interpretability in high-performing GLU architectures, addressing a known limitation of bilinear MLPs. The idea of modifying a quadratic form with a shrink factor to obtain a practical activation is interesting, and the paper provides a concrete, cheaply computable activation. However, the paper's core evidence for interpretability is not currently valid: no procedure is given to map a trained SQS-GLU to an interaction matrix, the hyperparameters underlying the interpretability figures were selected using the very same interpretability criterion, and the derivation of the activation is internally inconsistent. As such, the significance is currently speculative rather than demonstrated.","major_comments":[{"comment":"The paper's central interpretability claim is unsupported because the interaction matrix for an SQS-GLU is never defined. For a bilinear MLP, g_a(x)=(W_a^T x)(V_a^T x)=x^T A_a x with A_a symmetric, so eigendecomposition is immediate. For SQS-GLU, g_a(x)=(W_a^T x)σ(V_a^T x), where σ(v) is nonlinear and input-dependent (Algorithm 1). No fixed symmetric matrix A_a satisfies g_a(x)=x^T A_a x for all x. The text says 'we follow the same procedure' as Pearce et al., but for bilinear layers that procedure is unambiguous, whereas for SQS-GLU it is not. Without a concrete definition of how A_a is extracted from the trained weights, the eigenvectors in Figure 2 and Appendix B, and the cosine similarity comparisons, have no clear meaning. This is the load-bearing gap for the paper's main claim.","section":"Section 4.1"},{"comment":"The activation is not derived consistently with the stated motivation. The paper begins with a quadratic activation g(x)=((|x|+c)^2-c^2)/(1+(λ|x|)^p)^{1/p}, then states that factoring out |x| and adding directional information yields σ(x)=sign(x)(|x|-c)/(1+(λ|x|)^p)^{1/p}. Factoring |x| from the numerator would produce |x|(|x|+2c), not |x|-c. Algorithm 1 implements y=(x-c*s)/(1+λ*x*s), which is the p=1 case of the latter, not of the derived quadratic. Moreover, the claim in the text that σ is 'quasi-linear for |x|<<1 and |x|>10' is false for the implemented function: for large |x|, σ(x) tends to sign(x)/λ (a constant), and at x=0 there is a discontinuity (σ(0)=-c if sign(0)=1). This undermines the theoretical motivation for why SQS should preserve bilinear structure.","section":"Section 3.1 and Algorithm 1"},{"comment":"The hyperparameter selection is circular with respect to the interpretability claim. The paper states that λ=0.5, c=0.01 were chosen because they 'yield the most interpretable eigenvectors whilst maintaining performance,' and the same runs are then used to report the eigenvector interpretability and cosine similarity with bilinear MLPs. Since interpretability is assessed visually and by similarity to bilinear eigenvectors—criteria that were used to pick hyperparameters—the results in Figure 2 and Section 4.1 are selected on the outcome variable. A valid evaluation would require a separate validation set for hyperparameter selection and a pre-specified interpretability metric, or at least a demonstration that results are robust across a range of hyperparameters.","section":"Section 4"},{"comment":"The performance claims are weaker than the text suggests. On MNIST and FMNIST, SQS final loss is higher than GELU and SwiGLU (e.g., MNIST 0.0834 vs 0.0680 for GELU, 0.0728 for SwiGLU), and SQS only clearly outperforms ReLU and Bilinear MLPs. On Tiny Stories, SQS's advantages are tiny (e.g., final loss 1.9020 vs 1.9021 for ReLU) and no error bars, number of seeds, or significance tests are reported for Tiny Stories, unlike MNIST/FMNIST where 5-run averages are stated. The claim that SQS is 'first to reach 80%, 85%, and 90% accuracy' is only supported by Figure 3, not by a quantitative table. For a new activation function, the performance evidence is insufficient to support the stated competitiveness.","section":"Table 1 and Figure 3"}],"minor_comments":[{"comment":"The formula (Wx)⊙σ(Vx) = x/||x||· Wx ⊙ (|Vx|−c(Wx))/(...) appears to be a typographical error: it multiplies Wx by x/||x|| and subtracts c(Wx) in the numerator, which does not match the scalar definition of σ. It should be (Wx)⊙( sign(Vx)⊙(|Vx|-c)/(1+λ|Vx|) ) for p=1.","section":"Section 3.1, equation for GLU"},{"comment":"The comment 'parameter p is defined but not used in the forward pass' indicates that Algorithm 1 only implements p=1. If p is intended to be a general hyperparameter, an algorithm for p>1 should be provided, and the derivative for backpropagation should be stated. Also, using sign(0)=1 creates a discontinuity at 0; the implications for training and for the claim of quasi-linearity should be discussed.","section":"Algorithm 1"},{"comment":"There is a malformed citation 'Bilinear MLP [2? ]' in the text. Several figure references are duplicated or misnumbered: 'Figures 4 5 7 7 8' in Appendix B, and Figure 9 has only the caption 'Caption' with no actual caption text.","section":"Section 4.1"},{"comment":"Table 2 does not list the model dimension or embedding size for the Section 4.1 experiments, despite these being essential for reproducing the eigenvector visualizations. Table 3 lists model dim only for Section 4.2; please clarify whether Section 4.1 uses the same architecture.","section":"Appendix A"},{"comment":"Cosine similarity values are reported as averages over 5 runs but no error bars or standard deviations are shown, nor is it explained how eigenvectors are aligned across runs or across architectures. Without this, the claim that similarity is 'never below .5' is not well supported.","section":"Figure 2"},{"comment":"The paper claims 'weight-based interpretability' as a general property, but interpretability is only evaluated on MNIST and Fashion-MNIST; no interpretability analysis is presented for the Tiny Stories language model, so the broader claim is overreaching.","section":"Abstract and Conclusion"}],"recommendation":"reject","confidential_remarks":"The core issue is not a minor omission but a missing definition of the very object that the paper claims to analyze. Without an extraction procedure for the interaction matrix of a nonlinear-gated GLU, the eigenvector plots and cosine similarities are not meaningful, and the hyperparameter selection compounds the problem. The definitional inconsistency between the quadratic derivation and the implemented activation further undermines the paper's motivation. While a revised version could potentially address some of these points, the interpretability claim would require a substantially new analysis, and the performance advantage is marginal. I cannot recommend acceptance or minor revision in the current state."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2509.01874. The SQS activation is a genuinely new element-wise function, and on the small benchmarks reported it performs competitively with GELU/SwiGLU at the end of training. The second thing is that the central claim—that SQS-GLU preserves the bilinear weight spectrum that makes Bilinear MLPs interpretable—is asserted, not shown. The authors never define how a fixed interaction matrix can be extracted from a GLU whose gate is nonlinear and input-dependent. For a true bilinear layer, the matrix A_a = (W_a V_a^T + V_a W_a^T)/2 is immediate. For SQS, no such matrix exists in general, because the logit is x^T W σ(V x) with a nonlinear σ. The paper says it 'follows the same procedure' as Pearce et al., but that procedure only applies when the logit is already a quadratic form. Without a definition of the extraction, the eigenvectors in Figure 2 and the cosine similarities are not interpretable as evidence.\n\nThere are also internal inconsistencies. The text says SQS is quasi-linear for small and large |x|, but the implemented function σ(x) = sign(x)(|x|-c)/(1+λ|x|) is a step near zero and saturates to 1/λ for large arguments—neither regime is linear. Algorithm 1 defines a parameter p that is unused in the forward pass. The hyperparameters λ=0.5, c=0.01 were explicitly chosen to maximize interpretability on the same runs later reported, which makes the interpretability result circular. Baselines are not tuned, and the paper provides no error bars, code, or data.\n\nTo give credit where it is due: the authors ran real experiments, reported multiple checkpoints, and shipped a Triton implementation that keeps SQS competitive in speed. The performance numbers, while not remarkable, are plausible. The underlying idea—modifying a quadratic activation so a GLU approximately retains a bilinear spectral structure—is worth exploring, and a rigorous version of this paper would be useful for weight-based mechanistic interpretability.\n\nWho should read it: researchers working on interpretability through weights, particularly those who want a concrete example of how an activation function can break the quadratic-form assumption. As it stands, the paper is not ready for acceptance, but it is not a crackpot submission either. I would send it to peer review with the explicit expectation that the authors define their extraction procedure, address the quasi-linearity error, and release code/data. Without those changes, the interpretability claim remains an assertion.\n\nFor your own work: I would not cite it yet, but it is worth a skim if you are in this area.","headline":"SQS is a new, plausibly competitive activation, but the paper never defines the matrix it eigen-decomposes, so the central interpretability claim is unsupported.","tokens_in":7940,"tokens_out":5457,"would_cite":false,"duration_ms":50448,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that SQS, a signed and shrunk quadratic activation for gated linear units, preserves the bilinear weight spectrum that makes eigenvector interpretability possible while matching state-of-the-art performance.","keywords":["Signed Quadratic Shrink","Gated Linear Units","bilinear MLP","weight-based interpretability","eigendecomposition","mechanistic interpretability","activation functions"],"falsifier":"Run an SQS-GLU and a bilinear MLP on the same data, extract each logit's interaction matrix, then rank each logit by how much dropping its least-important eigenvectors changes classification decisions; if SQS-GLU's top eigenvectors do not carry the class information, the weight-spectra interpretability claim is refuted. Alternatively, if SQS-to-bilinear eigenvector cosine similarity drops consistently below about 0.5 across many seeds, the claim that SQS preserves bilinear spectra fails.","tokens_in":7074,"feed_emoji":"🧠","tokens_out":8019,"duration_ms":72981,"temperature":0.7,"pith_summary":"The paper introduces Signed Quadratic Shrink (SQS), an activation function for gated linear units designed to give neural networks the weight-based interpretability of bilinear MLPs without the performance and data-efficiency costs that bilinear layers incur. It claims that SQS makes the quadratic-form structure of a bilinear layer—where each logit reads as x^T A x and can be decomposed into weighted eigen-directions—survive in a nonlinear, trainable gate. On MNIST, Fashion-MNIST, and TinyStories, SQS matches the loss and convergence of GELU and SwiGLU, beats ReLU and bilinear layers, and produces eigenvectors with high cosine similarity to bilinear eigenvectors. If right, it means interpretability need not be traded away when switching to competitive activations.","feed_headline":"New activation keeps neural weights readable at full performance","feed_subtitle":"The shifted quadratic matches GELU and SwiGLU on vision and language while keeping weight spectra interpretable.","key_machinery":"The central object is the signed, shifted, shrunk quadratic activation SQS. With p = 1 it reduces to σ(x) = (x − c·sgn(x))/(1 + λ|x|): quasi-linear for very small and very large inputs, approximately quadratic in between while retaining sign. Inside a GLU, (Wx)⊙σ(Vx), it keeps each logit close to a sum of squared projections along eigenvectors, so an interaction matrix A extracted from the trained weights retains the spectral structure of a bilinear layer. The shrink term bounds large activations and the signed shift avoids the flat gradients of a pure quadratic; hyperparameters c and λ control the shift and shrinkage, and the paper sets λ = 0.5, c = 0.01, p = 1 for its experiments.","core_discovery":"On the paper's terms: SQS is defined as σ(x) = (x/|x|)·(|x| − c)/(1 + (λ|x|)^p)^(1/p), which for p = 1 becomes (x − c·sgn(x))/(1 + λ|x|). This replaces the identity gate of a bilinear GLU with a signed, shifted, shrinking quadratic that is quasi-linear for very small and very large inputs. The discovery claim is that the bilinear layer's interpretability mechanism—each output logit computed as x^T A x and decomposable into weighted eigen-directions—survives inside this nonlinear gate, so the trained model's weights can be read as class-specific eigenfeatures. The experimental evidence is that SQS-GLU eigenvectors are highly similar to those of a bilinear MLP (cosine similarity above 0.5, nea","pith_inferences":["A formal, published definition of how the interaction matrix is extracted from a trained SQS-GLU would let others verify the eigenvector-similarity result without reimplementing the paper's undocumented procedure in Section 4.1.","Because SQS's shrink term bounds large activations, it likely acts as an implicit regularizer; one testable extension is measuring weight spectra and generalization under lower weight decay to see whether λ can substitute for part of explicit regularization.","The p parameter, kept at 1 for speed, could interpolate between a smooth quadratic and a near-linear gate, potentially exposing a dataset-specific interpretability–performance trade-off curve.","A testable extension is applying SQS to transformer MLP blocks and checking whether the eigenvectors of the combined weight matrix correspond to interpretable features such as syntactic roles in generated stories."],"forward_implications":["Weight-based mechanistic interpretability becomes available in a performant GLU, without the activation-space machinery of sparse-autoencoder analyses.","SQS-GLU reaches competitive loss and perplexity on language modeling with TinyStories, offering a drop-in transformer activation that keeps training speed comparable to ReLU, GELU, and SwiGLU.","On vision benchmarks, SQS converges faster than ReLU-GLU and bilinear MLPs, being first to reach 80%, 85%, and 90% accuracy.","The eigenvectors of an SQS-GLU can be used to explain individual classification decisions, inheriting the bilinear layer's decomposition into ranked eigen-directions."],"supporting_citations":[{"why":"Supplies the bilinear-MLP eigen-decomposition method and the interpretability results that SQS seeks to preserve.","marker":"[2]"},{"why":"Introduces bilinear layers for interpretability, the architectural starting point for SQS-GLU.","marker":"[1]"},{"why":"Defines gated linear units, the layer family SQS is designed for.","marker":"[3]"},{"why":"Provides SwiGLU and GEGLU baselines and the evidence that standard GLU activations outperform bilinear layers in performance and data efficiency.","marker":"[11]"},{"why":"Provides GELU, a state-of-the-art baseline whose loss and convergence SQS must match.","marker":"[4]"},{"why":"Provides ReLU, the simplest GLU-gate baseline used in all three benchmarks.","marker":"[5]"},{"why":"Supplies the MNIST dataset used for eigenvector interpretability and performance tests.","marker":"[6]"},{"why":"Supplies Fashion-MNIST, the second vision benchmark and eigenvector testbed.","marker":"[7]"},{"why":"Supplies TinyStories, the language-modeling benchmark for final-loss and perplexity results.","marker":"[8]"}],"fun_headline_variants":["SQS activation decodes neural weights without performance loss","Read weights directly: SQS keeps GLU accuracy and interpretability","Signed quadratic shrink: interpretable GLUs at GELU-level performance","Weight spectra stay readable with SQS's signed quadratic gate","GLU weights get eigenfeatures via SQS, no accuracy drop"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that a usable, meaningful interaction matrix can be extracted from a trained SQS-GLU by treating it like a bilinear layer, even though the SQS gate is nonlinear and input-dependent; the paper does not specify how this extraction is done.","fun_headline_variants_meta":{"raw":{"variants":["SQS activation decodes neural weights without performance loss","Read weights directly: SQS keeps GLU accuracy and interpretability","Signed quadratic shrink: interpretable GLUs at GELU-level performance","Weight spectra stay readable with SQS's signed quadratic gate","GLU weights get eigenfeatures via SQS, no accuracy drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1051,"prompt_tokens":678,"completion_tokens":373,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":422,"tokens_out":373,"duration_ms":4925,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:06:36.889694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an SQS-GLU and a bilinear MLP on the same data, extract each logit's interaction matrix, then rank each logit by how much dropping its least-important eigenvectors changes classification decisions; if SQS-GLU's top eigenvectors do not carry the class information, the weight-spectra interpretability claim is refuted. Alternatively, if SQS-to-bilinear eigenvector cosine similarity drops consistently below about 0.5 across many seeds, the claim that SQS preserves bilinear spectra fails.","supporting_citations":[{"cited_title":"The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]","cited_arxiv_id":null,"evidence_quote":"Supplies the MNIST dataset used for eigenvector interpretability and performance tests."}],"review_version":1}