Pith. sign in

REVIEW 4 major objections 5 minor 15 references

Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A closed-tail gate matches SwiGLU's language-modeling loss at 9M and 30M, so the open positive tail is not necessary at the tested scales.

desk verdict A clean small-scale test of a focused claim, but the counterexample lacks statistical anchoring and the corpus is unnamed. read the letter →

arxiv 2608.07323 v1 pith:GD5VZGMZ submitted 2026-08-07 cs.LG

Yuting Ge , Pengju Yang , Mingkai Nie This is my paper · ORCID
classification cs.LG MSC 68T07
keywords SwiGLUMemclosed-tailgatinggatedfeed-forwardnetworksdecoder-onlylanguagemodelsactivationfunctionsmemristormodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether decoder-only language-model feed-forward networks require SwiGLU's open positive tail, and answers no at 9M and 30M parameter scales. It introduces MemGLU, a closed-tail gate $\phi_{\mathrm{MemGLU}}(g)=c_0\tanh(g)\operatorname{sech}(g)$ derived from memristor branch geometry, and swaps only the scalar gate in otherwise identical networks. Across three paired seeds per scale, RMS-matched MemGLU stays within about 0.1% of SwiGLU in validation NLL, with the difference reversing sign between scales. Mechanism diagnostics show the models use their gates differently despite similar losses, and suppressing the positive tail on trained SwiGLU checkpoints hurts performance. The paper concludes that pretraining adapts to the available gate geometry, so the open positive tail is not a functional requirement at these scales.

What carries the argument

The load-bearing object is MemGLU's closed-tail gate, $\phi_{\mathrm{MemGLU}}(g)=c_0\tanh(g)\operatorname{sech}(g)$, obtained from the normalized antisymmetric branch separation of a first-order memristor model after mapping the bounded coordinate $\xi\in[-1,1]$ to the real gate preactivation via $\xi=\tanh(g)$. The gate carries the argument because it is the only difference between otherwise identical FFNs: SwiGLU's $\phi_{\mathrm{SiLU}}(g)=g\sigma(g)$ is replaced while all projections, dimensions, and parameter counts stay fixed, and the scale $c_0$ is chosen per scale by matching initialization-time gate-output RMS. The paired-protocol machinery (shared initial weights, data order, token schedule, three seeds) isolates the gate as the experimental variable, and the diagnostics define the beyond-peak region $g>g^\star$ with $g^\star=\operatorname{arsinh}(1)$ to separate occupancy (fraction of preactivations beyond the peak) from energy share (fraction of squared gate or GLU-product magnitude coming from that region).

What would settle it

Run the same paired SwiGLU versus RMS-matched MemGLU comparison on a named, publicly released corpus with standard tokenization at 9M and 30M using fresh seeds; if MemGLU consistently trails SwiGLU by more than roughly 0.2% in final validation NLL at either scale, the near-parity claim fails. A companion test at 100M or more parameters would show whether the conclusion persists beyond the tested sizes.

Watch

Extended reading notes

Core claim

The paper's central claim is that a closed-tail gate can match SwiGLU's language-modeling loss in decoder-only transformers at 9M and 30M parameters, making SwiGLU's open positive tail not necessary at those scales. The controlled comparison uses MemGLU, $\phi_{\mathrm{MemGLU}}(g)=c_0\tanh(g)\operatorname{sech}(g)$, which peaks at $g^\star=\operatorname{arsinh}(1)$ and decays to zero on both sides. Paired runs with shared initial weights, data order, and hyperparameters give mean final-NLL differences of $-0.111\%$ at 9M and $+0.122\%$ at 30M relative to SwiGLU, and the reversed sign with similar magnitude is taken as evidence that neither tail geometry carries an inherent advantage. Occupancy-energy diagnostics show MemGLU still enters the region beyond the peak but draws much less gate-output and GLU-product energy from it, while evaluation-only interventions on trained SwiGLU checkpoints show that removing, capping, or attenuating the positive tail degrades validation NLL. The paper reads these observations as adaptation: pretraining shapes how the FFN uses its gate, so the open tail can influence internal routing without determining the final loss.

Load-bearing premise

The single load-bearing premise is that the unnamed pre-tokenized corpus and tokenization are representative, and that sharing initial weights and data order between paired runs isolates the gate's effect; if either fails, the near-match between MemGLU and SwiGLU at these scales may not transfer to other settings.

Editorial extensions

If this is right

  • At 9M and 30M, a closed-tail gate reaches near-SwiGLU validation NLL, so the open positive tail is not required at these scales.
  • Trained SwiGLU checkpoints are sensitive to post-hoc positive-tail suppression, which means the tail gets encoded into the trained model even though starting without it costs little.
  • MemGLU and SwiGLU reach similar losses while using the gate differently, so loss parity does not imply identical hidden behavior.
  • The paired difference magnitude remains near 0.1% while its sign reverses across scales, suggesting no systematic advantage of either tail geometry in this size range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the adaptation account generalizes to larger scales, the open positive tail is a design convenience rather than a functional constraint, and gate search should focus on peak location, boundedness, and curvature instead of tail openness.
  • Editorial inference: the sign reversal between 9M and 30M is not predicted by the paper's account; estimating the paired difference at intermediate and larger scales would show whether the 0.1% magnitude persists or drifts.
  • Editorial inference: the occupancy-energy pair is a transferable diagnostic for any gated network, since it separates whether a gate region is visited from whether it carries signal, and could inform post-training pruning or quantization choices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper asks whether SwiGLU's open positive tail is necessary in decoder-only language-model FFNs. It introduces MemGLU, a closed-tail gate φ(g)=c0 tanh(g) sech(g), derived from a memristive branch-geometry calculation, and compares it against SwiGLU in paired 9M and 30M decoder-only pretraining runs with three seeds. The primary RMS-matched comparison reports relative validation-NLL differences of −0.111% at 9M and +0.122% at 30M. The paper also reports mechanism diagnostics showing that the two gates populate the beyond-peak region differently and that trained SwiGLU checkpoints are sensitive to positive-tail suppression. The conclusion is that, at the tested scales, SwiGLU's open positive tail is not necessary.

Significance. If the empirical claim is statistically sound, the paper provides a clean controlled counterexample to the default assumption that an open positive gate tail is functionally required in decoder-only FFNs. The paired design, shared initial weights and data order, the fixed RMS calibration constant, and the controlled replacement of only the scalar gate are genuine strengths, as is the honest acknowledgment of the limited scale and the kernel-efficiency cost. The result is directly relevant to the design space of GLU gates in language models. However, the central near-equivalence currently rests on three seeds per scale and an unidentified pretraining corpus, and no significance or equivalence testing is reported; the paper's own numbers are consistent with a null result within sampling noise. The mechanistic diagnostics are novel and useful, but they are descriptive rather than statistically anchored.

major comments (4)
  1. [Section 5.2, Table 1] The central claim of near-equivalence is not statistically anchored. The paper reports only means and sample standard deviations for three seeds, with no per-pair deltas, paired confidence intervals, or equivalence tests. At 9M the absolute difference is 0.006356 in NLL, which is smaller than the SwiGLU sample SD of 0.007679; at 30M the absolute difference is 0.004333, which is larger than the SwiGLU SD of 0.000169 but comparable to the MemGLU SD of 0.003820. The 3/3 and 0/3 paired-win counts each correspond to a two-sided sign-test p-value of 0.25 and a one-sided p-value of 0.125, so the win counts alone do not establish a systematic difference. Given that the headline is 'within about 0.1%,' the paper needs to rule out the possibility that the observed differences are within paired run-to-run noise. Please report per-seed paired deltas, paired confidence intervals for the relative differences, and preferably a pre-specified TOST equivalence test (or another explicit equivalence bound) at both scales.
  2. [Appendix B, Data paragraph] The pretraining corpus is never identified. Appendix B states only that 'both scales use the same pre-tokenized corpus with disjoint training and validation shards' and gives a vocabulary size of 32,768; it does not name the corpus, the tokenizer, the shard construction, or a release location. This is a load-bearing reproducibility gap for an empirical counterexample claim: if the corpus or tokenization is unusual, the near-match between MemGLU and SwiGLU may not transfer to standard decoder-only pretraining settings. Please identify the corpus and tokenizer (or release the pre-tokenized shards), and report the corpus size, the validation split criterion, and the exact tokenization scheme.
  3. [Section 7.1 and Figure 4] The 'models adapt to gate geometry' conclusion is inferred from matched trajectories plus descriptive diagnostics, but the diagnostics are not statistically supported. Figure 4 reports means and standard deviations for occupancy and energy shares without any paired tests or effect sizes, and Figure 5 reports intervention ΔNLL values without confidence intervals. Because the adaptation account is one of the paper's main contributions, the evidence that the two models 'use their gates differently' should be quantified with seed-level paired comparisons (e.g., paired tests on occupancy, energy share, and intervention deltas) and reported with intervals, rather than relying on visual separation of means.
  4. [Section 3.1 and Section 4] The paper's counterexample is for a single closed-tail gate family. The derivation of MemGLU depends on the design mapping ξ=tanh(g), on the scaling constant c0 (chosen to match SwiGLU's gate-output RMS at initialization), and later on the intervention parameters w and β. These choices are reasonable for a controlled comparator, but the abstract and conclusions should explicitly state that the evidence bears on this particular closed-tail shape and on the tested scales, not on all possible closed-tail gates. The current wording is mostly careful, but a sentence in the abstract or introduction clarifying the single-gate scope would prevent overgeneralization.
minor comments (5)
  1. [Table 1] The metric 'final-five NLL' is used without a definition in the main text or appendix; please specify whether it is the mean of the last five validation evaluations, the last five checkpoints, or something else, and report the evaluation interval.
  2. [Appendix C and Figure C1] The paired-difference plots show only the mean difference curve with no per-seed spread or confidence band; adding per-seed lines or a shaded interval would let readers assess whether the apparent trajectory differences are within noise.
  3. [Appendix B, Optimization paragraph] The 9M runs use FP16 autocast while the 30M runs use BF16; please state whether this mixed-precision difference was tested for its effect on the calibration constant or on the final NLL comparisons, or add it to the limitations list.
  4. [Section 3.1] A sentence clarifying that the memristive derivation is a mathematical motivation for the static gate shape, and that no device dynamics or path dependence are carried into the Transformer forward pass, would help avoid confusion with genuinely dynamic memristive models; the point is stated in Section 3.2 but could be made earlier.
  5. [Appendix D, Table D1] The RMS ratios in Table D1 would be more interpretable if the raw RMS values or the number of accumulated scalar elements were also reported, since the ratios alone do not show the magnitudes being compared.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MemGLU's gate is derived from an external memristor model, and the near-match with SwiGLU is an empirical measurement rather than a constructed consequence.

full rationale

The paper's central claim is an empirical counterexample: a closed-tail gate, MemGLU, reaches within about 0.1% validation NLL of SwiGLU at 9M and 30M. MemGLU is not defined in terms of the target outcome. Section 3.1 and Appendix A derive tanh(g)sech(g) from the normalized antisymmetric branch separation of a first-order memristor under sinusoidal excitation, (v_-(ξ)-v_+(ξ))/(2Γ) = ξ√(1-ξ²), followed by the explicit mapping ξ=tanh(g); this is a stated design choice, not a hidden assumption. The only calibrated scalar is c0, fixed to match SwiGLU's initialization-time gate-output RMS in Section 4, and it is not fitted to final validation NLL. The unscaled variant (c0=1) achieves the same qualitative outcome at 9M (Table 1), so the near-match is not forced by the calibration. The conclusion that the open positive tail is not necessary rests on paired pretraining measurements and checkpoint interventions, not on a definition or a self-citation. No load-bearing self-citations or imported uniqueness theorems appear; known sech-based functions (ReSech, Fujita et al.) are cited as precedents rather than renamed. Statistical and reproducibility limitations (three seeds, unnamed corpus) bear on evidential strength, not on circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The ledger shows one fitted scale constant (c0) in the primary comparison, plus two hand-chosen intervention parameters that do not affect the central claim. The memristor derivation is motivating background, not a result. The main unstated dependency is the identity and representativeness of the pretraining corpus.

free parameters (3)
  • c0 (RMS calibration constant) = 0.6039096539628113 (9M), 0.6990967290796867 (30M)
    Chosen so that MemGLU's initialization-time gate output RMS matches SwiGLU's; used in the primary comparison (Section 4, Appendix B).
  • w (transition width) = 0.5
    Hand-chosen for the smooth-attenuation intervention in Appendix E; affects only the intervention diagnostic, not the main claim.
  • beta (decay parameter) = 1.0
    Hand-chosen for the smooth-attenuation intervention in Appendix E; affects only the intervention diagnostic, not the main claim.
assumptions (4)
  • standard math First-order memristor model equations v = R(w)i, dw/dt = kappa*i, R(w) = R0 + beta*(w-w0)
    Used in Appendix A to derive the branch separation that motivates MemGLU; standard circuit-theory background.
  • ad hoc to paper The mapping xi = tanh(g) from the bounded memristor coordinate to the real-valued gate preactivation
    Introduced as a design mapping in Section 3.1; it is natural but not forced by the memristor physics.
  • domain assumption The unnamed pre-tokenized corpus is representative of typical language-model pretraining data
    Appendix B states both scales use the same pre-tokenized corpus without naming it; the empirical claim depends on this data being representative.
  • domain assumption Paired training with shared initial weights and data order isolates the gate function as the only difference
    Standard paired-experiment design assumption in Section 5; depends on seed pairing not introducing hidden interactions.
invented entities (1)
  • MemGLU gate (phi(g) = c0 * tanh(g) * sech(g)) independent evidence
    purpose: Closed-tail comparator to test whether SwiGLU's open positive tail is necessary
    The paper provides empirical validation-loss comparisons and mechanism diagnostics against SwiGLU, giving the gate a falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU." pith.science (2026). https://pith.science/paper/GD5VZGMZ

@misc{pith2026260807323,
  author       = {Pith},
  title        = {Pith review of: Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GD5VZGMZ}},
  note         = {Machine review of arXiv:2608.07323}
}
read the original abstract

We test whether decoder-only language-model FFNs require SwiGLU's open positive tail. We introduce MemGLU as a closed-tail comparator derived from a memristive branch geometry. Across paired 9M and 30M pretraining runs with three seeds, MemGLU remains within about 0.1% of SwiGLU in validation NLL. Trained SwiGLU checkpoints are sensitive to positive-tail suppression, while mechanism diagnostics show that the two models use their gates differently despite similar losses. These results suggest that models adapt to the gate geometry available during pretraining. At the tested scales, SwiGLU's open positive tail is not necessary for decoder-only language-model FFNs.

Figures

Figures reproduced from arXiv: 2608.07323 by the authors.

Figure 1
Figure 1. Gate geometry of SwiGLU and MemGLU. (a) The SiLU gate and its open positive tail. (b) The MemGLU gate and its closed positive tail, with g⋆ = arsinh(1) marking the positive maximum. (c) Direct comparison of the two scalar gate responses. 2 Related Work 2.1 Gated Feed-Forward Networks and Gate Geometry Gated linear units introduce multiplicative interactions by modulating a value branch with an activation-controlled … view at source ↗
Figure 2
Figure 2. Controlled replacement of the scalar gate in the Transformer FFN. SwiGLU and MemGLU use the same projection structure and dimensions. Only the scalar gate function is changed. We evaluate an unscaled form with c0 = 1 and use the RMS-matched form as the primary comparison. The calibration procedure is described in Section 4. 4 Experimental Design We use paired training runs to isolate the effect of the scalar gate at… view at source ↗
Figure 3
Figure 3. Paired pretraining trajectories at 9M and 30M scales. (a) 9M training loss. (b) 9M validation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Occupancy–energy decoupling beyond the MemGLU peak. (a) The region [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Positive-tail interventions on trained SwiGLU checkpoints. (a) Hard removal. (b) Positive [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages

  1. [9]

    Allen Hao Huang and Imanol Schlag

    URLhttps://arxiv.org/abs/2405.20768. Allen Hao Huang and Imanol Schlag. Deriving activation functions using integration.arXiv preprint arXiv:2411.13010,

    arXiv

  2. [10]

    Introduces the Expanded Integral of the Exponential Linear Unit (xIELU)

    URL https://arxiv.org/abs/2411.13010. Introduces the Expanded Integral of the Exponential Linear Unit (xIELU). Peijie Jiang, Yuqi Feng, Cunyin Peng, Qian Zhao, Jia Liu, KunLong Chen, Zhiqiang Zhang, and Jun Zhou. PowLU: An activation function for stable pre-training of LLMs.arXiv preprint arXiv:2605.25704,

    arXiv

  3. [12]

    Kun Li, Yi Sun, Wei Wang, Xi Zhu, Bing Song, Rongrong Cao, Sen Liu, and Qingjiang Li

    URLhttps://arxiv.org/abs/2404.08763. Kun Li, Yi Sun, Wei Wang, Xi Zhu, Bing Song, Rongrong Cao, Sen Liu, and Qingjiang Li. Config- urable activation function realized by non-linear memristor for neural network.AIP Advances, 10 (8):085207,

    arXiv

  4. [15]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, et al

    URL https://arxiv.org/abs/2002.05202. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    arXiv 2002

  5. [16]

    org/abs/2302.13971

    URL https://arxiv. org/abs/2302.13971. Mingze Wang, Jinbo Wang, Yikuan Xia, Kai Shen, and Shu Zhong. More expressive feedforward layers: Part i. token-adaptive mixing of activations.arXiv preprint arXiv:2605.26647,

    arXiv

  6. [17]

    More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

    URL https://arxiv.org/abs/2605.26647. Introduces Mixture of Activations (MoA). Yongbin Yu, Kwabena Adu, Nyima Tashi, Patrick Anokye, Xiangxiang Wang, and Mighty Abra Ayidzoe. RMAF: ReLU-memristor-like activation function for deep learning.IEEE Access, 8: 72727–72741,

    work page Pith review arXiv

  7. [18]

    URL https://doi.org/10.1109/ ACCESS.2020.2987829

    doi: 10.1109/ACCESS.2020.2987829. URL https://doi.org/10.1109/ ACCESS.2020.2987829. Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial composi- tion activations: Unleashing the dynamics of large language models. InInternational Conference on Learning Representations,

    work page arXiv 2020

  8. [1971]

    URL https://doi.org/10.1109/TCT

    doi: 10.1109/TCT.1971.1083337. URL https://doi.org/10.1109/TCT. 1971.1083337. Leon O. Chua and Sung Mo Kang. Memristive devices and systems.Proceedings of the IEEE, 64 (2):209–223,

    work page arXiv 1971

Show all 15 references
  1. [1976]

    URL https://doi.org/10.1109/PROC

    doi: 10.1109/PROC.1976.10092. URL https://doi.org/10.1109/PROC. 1976.10092. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. InProceedings of the 34th International Conference on Machine Learning, volume 70 ofP...

    1976

  2. [2020]

    URLhttps://doi.org/10.1063/5.0013510

    doi: 10.1063/5.0013510. URLhttps://doi.org/10.1063/5.0013510. Aboubakar Nasser Samatin Njikam and Huan Zhao. A novel activation function for multi- layer feed-forward neural networks.Applied Intelligence, 45(1):75–82,

    doi

  3. [2022]

    URL https: //arxiv.org/abs/2204.02311. Leon O. Chua. Memristor—the missing circuit element.IEEE Transactions on Circuit Theory, 18 (5):507–519,

    arXiv

  4. [2023]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al

    URL https://arxiv.org/abs/2309.16609. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311,

    arXiv

  5. [2024]

    8 Tomohiro Fujita, Zhiwei Luo, Changqin Quan, Kohei Mori, and Sheng Cao

    URLhttps://arxiv.org/abs/2409.12517. 8 Tomohiro Fujita, Zhiwei Luo, Changqin Quan, Kohei Mori, and Sheng Cao. Performance evaluation of RNN with hyperbolic secant in gate structure through application of parkinson’s disease detection.Applied Sciences, 11(10):4361,

    arXiv

  6. [2025]

    arXiv:2411.03884

    URL https://openreview.net/forum?id=CbpWPbYHuv. arXiv:2411.03884. 9 A Full Derivation of the Memristive Branch Geometry Following the first-order memristor formulation [Chua, 1971, Chua and Kang, 1976], we consider v=R(w)i,˙w=κi, R(w) =R 0 +β(w−w 0). Here, v and i denote the d...

    1971 arXiv

  7. [2026]

    Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini

    URLhttps://arxiv.org/abs/2605.25704. Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Contextually-aware thresholding for sparsity in large language models.arXiv preprint arXiv:2404.08763,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.