{"id":"89a8eaa1-6fc1-4a9c-a8ad-cb3cd6302632","arxiv_id":"2608.07323","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"At 9M and 30M parameters, a closed-tail gate (MemGLU) matches SwiGLU's validation NLL within about 0.1%, so SwiGLU's open positive tail is not necessary at those scales.","lead":"This paper swaps SwiGLU's open-tailed gate for a closed-tail gate called MemGLU and trains small paired language models. MemGLU lands within about 0.1% of SwiGLU's validation loss at both 9M and 30M scales, so the open positive tail does not appear necessary at those scales.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The counterexample is not statistically anchored: n=3 paired runs, differences of only ~0.1%, no paired confidence intervals or equivalence tests, and an unidentified pretraining corpus.","rationale":"The reader's CONDITIONAL verdict is appropriate. I considered whether the more important flaw is that MemGLU differs from SwiGLU in several ways beyond tail closure: MemGLU is non-monotonic, bounded, and odd, whereas SiLU is monotonic and unbounded. However, the paper's claim is a counterexample claim: if some closed-tail gate reaches near-SwiGLU loss, then SwiGLU's open positive tail is not necessary in that setting. A confounded comparator does not invalidate that logical point, though it does weaken any mechanistic interpretation. The more load-bearing issue is whether the near-match is real and representative. With only three paired seeds per scale, no paired confidence intervals or equivalence tests, and an unnamed corpus, the observed 0.1% differences could be sampling noise or an artifact of an unusual data distribution. These are exactly the gaps that a CONDITIONAL verdict should require the authors to close. I therefore keep the reader's verdict unchanged while emphasizing the need for seed-level statistics and dataset disclosure.","tokens_in":9760,"tokens_out":12733,"duration_ms":126139,"concrete_test":"Obtain from the authors the seed-level paired validation-NLL differences for both scales, and require them to report paired 95% confidence intervals (or an equivalence test with a pre-specified margin such as ±0.1% relative NLL) and to identify the training corpus and tokenizer. If the corpus is not a standard public dataset, rerun the 30M paired comparison on a named standard corpus (e.g., C4 or OpenWebText) with three seeds. If the confidence intervals remain within the equivalence margin on the disclosed or standard corpus, the concern is resolved; if the intervals are wide or the result changes materially on a standard corpus, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical counterexample: a closed-tail gate matches SwiGLU closely enough to show the open positive tail is not necessary. That counterexample rests on paired 9M and 30M runs with three seeds each. The reported differences are small (-0.111% and +0.122% relative NLL), but the paper does not report paired seed-level deltas, confidence intervals, equivalence tests, or any measure of whether the 0.1% differences are distinguishable from run-to-run noise. From the shown means and standard deviations alone, rough unpaired t-tests would be non-significant at both scales. If the paired differences are similarly noisy, the 'near-match' could be a null result within sampling error rather than a robust demonstration that a closed-tail gate matches SwiGLU. Appendix B compounds this by never naming the pretraining corpus or tokenizer, saying only that both scales use the same pre-tokenized corpus with vocabulary size 32,768. If this corpus is idiosyncratic, the counterexample may not generalize to standard decoder-only LM pretraining settings, even at the tested scales. The headline claim therefore depends on evidential support that is currently neither statistically quantified nor reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether SwiGLU's open positive tail is necessary in decoder-only language-model FFNs. It introduces MemGLU, a closed-tail gate φ(g)=c0 tanh(g) sech(g), derived from a memristive branch-geometry calculation, and compares it against SwiGLU in paired 9M and 30M decoder-only pretraining runs with three seeds. The primary RMS-matched comparison reports relative validation-NLL differences of −0.111% at 9M and +0.122% at 30M. The paper also reports mechanism diagnostics showing that the two gates populate the beyond-peak region differently and that trained SwiGLU checkpoints are sensitive to positive-tail suppression. The conclusion is that, at the tested scales, SwiGLU's open positive tail is not necessary.","tokens_in":10039,"tokens_out":5249,"duration_ms":49816,"significance":"If the empirical claim is statistically sound, the paper provides a clean controlled counterexample to the default assumption that an open positive gate tail is functionally required in decoder-only FFNs. The paired design, shared initial weights and data order, the fixed RMS calibration constant, and the controlled replacement of only the scalar gate are genuine strengths, as is the honest acknowledgment of the limited scale and the kernel-efficiency cost. The result is directly relevant to the design space of GLU gates in language models. However, the central near-equivalence currently rests on three seeds per scale and an unidentified pretraining corpus, and no significance or equivalence testing is reported; the paper's own numbers are consistent with a null result within sampling noise. The mechanistic diagnostics are novel and useful, but they are descriptive rather than statistically anchored.","major_comments":[{"comment":"The central claim of near-equivalence is not statistically anchored. The paper reports only means and sample standard deviations for three seeds, with no per-pair deltas, paired confidence intervals, or equivalence tests. At 9M the absolute difference is 0.006356 in NLL, which is smaller than the SwiGLU sample SD of 0.007679; at 30M the absolute difference is 0.004333, which is larger than the SwiGLU SD of 0.000169 but comparable to the MemGLU SD of 0.003820. The 3/3 and 0/3 paired-win counts each correspond to a two-sided sign-test p-value of 0.25 and a one-sided p-value of 0.125, so the win counts alone do not establish a systematic difference. Given that the headline is 'within about 0.1%,' the paper needs to rule out the possibility that the observed differences are within paired run-to-run noise. Please report per-seed paired deltas, paired confidence intervals for the relative differences, and preferably a pre-specified TOST equivalence test (or another explicit equivalence bound) at both scales.","section":"Section 5.2, Table 1"},{"comment":"The pretraining corpus is never identified. Appendix B states only that 'both scales use the same pre-tokenized corpus with disjoint training and validation shards' and gives a vocabulary size of 32,768; it does not name the corpus, the tokenizer, the shard construction, or a release location. This is a load-bearing reproducibility gap for an empirical counterexample claim: if the corpus or tokenization is unusual, the near-match between MemGLU and SwiGLU may not transfer to standard decoder-only pretraining settings. Please identify the corpus and tokenizer (or release the pre-tokenized shards), and report the corpus size, the validation split criterion, and the exact tokenization scheme.","section":"Appendix B, Data paragraph"},{"comment":"The 'models adapt to gate geometry' conclusion is inferred from matched trajectories plus descriptive diagnostics, but the diagnostics are not statistically supported. Figure 4 reports means and standard deviations for occupancy and energy shares without any paired tests or effect sizes, and Figure 5 reports intervention ΔNLL values without confidence intervals. Because the adaptation account is one of the paper's main contributions, the evidence that the two models 'use their gates differently' should be quantified with seed-level paired comparisons (e.g., paired tests on occupancy, energy share, and intervention deltas) and reported with intervals, rather than relying on visual separation of means.","section":"Section 7.1 and Figure 4"},{"comment":"The paper's counterexample is for a single closed-tail gate family. The derivation of MemGLU depends on the design mapping ξ=tanh(g), on the scaling constant c0 (chosen to match SwiGLU's gate-output RMS at initialization), and later on the intervention parameters w and β. These choices are reasonable for a controlled comparator, but the abstract and conclusions should explicitly state that the evidence bears on this particular closed-tail shape and on the tested scales, not on all possible closed-tail gates. The current wording is mostly careful, but a sentence in the abstract or introduction clarifying the single-gate scope would prevent overgeneralization.","section":"Section 3.1 and Section 4"}],"minor_comments":[{"comment":"The metric 'final-five NLL' is used without a definition in the main text or appendix; please specify whether it is the mean of the last five validation evaluations, the last five checkpoints, or something else, and report the evaluation interval.","section":"Table 1"},{"comment":"The paired-difference plots show only the mean difference curve with no per-seed spread or confidence band; adding per-seed lines or a shaded interval would let readers assess whether the apparent trajectory differences are within noise.","section":"Appendix C and Figure C1"},{"comment":"The 9M runs use FP16 autocast while the 30M runs use BF16; please state whether this mixed-precision difference was tested for its effect on the calibration constant or on the final NLL comparisons, or add it to the limitations list.","section":"Appendix B, Optimization paragraph"},{"comment":"A sentence clarifying that the memristive derivation is a mathematical motivation for the static gate shape, and that no device dynamics or path dependence are carried into the Transformer forward pass, would help avoid confusion with genuinely dynamic memristive models; the point is stated in Section 3.2 but could be made earlier.","section":"Section 3.1"},{"comment":"The RMS ratios in Table D1 would be more interpretable if the raw RMS values or the number of accumulated scalar elements were also reported, since the ratios alone do not show the magnitudes being compared.","section":"Appendix D, Table D1"}],"recommendation":"major_revision","confidential_remarks":"I agree with the reader's conditional verdict. The paired design is a genuine strength, and the paper's controlled replacement of only the scalar gate is the right way to test the necessity claim. The two blocking issues are the missing statistical anchoring of the near-match and the unidentified corpus; both are fixable within the manuscript's scope. If the authors provide per-seed paired deltas, paired confidence intervals or equivalence tests, and identify the corpus/tokenizer, I would support acceptance after a careful revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a carefully run small-scale study with a clearly stated question, and the authors are honest about its limits. But the headline claim — that SwiGLU's open positive tail is not necessary — is supported by a paired comparison that is not statistically anchored, and the corpus is never named. I'd send it to review, not desk-reject, but it needs real work before publication.\n\nWhat's new: MemGLU (c0 tanh(g) sech(g)) used as a closed-tail GLU gate in a decoder-only LM. Prior work has sech-based activations (ReSech, Fujita et al.), but not this exact product as a Transformer FFN gate. The paper correctly notes the memristor derivation is motivation rather than the main event. The paired design — same initial weights and data order, three seeds per scale, same token schedule — is a clean way to isolate the gate. The diagnostics (occupancy–energy decoupling, positive-tail interventions on trained SwiGLU) are thoughtful and give the adaptation story some mechanism.\n\nWhere the soft spots are. The main one is statistical. With three paired seeds per scale, differences of ~0.1% relative NLL, and no paired confidence intervals, equivalence tests, or seed-level deltas, \"near-match\" is indistinguishable from run-to-run noise. A rough unpaired t-test on the reported means and SDs would be non-significant at both scales. The sign reversal between 9M and 30M reinforces that the effect is small relative to noise. The paper's own conclusion is appropriately scoped to \"at the tested scales\", but the claim of a counterexample requires demonstrating equivalence, not just failing to reject a difference.\n\nSecond, Appendix B says both scales use \"the same pre-tokenized corpus\" with vocab size 32,768, but never names the corpus or tokenizer. That is a reproducibility gap. No code or data are released either. The 30M runs use 500M tokens on an unnamed corpus; if that corpus is idiosyncratic, the near-match may not transfer.\n\nThird, the scale is small. The authors acknowledge this, so it's a limitation, not a concealed flaw. The unscaled 9M variant outperforming SwiGLU by 0.5% is interesting but auxiliary.\n\nWho this is for: people working on activation/gate design, hardware-friendly FFNs, and anyone who wants a controlled negative result to build on. It deserves a serious referee, but the authors should be pushed to disclose the corpus, add paired significance/equivalence testing (or at least seed-level deltas), and release code. I'd accept it for review with major revision.","headline":"A clean small-scale test of a focused claim, but the counterexample lacks statistical anchoring and the corpus is unnamed.","tokens_in":10560,"tokens_out":2789,"would_cite":false,"duration_ms":23950,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A closed-tail gate matches SwiGLU's language-modeling loss at 9M and 30M, so the open positive tail is not necessary at the tested scales.","keywords":["SwiGLU","MemGLU","closed-tail gating","gated feed-forward networks","decoder-only language models","activation functions","memristor","language modeling"],"falsifier":"Run the same paired SwiGLU versus RMS-matched MemGLU comparison on a named, publicly released corpus with standard tokenization at 9M and 30M using fresh seeds; if MemGLU consistently trails SwiGLU by more than roughly 0.2% in final validation NLL at either scale, the near-parity claim fails. A companion test at 100M or more parameters would show whether the conclusion persists beyond the tested sizes.","tokens_in":9558,"feed_emoji":"🧠","tokens_out":9453,"duration_ms":72090,"temperature":0.7,"pith_summary":"This paper tests whether decoder-only language-model feed-forward networks require SwiGLU's open positive tail, and answers no at 9M and 30M parameter scales. It introduces MemGLU, a closed-tail gate $\\phi_{\\mathrm{MemGLU}}(g)=c_0\\tanh(g)\\operatorname{sech}(g)$ derived from memristor branch geometry, and swaps only the scalar gate in otherwise identical networks. Across three paired seeds per scale, RMS-matched MemGLU stays within about 0.1% of SwiGLU in validation NLL, with the difference reversing sign between scales. Mechanism diagnostics show the models use their gates differently despite similar losses, and suppressing the positive tail on trained SwiGLU checkpoints hurts performance. The paper concludes that pretraining adapts to the available gate geometry, so the open positive tail is not a functional requirement at these scales.","feed_headline":"SwiGLU's open tail is not needed at tested scales","feed_subtitle":"A closed-tail MemGLU gate lands within 0.1 percent of SwiGLU on 9M and 30M language models, suggesting gate shape is adaptable.","key_machinery":"The load-bearing object is MemGLU's closed-tail gate, $\\phi_{\\mathrm{MemGLU}}(g)=c_0\\tanh(g)\\operatorname{sech}(g)$, obtained from the normalized antisymmetric branch separation of a first-order memristor model after mapping the bounded coordinate $\\xi\\in[-1,1]$ to the real gate preactivation via $\\xi=\\tanh(g)$. The gate carries the argument because it is the only difference between otherwise identical FFNs: SwiGLU's $\\phi_{\\mathrm{SiLU}}(g)=g\\sigma(g)$ is replaced while all projections, dimensions, and parameter counts stay fixed, and the scale $c_0$ is chosen per scale by matching initialization-time gate-output RMS. The paired-protocol machinery (shared initial weights, data order, token schedule, three seeds) isolates the gate as the experimental variable, and the diagnostics define the beyond-peak region $g>g^\\star$ with $g^\\star=\\operatorname{arsinh}(1)$ to separate occupancy (fraction of preactivations beyond the peak) from energy share (fraction of squared gate or GLU-product magnitude coming from that region).","core_discovery":"The paper's central claim is that a closed-tail gate can match SwiGLU's language-modeling loss in decoder-only transformers at 9M and 30M parameters, making SwiGLU's open positive tail not necessary at those scales. The controlled comparison uses MemGLU, $\\phi_{\\mathrm{MemGLU}}(g)=c_0\\tanh(g)\\operatorname{sech}(g)$, which peaks at $g^\\star=\\operatorname{arsinh}(1)$ and decays to zero on both sides. Paired runs with shared initial weights, data order, and hyperparameters give mean final-NLL differences of $-0.111\\%$ at 9M and $+0.122\\%$ at 30M relative to SwiGLU, and the reversed sign with similar magnitude is taken as evidence that neither tail geometry carries an inherent advantage. Occupancy-energy diagnostics show MemGLU still enters the region beyond the peak but draws much less gate-output and GLU-product energy from it, while evaluation-only interventions on trained SwiGLU checkpoints show that removing, capping, or attenuating the positive tail degrades validation NLL. The paper reads these observations as adaptation: pretraining shapes how the FFN uses its gate, so the open tail can influence internal routing without determining the final loss.","pith_inferences":["Editorial inference: if the adaptation account generalizes to larger scales, the open positive tail is a design convenience rather than a functional constraint, and gate search should focus on peak location, boundedness, and curvature instead of tail openness.","Editorial inference: the sign reversal between 9M and 30M is not predicted by the paper's account; estimating the paired difference at intermediate and larger scales would show whether the 0.1% magnitude persists or drifts.","Editorial inference: the occupancy-energy pair is a transferable diagnostic for any gated network, since it separates whether a gate region is visited from whether it carries signal, and could inform post-training pruning or quantization choices."],"forward_implications":["At 9M and 30M, a closed-tail gate reaches near-SwiGLU validation NLL, so the open positive tail is not required at these scales.","Trained SwiGLU checkpoints are sensitive to post-hoc positive-tail suppression, which means the tail gets encoded into the trained model even though starting without it costs little.","MemGLU and SwiGLU reach similar losses while using the gate differently, so loss parity does not imply identical hidden behavior.","The paired difference magnitude remains near 0.1% while its sign reverses across scales, suggesting no systematic advantage of either tail geometry in this size range."],"supporting_citations":[{"why":"Introduces SwiGLU and the GLU variant family; it defines the baseline gate whose open positive tail the paper tests.","marker":"[Shazeer, 2020]"},{"why":"Introduces gated linear units, the multiplicative gate architecture that both SwiGLU and MemGLU FFNs use.","marker":"[Dauphin et al., 2017]"},{"why":"Supplies the first-order memristor model whose two-branch separation Appendix A uses to derive the MemGLU gate.","marker":"[Chua, 1971]"},{"why":"Provides the memristive systems formulation used for the branch derivation and the normalized antisymmetric separation.","marker":"[Chua and Kang, 1976]"}],"fun_headline_variants":["Closed-tail MemGLU matches SwiGLU within 0.1% at 9M/30M","SwiGLU's open tail not needed: closed-tail gate ties at 9M/30M","Gate shape is flexible: closed-tail MemGLU matches SwiGLU","Closed tail? No problem: MemGLU ties SwiGLU at 9M/30M","MemGLU's closed tail matches SwiGLU within 0.1% at small scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single load-bearing premise is that the unnamed pre-tokenized corpus and tokenization are representative, and that sharing initial weights and data order between paired runs isolates the gate's effect; if either fails, the near-match between MemGLU and SwiGLU at these scales may not transfer to other settings.","fun_headline_variants_meta":{"raw":{"variants":["Closed-tail MemGLU matches SwiGLU within 0.1% at 9M/30M","SwiGLU's open tail not needed: closed-tail gate ties at 9M/30M","Gate shape is flexible: closed-tail MemGLU matches SwiGLU","Closed tail? No problem: MemGLU ties SwiGLU at 9M/30M","MemGLU's closed tail matches SwiGLU within 0.1% at small scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3080,"prompt_tokens":951,"completion_tokens":2129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2003}},"tokens_in":567,"tokens_out":2129,"duration_ms":12873,"temperature":1.0,"reasoning_tokens":2003,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:18:18.235328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same paired SwiGLU versus RMS-matched MemGLU comparison on a named, publicly released corpus with standard tokenization at 9M and 30M using fresh seeds; if MemGLU consistently trails SwiGLU by more than roughly 0.2% in final validation NLL at either scale, the near-parity claim fails. A companion test at 100M or more parameters would show whether the conclusion persists beyond the tested sizes.","supporting_citations":[],"review_version":1}