{"id":"f196e24c-01ed-4f5f-b53e-79e4b661ba1b","arxiv_id":"2607.10593","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Gate-freezing after warm-up stabilizes differentiable LN/DyT selection in Transformers, helping non-stationary NLP tasks while remaining competitive on vision.","lead":"AutoNorm-S freezes a Gumbel-Softmax gate after a short warm-up so Transformers can pick LayerNorm or Dynamic Tanh per layer without early training noise. The fix helps language tasks more than stationary vision tasks and gives a simple rule for when adaptive normalization is worth using.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The freeze schedule and informal stationarity definition leave the causal diagnosis of the Gumbel bottleneck under-specified.","rationale":"The Reader correctly isolates the weakest assumption: a single dataset-class freeze schedule plus an informal stationarity definition is asked to carry both the diagnosis and the remedy. That assumption is load-bearing for the strongest claim; without a quantitative non-stationarity metric or a controlled isolation of Gumbel variance, the causal story remains correlational. The concrete freeze-vs-RandomSelector test would settle whether the pathology is truly Gumbel-driven or merely an early-training stabilization effect that any schedule can mitigate. Because the paper already shows multi-seed gains, ablations, and no mathematical inconsistency, the appropriate stance remains CONDITIONAL rather than REJECT: accept the empirical observation and the practical schedule, but require the isolation experiment (and clearer stationarity metrics) before elevating the design principle. No stronger internal flaw is present, so the Reader’s verdict does not need to be moved.","tokens_in":10415,"tokens_out":637,"duration_ms":6242,"concrete_test":"On CIFAR-10 (and one NLP set) re-train three variants under identical seeds: (1) unfrozen AutoNorm, (2) AutoNorm-S with the published freeze, (3) RandomSelector that is also frozen after epoch 10. If the freeze lift for RandomSelector is within ~0.3 % of the lift for AutoNorm, or if entropy-thresholded freezing fails to improve a pure soft-blend (no Gumbel) baseline, the claimed Gumbel-specific bottleneck is not isolated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that high early Gumbel-Softmax gradient variance creates a stabilization bottleneck on stationary vision tasks (so learned gates underperform RandomSelector) while sustained entropy on non-stationary NLP lets AutoNorm-S learn useful layer-wise LN/DyT policies; freezing after T_freeze=10 (or when H<0.1) is presented as the fix that restores vision performance and yields the NLP gains (Abstract, §III-A, §III-D, §IV-A, §IV-F). The load-bearing link is that the authors’ informal stationarity notion—activation-statistic variance plus rapid entropy collapse—plus one fixed hyper-parameter pair applied uniformly to all vision sets is sufficient both to diagnose the bottleneck and to correct it without per-dataset retuning. Section III-A never supplies a quantitative non-stationarity measure, and the freeze-gate ablation is reported only as a 1.1 % lift on CIFAR-10; no controlled experiment isolates Gumbel estimator variance from other early-training effects (temperature schedule, MixUp/CutMix, LayerScale). If the same freeze schedule improves RandomSelector or a non-Gumbel soft-blend baseline by a comparable amount, the causal story that “Gumbel noise is the bottleneck” is weakened and the design principle becomes an empirical heuristic rather than a grounded optimization insight.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies differentiable selection between Layer Normalization (LN) and Dynamic Tanh (DyT) inside Transformer residual blocks via a lightweight NormSelector trained with Gumbel–Softmax (AutoNorm). Empirically, on relatively stationary vision benchmarks the learned gates often underperform a RandomSelector, which the authors attribute to high early gradient variance of the Gumbel estimator (a “stabilization bottleneck”); on PTB POS tagging and SST-2, sustained gating entropy yields layer-wise LN/DyT policies that improve accuracy and perplexity. AutoNorm-S freezes the selector after a short warm-up (T_freeze=10 or entropy H<0.1 on vision; no freeze on NLP) and reports competitive vision results, a +1.1% freeze-gate lift on CIFAR-10, statistically significant gains on PTB/SST-2 and CIFAR-10-C, plus ablations on temperature, selector capacity, and hard vs. soft selection, with five-seed means, standard deviations, and paired t-tests.","tokens_in":10789,"tokens_out":1249,"duration_ms":22737,"significance":"If the stationarity–gating interaction and the freeze schedule hold under stronger controls, the work supplies a simple, low-overhead training recipe and a transferable design principle for when adaptive architectural selection is useful in Transformers. Credit is due for multi-seed evaluation with significance tests, the freeze-gate intervention, temperature/capacity/hard–soft ablations, and the mechanistic entropy and layer-wise weight analyses (Figs. 3–4). The contribution is primarily empirical and methodological rather than theoretical; relative to SwitchNorm and related conditional normalizations, the distinctive claim is the optimization diagnosis and the freeze schedule, not the mixture idea itself. That claim is interesting for adaptive modules more broadly (routing, MoE) if the causal link to Gumbel variance is tightened.","major_comments":[{"comment":"§III-A defines input-distribution stationarity only informally via “variance of activation statistics” and speed of gating-entropy collapse, with no quantitative non-stationarity statistic, threshold, or pre-training diagnostic. The central design principle (§IV-G) and the decision when to freeze (§III-D) rest on this notion; without a measurable criterion the principle remains a post-hoc narrative rather than a falsifiable rule that can be applied to a new dataset before training.","section":null},{"comment":"§III-D and §IV-F present the freeze-gate schedule as the fix for Gumbel-induced early variance, but the only controlled number given is a +1.1% accuracy lift on CIFAR-10 for freezing the learned selector after 10 epochs. There is no parallel freeze applied to RandomSelector or to a non-Gumbel soft-blend baseline, nor an isolation of Gumbel estimator variance from concurrent early-training factors (fixed τ=0.5, MixUp/CutMix, LayerScale, cosine schedule). If freezing improves those controls by a similar margin, the causal attribution to Gumbel noise is not established and the design principle reduces to an empirical heuristic.","section":null},{"comment":"Table I and §IV-A: on Fashion-MNIST and SVHN, AutoNorm-S is matched or beaten by FrozenLN or AdaNorm, which the text treats as theory-consistent with stationarity. Because the same fixed (T_freeze, ε_H) pair is applied uniformly to all vision sets without reporting per-dataset entropy trajectories or a RandomSelector+freeze control, it remains unclear whether the schedule is correctly diagnosing the bottleneck or simply regularizing some datasets. A small controlled table (entropy at epoch 10, frozen vs. unfrozen learned gate, frozen RandomSelector) across MNIST/CIFAR-10/Fashion/SVHN would make the claim load-bearing rather than selective.","section":null}],"minor_comments":[{"comment":"Abstract and §I refer to “language modeling,” but the NLP experiments are PTB POS tagging and SST-2 classification; §IV-C correctly notes that the reported PTB perplexity is over tag sequences and not LM-comparable. Align the abstract wording with the actual tasks.","section":null},{"comment":"Eq. (1) writes w_DyT(X)·DyT(X) + w_LN(X)·LN(X) while Eq. (3) uses w0/w1; keep a single notation throughout.","section":null},{"comment":"Related work (§II-A.7) contrasts AutoNorm with SwitchNorm but does not report a same-backbone SwitchNorm (or LN+RMSNorm+DyT) baseline in the main tables; even a short appendix comparison would sharpen the novelty claim.","section":null},{"comment":"Fig. 4 caption and §IV-F: state the exact entropy threshold and freeze epoch used in the plotted AutoNorm-S curve so the figure is reproducible from the text alone.","section":null},{"comment":"§III-E claims <1% parameters and <0.1% FLOPs; Table I latency/FLOPs are essentially identical across methods—briefly note whether the selector is disabled or hard-argmaxed at inference in those measurements.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical systems paper with a clear story, but the causal “Gumbel bottleneck” claim is currently under-supported relative to how strongly it is sold in the abstract and conclusion. I would accept after a revision that either (i) adds the freeze controls on RandomSelector/soft-blend and a quantitative stationarity diagnostic, or (ii) softens the causal language to an empirical training recipe. Scope is appropriate for a solid ML venue; novelty vs. SwitchNorm should be watched in the revision."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that they actually caught a concrete failure mode: on stationary vision, learned Gumbel gates for LN vs DyT underperform a plain RandomSelector, while on PTB/SST-2 the same machinery finds useful layer-wise policies. AutoNorm-S is just a short warm-up then freeze (T_freeze=10, H<0.1 on vision; free-run on NLP). That is the new bit—not the Gumbel router or the DyT candidate themselves.\n\nThey do the work cleanly. Five seeds, stds, paired t-tests (p<0.01 PTB, p<0.05 CIFAR-10-C), RandomSelector and FrozenLN controls, temperature/capacity/hard-vs-soft ablations, and a freeze-gate lift of +1.1 % on CIFAR-10. Layer-wise weight plots and entropy curves make the story readable: vision collapses fast to LN, NLP keeps mixed gates deeper. Overhead is negligible. No circular math; claims sit on held-out accuracy/perplexity.\n\nSoft spots are real but proportional. Stationarity is defined by “activation-statistic variance + rapid entropy collapse” with no quantitative measure, and one fixed hyper-parameter pair is applied to every vision set. The freeze ablation is only reported for CIFAR-10; they never show that the same freeze leaves RandomSelector or a non-Gumbel soft blend untouched, so the causal link “Gumbel variance is the bottleneck” is still under-specified. Gains on clean vision are modest and sometimes inside CI; everything is ViT-Lite scale. Those are the limits of the evidence, not hidden disasters.\n\nThis is for people already experimenting with adaptive or DyT-style normalization who want a practical schedule and a caution about early Gumbel noise. It is not a new technology class. I would bring it to reading group as a short empirical note, cite the freeze schedule and the RandomSelector failure if I am writing on routing stability, and I would send it to referees—they can demand a proper non-stationarity metric and a larger-scale check without the paper collapsing. Worth engaging.","headline":"Solid empirical diagnosis of a real Gumbel-gating pathology plus a cheap freeze fix; useful for people already playing with DyT/SwitchNorm, but the stationarity story is still informal and the models are small.","tokens_in":11378,"tokens_out":545,"would_cite":true,"duration_ms":5516,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Learned normalization routing fails on stationary vision tasks because early Gumbel noise drowns the signal; freezing the gates after a short warm-up restores competitive performance and yields clear gains on language tasks.","keywords":["adaptive normalization","Transformers","Gumbel-Softmax","differentiable gating","LayerNorm","Dynamic Tanh","gate freezing","training dynamics"],"falsifier":"Train the identical architecture on CIFAR-10 and PTB both with and without the freeze schedule; if the unfrozen selector still matches or exceeds the frozen AutoNorm-S accuracy on CIFAR-10 while the language gains disappear, the claimed stationarity-dependent bottleneck is false.","tokens_in":11331,"feed_emoji":"🔀","tokens_out":938,"duration_ms":10998,"temperature":0.7,"pith_summary":"Transformers usually lock every layer to one fixed normalization rule such as LayerNorm. This paper asks whether a lightweight network can instead choose, for each layer and each input, between ordinary LayerNorm and a learnable Dynamic-Tanh alternative. The surprising finding is that the choice process itself is fragile: on clean image datasets whose statistics barely change, the random noise of the Gumbel-Softmax sampler prevents the selector from settling, so a purely random mixture often beats a learned one. On language tasks, where feature statistics keep shifting, the same selector stays exploratory long enough to discover useful layer-wise policies. The authors therefore introduce AutoNorm-S, a schedule that simply freezes the selector after a short warm-up once its entropy has collapsed. The resulting models match or beat static LayerNorm on vision benchmarks and post statistically significant gains on PTB part-of-speech tagging and SST-2 sentiment classification. The practical message is that adaptive architectural choices must be insulated from early optimization noise when the data distribution is stable.","feed_headline":"Freeze the gates: why learned norm routing fails on clean images","feed_subtitle":"Early Gumbel noise drowns the selector on stationary vision data; a short freeze restores gains on language.","key_machinery":"AutoNorm-S: a NormSelector MLP that produces Gumbel-Softmax weights blending LayerNorm and Dynamic Tanh, combined with a one-time gate-freezing schedule (T_freeze = 10 epochs, entropy threshold 0.1) applied uniformly to all vision datasets once the selector’s entropy collapses.","core_discovery":"On stationary vision tasks the high gradient variance of Gumbel-Softmax gating creates a stabilization bottleneck that makes learned normalization gates underperform simple random selection; on non-stationary language tasks sustained gating entropy allows the same mechanism to learn effective layer-wise LN/DyT policies; a uniform gate-freezing schedule after warm-up removes the bottleneck and produces competitive vision accuracy together with clear NLP gains.","pith_inferences":["The same entropy-collapse diagnostic could serve as an automatic detector of whether a new dataset needs free or frozen routing, removing the need for the authors’ manual vision-versus-NLP distinction.","If the principle generalizes, mixture-of-experts and dynamic-depth Transformers may also benefit from an early freeze phase rather than continuous soft routing throughout training.","A formal non-stationarity measure based on activation-statistic variance would turn the current empirical schedule into a theoretically justified adaptive rule."],"forward_implications":["When input statistics are stable, any early-stage differentiable router should be frozen or heavily annealed; free exploration is useful only when features keep evolving.","Layer-wise normalization policy can be treated as a learnable inductive bias rather than a fixed hyper-parameter, provided the selector is protected from early noise.","The same freeze-after-warm-up pattern is predicted to improve other Gumbel-style architectural choices (attention routing, expert selection) on stationary data.","Robustness under distribution shift improves when the model retains the ability to re-weight normalization once training has stabilized."],"fun_headline_variants":["Gumbel variance sinks learned norm gates on stationary vision","Gate freeze after warmup fixes routing noise on clean images","Learned LN routing lags random select under high Gumbel noise","Sustained gate entropy helps language; freeze stabilizes vision","AutoNorm-S: freeze gates to cut optimization noise in Transformers"],"cache_read_input_tokens":128,"weakest_assumption_plain":"A single fixed freeze-epoch and entropy threshold, plus an informal notion of “stationarity” measured by how fast gating entropy collapses, is enough to diagnose and correct the bottleneck on every vision dataset without per-dataset retuning.","fun_headline_variants_meta":{"raw":{"variants":["Gumbel variance sinks learned norm gates on stationary vision","Gate freeze after warmup fixes routing noise on clean images","Learned LN routing lags random select under high Gumbel noise","Sustained gate entropy helps language; freeze stabilizes vision","AutoNorm-S: freeze gates to cut optimization noise in Transformers"]},"model":"grok-4.5","effort":"low","cost_usd":0.0039,"raw_usage":{"total_tokens":1178,"prompt_tokens":744,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":39000000,"prompt_tokens_details":{"text_tokens":744,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":361,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":744,"tokens_out":73,"duration_ms":3913,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T10:35:02.481013+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the identical architecture on CIFAR-10 and PTB both with and without the freeze schedule; if the unfrozen selector still matches or exceeds the frozen AutoNorm-S accuracy on CIFAR-10 while the language gains disappear, the claimed stationarity-dependent bottleneck is false.","supporting_citations":[],"review_version":1}