{"id":"9de5f398-7288-487a-8a18-df6963690ac7","arxiv_id":"2411.13010","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A gradient-derived trainable activation, xIELU, lowers language model perplexity versus ReLU2 and SwiGLU at matched parameter counts in 125B-token trainings.","lead":"This paper introduces xIELU, a trainable activation function created by integrating a carefully chosen gradient. In 1.1B and 3B language models it reaches lower perplexity than ReLU2 and SwiGLU, though wall-clock speed and downstream task gains lag the headline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Compute-cost match is asserted via FLOP equivalence but not demonstrated in wall-clock; Table 1 shows xIELU slower, so the headline perplexity advantage may not hold at fixed time budget.","rationale":"The reader identified the same load-bearing assumption: that the 1.5x hidden-dimension scaling for standard MLPs versus gated MLPs actually delivers matched compute cost. This is the correct weak point. The paper's own Table 1 shows a wall-clock discrepancy of about 2% against SwiGLU and 5% against ReLU2, and the authors acknowledge the implementation is unoptimized. Since the abstract promises equality of compute cost, and the only empirical evidence of equality is a FLOP arithmetic identity for the linear layers, the headline claim is conditional on an implementation that does not yet exist. The derivation itself is internally consistent: the piecewise function is continuous and differentiable at zero (in the mathematical form), the integral is correct, and the parameter counts are negligible. The omission of a 3B ReLU2 baseline weakens the abstract's sweeping claim but is secondary. A fixed-wall-clock retraining experiment with optimized kernels would settle whether the small perplexity advantage is real after accounting for actual time cost. The paper's honesty about the unoptimized implementation and its explicit limitation statement support a conditional rather than a rejection verdict; the concern is a gap in evidence, not a discovered error.","tokens_in":14733,"tokens_out":6964,"duration_ms":66020,"concrete_test":"Train 1.1B Llama models with xIELU and SwiGLU/ReLU2 for a fixed wall-clock budget (e.g., 100k GPU-seconds) using optimized, fused CUDA kernels for each activation, with the same architecture and hidden dims as Table 1. Compare final validation perplexity. Also run a 3B xIELU vs ReLU2 comparison under the same protocol. If xIELU's perplexity advantage persists under fixed-time training, the concern is resolved; if not, the compute-matched claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract, Table 2) is that xIELU beats ReLU2 and SwiGLU 'when matched for the same compute cost and parameter count.' The parameter match is exact by construction (standard MLP hidden 9216 vs gated 6144 gives equal weight counts and equal matmul FLOPs). The compute match, however, is only a theoretical FLOP equivalence for the linear layers. Section 4's own efficiency table (Table 1) reports 560 ms/iteration for xIELU vs 549 ms for SwiGLU and 534 ms for ReLU2 on the same hardware, with the paper conceding that the xIELU implementation is unoptimized. Thus the experiments compare models trained on identical token counts but unequal wall-clock time; if compute cost is measured as time or energy, xIELU has not been shown to match SwiGLU or ReLU2. Because the 0.14 perplexity gap over ReLU2 at 1.1B is small, a 2–5% training-time penalty could erase the practical benefit. Additionally, the 3B comparison omits a ReLU2 baseline, so the abstract's claim of superiority over ReLU2 at 3B is unsupported. The soft spot is not the derivation—which is algebraically sound—but the operational meaning of 'matched compute cost' in the empirical comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a gradient-first method for designing activation functions: choose a desired gradient and integrate to obtain the activation. It introduces xIELU (Eq. 9), a trainable piecewise function with a linearly increasing positive gradient and a negative-component gradient that can take negative values, and xIPReLU (Eq. 12) as a cheaper linear-gradient variant. Experiments on 1.1B and 3B Llama-style models trained on 125B FineWeb Edu tokens report lower validation perplexity for xIELU than ReLU2 and SwiGLU at 1.1B and than SwiGLU at 3B, with comparable aggregate downstream scores at 3B. Ablations on 4B tokens support the main design choices, and the limitations section candidly notes that the current xIELU/xIPReLU implementations are unoptimized.","tokens_in":15048,"tokens_out":6219,"duration_ms":59442,"significance":"If the perplexity gains hold under genuinely matched compute and are reproducible, xIELU would be a useful drop-in activation for dense transformer LLMs. The main empirical comparison is carefully designed on the parameter-count and FLOP side: standard MLPs with 1.5x hidden dimension are compared against gated MLPs, and the paper reports token-matched training runs. The calculus is transparent and correct, continuity is enforced, the ablations cover the relevant design axes, and a reference implementation is provided. The 'derivation by integration' framing is more a constructive design method than a theoretical derivation, but that is an acceptable contribution if the empirical claims are accurate. The practical significance is currently weakened by the wall-clock mismatch in Table 1 and by the absence of a ReLU2 baseline at 3B.","major_comments":[{"comment":"The load-bearing claim that xIELU is matched 'for the same compute cost' is not supported by the paper's own efficiency measurement. Table 1 reports 560 ms/iteration for xIELU versus 549 ms for SwiGLU and 534 ms for ReLU2 on the same hardware, while the text concedes that the xIELU implementation is unoptimized. The 1.5x hidden-dimension construction matches parameter counts and linear-layer FLOPs, but not wall-clock time or energy, so the 0.14 perplexity improvement over ReLU2 at 1.1B may be a cost-bearing rather than cost-matched improvement. Please either provide timings with optimized CUDA kernels, or explicitly qualify the abstract and Section 4.1 claims as 'matched in parameter count and theoretical FLOPs' rather than 'same compute cost.'","section":"Abstract and Section 4, Table 1"},{"comment":"The abstract claims that xIELU outperforms ReLU2 and SwiGLU when matched for the same compute cost and parameter count, but Table 2 contains no ReLU2 baseline at 3B; only SwiGLU is compared at that scale. Since the 1.1B perplexity gap over ReLU2 (10.21 vs 10.35) is small and ReLU2 is the closer baseline, the claim of superiority over ReLU2 at 3B is unsupported. Please add a ReLU2 run at 3B or restrict the scalability claim to SwiGLU.","section":"Table 2 and Abstract, 3B results"}],"minor_comments":[{"comment":"The caption contains a typo: 'across different activations activation functions' should read 'across different activation functions.'","section":"Table 1 caption"},{"comment":"Equation (5) uses alpha for the generic gradient scale while Section 3.1 uses alpha_p and alpha_n for the trainable parameters; please make the notation less overloaded, for example by using a different symbol in Eq. (5).","section":"Section 3 and Appendix A.3"},{"comment":"The 3B rows do not report uncertainty or the number of runs, unlike the 1.1B rows; please add standard errors or at least state that these are single runs.","section":"Table 2"},{"comment":"The axis labels in the extracted figure are garbled and should be checked; the reader should be able to tell immediately that the horizontal axis is normalized depth and the vertical axis is the learned alpha value.","section":"Figure 3b"},{"comment":"The aggregate downstream scores of 39.6 and 39.7 are presented as comparable, but no error bars or significance tests are given; please add a caveat that these differences are within the noise typical of few-shot evaluations.","section":"Section 4.2, Table 3"},{"comment":"In the xIPReLU forward, alpha_n = F.softplus(self.alpha_n) while the constraint statement in Appendix A.5 says alpha_n > 0; the implementation and the constraint statement are consistent, but the constraint text should also explicitly mention that alpha_n > 0 permits negative-valued gradients for x < 0 rather than merely 'allow for negative-valued gradients.'","section":"A.6, xIPReLU implementation"}],"recommendation":"major_revision","confidential_remarks":"The mathematical content is simple and the main contribution is the empirical comparison. The paper relies heavily on Huang (2024), the first author's own prior work, for the theoretical motivation, so the editor may want to confirm that the overlap is appropriately bounded by citation and not by reused text. The anonymous GitHub link is fine for double-blind review. I see no correctness issue in the derivations; the revision hinges on the compute-match claim and the 3B baseline coverage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clean, honestly written activation-function paper. The math is right, the ablations are informative, and the authors are upfront that the new activations are currently slower in wall-clock than their FLOP-matched baselines. The empirical win is real but modest, and the headline 'matched compute cost' needs a footnote.\n\nWhat's new: xIELU and xIPReLU, obtained by integrating trainable affine transforms of the ELU and PReLU gradients. The integration template is from Huang (2024), so the novelty is applying it to a ReLU2-style positive branch and an exponential/linear negative branch. That is a reasonable incremental step, not a revolution. The derivation in Section 3 is correct, continuity is imposed, and the numerical-stability note about expm1 is a nice practical touch.\n\nWhat's good: Table 2 shows consistent perplexity improvements at 1.1B over ReLU2 and SwiGLU, and at 3B over SwiGLU. The 4B-token ablations in Table 4 support the design choices: linear-increasing positive gradient, trainable negative gradient, beta=0.5. The downstream aggregate at 3B is a tie (39.6 vs 39.7), which the paper reports without spin. Appendix code and hyperparameters are there.\n\nSoft spots: (1) The 'same compute cost' claim is only FLOP-equivalence, not wall-clock. Table 1 shows xIELU at 560 ms/iter vs 549 for SwiGLU and 534 for ReLU2. The paper concedes the implementation is unoptimized and says CUDA kernels would close the gap. Fair, but the abstract should say 'FLOP-matched' rather than 'same compute cost' until that is demonstrated. (2) The 3B comparison omits ReLU2 entirely, so the abstract's superiority-over-ReLU2 claim at 3B is unsupported. (3) The perplexity deltas are small (0.10-0.31 at 1.1B), no multiple seeds are reported, and the downstream tasks show no overall gain. These are limitations, not fatal flaws.\n\nBottom line: this is a legitimate candidate for peer review. It would benefit from a custom CUDA kernel, a 3B ReLU2 run, and a more careful definition of compute matching. If those are addressed, the contribution is solid incremental work. My verdict is a conditional accept, not a reject.","headline":"Solid, honest activation-function paper with a real but modest perplexity win; the compute-match claim is FLOP-only, and the 3B ReLU2 baseline is missing.","tokens_in":15601,"tokens_out":3205,"would_cite":false,"duration_ms":29949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A gradient-first design method yields xIELU, a trainable activation that beats ReLU2 and SwiGLU in LLM perplexity at matched cost.","keywords":["gradient-based activation design","xIELU","xIPReLU","trainable activation functions","large language models","perplexity","ReLU2","SwiGLU"],"falsifier":"A wall-clock-matched training run is the decisive check: fix total GPU-hours, train 1.1B models with xIELU and SwiGLU to the same time budget, and compare final perplexities. Table 1 already reports 560 ms per iteration for xIELU versus 549 ms for SwiGLU; if xIELU's perplexity edge disappears when steps are normalized by time rather than by a FLOP formula, the matched-cost claim fails.","tokens_in":14520,"feed_emoji":"📉","tokens_out":12110,"duration_ms":98744,"temperature":0.7,"pith_summary":"The paper's central claim is that activation functions for large language models are better designed from their gradients than from their output shapes: choose a gradient with desirable properties, integrate it, and the resulting activation inherits those properties. Applying this recipe to the Exponential Linear Unit produces xIELU, a piecewise trainable activation whose positive-input gradient grows linearly, as in ReLU2, and whose negative-input gradient is trainable and can take negative values, as in xSiLU. In 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU reaches final perplexities of 10.21 and 9.56, compared with 10.35 for ReLU2, 10.52 for SwiGLU at 1.1B, and 9.71 for SwiGLU at 3B, under the paper's parameter- and compute-matching convention. A cheaper sibling, xIPReLU, also beats both baselines at 1.1B. The paper also reports that the learned per-layer parameters shrink with depth, which it interprets as the network adaptively reducing nonlinearity for higher-level representations.","feed_headline":"New activation beats ReLU2 and SwiGLU in language-model perplexity","feed_subtitle":"At 1.1B scale xIELU reaches 10.21 perplexity; ReLU2 gets 10.35 and SwiGLU 10.52 on the same run.","key_machinery":"The load-bearing machinery is the gradient-first construction rule $f(x)=\\int(\\alpha\\,g(x)+\\beta)\\,dx$, applied with ELU as the base function $g$. For xIELU the positive branch integrates $2\\alpha_p x+0.5$ and the negative branch integrates $\\alpha_n(e^x-1)+0.5$; $\\beta=0.5$ is fixed to keep gradient and function continuous through zero, and softplus constraints keep $\\alpha_p$ positive and $\\alpha_n$ above $\\beta_n$ so the negative gradient can go negative. This construction does two jobs: it transfers the desired gradient shape into the activation by calculus rather than by curve-fitting, and it gives each layer two trainable scalars that control how much curvature the activation applies. The authors' 1.5x hidden-dimension convention for standard MLPs versus gated MLPs is what makes ReLU2 and xIELU comparable to SwiGLU in parameters and FLOPs.","core_discovery":"On the paper's own terms, the discovery is that integrating a trainable affine transformation of a base gradient function yields a competitive activation, and that the specific combination of two gradient properties is what drives better language modeling. xIELU is defined by $\\mathrm{xIELU}(x)=\\alpha_p x^2 + 0.5x$ for $x>0$ and $\\mathrm{xIELU}(x)=\\alpha_n(e^x-1)-\\alpha_n x+0.5x$ for $x\\leq 0$; both $\\alpha_p$ and $\\alpha_n$ are trainable, with $\\alpha_p>0$ and $\\alpha_n>\\beta_n=0.5$ enforced by softplus. Its gradient is linearly increasing for positive inputs and, for negative inputs, is a trainable exponential that can produce negative values, so negative pre-activations still receive learning signal. Trained on 125B tokens, this combination lowers validation perplexity relative to ReLU2 and SwiGLU at equal parameter count under a 1.5x standard-MLP-width matching rule, while the learned $\\alpha$ values fall in deeper layers. The authors frame this as evidence that gradient properties such as linearly increasing positive slope and negative-valued negative slope are the operative cause of activation-quality differences.","pith_inferences":["If the wall-clock gap in Table 1 persists, equal-FLOP matching may not mean equal-cost matching: under a fixed GPU-hour budget the current xIELU implementation would finish fewer steps, so the practical efficiency gain remains an open engineering question.","If the perplexity advantage survives a wall-clock-matched run and lower-precision training, gradient-designed activations of this type could become a default choice for dense LLM MLPs, shifting attention from gating structure to gradient shape.","A natural testable extension is to treat the learned depth profile of alpha_p and alpha_n as a prior: fix the profile from a small run, apply it to a larger run, and check whether the initialization explains part of the gain.","The recipe may transfer to non-transformer architectures or to quantization-aware training, where clean gradient shapes are especially valuable, though the paper does not test those settings."],"forward_implications":["At 1.1B and 3B scale, swapping SwiGLU for xIELU in a standard MLP lowers final perplexity by roughly 0.15 to 0.31 points under matched parameters and FLOPs, so dense MLPs with gradient-designed activations can compete with gated MLPs.","xIPReLU nearly matches xIELU with a linear negative branch, implying that the exponential in xIELU is not the source of the gain and a cheaper polynomial activation may capture most of it.","The depth-decreasing learned alpha values suggest per-layer nonlinearity budgets, so training procedures that initialize or constrain alpha by depth might train faster or better.","Because the construction is generic, the same integrate-the-gradient recipe can be applied to other base functions to generate new activation families beyond ELU and PReLU.","The unoptimized xIELU still runs slower per iteration than SwiGLU (560 ms versus 549 ms in Table 1), so realizing the theoretical efficiency requires custom fused kernels; the paper states this limitation explicitly."],"supporting_citations":[{"why":"Supplies ELU, whose integral forms the negative branch of xIELU.","marker":"(Clevert et al., 2015)"},{"why":"Defines ReLU2 and the linearly increasing positive-gradient property xIELU extends.","marker":"(So et al., 2021)"},{"why":"Introduces xSiLU and the gradient-first, integrate-the-transform design approach.","marker":"(Huang, 2024)"},{"why":"Defines SwiGLU, the gated-MLP baseline compared under the 1.5x width parity rule.","marker":"(Shazeer, 2020)"},{"why":"Defines the Llama decoder-only backbone used in all training runs.","marker":"(Touvron et al., 2023)"},{"why":"Provides FineWeb Edu, the 125B-token dataset for the main comparisons.","marker":"(Penedo et al., 2024)"},{"why":"Sets the near-optimal batch sizes used at the 1.1B and 3B scales.","marker":"(DeepSeek AI, 2024)"},{"why":"Supplies the warmup-stable-decay schedule with 1-sqrt cooldown used in training.","marker":"(Hägele et al., 2024)"}],"fun_headline_variants":["Integration-derived activation xIELU beats ReLU2 and SwiGLU","New activation via integration lowers perplexity on Llama","Gradient-based design yields xIELU, beating ReLU2 and SwiGLU","xIELU: integration-built activation outperforms popular choices","From gradient to activation: integration yields better perplexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that widening a standard MLP to 1.5 times the gated MLP's hidden width costs the same compute and parameters; if that assumption overstates the true cost, the lower perplexity may be a trade for extra computation rather than a free gain.","fun_headline_variants_meta":{"raw":{"variants":["Integration-derived activation xIELU beats ReLU2 and SwiGLU","New activation via integration lowers perplexity on Llama","Gradient-based design yields xIELU, beating ReLU2 and SwiGLU","xIELU: integration-built activation outperforms popular choices","From gradient to activation: integration yields better perplexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3112,"prompt_tokens":1048,"completion_tokens":2064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":1974}},"tokens_in":664,"tokens_out":2064,"duration_ms":29514,"temperature":1.0,"reasoning_tokens":1974,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:55:53.609064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A wall-clock-matched training run is the decisive check: fix total GPU-hours, train 1.1B models with xIELU and SwiGLU to the same time budget, and compare final perplexities. Table 1 already reports 560 ms per iteration for xIELU versus 549 ms for SwiGLU; if xIELU's perplexity edge disappears when steps are normalized by time rather than by a FLOP formula, the matched-cost claim fails.","supporting_citations":[{"cited_title":"Fast and accurate deep network learning by exponential linear units (elus), 2015","cited_arxiv_id":null,"evidence_quote":"Supplies ELU, whose integral forms the negative branch of xIELU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces xSiLU and the gradient-first, integrate-the-transform design approach."},{"cited_title":"Llama 2: Open foundation and fine-tuned chat models, 2023","cited_arxiv_id":null,"evidence_quote":"Defines the Llama decoder-only backbone used in all training runs."},{"cited_title":"B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L","cited_arxiv_id":null,"evidence_quote":"Provides FineWeb Edu, the 125B-token dataset for the main comparisons."}],"review_version":1}