Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Deriving Activation Functions Using Integration

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A gradient-first design method yields xIELU, a trainable activation that beats ReLU2 and SwiGLU in LLM perplexity at matched cost.

desk verdict Solid, honest activation-function paper with a real but modest perplexity win; the compute-match claim is FLOP-only, and the 3B ReLU2 baseline is missing. read the letter →

arxiv 2411.13010 v3 pith:XQXDBJKQ submitted 2024-11-20 cs.LG cs.NE

Allen Hao Huang , Imanol Schlag This is my paper · ORCID
classification cs.LGcs.NE
keywords gradient-basedactivationdesignxIELUIPReLUtrainablefunctionslargelanguagemodelsperplexityReLU2SwiGLU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that activation functions for large language models are better designed from their gradients than from their output shapes: choose a gradient with desirable properties, integrate it, and the resulting activation inherits those properties. Applying this recipe to the Exponential Linear Unit produces xIELU, a piecewise trainable activation whose positive-input gradient grows linearly, as in ReLU2, and whose negative-input gradient is trainable and can take negative values, as in xSiLU. In 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU reaches final perplexities of 10.21 and 9.56, compared with 10.35 for ReLU2, 10.52 for SwiGLU at 1.1B, and 9.71 for SwiGLU at 3B, under the paper's parameter- and compute-matching convention. A cheaper sibling, xIPReLU, also beats both baselines at 1.1B. The paper also reports that the learned per-layer parameters shrink with depth, which it interprets as the network adaptively reducing nonlinearity for higher-level representations.

What carries the argument

The load-bearing machinery is the gradient-first construction rule $f(x)=\int(\alpha\,g(x)+\beta)\,dx$, applied with ELU as the base function $g$. For xIELU the positive branch integrates $2\alpha_p x+0.5$ and the negative branch integrates $\alpha_n(e^x-1)+0.5$; $\beta=0.5$ is fixed to keep gradient and function continuous through zero, and softplus constraints keep $\alpha_p$ positive and $\alpha_n$ above $\beta_n$ so the negative gradient can go negative. This construction does two jobs: it transfers the desired gradient shape into the activation by calculus rather than by curve-fitting, and it gives each layer two trainable scalars that control how much curvature the activation applies. The authors' 1.5x hidden-dimension convention for standard MLPs versus gated MLPs is what makes ReLU2 and xIELU comparable to SwiGLU in parameters and FLOPs.

What would settle it

A wall-clock-matched training run is the decisive check: fix total GPU-hours, train 1.1B models with xIELU and SwiGLU to the same time budget, and compare final perplexities. Table 1 already reports 560 ms per iteration for xIELU versus 549 ms for SwiGLU; if xIELU's perplexity edge disappears when steps are normalized by time rather than by a FLOP formula, the matched-cost claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that integrating a trainable affine transformation of a base gradient function yields a competitive activation, and that the specific combination of two gradient properties is what drives better language modeling. xIELU is defined by $\mathrm{xIELU}(x)=\alpha_p x^2 + 0.5x$ for $x>0$ and $\mathrm{xIELU}(x)=\alpha_n(e^x-1)-\alpha_n x+0.5x$ for $x\leq 0$; both $\alpha_p$ and $\alpha_n$ are trainable, with $\alpha_p>0$ and $\alpha_n>\beta_n=0.5$ enforced by softplus. Its gradient is linearly increasing for positive inputs and, for negative inputs, is a trainable exponential that can produce negative values, so negative pre-activations still receive learning signal. Trained on 125B tokens, this combination lowers validation perplexity relative to ReLU2 and SwiGLU at equal parameter count under a 1.5x standard-MLP-width matching rule, while the learned $\alpha$ values fall in deeper layers. The authors frame this as evidence that gradient properties such as linearly increasing positive slope and negative-valued negative slope are the operative cause of activation-quality differences.

Load-bearing premise

The comparison assumes that widening a standard MLP to 1.5 times the gated MLP's hidden width costs the same compute and parameters; if that assumption overstates the true cost, the lower perplexity may be a trade for extra computation rather than a free gain.

Editorial extensions

If this is right

  • At 1.1B and 3B scale, swapping SwiGLU for xIELU in a standard MLP lowers final perplexity by roughly 0.15 to 0.31 points under matched parameters and FLOPs, so dense MLPs with gradient-designed activations can compete with gated MLPs.
  • xIPReLU nearly matches xIELU with a linear negative branch, implying that the exponential in xIELU is not the source of the gain and a cheaper polynomial activation may capture most of it.
  • The depth-decreasing learned alpha values suggest per-layer nonlinearity budgets, so training procedures that initialize or constrain alpha by depth might train faster or better.
  • Because the construction is generic, the same integrate-the-gradient recipe can be applied to other base functions to generate new activation families beyond ELU and PReLU.
  • The unoptimized xIELU still runs slower per iteration than SwiGLU (560 ms versus 549 ms in Table 1), so realizing the theoretical efficiency requires custom fused kernels; the paper states this limitation explicitly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the wall-clock gap in Table 1 persists, equal-FLOP matching may not mean equal-cost matching: under a fixed GPU-hour budget the current xIELU implementation would finish fewer steps, so the practical efficiency gain remains an open engineering question.
  • If the perplexity advantage survives a wall-clock-matched run and lower-precision training, gradient-designed activations of this type could become a default choice for dense LLM MLPs, shifting attention from gating structure to gradient shape.
  • A natural testable extension is to treat the learned depth profile of alpha_p and alpha_n as a prior: fix the profile from a small run, apply it to a larger run, and check whether the initialization explains part of the gain.
  • The recipe may transfer to non-transformer architectures or to quantization-aware training, where clean gradient shapes are especially valuable, though the paper does not test those settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes a gradient-first method for designing activation functions: choose a desired gradient and integrate to obtain the activation. It introduces xIELU (Eq. 9), a trainable piecewise function with a linearly increasing positive gradient and a negative-component gradient that can take negative values, and xIPReLU (Eq. 12) as a cheaper linear-gradient variant. Experiments on 1.1B and 3B Llama-style models trained on 125B FineWeb Edu tokens report lower validation perplexity for xIELU than ReLU2 and SwiGLU at 1.1B and than SwiGLU at 3B, with comparable aggregate downstream scores at 3B. Ablations on 4B tokens support the main design choices, and the limitations section candidly notes that the current xIELU/xIPReLU implementations are unoptimized.

Significance. If the perplexity gains hold under genuinely matched compute and are reproducible, xIELU would be a useful drop-in activation for dense transformer LLMs. The main empirical comparison is carefully designed on the parameter-count and FLOP side: standard MLPs with 1.5x hidden dimension are compared against gated MLPs, and the paper reports token-matched training runs. The calculus is transparent and correct, continuity is enforced, the ablations cover the relevant design axes, and a reference implementation is provided. The 'derivation by integration' framing is more a constructive design method than a theoretical derivation, but that is an acceptable contribution if the empirical claims are accurate. The practical significance is currently weakened by the wall-clock mismatch in Table 1 and by the absence of a ReLU2 baseline at 3B.

major comments (2)
  1. [Abstract and Section 4, Table 1] The load-bearing claim that xIELU is matched 'for the same compute cost' is not supported by the paper's own efficiency measurement. Table 1 reports 560 ms/iteration for xIELU versus 549 ms for SwiGLU and 534 ms for ReLU2 on the same hardware, while the text concedes that the xIELU implementation is unoptimized. The 1.5x hidden-dimension construction matches parameter counts and linear-layer FLOPs, but not wall-clock time or energy, so the 0.14 perplexity improvement over ReLU2 at 1.1B may be a cost-bearing rather than cost-matched improvement. Please either provide timings with optimized CUDA kernels, or explicitly qualify the abstract and Section 4.1 claims as 'matched in parameter count and theoretical FLOPs' rather than 'same compute cost.'
  2. [Table 2 and Abstract, 3B results] The abstract claims that xIELU outperforms ReLU2 and SwiGLU when matched for the same compute cost and parameter count, but Table 2 contains no ReLU2 baseline at 3B; only SwiGLU is compared at that scale. Since the 1.1B perplexity gap over ReLU2 (10.21 vs 10.35) is small and ReLU2 is the closer baseline, the claim of superiority over ReLU2 at 3B is unsupported. Please add a ReLU2 run at 3B or restrict the scalability claim to SwiGLU.
minor comments (6)
  1. [Table 1 caption] The caption contains a typo: 'across different activations activation functions' should read 'across different activation functions.'
  2. [Section 3 and Appendix A.3] Equation (5) uses alpha for the generic gradient scale while Section 3.1 uses alpha_p and alpha_n for the trainable parameters; please make the notation less overloaded, for example by using a different symbol in Eq. (5).
  3. [Table 2] The 3B rows do not report uncertainty or the number of runs, unlike the 1.1B rows; please add standard errors or at least state that these are single runs.
  4. [Figure 3b] The axis labels in the extracted figure are garbled and should be checked; the reader should be able to tell immediately that the horizontal axis is normalized depth and the vertical axis is the learned alpha value.
  5. [Section 4.2, Table 3] The aggregate downstream scores of 39.6 and 39.7 are presented as comparable, but no error bars or significance tests are given; please add a caveat that these differences are within the noise typical of few-shot evaluations.
  6. [A.6, xIPReLU implementation] In the xIPReLU forward, alpha_n = F.softplus(self.alpha_n) while the constraint statement in Appendix A.5 says alpha_n > 0; the implementation and the constraint statement are consistent, but the constraint text should also explicitly mention that alpha_n > 0 permits negative-valued gradients for x < 0 rather than merely 'allow for negative-valued gradients.'

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity; the integration construction is definitional but the perplexity claim rests on external benchmarks.

  1. self definitional [Section 3.1 (Eq. 6) to Section 3.2 (Eq. 9)]
    "We derive xIELU by taking the integral of trainable affine transformations applied to the ELU activation function. ... d/dx xIELU(x) = ( 2α_p x + β_p if x > 0; α_n(e^x − 1) + β_n if x ≤ 0 ) ... xIELU(x) = ( α_p x^2 + 0.5x if x > 0; α_n(e^x − 1) − α_n x + 0.5x if x ≤ 0 )"

    By construction, Eq. 9 is the indefinite integral of Eq. 6 with the constants chosen for continuity, so xIELU is defined as exactly the function whose gradient is the one the paper selected. The advertised gradient properties (linearly increasing positive gradient, negative-capable negative gradient) are therefore true by definition and cannot fail; the 'derivation' is the tautology f(x)=∫f'(x)dx+C. This is a definitional construction rather than a derivation of an independent result. Crucially, the paper's central empirical claim—lower perplexity than ReLU2 and SwiGLU—is tested against external baselines and does not reduce to this construction, so the circularity is confined to the framing and is mild.

full rationale

The paper's central claim is benchmark evidence: Table 2 compares xIELU, xIPReLU, ReLU2, and SwiGLU on 1.1B and 3B Llama models trained on the same 125B tokens, with reported losses, perplexities, and downstream evaluations. Those comparisons are external to the construction and are not forced by the 'integration' step. The only self-citation is Huang (2024) for xSiLU and for the principle that negative-valued gradients matter; that citation is motivational and is independently supported here by the paper's own ablation results (e.g., Table 4 includes SiLU, xSiLU, and negative-component ablations). The compute-cost matching concern visible in Table 1—xIELU at 560 ms/iteration versus SwiGLU at 549 ms and ReLU2 at 534 ms—is an empirical measurement/correctness issue about the operational meaning of 'matched compute cost', not a circularity: it does not make the perplexity result equivalent to its inputs. The paper also candidly lists the unoptimized implementation and the need for larger-scale validation as limitations. Overall, no prediction in the paper reduces to a fitted parameter renamed as a result, and no load-bearing argument rests solely on a self-citation chain; the only circular-adjacent element is the definitional character of 'deriving by integration', which does not drive the empirical conclusion.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces two new piecewise activation functions whose parameters are trained; the only hand-chosen constants are beta=0.5 and the alpha initialization. No new physical or theoretical entities are postulated.

free parameters (4)
  • alpha_p (trainable positive gain) = learned per layer; initialized to 0.8
    Controls the slope of the linearly increasing positive gradient. Listed as a free parameter because its initialization is chosen from small-scale experiments and it is fit to data during training.
  • alpha_n (trainable negative gain) = learned per layer; initialized to 0.8
    Controls the magnitude and range of the negative-valued gradient. Also a trainable scalar.
  • beta (gradient y-intercept) = 0.5 (fixed)
    Set to 0.5 to match GELU/SiLU gradient intercept; ablation in Table 4 shows 0.5 beats 0.0 and 1.0 and trainable beta gives no improvement.
  • alpha_p_init and alpha_n_init = 0.8
    Initial values chosen by inspecting learned values from smaller-scale experiments (Section 4.1), a hand-tuned hyperparameter.
assumptions (4)
  • standard math Standard integral calculus (integration of polynomials, exponentials, and constants).
    Used in Section 3.2 to derive Equations 7 and 8. Uncontroversial.
  • domain assumption Linearly increasing positive gradients improve learning in deep networks.
    Section 2.2 states this as the explanation for ReLU2's effectiveness and bases the xIELU positive component on it; no proof is provided.
  • domain assumption Negative-valued gradients for negative inputs are crucial for activation performance.
    Section 2.3 inherits this claim from Huang (2024); xIELU's negative component is designed around it.
  • domain assumption A standard MLP with hidden dimension 1.5x a gated MLP's hidden dimension is matched in parameter count and compute cost.
    Section 4 sets hidden sizes 9216 vs 6144 and 12288 vs 8192 to compare SwiGLU (gated) to xIELU/ReLU2 (standard). The equivalence is approximate and is contradicted by measured wall-clock throughput in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deriving Activation Functions Using Integration." pith.science (2026). https://pith.science/paper/XQXDBJKQ

@misc{pith2026241113010,
  author       = {Pith},
  title        = {Pith review of: Deriving Activation Functions Using Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQXDBJKQ}},
  note         = {Machine review of arXiv:2411.13010}
}
abstract

Our work proposes a novel approach to designing activation functions by focusing on their gradients and deriving the corresponding activation functions using integration. We introduce the Expanded Integral of the Exponential Linear Unit (xIELU), a trainable piecewise activation function derived by integrating trainable affine transformations applied to the Exponential Linear Unit (ELU). xIELU combines two key properties for the gradient: (1) a trainable and linearly increasing gradient for positive inputs, similar to Squared ReLU (ReLU$^2$), and (2) a trainable gradient that can take negative values for negative inputs, inspired by Expanded SiLU (xSiLU). Conceptually, xIELU can be viewed as an extension of ReLU$^2$ to handle negative inputs. The trainable parameters in xIELU allow it to adaptively reduce its nonlinearity for higher-level representations deeper in the network. In experiments with 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU achieves lower perplexity compared to popular activation functions like ReLU$^2$ and SwiGLU when matched for the same compute cost and parameter count. A reference implementation is available at https://github.com/Anonymous5823/xielu.

Figures

Figures reproduced from arXiv: 2411.13010 by the authors.

Figure 1
Figure 1. Comparison of activation functions related to xIELU and their gradients. ELU is linearly increasing for positive inputs and is bounded below for negative inputs. ReLU2 has a linearly increasing gradient for positive inputs and zero gradient for negative inputs. xSiLU introduces a trainable parameter α that controls the magnitude and range of negative-valued gradients by expanding the gradient limits of SiLU from (0,… view at source ↗
Figure 2
Figure 2. Visualization of xIELU and its gradients. The parameters αp and αn control the magnitude and range of the gradients. Larger values of either parameter increase the nonlinearity of xIELU. For the positive component, constraining αp > 0 ensures a linearly increasing gradient. For the negative component, the gradient is bounded within the range (βn − αn, βn] and constraining αn > βn ensures the presence of negative-val… view at source ↗
Figure 3
Figure 3. Perplexity comparison and parameter analysis of xIELU. (a) Perplexity for activation functions in 1.1B and 3B Llama models trained on 125B tokens. While xIELU initially shows higher perplexity, it progressively outperforms other functions as training continues. (b) Learned parameters αp and αn across normalized network depth (0 to 1). Both parameters decrease in deeper layers, suggesting xIELU adaptively reduces its… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Adaptive behavior of xIELU across network depth in 1.1B model. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Adaptive behavior of xIELU across network depth in 3B model. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Adaptive behavior of xIPReLU across network depth in 1.1B model. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU

    cs.LG 2026-08 conditional novelty 5.0 of 10

    At 9M and 30M parameters, a closed-tail gate (MemGLU) matches SwiGLU's validation NLL within about 0.1%, so SwiGLU's open positive tail is not necessary at those scales.

Reference graph

Works this paper leans on

28 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gqa: Training generalized multi-query transformer models from multi-head check- points, 2023

    Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points, 2023

    work page 2023

  2. [2]

    and Li, Y

    Allen-Zhu, Z. and Li, Y . Physics of language models: Part 3.3, knowledge capacity scaling laws, 2024

    work page 2024

  3. [3]

    Palm: Scaling language modeling with pathways, 2022

    Chowdhery, A., Narang, S., Devlin, J., et al. Palm: Scaling language modeling with pathways, 2022

    work page 2022

  4. [4]

    Fast and accurate deep network learning by exponential linear units (elus), 2015

    Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus), 2015. DeepSeek AI. Deepseek llm: Scaling open-source language models with longtermism, 2024

    work page 2015

  5. [5]

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017

    work page 2017

  6. [6]

    Scal- ing fp8 training to trillion-token llms, 2024

    Fishman, M., Chmiel, B., Banner, R., and Soudry, D. Scal- ing fp8 training to trillion-token llms, 2024

    work page 2024

  7. [7]

    What every computer scientist should know about floating-point arithmetic, 1991

    Goldberg, D. What every computer scientist should know about floating-point arithmetic, 1991

    work page 1991

  8. [8]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), 2016

    work page 2016

Show all 28 references
  1. [9]

    Minicpm: Unveil- ing the potential of small language models with scalable training strategies, 2024

    Fang, Y ., Huang, Y ., Zhao, W., et al. Minicpm: Unveil- ing the potential of small language models with scalable training strategies, 2024

    2024

  2. [10]

    Huang, A. H. Expanded gating ranges improve activation functions, 2024. H¨agele, A., Bakouch, E., Kosson, A., Allal, L. B., Werra, L. V ., and Jaggi, M. Scaling laws and compute-optimal training beyond fixed training durations, 2024

    2024

  3. [11]

    Self-normalizing neural networks, 2017

    Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. Self-normalizing neural networks, 2017

    2017

  4. [12]

    Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks, 2012. Llama Team. The llama 3 herd of models, 2024

    2012

  5. [13]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization, 2019

    2019

  6. [14]

    L., Hannun, A

    Maas, A. L., Hannun, A. Y ., and Ng, A. Y . Rectifier nonlin- earities improve neural network acoustic models, 2013

    2013

  7. [15]

    McCulloch, W. S. and Pitts, W. A logical calculus of the ideas immanent in nervous activity, 1943. Mistral AI. Mistral NeMo: Frontier AI in your hands, 2024. URL https://mistral.ai/news/ mistral-nemo/

    1943

  8. [16]

    and Hinton, G

    Nair, V . and Hinton, G. E. Rectified linear units improve restricted boltzmann machines, 2010

    2010

  9. [17]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydl´ıˇcek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V ., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    2024

  10. [18]

    Ramachandran, P., Zoph, B., and Le, Q. V . Searching for activation functions, 2017

    2017

  11. [19]

    Glu variants improve transformer, 2020

    Shazeer, N. Glu variants improve transformer, 2020

    2020

  12. [20]

    R., Mani, W., Liu, Y ., Dai, Z., Shleifer, S., Landolfi, N

    So, D. R., Mani, W., Liu, Y ., Dai, Z., Shleifer, S., Landolfi, N. C., He, A., and Le, Q. V . Primer: Searching for efficient transformers for language modeling, 2021

    2021

  13. [21]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Su, J., Lu, Y ., Pan, S., Murtadha, A., Wen, B., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding, 2023

    2023

  14. [22]

    Going deeper with convolutions, 2014

    Anguelov, D., Erhan, D., Vanhoucke, V ., and Rabinovich, A. Going deeper with convolutions, 2014

    2014

  15. [23]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Touvron, H., Martin, L., Stone, K., et al. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  16. [24]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2017

    2017

  17. [25]

    Aggre- gated residual transformations for deep neural networks, 2017

    Xie, S., Girshick, R., Doll´ar, P., Tu, Z., and He, K. Aggre- gated residual transformations for deep neural networks, 2017

    2017

  18. [26]

    Scaling vision transformers, 2022

    Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers, 2022

    2022

  19. [27]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normal- ization, 2019

    2019

  20. [28]

    Relu 2 wins: Discovering efficient activation functions for sparse llms, 2024

    Zhang, Z., Song, Y ., Yu, G., Han, X., Lin, Y ., Xiao, C., Song, C., Liu, Z., Mi, Z., and Sun, M. Relu 2 wins: Discovering efficient activation functions for sparse llms, 2024. 9 Deriving Activation Functions Using Integration A. Appendix A.1. Main Experiment 1.1B and 3B Setup...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.