Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Rational neural networks with trainable low-degree activations are exponentially more parameter-efficient than fixed smooth activations such as GELU and SiLU when approximating on a compact domain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Rational-activation networks can emulate GELU-type networks with O((log log 1/ε)^3) overhead, while emulating hard rational functions with GELU networks costs Ω(log 1/ε) parameters, an exponential gap.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection The empirical work is solid, but a load-bearing gap in the log-block construction and a wrong pole-location equation undermine the theoretical core as written. the 3 major comments →

arxiv 2602.12390 v2 pith:DYVQII2H submitted 2026-02-12 cs.LG cs.AIcs.NAmath.NA

Rational Neural Networks have Expressivity Advantages

Maosen Tang , Alex Townsend This is my paper
classification cs.LG cs.AIcs.NAmath.NA MSC 41A2068T07
keywords rational neural networkstrainable activation functionsapproximation theoryparameter complexityGELUtransformer architecturesnormalizationexpressivity separation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that neural networks whose scalar activation is a trainable low-degree rational function are strictly more parameter-efficient than networks using standard fixed activations—GELU, SiLU, Mish, ELU, tanh, sigmoid, Softplus, softmax, and the rest—when the goal is uniform approximation on a compact domain. Specifically, any network built from those fixed activations can be uniformly approximated by a rational-activation network with only poly(log log(1/ε)) overhead in size, while the reverse direction provably costs Ω(log(1/ε)) parameters for some rational targets. The gap is exponential in the error tolerance, and the paper shows it persists at the level of full feedforward networks, gated activations, and transformer-style MLP blocks. The same theoretical motivation is tested empirically: rational activations match or beat fixed activations on CIFAR-10, offline reinforcement learning benchmarks, and Tiny ImageNet transformers, and they train successfully in a normalization-free ViT variant where fixed activations fail. A sympathetic reader would take the paper to establish that rational activations are not merely competitive but a mathematically natural and parameter-efficient choice.

Core claim

On the paper's own terms, the discovery is an asymmetry in approximation complexity. For an error ε, a constant-width rational network approximates GELU (and a broad family of transcendental activations) to uniform error ε with O(log³ log(1/ε)) size, and no rational network of size o(log log(1/ε)) can do better; conversely, any bounded-parameter smooth-activation network needs Ω(log(1/ε)) size to approximate a worst-case rational function such as rη(x)=1/(x²+η²) with η=ε^(1/4). The upper bound is built from rational blocks for square roots, logarithms, artanh, and tanh that converge doubly exponentially; the lower bound comes from the curvature rη develops near zero, which bounded smooth net

What carries the argument

The central object is the trainable rational activation r(x)=P(x)/Q(x), low-degree in both numerator and denominator, composed across layers; the paper uses a 'safe' parameterization with denominator 1+|Q̃(x)| to exclude real poles. The upper bound is carried by four rational blocks—composite p-th root, arithmetic-geometric-mean logarithm, artanh, and tanh—each with doubly exponential convergence in depth. The lower bound is carried by rη(x)=1/(x²+η²): its second derivative at 0 is ~1/η⁴, while bounded-parameter smooth networks have curvature growing only polynomially with width and exponentially with depth. The reverse constructive bound uses a Chebyshev truncation evaluated by an inexact C

Load-bearing premise

The full-network separation is proven under the condition that every node's pre-activation stays in [−1,1] (enforced by ∥a∥₁+|b|≤1 on each affine map); the main theorem states the result without that condition, so if pre-activations leave that box in general networks, the claimed O(kM log³ log) and Ω(kM log) network-level bounds do not follow from the proofs as written.

What would settle it

Construct a family of bounded-parameter GELU networks whose size is constant in ε and yet uniformly approximates rε(x)=1/(x²+ε^(1/2)) (or any rη with η≈ε^(1/4)) to error ε on [−1,1]; Theorem 3.2 predicts any such family must grow like Ω(log(1/ε)), so a constant-size family would settle the claim false. Conversely, a rational network of size o(log log(1/ε)) that uniformly approximates GELU to error ε for all ε would falsify the lower bound in Theorem 3.1.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any architecture using GELU, Swish, SiLU, Mish, ELU, tanh, sigmoid, or Softplus can be swapped to rational activations at negligible size overhead—O(log³ log(1/ε)) multiplicative—while retaining the same uniform approximation accuracy.
  • The reverse swap is exponentially expensive in the worst case: there exist rational functions realizable by small networks that any bounded-parameter smooth-activation network needs Ω(log(1/ε)) size to approximate, so fixed-activation networks cannot absorb the gap by simple rescaling.
  • The separation extends to gated activations and transformer MLP blocks, so replacing the MLP nonlinearity inside a transformer with a rational one is expected to preserve the expressivity advantage under identical wiring.
  • Normalization layers are not a neutral addition for adaptive rationals: they create non-identifiability and conditioning pathologies, and removing them (e.g., a no-LayerNorm ViT) can let rational activations train where fixed activations fail.
  • The learned rational shapes reallocate curvature toward high-density regions, explaining faster convergence and higher plateaus on CIFAR-10 and offline continuous-control benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the parameter-efficiency gap should be most valuable in low-budget settings—mobile models, few-shot fine-tuning, or compressed transformers—since the advantage is stated in parameter count rather than FLOPs; a direct test would be to compare rational versus GELU activations under a fixed parameter budget on a memory-constrained task.
  • The network-level theorem as proven in the appendix requires every affine map to satisfy ∥a∥₁+|b|≤1 so pre-activations stay in [−1,1]; a natural testable extension is to check whether the O(kM log³ log) bound survives in deep networks without that constraint, since real trained transformers do not enforce it.
  • Because the lower bound rests on curvature near a pole, the same argument should apply to any smooth activation with uniformly bounded derivatives; one could verify empirically whether gated rational networks retain the gap when weight norms are explicitly constrained.
  • The paper's own learned-shape observations suggest a further architectural idea: if rationals are mostly evaluated in smooth low-curvature regions, an input-dependent routing that drives signals through the high-curvature parts of the learned rational could convert the theoretical expressivity into larger empirical gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper claims that neural networks with trainable low-degree rational activations are exponentially more parameter-efficient than networks with standard fixed smooth activations (GELU, Swish, SiLU, etc.). Specifically, it asserts that any fixed-activation network can be uniformly approximated on a compact domain by a rational-activation network of size poly(log log(1/ε)), while the converse requires Ω(log(1/ε)) parameters. The main theoretical supports are: (i) Theorem 3.1, a construction of a rational network of size O((log log 1/ε)^3) approximating GELU on [-1,1]; (ii) Theorems 3.2 and 3.3, an Ω(log 1/ε) lower bound and an O(log^2 1/ε) upper bound for GELU networks approximating rational functions; and (iii) Theorem 3.4, a lifting of these scalar results to full networks. The paper also presents experiments on CIFAR-10, Tiny ImageNet transformers, and offline MuJoCo RL, and analyzes interactions with normalization layers.

Significance. If the central separation were proved, it would be a substantial contribution to neural-network approximation theory and to the practical understanding of rational activations: an exponential gap in parameter complexity between rational and smooth-activation networks is a strong and surprising claim. The paper has several sound components. The GELU-to-rational upper bound of O(log^2 1/ε) via Chebyshev truncation and an inexact Clenshaw recurrence (Appendix B) is plausible and constructive. The curvature-based lower bound of Ω(log 1/ε) for bounded-parameter GELU networks approximating rationals (Theorem 3.2) is carefully argued. The experiments are reproducible and honestly scoped, with code provided. However, the headline upper bound for rational approximation of GELU (Theorem 3.1) rests on a flawed step in Appendix A, and the network-level lifting (Theorem 3.4) is proved only under an additional norm constraint not present in the statement. Since the main theoretical novelty depends on these results, the paper in its current form does not establish the claimed exponential separation.

major comments (3)
  1. [Appendix A, Step 3 (pp. 16–17)] The construction of the modified logarithm block eRlog on [7/8,9/8] is invalid. Lemma A.2 approximates log only on Dσ,τ = {x>0 : |x−1|≥σ, τ≤x≤1/τ}, i.e., with a neighborhood of 1 removed. Step 3 claims that for u∈[7/8,9/8] one can write u=ab with a,b∈Dσ,τ via a fixed factorization, hard-wiring log(1)=0 when needed. This is impossible: for u=1, if |a−1|≥σ and |b−1|≥σ, then a and b lie on opposite sides of 1, forcing |ab−1|≥σ²>0. By continuity, the same obstruction affects u arbitrarily close to 1. Moreover a rational network computes a continuous function, so it cannot contain an isolated hard-wired value at u=1. Consequently Lemma A.2 does not yield a uniform ε-approximation of log on [7/8,9/8] with size Θ((log log 1/ε)^2). Since Rartanh, Rtanh, and RGELU all call this block, the proof of Theorem 3.1's upper bound O(log³ log 1/ε) collapses. This is the foundation for the paper's main exp
  2. [Section 3.3 / Appendix C, Theorem 3.4 vs Theorem C.1] Theorem 3.4 as stated in the main text imposes no restriction on the affine maps. The proof in Appendix C, however, assumes ∥a∥₁+|b|≤1 on every affine map, which enforces that all pre-activations remain in [−1,1]. Without this condition, pre-activations can leave the interval where the scalar approximation theorems apply, and the layerwise error propagation using Lipschitz constants on [−1,1] is not justified. Thus the O(kM log³ log(1/ε)) and Ω(kM log(1/ε)) network-level bounds do not follow for general bounded-parameter networks. The theorem must either be restricted to the norm-one-affine-map setting or be supplied with a separate argument controlling pre-activation growth.
  3. [Appendix A, Eq. (11) and following paragraph] The claim that minimizing W L subject to Cdeg W³ L ≥ c log(1/ε) gives W L = Ω(log log 1/ε) is not correct as stated. Treating W L as the size and minimizing over integer W,L≥1, the minimum is Θ((log 1/ε)^{1/3}) (attained near L=1, W≈(log 1/ε)^{1/3}), not Ω(log log 1/ε). The stated Ω(log log 1/ε) lower bound is nevertheless true by a simpler argument (if W L = o(log log 1/ε), then W³ L ≤ (W L)^3 = o(log 1/ε), contradicting Eq. (11)). The proof should be corrected to avoid the false minimization statement.
minor comments (6)
  1. [Throughout] The notation 'O(log³ log(1/ε))' is ambiguous; it should be written as O((log log(1/ε))^3) to match the intended meaning.
  2. [Appendix A] The text repeatedly refers to 'Theorem A.1', 'Theorem A.2', and 'Theorem A.3' where the statements are lemmas. Please renumber or relabel.
  3. [Section 3.4] The extensions to gated activations and transformers are asserted in prose without formal theorem statements or proofs. If kept, they need precise model definitions and explicit simulation arguments; as written they are not load-bearing for the main theorems but should be tightened.
  4. [Figure 6] The numerical verification in Figure 6(b) demonstrates convergence of the AGM-theta log block away from 1, but does not test the problematic Step 3 regime where the argument approaches 1. A numerical test on [7/8,9/8] including 1 would have highlighted the issue.
  5. [Section 2] The theoretical construction uses generic low-degree rationals, while the experiments use a specific 'safe' parameterization with P=5, Q=4 from Delfosse et al. The relationship between the theoretical rational networks and the implemented safe rationals is not discussed.
  6. [Tables 1–5] CIFAR-10 results report best test accuracy over training. This selection protocol can overestimate performance relative to a fixed-checkpoint evaluation; please state whether any test-set-based selection was used and consider reporting final-epoch accuracy as well.

Circularity Check

0 steps flagged

No significant circularity: the central rational/GELU separation proofs are self-contained constructions; self-citations are background, not load-bearing.

full rationale

The central derivation chain is not circular. The scalar upper bound (Theorem 3.1, Appendix A) is a constructive composition of independent building blocks: the Gawlik-Nakatsukasa pth-root iteration, the Sasaki-Kanada/Brent AGM-theta logarithm, an artanh block built from that logarithm, a Halley iteration for tanh, and a fixed double-angle ladder. None of these steps fits a parameter to the target or assumes the conclusion; error targets are propagated through explicit stability bounds (Lemmas A.1-A.3). The scalar lower bound for rational approximation of GELU (Theorem A.4) invokes Gonchar's condenser-capacity theorem and the explicit pole locations of GELU, an external classical result, not a self-citation. The converse lower bound (Theorem B.2) constructs the explicit hard instance R(x)=1/(x^2+eta^2) and compares its second derivative with a derived curvature bound on bounded-parameter GELU networks; the Omega(log(1/epsilon)) parameter count follows from an AM-GM argument, not from the statement being proven. The full-network lift (Appendix C) is a layer-by-layer Lipschitz error propagation; it is under-proved as stated because Theorem 3.4 drops the ||a||_1+|b|<=1 condition used in the appendix, but that is a proof-completeness/correctness gap, not a circular reduction. Self-citations to Boulle et al. (2020) occur as background (the rational-vs-ReLU separation) and for the degree bound deg <= C W^3 L for constant-width rational networks; both are stated theorems with assumptions that do not include the present target results, so under the review rules they count as real evidence and do not raise the circularity score. The only other passage that invites scrutiny, Appendix A Step 3's fixed-factorization claim for log on [7/8,9/8], is asserted tersely and would need a uniform construction to be fully rigorous, but that is again a proof-completeness issue, not a reduction of the theorem to its inputs. The claimed empirical gains are experimental measurements, not predictions derived from the theorems by construction. No fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' prior work. I set the score to 1 only to register the presence of non-load-bearing self-citations, not because any derivation reduces to its inputs.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 0 invented entities

The theoretical results rest on classical approximation theory and prior rational-network iteration results, not on new fitted constants. The paper introduces no new physical or mathematical entities. The main unstated premises are the bounded-pre-activation condition for network lifting and the bounded-parameter condition for the smooth-network lower bound.

free parameters (3)
  • η in hard rational instance = η ∈ (0, ε^{1/4})
    Ad hoc scale in R(x)=1/(x²+η²) used to force Ω(1/ε) curvature; it is a proof device, not a fitted constant.
  • Richardson scale parameters s*, γ = s* ∈ (0, s0], γ > 1
    Hand-chosen constants in the Appendix B square-block construction; they affect constant factors but not the asymptotic rate.
  • Rational activation degrees (P,Q) = P=5, Q=4
    Chosen to match Delfosse et al. (2020) defaults in experiments; theory only requires fixed low-degree rationals.
axioms (7)
  • standard math Classical rational approximation theory: best uniform rational error decays geometrically with degree when the target is meromorphic with nearest singularities at distance ρ; condenser-capacity formulas apply.
    Used in Appendix A to prove the Ω(log log(1/ε)) lower bound for rational networks approximating GELU (Theorem A.4).
  • standard math Gawlik–Nakatsukasa composite p-th root iteration converges doubly exponentially: error ≤ C_p exp(−c_p 2^k).
    Invoked in Appendix A Step 1 to construct the square-root block.
  • standard math Sasaki–Kanada AGM-theta identity log(1/q) = (π/4)/AGM(θ₂², θ₃²), with classical AGM convergence.
    Invoked in Appendix A Step 2 to construct the logarithm block.
  • standard math Halley's method for artanh(t)=s converges with cubic order on a compact interval around the root.
    Invoked in Appendix A Step 4 to construct the tanh block.
  • domain assumption For networks with ∥a∥₁+|b|≤1 and activations mapping [−1,1] into [−1,1], all pre-activations remain in [−1,1].
    Appendix C relies on this to lift scalar approximations layer-by-layer; Theorem 3.4 as stated omits this condition.
  • domain assumption GELU is represented by the tanh-approximation G(x)=x/2(1+tanh(α(x+βx³))) and is treated as meromorphic with poles at solutions of αz+βz³=i(π/2+kπ).
    The lower-bound pole analysis is for this tanh-approximate GELU, not the exact erf-based GELU.
  • domain assumption The lower bound for GELU networks approximating rationals requires weights and biases bounded by B independent of ε.
    The bound is stated in Theorem 3.2's proof and is essential for the curvature argument, but is omitted from the abstract's converse claim.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Rational Neural Networks have Expressivity Advantages." pith.science (2026). https://pith.science/paper/DYVQII2H

@misc{pith2026260212390,
  author       = {Pith},
  title        = {Pith review of: Rational Neural Networks have Expressivity Advantages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYVQII2H}},
  note         = {Machine review of arXiv:2602.12390}
}
Share X Bluesky LinkedIn Reddit HN
abstract

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $\Omega(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Figures

Figures reproduced from arXiv: 2602.12390 by Alex Townsend, Maosen Tang.

Figure 1
Figure 1. Figure 1: visualizes the test-accuracy score curves for VGG4 in the baseline setting and in the augemented pipeline with￾out GroupNorm. In both cases, rational activations reach strong accuracy earlier in training and converge to a better fi￾nal value than the fixed baselines, so the score curves corrob￾orate the advantages observed in the best-over-training sum￾maries. Additional score curves for the remaining mode… view at source ↗
Figure 2
Figure 2. Figure 2: Example learned Rational shape in VGG8 (representative feature-layer snapshot). Curves show the learned scalar nonlin￾earity at epochs 0, 5, 30, and 60; the light shaded overlay is the empirical density of the corresponding layer’s pre-activation inputs estimated from held-out mini-batches and plotted on a secondary axis. The Rational quickly reallocates curvature to the high-density input region and devel… view at source ↗
Figure 3
Figure 3. Figure 3: highlights a representative learning curve for IQL 0 50000 100000 150000 200000 250000 300000 0 20 40 60 80 100 0 50000 100000 150000 200000 250000 300000 20 40 60 80 GELU ReLU SiLU Rational [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Example learned Rational shape in offline RL. Represen￾tative Rational layer from the IQL actor on HalfCheetah-medium with Rational initialized to ReLU. Curves show the learned scalar nonlinearity at selected training checkpoints and the light shaded overlay is the empirical density of the corresponding layer pre￾activation inputs estimated from offline mini-batches. Supplemen￾tary material provides broade… view at source ↗
Figure 5
Figure 5. Figure 5: plots the full EMA accuracy trajectories: for CaiT￾S24 the rational run separates clearly and attains a higher plateau, for Swin-T the GELU and rational curves remain very close and repeatedly trade the lead over training, and for ViT-Small the rational run improves over GELU, with the normalization-free ViT-Small variant converging earlier and reaching its plateau sooner, allowing a smaller ViT-Small mode… view at source ↗
Figure 6
Figure 6. Figure 6: Numerical verification of the convergence behavior of the core blocks used in the GELU construction. Panels (a,d) correspond to the composite pth-root iteration from Step 1, (b,e) to the AGM-theta logarithm construction from Step 2, and (c,f) to the Halley-based tanh construction from Step 4. The top row reports log10 error decay together with the corresponding theoretical rate guides, while the bottom row… view at source ↗
Figure 7
Figure 7. Figure 7: Pole barrier for GELU(z) in the complex plane. The nearest poles to the interval [−1, 1] restrict the largest Bernstein ellipse on which GELU is analytic. This analytic domain determines the geometric factor ζ + p 1 + ζ 2 that governs the exponential lower bound in Equation (9) for uniform type-n rational approximation on [−1, 1]. on [−1, 1]. Set P(x) = α(x+βx3 ), so P is a fixed polynomial map of constant… view at source ↗
Figure 8
Figure 8. Figure 8: CIFAR-10 score curves. Top-1 test accuracy versus epoch. Curves show mean across five seeds and the shaded region is one standard deviation. Columns are baseline, augmented without GroupNorm, and augmented with GroupNorm. Rows are VGG4 (top) and VGG8 (bottom). 4 2 0 2 4 input 0 2 4 000:features.1 epoch 0 4 2 0 2 4 input 0.25 0.00 0.25 0.50 epoch 5 4 2 0 2 4 input 0.0 0.5 1.0 epoch 30 4 2 0 2 4 input 0.5 0.… view at source ↗
Figure 9
Figure 9. Figure 9: Layerwise Rational activation snapshots for VGG8 (pages 1–4). Each panel shows the learned Rational nonlinearity at epochs 0, 5, 30, and 60. The light shaded overlay is the empirical histogram (density) of the corresponding layer input, estimated from held-out batches and plotted on a secondary axis. 4 2 0 2 4 input 0 2 4 004:features.11 epoch 0 4 2 0 2 4 input 0 1 2 3 epoch 5 4 2 0 2 4 input 0 2 4 epoch 3… view at source ↗
Figure 10
Figure 10. Figure 10: Layerwise Rational activation snapshots for VGG8 (pages 5–8). Each panel shows the learned Rational nonlinearity at epochs 0, 5, 30, and 60. The light shaded overlay is the empirical histogram (density) of the corresponding layer input, estimated from held-out batches and plotted on a secondary axis. F. Offline reinforcement learning experiments We study offline reinforcement learning on continuous-contro… view at source ↗
Figure 11
Figure 11. Figure 11: Offline reinforcement learning curves on medium locomotion benchmarks (Todorov et al., 2012; Kostrikov et al., 2021; Fujimoto & Gu, 2021). Each panel plots the v5-normalized evaluation score versus gradient updates. For each activation, the solid curve is the mean over five random seeds and the shaded band corresponds to one standard deviation. Columns correspond to environments (HalfCheetah medium, Hoppe… view at source ↗
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p034_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Rational activation snapshots for IQL on HalfCheetah-medium with Rational initialized to ReLU, value and actor networks. Each panel shows the learned Rational nonlinearity at updates 50k, 100k, 150k, and 300k. The light shaded overlay is the empirical histogram (density) of the corresponding layer pre-activation input, estimated from held-out mini-batches of offline transitions. From top-left to bottom-ri… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rational Sparse Autoencoder

    cs.LG 2026-06 unverdicted novelty 7.0

    RSAE replaces fixed SAE encoder activations (ReLU, JumpReLU, TopK) with trainable rational functions, initialized from baselines and fine-tuned to improve reconstruction and downstream metrics on language-model residu...

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    The first evaluates the Jacobi theta functions θ2(0, q4) and θ3(0, q4) via the complete elliptic integral, using the Sasaki–Kanada framework. To describe this stage, recall the standard relations between the theta functions, the elliptic modulus k∈(0,1) corresponding toq, and the complete elliptic integralK(k); see Brent (2020, Sec. 2). One has θ3(0, q)2 ...

    2020

  2. [2]

    We then iterate alog j+1 = alog j +b log j 2 , b log j+1 = q alog j blog j , j≥0

    In the second stage we apply the real AGM directly to the pair alog 0 =θ 2(0, q4)2, b log 0 =θ 3(0, q4)2, where in practice we insert the outputs of the first stage. We then iterate alog j+1 = alog j +b log j 2 , b log j+1 = q alog j blog j , j≥0. The Sasaki–Kanada identity log 1 q = π/4 AGM θ2(0, q4)2, θ3(0, q4)2 ,0< q <1(7) then states that in exact ari...

    2020

  3. [7]

    ,0, x∈[−1,1], for a constant C ⋆ e >0 depending only on the gadget bounds and independent of d and ϵ

    Standard comparison with the fundamental solutions rk yields an inequality of the form |ek(x)| ≤C⋆ e λ d−k C ′ × δB ′2, k=d, . . . ,0, x∈[−1,1], for a constant C ⋆ e >0 depending only on the gadget bounds and independent of d and ϵ. The inexact output error can now be written as yd(x)−p d(x) = Multδ(x, ˜b1(x))−x ˜b1(x) +xe 1(x)− 1 2 e2(x). The first term ...

  4. [8]

    There exists a rational functionR: [−1,1]→R such that for any scalar-input, scalar-output GELU network, F , with uniformly bounded weights and biases that satisfies ∥F−R∥ L∞([−1,1]) ≤ε , the size of the network isΩ log(1/ε) . Proof. Let 0< η < ε1/4 and consider R(x) = 1 x2+η2 . Then R∈C ∞([−1,1]) and R′′(0) =−2/η 4, so |R′′(0)|= 2/η 4 > 2/ε. We quantify t...

  5. [9]

    Then there exists a GELU networkf: [−1,1] d →[−1,1]of size O kMlog 2(SM (L)/ε) , S M (L) := M−1X j=0 Lj, such thatmax x∈[−1,1]d |R(x)−f(x)| ≤ε

    Let R: [−1,1] d →[−1,1] be a rational network with M layers and at most k nodes per layer, where each node computes x7→r(a ⊤x+b) with a rational function r: [−1,1]→[−1,1] whose Lipschitz constant on [−1,1] is at most L, and where ∥a∥1 +|b| ≤1. Then there exists a GELU networkf: [−1,1] d →[−1,1]of size O kMlog 2(SM (L)/ε) , S M (L) := M−1X j=0 Lj, such tha...

  6. [10]

    Let LG := sup{|G′(t)|:|t| ≤1}

    Let f: [−1,1] d →[−1,1] be a GELU network with M layers and at most k nodes per layer, where each node computes x7→G(a ⊤x+b)with G(z) = z 2 1 + tanh(α(z+βz 3)) , α= p 2/π, β= 0.044715, and where ∥a∥1 +|b| ≤1. Let LG := sup{|G′(t)|:|t| ≤1}. Then there exists a rational network R: [−1,1] d →[−1,1] of size O kMlog log 3(SM (LG)/ε) such thatmax x∈[−1,1]d |f(x...

  7. [11]

    (Rational → GELU) Fix a layer index J. Let H be the subnetwork of the rational network R up to layer J, and let HG be obtained from H by replacing, node by node, each rational activation ri,j by a GELU subnetwork fi,j with tolerance ϵj >0 : max t∈[−1,1] ri,j(t)−f i,j(t) ≤ϵ j,1≤j≤J,1≤i≤k j. 27 Rational Neural Networks have Expressivity Advantages Write yi,...

  8. [12]

    (GELU → rational) Fix a layer index J. Let H be the subnetwork of the GELU network f up to layer J, and let HR be obtained fromHby replacing eachGwith a rational network eRδ such that max t∈[−1,1] G(t)− eRδ(t) ≤δ, eRδ([−1,1])⊂[−1,1]. Write yi,j+1(x) :=G a⊤ i,j+1H(x) +b i,j+1 ,by i,j+1(x) := eRδ a⊤ i,j+1HR(x) +b i,j+1 , and define errors as before. Add and...

    2015

  9. [1997]

    URL https://link.springer.com/book/10

    doi: 10.1007/978-3-662-03329-6. URL https://link.springer.com/book/10. 1007/978-3-662-03329-6. Sandler, M., Howard, A. G., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and lin- ear bottlenecks.2018 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 4510–4520,

    work page doi:10.1007/978-3-662-03329-6 2018

  10. [2009]

    Liu, Z., Lin, Y ., Cao, Y ., Hu, H., Wei, Y ., Zhang, Z., Lin, S., and Guo, B

    URL https://www.cs.toronto.edu/ ˜kriz/learning-features-2009-TR.pdf. Liu, Z., Lin, Y ., Cao, Y ., Hu, H., Wei, Y ., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https: //arxiv.org/abs/2103.14030. Maas, A. L., Hannun, A. Y ., and Ng, A. Y . Rectifier nonlin- earities improve neural netwo...

    Pith/arXiv arXiv 2009

  11. [2018]

    org/CorpusID:4555207

    URL https://api.semanticscholar. org/CorpusID:4555207. Surdej, R., Bortkiewicz, M., Lewandowski, A., Ostaszewski, M., and Lyle, C. Balancing expressivity and robustness: Constrained rational activations for reinforcement learn- ing, 2025. URL https://arxiv.org/abs/2507. 14736. Telgarsky, M. Neural networks and rational functions, 2017. URLhttps://arxiv.or...

    Pith/arXiv arXiv 2025

  12. [9045]

    URL https://www.sciencedirect.com/ science/article/pii/S002190452100040X

    doi: https://doi.org/10.1016/j.jat.2021.105577. URL https://www.sciencedirect.com/ science/article/pii/S002190452100040X. Gonchar, A. A. Rational approximation of analytic func- tions.Proceedings of the Steklov Institute of Math- ematics, 272(Suppl 2):44–57, 2011. doi: 10.1134/ S0081543811030047. URL https://doi.org/10. 1134/S0081543811030047. 9 Rational ...

    arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.