Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that trainable rational activation functions, despite improving plasticity, cause activation explosions and critic overestimation in high-update-to-data continuous control, and that a constrained variant with asymptotic…

desk verdict A useful empirical fix and a genuinely new failure mode, but the causal story is confounded because the constrained variant also changes the asymptotic degree; the paper still deserves a serious referee. read the letter →

arxiv 2507.14736 v1 pith:DTJIJLJE submitted 2025-07-19 cs.LG

Rafa{l} Surdej , Micha{l} Bortkiewicz , Alex Lewandowski , Mateusz Ostaszewski , Clare Lyle This is my paper
classification cs.LG
keywords rationalactivationfunctionstrainableactivationsreinforcementlearningcontinuouscontroloverestimationplasticitycontinualexplosion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether trainable rational activation functions—neuron activations defined as ratios of polynomials with learned coefficients—are safe to use in reinforcement learning. It argues they are not, in their original unconstrained form, for high-update-to-data continuous control: the numerator coefficients can outgrow the denominator, the activation produces enormous outputs, gradient norms climb, and the critic overestimates values until training diverges. To fix this, the paper proposes a constrained variant that adds a term $\left|x/c\right|^{n+1}$ to the denominator so the activation decays to zero for large inputs, and drops the constant numerator term so $f(0)=0$. Across MetaWorld and DeepMind Control Suite, the constrained variant with periodic resets matches or beats ReLU with resets, while the unconstrained version fails in several MetaWorld tasks. The paper also shows the constraint costs plasticity in continual learning, revealing a trade-off between expressivity and stability in trainable activations.

What carries the argument

The load-bearing object is the constrained rational activation function, a ratio of polynomials in which every numerator coefficient is trainable but the function is forced to behave like a bounded activation: the denominator contains the internal-regularization term $\left|x/c\right|^{d}$ with $d=n+1$, so the highest power of $x$ in the denominator exceeds that of the numerator and $f(x)\to 0$ as $|x|\to\infty$, and the constant term $a_0$ is removed so $f(0)=0$. This structural constraint is what prevents coefficient imbalance from translating into huge activation outputs; the paper's evidence is that constrained rationals keep activation-coefficient gradient norms low and avoid critic overestimation, whereas original rationals show activation explosions under the same conditions. A secondary mechanism is the interplay of coefficient initialization, weight decay, and periodic resets, which the paper studies through Neural Tangent Kernel measurements and gradient-norm comparisons.

What would settle it

Train SAC with UTD 10 on a MetaWorld task such as Sweep using original rational activations but with the activation output hard-clamped to a modest range; if overestimation and divergence still occur, unbounded output is not the root cause. Conversely, keep the constrained decay term but raise the numerator degree back above the denominator; if training stays stable, the decay term alone, not the degree balance, is doing the work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that rational activations exhibit a previously unreported failure mode in high-UTD continuous control: an "activation explosion" in which learned numerator coefficients grow large relative to the denominator, outputs become extremely large, gradient norms spike, and the critic overestimates values, producing divergent learning. This happens under dynamic input distributions in MetaWorld and some DeepMind Control Suite tasks, is worsened by high replay ratios, and is not seen in discrete-action Atari. The paper's proposed constrained rational activation $f(x)=\frac{a_n x^n+\cdots+a_1 x}{|b_m x^m|+\cdots+|b_1 x|+1+\left|x/c\right|^{d}}$ with $d=n+1$ removes the constant numerator term and forces asymptotic decay to zero, which empirically stabilizes training. Combined with periodic resets, it reaches performance comparable to ReLU with resets while retaining faster early learning; however, it reduces long-run plasticity on shuffled-label MNIST, and weight decay on activation coefficients partially restores that plasticity. The paper frames this as a fundamental expressivity–stability trade-off in trainable activation design.

Load-bearing premise

The fix assumes that the instability comes from the rational activation's output growing too large; the evidence is correlational, and if the true cause is something else about the added expressivity, the constrained variant's success on these benchmarks may not generalize.

Editorial extensions

If this is right

  • High-UTD continuous-control agents should not use unconstrained rational activations as a drop-in replacement for ReLU; in MetaWorld they produce activation explosions and severe critic overestimation, even with resets.
  • A structurally bounded rational activation—one that decays for large inputs and passes through zero—is a practical alternative: with periodic resets it matches or outperforms ReLU with resets on aggregated MetaWorld and DeepMind Control Suite scores.
  • Stability comes at a measurable price: the constrained variant loses long-term plasticity on shuffled-label MNIST, so users face an expressivity–stability trade-off rather than a free win.
  • Weight decay applied only to activation coefficients counteracts coefficient drift and preserves adaptability in continual learning, but strong decay harms RL performance, so the right amount of regularization depends on the regime.
  • The absence of explosions in Atari 100k suggests the failure mode is specific to continuous control with dynamic input distributions, not a universal property of rational activations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the output-scaling diagnosis is right, the design principle extends beyond rational functions: any trainable or parametric activation used in high-UTD RL should have controlled asymptotic behavior, which would predict that learned polynomial or Fourier activations without such bounds will hit similar explosions.
  • The choices $d=n+1$ and $c=2$ are hand-set; an adaptive schedule that starts with a mild constraint and tightens it during training could recover some of the plasticity the fixed constraint gives up, and is directly testable on the same benchmarks.
  • A sharper test of the mechanism would manipulate reward scale in a fixed environment: since the paper links explosions to high-magnitude losses in MetaWorld, scaling DeepMind Control Suite rewards up or MetaWorld rewards down should move where and whether explosions appear.
  • The continual-learning results suggest coefficient initialization is a control knob: small initial coefficients preserve plasticity, while large coefficients improve RL stability, so per-environment initialization rules could be tuned without changing the architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies trainable rational activation functions in reinforcement learning (RL) and continual learning (CL). It reports that unconstrained rational activations (OR) with numerator degree n=3 and denominator degree m=2 cause instability in high-update-to-data (UTD) continuous-control RL, manifesting as large activation outputs and critic overestimation. To address this, the authors propose a constrained rational activation (CR) in Eq. (1) that removes the numerator constant term a0 and adds a denominator term |x/c|^{n+1}, making the function decay to zero as |x| grows. Experiments on MetaWorld, DeepMind Control Suite, MNIST with shuffled labels, Split CIFAR-100, and Atari 100k show that CR stabilizes training and improves or matches ReLU-based baselines in continuous control, while the original rationals sometimes fail. The paper also reports a trade-off: CR loses plasticity relative to OR on the MNIST continual-learning benchmark, and it analyzes the effect of coefficient initialization, weight decay, and NTK properties.

Significance. If the empirical findings hold, the paper identifies a practically important failure mode of trainable rational activations in high-UTD continuous control and proposes a simple, effective fix. The study is broad in scope, covering 15 MetaWorld and 15 DMC tasks, several continual-learning settings, and Atari 100k, and it provides open-source code. Strengths include the explicit documentation of activation and gradient explosions (e.g., Figure 17), the NTK-based analysis of initialization effects, and the demonstration that Layer Normalization and resets alone do not cure the instability. However, the central causal claim—that unconstrained flexibility causes the instability and that the proposed structural constraints are the reason for stabilization—is not fully isolated, and the claimed expressivity-plasticity trade-off is contradicted by part of the paper's own continual-learning evidence. These issues require additional controls and more careful framing before the main conclusions are fully supported.

major comments (4)
  1. [Section 3.1, Eq. (1)] The comparison between Original Rationals (OR) and Constrained Rationals (CR) changes two properties simultaneously: the constraint set (removal of a0, addition of |x/c|^{n+1}) and the asymptotic degree of the function. With n=3,m=2, OR grows linearly as |x| tends to infinity, whereas CR with d=n+1 decays to zero. The manuscript's own degree sweep in Appendix A shows that degree imbalance is decisive for stability ((7,1) and (5,0) explode on MNIST), and Figure 15 shows that increasing both degrees in the original form does not help. To support the causal claim that unconstrained flexibility/coefficient imbalance, rather than degree imbalance, is the root cause of the instability, the authors should test an unconstrained rational with denominator degree exceeding numerator degree (e.g., n=2,m=3 or n=3,m=4) in the same RL setups. Without such a control, the stabilization could simply be a manifestation of the known Padé degree-balance effect, not of the proposed structural constraints.
  2. [Section 4.3-4.4 and Appendix D.1] The claimed 'fundamental trade-off between expressivity and plasticity' is stated as a main result, but the evidence is mixed. In the MNIST shuffled-labels experiment, CR degrades faster than OR (Figure 8), yet in Split CIFAR-100 (Appendix D.1, Figure 31) CR outperforms OR and ReLU on later tasks, which the authors describe as 'improved adaptability in later tasks.' The trade-off claim should be restricted to the MNIST benchmark or reconciled with the Split CIFAR-100 results; as written, the general conclusion in Section 4.4 overstates the evidence.
  3. [Section 3.1, Figures 3 and 5] The paper's key instability diagnostic is 'overestimation,' but this metric is never formally defined. The reader cannot tell whether it is a TD error, Q_target minus Q_current, the difference between the learned Q and a Monte Carlo return, or something else. Figures 3, 5, and 9 use this quantity to support the central claim, so the definition and the computation details (including how the target is estimated and over which states or timesteps it is averaged) should be stated in the main text or an appendix.
  4. [Eq. (1), Section 4.1] The proposed constrained variant has two hand-set hyperparameters, d=n+1 and c=2. While d is justified asymptotically, c controls the input scale at which the |x/c|^{n+1} term becomes active, and no sensitivity analysis is provided. If the stabilization depends critically on c=2, the design principle is less 'structural' than claimed. The authors should report an ablation over c (or argue that the results are insensitive) to support the generality of the proposed fix.
minor comments (6)
  1. [Abstract vs. Section 4.4] The abstract says the constrained variant improves stability 'while preserving adaptability,' but Section 4.4 concludes that the modifications come at the cost of plasticity; please reconcile this phrasing with the trade-off result.
  2. [Figures 1 and 4] Pooling MetaWorld success rates (0-1) with DMC returns divided by 1000 is an arbitrary rescaling, and the pooled IQM ranking can change under different monotone transformations of the DMC returns; please report per-domain results as primary and justify the pooled aggregate, or present it as secondary.
  3. [Section 3.1] The term 'activation explosion' is defined informally as 'activations producing excessively large outputs,' without a threshold or a quantitative criterion; please provide an operational definition or at least specify the values used in the figures.
  4. [Appendix A] The text says 'coordinates (4,2) (i.e. (n=3,m=2))', which is confusing because (4,2) appears to count the number of coefficients while (n=3,m=2) counts polynomial degrees; please clarify the coordinate convention.
  5. [Section 3] There is a typo in 'the polynomial coeffecients' (should be 'coefficients'); similar minor typographical issues appear throughout the manuscript and should be fixed in a final proofread.
  6. [Section 4.3 and Appendix D.1] The differing results between MNIST shuffled-labels and Split CIFAR-100 are attributed to task length and number of examples in Appendix D.1; please make this explicit in the main text where the trade-off is discussed, so that the reader understands the scope of the claim.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: at most a minor definitional tautology in the 'activation explosion' diagnosis, with the central performance and trade-off claims independently tested.

  1. self definitional [Sec. 3.1 (Takeaway) defining 'activation explosion'; Sec. 4 (Eq. 1) constraining output decay]
    "We refer to this behavior as activation explosion, characterized by excessive activation outputs due to coefficient imbalance. ... To prevent uncontrolled growth, we ensure that the highest power of x in the denominator exceeds that of the numerator ( d > n). In our experiments, we set d = n+1, which forces the activation function to asymptotically decay to zero as|x| → ∞."

    The paper defines 'activation explosion' as excessive activation outputs, then designs Eq. (1) so that outputs decay by construction. Therefore the statement that Constrained Rationals 'eliminate the explosion problem' follows from the definition plus the formula rather than from an independent empirical test. This is a mild self-definitional component. The central claims—that unconstrained rationals are unstable in high-UTD continuous control, that CR improves performance, and that CR reduces plasticity in continual learning—are not forced by this definition; they are empirical and could have gone the other way. No fitted parameter is renamed as a prediction, so the load-bearing result remains independent.

full rationale

The paper's central result rests on direct experiments across MetaWorld, DMC, Atari, and continual-learning benchmarks, not on fitting a parameter to a target and calling it a prediction. The constrained activation in Eq. (1) is a structural proposal whose boundedness follows from the formula, but whether boundedness improves training stability and performance is an empirical question tested against external tasks. Self-citations such as Nauman et al. (2024a) and Lewandowski et al. (2025a) motivate the settings but are not load-bearing for the rational-activation failure mode or the proposed fix; no uniqueness theorem or cited ansatz supplies the core argument. The only mildly circular element is that 'activation explosion' is defined by the same large-output behavior that the constraint removes, but the measured RL performance gains, the continual-learning plasticity trade-off, and the Atari contrasts are independent of that tautology. The degree-choice confound noted by skeptics is a validity concern about causal attribution, not a circularity under the enumerated patterns.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies mostly on empirical observation. Free parameters: the constraint degree d and scale c are set by hand; initialization schemes are manually designed. Domain assumptions: high-UTD continuous control and shuffled-label MNIST are taken as representative of instability and plasticity respectively. No new physical entities are introduced.

free parameters (3)
  • Constraint denominator power d = n+1 (4 for n=3)
    Chosen by hand to force asymptotic decay; no tuning or theoretical derivation.
  • Constraint scale c = 2 (from |x/2|^4 in Figure 12)
    Set manually; not justified in the paper.
  • Rational coefficient initializations (Low/High init) = Table 2 values
    Manually designed to approximate Leaky ReLU or larger shapes; results depend on them (Figure 10), so the claim is conditional on this choice.
assumptions (3)
  • domain assumption High update-to-data ratio (UTD=10) is the regime where instability is most relevant for continuous control
    The paper focuses on UTD=10 based on prior work (D'Oro et al., Nikishin et al.) but does not justify that this regime is representative of practical RL deployments.
  • domain assumption MNIST with reshuffled labels is a valid proxy for plasticity in deep RL
    Used to measure plasticity loss; standard benchmark, but its connection to continuous-control RL plasticity is assumed.
  • standard math The rational function denominator with absolute values is differentiable and stable in the regimes tested
    The paper relies on this for gradient-based training without formal analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning." pith.science (2026). https://pith.science/paper/DTJIJLJE

@misc{pith2026250714736,
  author       = {Pith},
  title        = {Pith review of: Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DTJIJLJE}},
  note         = {Machine review of arXiv:2507.14736}
}
read the original abstract

Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainable activation functions defined as ratios of polynomials (rational functions) have been proposed to enhance plasticity in reinforcement learning. However, their impact on training stability remains unclear. In this work, we study trainable rational activations in both reinforcement and continual learning settings. We find that while their flexibility enhances adaptability, it can also introduce instability, leading to overestimation in RL and feature collapse in longer continual learning scenarios. Our main result is demonstrating a trade-off between expressivity and plasticity in rational activations. To address this, we propose a constrained variant that structurally limits excessive output scaling while preserving adaptability. Experiments across MetaWorld and DeepMind Control Suite (DMC) environments show that our approach improves training stability and performance. In continual learning benchmarks, including MNIST with reshuffled labels and Split CIFAR-100, we reveal how different constraints affect the balance between expressivity and long-term retention. While preliminary experiments in discrete action domains (e.g., Atari) did not show similar instability, this suggests that the trade-off is particularly relevant for continuous control. Together, our findings provide actionable design principles for robust and adaptable trainable activations in dynamic, non-stationary environments. Code available at: https://github.com/special114/rl_rational_plasticity.

Figures

Figures reproduced from arXiv: 2507.14736 by the authors.

Figure 1
Figure 1. Interquartile Mean (IQM) performance after 1M [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Critic overestimation and activation distributions from the second hidden layer of critic networks trained [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Mean overestimation on DMC benchmark for Original Rationals for different UTD settings. The higher UTD the less stable OR become leading to overestimation. under which we conducted the most of our experiments, the insta￾bility of rational activation functions was particularly noticeable in MetaWorld environments. In many of them, such as Sweep, training the agent did not yield any improvements. Even though we used a… view at source ↗
Figures from the paper (29 more)
Figure 4
Figure 4. Figure 4: IQM performance after 1M steps, aggregated [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overestimation on MW benchmark of different SAC setups. Original rational acti￾vation functions exhibit tendency to huge over￾estimation [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Activation distributions from the second hidden layer of critic networks using LayerNormalization [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: BroNet with different activation func￾tions on DMC & MW combined results. However, we discovered that this is not the case. Rational activation functions still exhibit explosive behavior ( [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: MNIST with labels reshuffling over 60 tasks: Original rational functions retain plasticity longer than modified version. WD helps our Con￾strained Rationals preserve adaptability. To evaluate the adaptability of our modified rational activation functions in a continual…
Figure 9
Figure 9. Figure 9: MetaWorld: Overestimation compar￾ison between Original Rationals (OR) and Con￾strained Rationals (CR) with large initialization scales. OR remains less stable and can lead to severe overestimation, while CR shows more controlled behavior, mitigating instability. Bringi…
Figure 10
Figure 10. Figure 10: Left: DMC: Impact of coefficient initialization on RL performance. No clear trend is observed. Right: [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: NTK analysis for different coefficient initializations in RL. Higher coefficient values increase NTK rank [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Impact of weight decay applied exclusively to rational activation coefficients. In top row. Left: Perfor [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Performance of trainable rational activation functions with varying numerator and denominator degrees on [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Training loss for a (7,1) rational activation on MNIST, showing severe instability. [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: IQM performance of our Constrained Rationals with different polynomial degrees across 1M steps aggree [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: Comparison of Constrained Rationals (CR) with and without [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Gradient norms of activation coefficients during MetaWorld training with UTD 10. We compare Original [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]
Figure 18
Figure 18. Figure 18: Impact of Spectral Normalization (SN) on RL performance across 1M steps under UTD=10. Left: DMC [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: IQM performance across 1M steps, aggregated over 15 MetaWorld (MW) and 15 DeepMind Control Suite [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20: Impact of applying Spectral Normalization to the outputs of activation functions during training. Left: [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]
Figure 21
Figure 21. Figure 21: Performance across 1M steps. Left: DMC returns. Right: MW goals. Comparison of activation function [PITH_FULL_IMAGE:figures/full_fig_p019_21.png]
Figure 22
Figure 22. Figure 22: Performance across 1M steps. Left: DMC returns. Right: MW goals. Comparison of activation function [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 23
Figure 23. Figure 23: Comparison of BroNet architecture with different activation functions: Original Rationals (OR), Con [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: Performance of BroNet architecture across 1M steps. Left: DMC returns. Right: MW success rate. [PITH_FULL_IMAGE:figures/full_fig_p020_24.png]
Figure 25
Figure 25. Figure 25: Results of 15 tasks from DeepMind Control Suite for different setups for 1M steps. We present the mean of rewards and 95% confidence intervals [PITH_FULL_IMAGE:figures/full_fig_p021_25.png]
Figure 26
Figure 26. Figure 26: Results of 15 tasks from MetaWorld for different setups for 1M steps. We present the mean of success rate and 95% confidence intervals. B.7 NTK ANALYSIS IN REINFORCEMENT LEARNING In this section, we provide additional details on the empirical computation of NTK mentio…
Figure 27
Figure 27. Figure 27: Visualization of empirical NTK computed on a random sampling of 256 observations. Each row corre [PITH_FULL_IMAGE:figures/full_fig_p022_27.png]
Figure 28
Figure 28. Figure 28: Visualization of empirical NTK computed on a random sampling of 256 observations. Each row corre [PITH_FULL_IMAGE:figures/full_fig_p022_28.png]
Figure 29
Figure 29. Figure 29: IQM performance across five Atari 100k games ( [PITH_FULL_IMAGE:figures/full_fig_p024_29.png]
Figure 30
Figure 30. Figure 30: Gradient norms of the numerator and denominator coefficients of Original Rational (OR) activations during [PITH_FULL_IMAGE:figures/full_fig_p024_30.png]
Figure 31
Figure 31. Figure 31: Training accuracy for the Split/CIFAR-100 experiment. Comparison of rational activation functions with [PITH_FULL_IMAGE:figures/full_fig_p025_31.png]
Figure 32
Figure 32. Figure 32: MNIST with non-stationary targets: Comparison of different coefficient initialization strategies. Functions [PITH_FULL_IMAGE:figures/full_fig_p025_32.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Activation Function Design Sustains Plasticity in Continual Learning

    cs.LG 2025-09 unverdicted novelty 5.0 of 10

    Smooth-Leaky and Randomized Smooth-Leaky activations mitigate loss of plasticity in continual learning by targeting negative-branch shape and saturation behavior.

Reference graph

Works this paper leans on

40 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  3. [3]

    Ball, Laura M

    Philip J. Ball, Laura M. Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proce...

    work page 2023

  4. [4]

    Bellemare, Yavar Naddaf, J

    Marc G. Bellemare, Yavar Naddaf, J. Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2012. doi:10.1613/jair.3912

    doi:10.1613/jair.3912 2012

  5. [5]

    Adaptive rational activations to boost deep reinforcement learning

    Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=g90ysX1sVs

    work page 2024

  6. [6]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023 a . URL https://openreview.net/forum?id=OpC-9aBBVJe

    work page 2023

  7. [7]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023 b . URL https://openreview.net/forum?id=OpC-9aBBVJe

    work page 2023

  8. [8]

    Abbeel, and S

    Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, 2018

    work page 2018

Show all 40 references
  1. [9]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence,...

    2018

  2. [10]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Neural Information Processing Systems, 2018

    2018

  3. [11]

    Maintaining plasticity via regenerative regularization

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv: 2308.11958, 2023

    2023 arXiv

  4. [12]

    Wurman, Jaegul Choo, Peter Stone, and Takuma Seno

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning, 2024. URL https://arxiv.org/abs/2410.09754

    2024 arXiv

  5. [13]

    Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado. Curvature explains loss of plasticity, 2024. URL https://openreview.net/forum?id=SkF7NZGVr5

    2024

  6. [14]

    Alex Lewandowski, Micha Bortkiewicz, Saurabh Kumar, Andr \'a s Gy \"o rgy, Dale Schuurmans, Mateusz Ostaszewski, and Marlos C. Machado. Learning continually by spectral regularization. In International Conference on Learning Representations, 2025 a

    2025

  7. [15]

    Alex Lewandowski, Dale Schuurmans, and Marlos C. Machado. Plastic learning with deep fourier features. In International Conference on Learning Representations, 2025 b

    2025

  8. [16]

    Functional regularization for reinforcement learning via learned fourier features

    Alexander Li and Deepak Pathak. Functional regularization for reinforcement learning via learned fourier features. Advances in Neural Information Processing Systems, 34: 0 19046--19055, 2021

    2021

  9. [17]

    Understanding and preventing capacity loss in reinforcement learning

    Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ZkC8wKoLbQ7

    2022

  10. [18]

    Understanding plasticity in neural networks

    Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40...

    2023

  11. [19]

    Understanding plasticity in neural networks

    Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp.\ 23190--23211. PMLR, 2023 b

    2023

  12. [20]

    Mavor-Parker, Matthew J

    Augustine N. Mavor-Parker, Matthew J. Sargent, Caswell Barry, Lewis Griffin, and Clare Lyle. Frequency and generalisation of periodic activation functions in reinforcement learning. arXiv preprint arXiv: 2407.06756, 2024

    2024 arXiv

  13. [21]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602, 2013

    2013 arXiv

  14. [22]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  15. [23]

    Pad \' e activation units: End-to-end learning of flexible activation functions in deep networks

    Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Pad \' e activation units: End-to-end learning of flexible activation functions in deep networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . Op...

    2020

  16. [24]

    Pad\'e activation units: End-to-end learning of flexible activation functions in deep networks, 2020 b

    Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Pad\'e activation units: End-to-end learning of flexible activation functions in deep networks, 2020 b . URL https://arxiv.org/abs/1907.06732

    2020 arXiv

  17. [25]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning, 2024 a

    Michal Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning, 2024 a . URL https://arxiv.org/abs/2403.00514

    2024 arXiv

  18. [26]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control, 2024 b

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control, 2024 b . URL https://arxiv.org/abs/2405.16158

    2024 arXiv

  19. [27]

    Courville

    Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre - Luc Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesv \' a ri, Gang Niu, and Sivan Sabato (eds.), International Conference on...

    2022

  20. [28]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. International Conference on Machine Learning, 2012

    2012

  21. [29]

    Data-efficient reinforcement learning with self-predictive representations, 2021

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations, 2021. URL https://arxiv.org/abs/2007.05929

    2021 arXiv

  22. [30]

    Courville, Marc G

    Max Schwarzer, Johan Samir Obando - Ceron, Aaron C. Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato...

    2023

  23. [31]

    Ghada Sokar, Rishabh Agarwal, P. S. Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. International Conference on Machine Learning, 2023 a . doi:10.48550/arXiv.2302.12902

    2023 doi arXiv

  24. [32]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th Internation...

    2023

  25. [33]

    Srinivasan, B

    Matthew Tancik, Pratul P. Srinivasan, B. Mildenhall, Sara Fridovich-Keil, N. Raghavan, Utkarsh Singhal, R. Ramamoorthi, J. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Neural Information Processing Systems, 2020

    2020

  26. [35]

    Deepmind control suite, 2018 b

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018 b . URL https://arxiv.org/abs/1801.00690

    2018 arXiv

  27. [36]

    Overcoming the spectral bias of neural value approximation

    Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approximation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=vIC-xLFuM6

    2022

  28. [37]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference...

    2019

  29. [38]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URL https://arxiv.org/abs/1910.10897

    2021 arXiv

  30. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  31. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  32. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.