Pith. sign in

REVIEW 1 major objections 5 minor 137 references

Reinforcement learning can work without absolute state values.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:26 UTC pith:AGIJK4UI

load-bearing objection Correct theory, honest experiments, but the headline 'competitive' claim rests on baseline numbers that appear to be quoted, not re-run. the 1 major comments →

arxiv 2607.21120 v1 pith:AGIJK4UI submitted 2026-07-23 cs.LG cs.AI

Relative Value Learning

Marc H\"oftmann , Jan Robine , Stefan Harmeling This is my paper
classification cs.LG cs.AI MSC 68T0590C40
keywords relative value learningantisymmetric value functionspairwise Bellman operatorgeneralized advantage estimationpolicy gradientgauge invarianceAtari benchmarksPPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that absolute state values are behaviorally redundant: only differences between values are needed to choose actions. It proposes Relative Value Learning, which trains an antisymmetric function Δ(s_i,s_j) to approximate Vπ(s_i)−Vπ(s_j) directly, and proves that a pairwise Bellman operator on such functions is a γ-contraction whose unique fixed point is exactly the true value differences. From that operator it derives well-posed 1-step, n-step, and λ-return bootstrapping targets that avoid needing absolute values at terminal states, and it reconstructs generalized advantage estimation from pairwise differences, obtaining an unbiased policy-gradient estimator when the learned model is exact. Replacing PPO's absolute critic with this relative critic produces scores competitive with standard PPO on 49 Atari games, supporting the claim that relative value estimation is an effective alternative.

Core claim

The paper's central discovery is that the additive offset in a value function is unneeded gauge freedom, so the critic's job can be redefined as learning pairwise differences rather than absolute magnitudes. Formally, the pairwise Bellman operator (T^πΔ)(s_i,s_j)=r^π(s_i)-r^π(s_j)+γ E[Δ(s'_i,s'_j)] is a γ-contraction on the space of bounded antisymmetric functions, and its unique fixed point equals Vπ(s_i)−Vπ(s_j). The paper derives bootstrapping targets (1-step, n-step, λ-return) that remain well-posed when successor states are terminal, and it reconstructs generalized advantage estimation from these pairwise differences. When the learned model equals the true difference function, the resul

What carries the argument

The load-bearing object is the pairwise Bellman operator T^π, defined on the Banach space of bounded antisymmetric pairwise functions by (T^πΔ)(s_i,s_j)=r^π(s_i)−r^π(s_j)+γ E[Δ(s'_i,s'_j)]. Because the reward-difference term cancels when two functions are compared, T^π contracts with factor γ and has a unique fixed point equal to the true value differences Vπ(s_i)−Vπ(s_j). Around this operator the paper builds relative TD residuals, 1-step/n-step/λ targets with explicit terminal-state case handling, and the R-GAE estimator; a trajectory-ranking step estimates start-state offsets to reduce the variance introduced by anchoring each rollout's relative values to zero.

Load-bearing premise

The guarantees assume the learned pairwise model Δθ matches the true value differences; since the paper's difference head is a linear function of the encoder and the paper admits this near-linear head may underfit complex value differences, a poor fit would make R-GAE biased and weaken the variance-reduction argument.

What would settle it

On a small tabular MDP with known Vπ, iterate the pairwise Bellman operator numerically and compare its fixed point to Vπ(s_i)−Vπ(s_j) for every pair; then estimate the policy gradient with R-GAE and compare with the true gradient. Any deviation beyond sampling noise would falsify the contraction and unbiasedness claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The offset of an absolute critic is a redundant degree of freedom; RV removes it by construction, so reward-shaping shifts and baseline offsets cannot move the learned critic's predictions.
  • GAE can be computed without knowing the absolute value of any state, and when the difference model is exact the policy-gradient estimator is unbiased.
  • Bootstrapping remains well-defined through terminal states using only observable rewards and nonterminal pairwise differences, so relative value learning is compatible with standard actor-critic training.
  • Relative value learning can replace an absolute critic in PPO with no loss of benchmark performance, beating the absolute critic on 30 of 49 Atari games.
  • Trajectory ranking provides a data-dependent initialization that reduces the variance inflation from the per-trajectory constant baseline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The pairwise target makes comparison-only feedback natural: in preference-based or human-in-the-loop RL, where the absolute reward scale is unidentifiable but pairwise comparisons remain well-defined, RV's loss is a plausible direct objective. The paper motivates this direction but does not test it.
  • The trajectory-ranking offset is a batch-level gauge anchor; one could learn these offsets end-to-end rather than with row-mean-minus-min, which would directly optimize away the residual B_t variance rather than reduce it heuristically.
  • The same contraction argument should extend to Q-learning-style relative critics or relative value iteration in off-policy settings, but pairwise state sampling is O(B²), so a structured or learned sampler would be needed to make that practical.
  • The linear difference head is the most likely capacity bottleneck; an antisymmetric nonlinear head could recover bias on games where RV trails PPO, a design choice the paper leaves open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes Relative Value Learning (RV), a critic that directly learns antisymmetric pairwise state-value differences Δ_θ(s_i,s_j) ≈ V^π(s_i) − V^π(s_j) instead of absolute values. It defines a pairwise Bellman operator T^π on bounded antisymmetric functions and proves it is a γ-contraction whose unique fixed point is the true value difference (Thm 3.1). It derives one-step, n-step, and λ-return targets that handle terminal successors using only observable rewards and pairwise differences (Eqs. 19–22), and reconstructs GAE from pairwise differences (R-GAE), showing that, when Δ_θ = Δ^π, the relative advantage differs from standard GAE by a trajectory-constant term that cancels in the policy gradient (Lemma 3.2, Cor. 3.3). To reduce the variance induced by that constant, the paper introduces a trajectory-ranking offset estimator (Sec. 4). Experiments replace the PPO critic with the RV critic on 49 Atari games and report that PPO+RV is competitive with PPO and DAE (Table 1, Fig. 3). The paper also includes ablations of the offset initialization (Fig. 4) and pair sampling (Table 4), hyperparameters, and a limitations section.

Significance. The theoretical core is a genuine contribution: Thm 3.1 is clean and correct; Eq. 20's four terminal cases are exact under the convention that terminal states have zero value; and Lemma 3.2/Cor. 3.3 establish that R-GAE is unbiased in the exact-model limit. The derivations are self-contained and checkable by hand, the code is released, and the Limitations section is candid about the restrictiveness of the linear difference head, the O(B^2) pairing cost, and the uninformative-ranking risk. If the empirical comparison were confirmed under a controlled protocol, the paper would show that a difference-based critic is a viable alternative to absolute critics with no practical penalty, which is of clear interest to the RL community. However, the headline empirical claim currently rests on baseline numbers whose provenance and evaluation protocol are not documented, so the practical significance is not yet established.

major comments (1)
  1. [§6.1, Table 1; §6 Compute Resources] The headline claim that PPO+RV 'attains competitive performance' and 'exceeds PPO on 30 out of 49 games' is not supported by the evidence as presented. The paper never states whether the PPO and DAE columns were produced in the same codebase with the same environment wrappers, no-op reset behavior, frame skip, evaluation window, and seeds as PPO+RV. The Compute Resources paragraph reports 490 runs (49 games × 10 seeds), which accounts only for PPO+RV; no compute is reported for the baseline columns. If those numbers are quoted from prior papers, differences in preprocessing and evaluation can easily exceed the reported standard deviations, in which case 30/49 wins is close to chance and the bootstrap intervals in Fig. 3 are not a valid comparison. Please re-run the baselines in the same pipeline or clearly state their provenance and protocol, and qualify the competitive claim accordingly
minor comments (5)
  1. [Cor. 3.3 proof] The proof writes B_t = C(1−(γλ)^{T−t+1})/(1−γλ), omitting the (1−γ) factor present in Lemma 3.2's definition B_t = (1−γ)C Σ_{l=0}^{T−t}(γλ)^l. The conclusion is unaffected because any t-dependent constant vanishes in the score-function identity, but the displayed formula should match Lemma 3.2.
  2. [Appendix A, Eqs. (32)–(34)] The derivations use '≈' where the equalities are in fact exact under the standard convention V(s_terminal)=0 (e.g., V(s_i)=r_i when s_{i+1} is terminal). Replacing ≈ with = would avoid the impression that the one-step targets are approximate.
  3. [Fig. 4 / §6.2] The ablation for value initialization is shown only as per-game curves with no aggregate statistic. A table or bootstrap confidence intervals over the 49 games would substantiate the claim that 'relative ranking is needed'.
  4. [Eq. (28) / Table 3] The critic loss is written as plain MSE, but Table 3 lists 'RV value clipping 0.15'. Please clarify how value clipping enters the critic update and how it interacts with the n-step targets.
  5. [Abstract / Cor. 3.3] The abstract says R-GAE is 'an unbiased policy-gradient estimator' without the qualifier that this holds when Δ_θ = Δ^π, as assumed in Cor. 3.3. Adding that qualifier would prevent overstatement, especially since the linear difference head in §5.1 may underfit complex value differences (as acknowledged in §7).

Circularity Check

0 steps flagged

No significant circularity: the theoretical derivations are self-contained and the central claims do not reduce to fitted inputs or self-citations.

full rationale

The paper's load-bearing theory is derived, not assumed. Theorem 3.1 defines the pairwise Bellman operator T^pi from reward differences and the transition kernel, proves gamma-contraction on the sup-norm space of bounded antisymmetric functions, and identifies the unique fixed point with V^pi(s_i)-V^pi(s_j) by subtracting the Bellman equations; uniqueness follows from the Banach fixed-point theorem, so the result is a proof rather than a definitional restatement. Lemma 3.2 and Corollary 3.3 are algebraic telescoping arguments plus the standard score-function identity; the only idealization is the explicitly stated assumption Delta_theta = Delta^pi, which is a consistency condition, not a fitted parameter renamed as a prediction. The 1-step/n-step/lambda targets are TD-style bootstraps with terminal-case corrections expressed in observable rewards and non-terminal pairwise terms, so they do not collapse into the predicted quantity by construction. Section 4.1's trajectory-ranking offset is estimated from the learned model itself, but it is presented as a heuristic initialization, and the Limitations section candidly acknowledges the risks of the near-linear head and uninformative ranking; this affects robustness, not circularity. The empirical comparison with PPO/DAE may be open to methodological criticism because baseline provenance is not stated, but that is an experimental-validity concern, not a circularity concern. There are no load-bearing self-citations, no imported uniqueness claims from the authors' prior work, and no ansatz smuggled in via citation. The derivations are self-contained against the stated assumptions, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

The theory is parameter-free: Theorem 3.1, the Eq. 20 targets, and R-GAE hold for any MDP satisfying the stated axioms, with no constant fitted to make the proofs work. The empirical method has standard RL hyperparameters (Appendix D), of which n=5, c_v=1.25, value-clip 0.15 and p=0.33 most directly shape the critic. The important non-numeric choice is the anchoring rule (§4.1): row-averaging pairwise predictions then subtracting the batch minimum — a gauge convention, not a fitted constant, and the element that re-introduces the absolute scale the paper argues is unnecessary. No invented entities: the trajectory constant C=V^π(s_0) and the pairwise function ∆ are derived quantities; trajectory-ranking offsets are estimated from data, not postulated.

free parameters (5)
  • n-step target horizon n = 5
    Horizon for the critic target y^(n)_ij (Eq. 21, Appendix D). Chosen by hand; bias-variance trade-off; not derived.
  • RV critic loss coefficient c_v = 1.25
    Weight on L_critic in Eq. 29 (Appendix D). Standard PPO-scale value-loss weight applied to the pairwise loss.
  • RV value clipping threshold = 0.15
    Clips pairwise TD targets (Appendix D); stability hyperparameter with no theoretical role.
  • same-episode pairing probability p = 0.33
    Probability that the paired state s_j is drawn from the same episode as s_i (Appendix D; ablated in Appendix E). Affects the realized distribution µ over pairs; only mild performance influence across p ∈ {random, 0.33, 0.66}.
  • PPO hyperparameter set (γ=0.99, λ=0.95, ϵ=0.1, minibatch 128, rollout T=128, LR 2.5e-4, entropy coef 0.01, epochs 5) = as listed
    Inherited unchanged from standard PPO (Schulman et al. 2017); identical across all 49 games; not fitted to the theoretical claims.
axioms (7)
  • domain assumption MDP with bounded rewards and γ ∈ [0,1) under a stochastic policy π
    §3.1 defines the setting; boundedness is used for the sup-norm contraction and existence of V^π.
  • domain assumption Terminal states have value zero (V(terminal) = 0)
    Used to make pairwise bootstrapping well-posed in Eq. 20 and Appendix A; this pins the gauge the paper's §1 motivation claims is free.
  • domain assumption Successor states s'_i ~ P^π(·|s_i) and s'_j ~ P^π(·|s_j) are drawn independently
    Eqs. 5 and 7 define the operator via the product transition. In training, pairs come from a batch (asymptotically consistent), and 33% same-episode pairs introduce correlation, affecting variance rather than bias.
  • standard math Banach fixed-point theorem (contraction mapping principle)
    Invoked in Theorem 3.1 to guarantee a unique fixed point.
  • standard math Score-function identity E_{a~π}[∇ log π(a|s)] = 0
    Invoked in Corollary 3.3 to show the B_t term vanishes in expectation.
  • standard math GAE (Schulman et al. 2015b) is the reference estimator being reconstructed
    §3.3 compares R-GAE to standard GAE; the paper takes GAE's form and unbiasedness as given external results.
  • domain assumption Telescoping Ṽ_θ(s_t) = Σ_{k<t} ∆_θ(s_{k+1}, s_k) equals V^π(s_t) − V^π(s_0) only if ∆_θ = ∆^π
    Eq. 9 and Lemma 3.2; an idealization that the learned model must approximate for the unbiasedness statement to hold in practice.

pith-pipeline@v1.3.0-alltime-deepseek · 16345 in / 36936 out tokens · 310070 ms · 2026-08-01T08:26:12.861142+00:00 · methodology

0 comments
read the original abstract

In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $\gamma$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $\lambda$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Figures

Figures reproduced from arXiv: 2607.21120 by Jan Robine, Marc H\"oftmann, Stefan Harmeling.

Figure 1
Figure 1. Figure 1: Relative Value vs. Absolute Value Learning.. RV (left) learns value differences be￾tween states for decision making while AV (right) learns the value for each state in isolation and then decides for the best decision (e.g. by taking maximum in Q-learning). In control, actions are chosen by comparisons, not by absolute magnitudes. What matters is how good one state (or action) is relative to another. Formal… view at source ↗
Figure 2
Figure 2. Figure 2: Trajectory Ranking. When training batches contain samples from more than one episode, the initialization V˜ (s0) = 0 for each trajectory τi is not correct. The trajectories need to be ranked relative to each other by adding an offset that is calculated with ∆(si , sj ). Note that s0, s0, s0 are start states of τ1, τ2, τ3 indicated by color. For simplicity, assume in this figure that start states are only p… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison between Methods. Aggregate metrics with 95% stratified bootstrap con￾fidence intervals (Agarwal et al., 2021). Higher median, interquartile mean (IQM), and mean, but lower optimality gap indicate better performance. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation for Value Initialization. This figure compares the performance difference between zero (see Equation 9) and offset initialization (see Equation 26) for relative values. Usually the proposed offset initialization is needed to improve credit assignment, but for some games the algorithm can handle zero initialization as well. 6.1 ARCADE LEARNING ENVIRONMENT (ALE) [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

137 extracted references · 35 linked inside Pith

  1. [1]

    Markov Decision Processes: Discrete Stochastic Dynamic Programming , author =

  2. [2]

    Proceedings of the Annual Allerton Conference on Communication Control and Computing , volume=

    Differential training of rollout policies , author=. Proceedings of the Annual Allerton Conference on Communication Control and Computing , volume=. 1997 , organization=

    1997

  3. [3]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning at the edge of the statistical precipice , author=. Advances in neural information processing systems , volume=

  4. [4]

    Encyclopedia of optimization , pages=

    Neuro-dynamic programming , author=. Encyclopedia of optimization , pages=. 2025 , publisher=

    2025

  5. [5]

    SIAM Journal on Control and Optimization , volume=

    Learning algorithms for Markov decision processes with average cost , author=. SIAM Journal on Control and Optimization , volume=. 2001 , publisher=

    2001

  6. [6]

    2014 , publisher=

    Markov decision processes: discrete stochastic dynamic programming , author=. 2014 , publisher=

    2014

  7. [7]

    Icml , volume=

    Policy invariance under reward transformations: Theory and application to reward shaping , author=. Icml , volume=. 1999 , organization=

    1999

  8. [8]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  9. [9]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Deep reinforcement learning with double q-learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  10. [10]

    International conference on machine learning , pages=

    Implicit quantile networks for distributional reinforcement learning , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  11. [11]

    International conference on machine learning , pages=

    A distributional perspective on reinforcement learning , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  12. [12]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  13. [13]

    International conference on machine learning , pages=

    Asynchronous methods for deep reinforcement learning , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  14. [14]

    International conference on machine learning , pages=

    Dueling network architectures for deep reinforcement learning , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  15. [15]

    arXiv preprint arXiv:1611.01224 , year=

    Sample efficient actor-critic with experience replay , author=. arXiv preprint arXiv:1611.01224 , year=

    Pith/arXiv arXiv

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    Direct advantage estimation , author=. Advances in Neural Information Processing Systems , volume=

  17. [17]

    Weng, Jiayi and Lin, Min and Huang, Shengyi and Liu, Bo and Makoviichuk, Denys and Makoviychuk, Viktor and Liu, Zichen and Song, Yufan and Luo, Ting and Jiang, Yukun and Xu, Zhongwen and Yan, Shuicheng , booktitle =. Env

  18. [18]

    arXiv preprint arXiv:1506.02438 , year=

    High-dimensional continuous control using generalized advantage estimation , author=. arXiv preprint arXiv:1506.02438 , year=

    Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2403.03950 , year=

    Stop regressing: Training value functions via classification for scalable deep rl , author=. arXiv preprint arXiv:2403.03950 , year=

    Pith/arXiv arXiv

  20. [20]

    Proceedings of the IEEE international conference on computer vision , pages=

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification , author=. Proceedings of the IEEE international conference on computer vision , pages=

  21. [21]

    arXiv preprint arXiv:2305.12073 , year=

    Gelu activation function in deep learning: a comprehensive mathematical analysis and performance , author=. arXiv preprint arXiv:2305.12073 , year=

    Pith/arXiv arXiv

  22. [22]

    Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages=

    Deep sparse rectifier neural networks , author=. Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages=. 2011 , organization=

    2011

  23. [23]

    arXiv preprint arXiv:2404.00781 , year=

    Addressing loss of plasticity and catastrophic forgetting in continual learning , author=. arXiv preprint arXiv:2404.00781 , year=

    Pith/arXiv arXiv

  24. [24]

    Advances in neural information processing systems , volume=

    Towards deeper deep reinforcement learning with spectral normalization , author=. Advances in neural information processing systems , volume=

  25. [25]

    Artificial Intelligence Review , volume=

    A review of convolutional neural networks in computer vision , author=. Artificial Intelligence Review , volume=. 2024 , publisher=

    2024

  26. [26]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Mobilenetv2: Inverted residuals and linear bottlenecks , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  27. [27]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Inception-v4, inception-resnet and the impact of residual connections on learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  28. [28]

    Conference on Lifelong Learning Agents , pages=

    Loss of plasticity in continual deep reinforcement learning , author=. Conference on Lifelong Learning Agents , pages=. 2023 , organization=

    2023

  29. [29]

    Deep Reinforcement Learning Workshop NeurIPS 2022 , year=

    Sample-efficient reinforcement learning by breaking the replay ratio barrier , author=. Deep Reinforcement Learning Workshop NeurIPS 2022 , year=

    2022

  30. [30]

    arXiv preprint arXiv:2406.02596 , year=

    Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks , author=. arXiv preprint arXiv:2406.02596 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2106.01151 , year=

    Towards deeper deep reinforcement learning , author=. arXiv preprint arXiv:2106.01151 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:1902.05605 , volume=

    Crossnorm: Normalization for off-policy td reinforcement learning , author=. arXiv preprint arXiv:1902.05605 , volume=

    Pith/arXiv arXiv 1902

  33. [33]

    arXiv preprint arXiv:2407.01800 , year=

    Normalization and effective learning rates in reinforcement learning , author=. arXiv preprint arXiv:2407.01800 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:1502.03167 , year=

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=. arXiv preprint arXiv:1502.03167 , year=

    Pith/arXiv arXiv

  35. [35]

    Neurocomputing , volume=

    Activation functions in deep learning: A comprehensive survey and benchmark , author=. Neurocomputing , volume=. 2022 , publisher=

    2022

  36. [36]

    Neural Networks , volume=

    A survey on modern trainable activation functions , author=. Neural Networks , volume=. 2021 , publisher=

    2021

  37. [37]

    international conference on machine learning , pages=

    Understanding and improving convolutional neural networks via concatenated rectified linear units , author=. international conference on machine learning , pages=. 2016 , organization=

    2016

  38. [38]

    arXiv preprint arXiv:1710.05941 , year=

    Searching for activation functions , author=. arXiv preprint arXiv:1710.05941 , year=

    Pith/arXiv arXiv

  39. [39]

    arXiv preprint arXiv:1908.08681 , year=

    Mish: A self regularized non-monotonic activation function , author=. arXiv preprint arXiv:1908.08681 , year=

    Pith/arXiv arXiv 1908

  40. [40]

    arXiv preprint arXiv:1511.07289 , year=

    Fast and accurate deep network learning by exponential linear units (elus) , author=. arXiv preprint arXiv:1511.07289 , year=

    Pith/arXiv arXiv

  41. [41]

    Neural networks , volume=

    Sigmoid-weighted linear units for neural network function approximation in reinforcement learning , author=. Neural networks , volume=. 2018 , publisher=

    2018

  42. [42]

    arXiv preprint arXiv:1606.08415 , year=

    Gaussian error linear units (gelus) , author=. arXiv preprint arXiv:1606.08415 , year=

    Pith/arXiv arXiv

  43. [43]

    Rectifier nonlinearities improve neural network acoustic models , author=. Proc. icml , volume=. 2013 , organization=

    2013

  44. [44]

    Proceedings of the 27th international conference on machine learning (ICML-10) , pages=

    Rectified linear units improve restricted boltzmann machines , author=. Proceedings of the 27th international conference on machine learning (ICML-10) , pages=

  45. [45]

    Neural networks: Tricks of the trade , pages=

    Efficient backprop , author=. Neural networks: Tricks of the trade , pages=. 2002 , publisher=

    2002

  46. [46]

    arXiv preprint arXiv:2209.07550 , year=

    Human-level Atari 200x faster , author=. arXiv preprint arXiv:2209.07550 , year=

    Pith/arXiv arXiv

  47. [47]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    A convnet for the 2020s , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  48. [48]

    Architecture , volume=

    In value-based deep reinforcement learning, a pruned network is a good network , author=. Architecture , volume=

  49. [49]

    International Conference on Machine Learning , pages=

    Bigger, better, faster: Human-level atari with human-level efficiency , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  50. [50]

    International conference on machine learning , pages=

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  51. [51]

    Journal of Artificial Intelligence Research , volume=

    Towards continual reinforcement learning: A review and perspectives , author=. Journal of Artificial Intelligence Research , volume=

  52. [52]

    Advances in Neural Information Processing Systems , volume=

    Plastic: Improving input and label plasticity for sample efficient reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  53. [53]

    Advances in neural information processing systems , volume=

    On warm-starting neural network training , author=. Advances in neural information processing systems , volume=

  54. [54]

    Advances in Neural Information Processing Systems , volume=

    Deep reinforcement learning with plasticity injection , author=. Advances in Neural Information Processing Systems , volume=

  55. [55]

    International Conference on Machine Learning , pages=

    The dormant neuron phenomenon in deep reinforcement learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  56. [56]

    Nature , volume=

    Loss of plasticity in deep continual learning , author=. Nature , volume=. 2024 , publisher=

    2024

  57. [57]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  58. [58]

    Trends in cognitive sciences , volume=

    Catastrophic forgetting in connectionist networks , author=. Trends in cognitive sciences , volume=. 1999 , publisher=

    1999

  59. [59]

    Psychology of learning and motivation , volume=

    Catastrophic interference in connectionist networks: The sequential learning problem , author=. Psychology of learning and motivation , volume=. 1989 , publisher=

    1989

  60. [60]

    The Twelfth International Conference on Learning Representations , year=

    CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity , author=. The Twelfth International Conference on Learning Representations , year=

  61. [61]

    arXiv preprint arXiv:2402.18762 , year=

    Disentangling the causes of plasticity loss in neural networks , author=. arXiv preprint arXiv:2402.18762 , year=

    Pith/arXiv arXiv

  62. [62]

    arXiv 2018 , author=

    Scalable agent alignment via reward modeling: A research direction. arXiv 2018 , author=. arXiv preprint arXiv:1811.07871 , year=

    Pith/arXiv arXiv 2018

  63. [63]

    Advances in neural information processing systems , volume=

    Reward learning from human preferences and demonstrations in atari , author=. Advances in neural information processing systems , volume=

  64. [64]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  65. [65]

    Advances in neural information processing systems , volume=

    Cooperative inverse reinforcement learning , author=. Advances in neural information processing systems , volume=

  66. [66]

    1982 , publisher=

    The growth of biological thought: Diversity, evolution, and inheritance , author=. 1982 , publisher=

    1982

  67. [67]

    The Sciences , volume=

    Darwin's dangerous idea , author=. The Sciences , volume=. 1995 , publisher=

    1995

  68. [68]

    arXiv preprint arXiv:2310.17688 , year=

    Managing ai risks in an era of rapid progress , author=. arXiv preprint arXiv:2310.17688 , year=

    Pith/arXiv arXiv

  69. [69]

    , author=

    Thinking, fast and slow. , author=. 2011 , publisher=

    2011

  70. [70]

    Conference on robot learning , pages=

    Learning latent plans from play , author=. Conference on robot learning , pages=. 2020 , organization=

    2020

  71. [71]

    Advances in Neural Information Processing Systems , volume=

    Planning with goal-conditioned policies , author=. Advances in Neural Information Processing Systems , volume=

  72. [72]

    International conference on machine learning , pages=

    Automatic goal generation for reinforcement learning agents , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  73. [73]

    arXiv preprint arXiv:1903.03698 , year=

    Skew-fit: State-covering self-supervised reinforcement learning , author=. arXiv preprint arXiv:1903.03698 , year=

    Pith/arXiv arXiv 1903

  74. [74]

    The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages=

    Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction , author=. The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages=

  75. [75]

    IJCAI , volume=

    Learning to achieve goals , author=. IJCAI , volume=. 1993 , organization=

    1993

  76. [76]

    International conference on machine learning , pages=

    Universal value function approximators , author=. International conference on machine learning , pages=. 2015 , organization=

    2015

  77. [77]

    Advances in neural information processing systems , volume=

    Hindsight experience replay , author=. Advances in neural information processing systems , volume=

  78. [78]

    1988 , publisher=

    Society of mind , author=. 1988 , publisher=

    1988

  79. [79]

    arXiv preprint arXiv:2401.12963 , year=

    Autort: Embodied foundation models for large scale orchestration of robotic agents , author=. arXiv preprint arXiv:2401.12963 , year=

    arXiv

  80. [80]

    Foundations and Trends

    Model-based reinforcement learning: A survey , author=. Foundations and Trends. 2023 , publisher=

    2023

Showing first 80 references.