Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Exploration by Random Reward Perturbation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding zero-mean noise to rewards provably expands the variance of RL trajectories, and the paper shows the trick boosts PPO and SAC on nine continuous-control tasks.

desk verdict A cheap exploration trick with believable empirical curves, but the theory is broken and the closest prior work is never compared. read the letter →

arxiv 2506.08737 v1 pith:2N2DMZRL submitted 2025-06-10 cs.LG cs.AI

Haozhe Ma , Guoji Fu , Zhengding Luo , Jiele Wu , Tze-Yun Leong This is my paper
classification cs.LGcs.AI
keywords reinforcementlearningexplorationrewardperturbationGaussiannoisepolicydiversitysparserewardsPPOSAC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Random Reward Perturbation (RRP) is the claim that adding zero-mean Gaussian noise to the reward signal during training widens the range of states and actions an agent explores, and that this can be added to any model-free reinforcement learning algorithm at negligible cost. The paper proves, for a one-step SGD update, that noise-perturbed labels increase the variance of a model's outputs, and then extends this to the variance of trajectories in DQN and A2C by treating transitions as a deterministic map. On nine continuous-control tasks, RRP-PPO and RRP-SAC improve sample efficiency and final returns over their vanilla counterparts, with the largest gains under sparse rewards. The contribution, taken sympathetically, is that random reward perturbation itself is a valid exploration strategy, complementary to action-level randomness and cheaper than novelty-based intrinsic motivation.

What carries the argument

The load-bearing object is the variance of model outputs, defined as the trace of the covariance matrix over a dataset, and the variance of state trajectories over a horizon. Lemma 3.2 is the engine: for a single SGD step with Gaussian label noise, the output covariance grows by $\frac{\alpha^2 B \sigma^2}{N} \sum_j \mathrm{Tr}(A_j A_j^T)$, where $A_j$ measures how the Jacobian products vary across the data, while the expected output mean is unchanged. Theorems 3.5 and 3.6 feed this into DQN and A2C TD targets, and Lemma 3.4 transfers the per-step variance dominance to trajectory-level variance by composing state maps. The mechanism's practical role is to make the optimization target itself wander, so the policy is pushed in different directions each update instead of locking onto a single reward surface.

What would settle it

Run two tabular Q-learning agents on the grid-maze of Figure 1b with identical seeds, one with original rewards and one with zero-mean noise added to rewards, and record the variance of visited states at each episode. If the noise-augmented agent does not exhibit strictly greater state-visitation variance than the baseline at the same training step, the trajectory-variance claim would be falsified.

Watch

Extended reading notes

Core claim

RRP defines a perturbed reward $R_{\mathrm{RRP}}(s) = R_{\mathrm{env}}(s) + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0, \sigma^2)$, and anneals $\sigma$ linearly during training so the agent eventually optimizes the true environment reward. The central theoretical claim is that this makes the variance of sampled trajectories strictly larger than without perturbation, which the paper identifies with broader exploration. The argument goes through a sequence of steps: Lemma 3.2 shows one SGD step on noise-perturbed labels increases output variance while preserving the expected output mean; Theorem 3.5 applies this to DQN's TD targets, and Theorem 3.6 to A2C's value function and advantage estimates; Lemma 3.4 then converts higher output variance into higher trajectory variance by composing the policy with the transition function. Experiments with PPO and SAC support the claim: the perturbed variants consistently beat their backbones, especially in sparse-reward settings, and reach final performance comparable to RND, NovelD, and ExploRS with far less computation.

Load-bearing premise

The load-bearing premise is that the variance increase proven for a single SGD step on a fixed dataset carries over to the state-visitation variance of a deep RL agent across the whole training run; the paper asserts this transfer rather than proving it.

Editorial extensions

If this is right

  • RRP can be plugged into any model-free algorithm by editing only the reward line, requiring no extra networks, buffers, or density models.
  • It composes additively with action-level exploration: stochastic policies, $\epsilon$-greedy, and entropy regularization continue to work and RRP adds further gains.
  • In sparse-reward environments, where most rewards are zero, the noise prevents the TD target from collapsing onto a constant, delaying premature convergence.
  • Because the noise is annealed to zero, the final policy is still optimized against the true environment reward, so exploration does not permanently distort exploitation.
  • On the paper's test suite, RRP improves both convergence speed and final returns of PPO and SAC, and beats NoisyNet while matching or approaching the final performance of RND, NovelD, and ExploRS at lower computational cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is right, reward noise is a generic 'optimization wobble' applicable to any gradient-trained predictor, though whether that wobble helps depends on the loss landscape; this is a testable conjecture outside RL.
  • The paper equates exploration with variance of visits, but the proof transfers variance from function outputs to trajectories only by assuming the per-step inequality it needs; a direct measurement of state-visitation variance during training would settle the link.
  • RRP's noise is agnostic to the state, so it could be layered on top of novelty or shaped rewards, turning intrinsic motivation into a slowly-wandering target rather than a fixed bonus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Random Reward Perturbation (RRP), a method that adds zero-mean Gaussian noise to environmental rewards and anneals the noise magnitude during training, as a lightweight and generally applicable exploration booster for model-free RL. The authors provide a theoretical analysis aiming to show that reward perturbation increases the variance of learned policies and of sampled state trajectories (Lemma 3.2, Lemma 3.4, Theorems 3.5 and 3.6), and support the claim with experiments integrating RRP into SAC and PPO across nine continuous-control tasks under dense and sparse rewards, including ablation studies of the noise scale and decay period.

Significance. If the theoretical claims were valid, RRP would be an attractive method: it is conceptually simple, requires almost no additional computation, and the experiments show consistent improvements over vanilla SAC and PPO across several challenging domains. The empirical section is broad and includes a useful MountainCar case study of visited-state density. However, the advertised theoretical contribution — the guarantee that trajectory variance, and hence exploration, strictly increases under RRP — is not established. The load-bearing proof chain is invalid at its central step (Lemma 3.4), and the transfer from value/advantage variance to policy and trajectory variance is asserted rather than derived. The method also relies on two tuned hyperparameters (σ0 and λ), so it is not parameter-free. No code release is indicated. The empirical results may still support RRP as a heuristic, but the paper's central theoretical claim, as stated in the abstract, is unsupported.

major comments (5)
  1. [§3.2, Lemma 3.4, Eqs. (14)–(16)] The proof of Lemma 3.4 is not valid. Equation (14) defines s_h^{(k)} = p(s_0) ∏_{h=1}^H g^{(k)}(s_{h-1}^{(k)}), which mixes a probability distribution with iterated function evaluations and is not a well-defined state. Equation (15) then simply asserts the per-step variance inequality 1/N ∑ |s_{h,i}^{(2)} − mean|² > 1/N ∑ |s_{h,i}^{(1)} − mean|² that the lemma is supposed to prove; no derivation from V(g^{(2)}) > V(g^{(1)}) is given. The statement itself is false: a function with high one-step output variance can be followed by a contracting iteration that collapses the spread of trajectories, while a lower-variance function can preserve it. Since Theorems 3.5 and 3.6 both invoke Lemma 3.4 to conclude V(T_RRP) > V(T_ori), the trajectory-level exploration claim collapses.
  2. [§3.2, Lemma 3.2 and text after Eq. (11)] Lemma 3.2 is a one-step SGD calculation on a fixed initial parameter vector θ0, but the paper immediately states that the variance increase 'accumulates over successive training steps' and uses this to justify the RL theorems. No induction is provided: after the first step, θ1 differs between the original and perturbed scenarios, and the TD targets y_ori and y_RRP in Eqs. (17) and (20) depend on each scenario's own target network or value network. Therefore Lemma 3.2 cannot be iterated to bound the variance of a multi-step RL training process.
  3. [§3.2, Theorem 3.5, Eqs. (18)–(19)] The step 'Since V(Q_RRP) > V(Q_ori), it follows that V(π_RRP) > V(π_ori)' is asserted without proof. Variance of Q-function outputs does not monotonically determine variance of the softmax policy or of actions sampled from it. In addition, the mapping g_k(s) = π_k(a|s) T(s'|s,a) defined in Eq. (19) is a transition kernel and not a deterministic function S → S as required by Lemma 3.4. Thus Theorem 3.5 does not establish V(T_RRP) > V(T_ori).
  4. [§3.2, Theorem 3.6, Eqs. (21)–(22)] Equation (22) decomposes Var[A_RRP] − Var[A_ori] as the sum of value-function variance differences plus σ², which assumes that R_env, V_ϕ(s), V_ϕ(s′), and ε are uncorrelated. No such independence is established, and V_ϕ is itself trained on the perturbed rewards, so this covariance omission is not benign. Moreover, even if Var[A_RRP] > Var[A_ori] held, the inference to V(π_RRP) > V(π_ori) and then to V(T_RRP) > V(T_ori) is again asserted rather than proved.
  5. [§4, Algorithm 1, line 10] The annealing update ε_{t,i} ← max{0, ε_t − ε_t t/(λT)} clips every negative noise sample to zero, so the perturbation actually applied in RRP-SAC has a positive mean whenever the noise scale is nonzero. This contradicts the zero-mean noise assumption in Eq. (1) and in the theoretical analysis. The implementation should scale the noise multiplicatively, e.g., ε_{t,i} ← ε_t · max{0, 1 − t/(λT)}, if a zero-mean annealed perturbation is intended.
minor comments (6)
  1. [§3.2, Eq. (7)] The Taylor expansion writes the higher-order term as o(∥θ0 − θ0∥), which is identically zero; it should be o(∥θ1 − θ0∥).
  2. [§3.2, Eq. (12)] Equation (12) in the main text sums M(x_n, x_i) over i but not over n, making the displayed expression depend on an unspecified n; the corrected version appears in Appendix A.1, Eq. (40).
  3. [§3.2, Definition 3.3 and Lemma 3.4] The variance V(g) from Definition 3.1 is defined over a fixed dataset, but in the sequential setting the input distribution to g changes at every step; the lemma should specify over which input distribution the output variances are evaluated and how that distribution is updated.
  4. [§3.2, Eq. (18)] DQN's behavior policy is ε-greedy over Q-values, not the softmax policy written in Eq. (18); the softmax policy is not used by DQN in the paper's own setup.
  5. [Figures 2, 3, 5, and 6] Several figure captions and axis labels contain font-encoding artifacts (strings such as '/uni00000013/uni00000015...'), making them unreadable in the provided manuscript; these need to be regenerated.
  6. [§5.1, Table 1] Many reported standard deviations are exactly 0.00 despite averaging over 5 seeds, which is implausible for continuous-control tasks; the authors should clarify the number of seeds and the presentation of confidence intervals.

Circularity Check

1 steps flagged · score 6.0 of 10

The exploration theorem is partly circular: Lemma 3.4 inserts the desired trajectory-variance inequality as Eq. (15), so Theorems 3.5 and 3.6 sum an assumption rather than derive it from the SGD variance calculation.

  1. other [Section 3.2, Lemma 3.4 proof, Eq. (15); repeated verbatim in Appendix A.2, Eq. (43)]
    "Given V(g(2)) > V(g(1)), for N independent trajectories: 1/N \sum_i |s_{h,i}^{(2)} - \bar{s}_h^{(2)}|^2 > 1/N \sum_i |s_{h,i}^{(1)} - \bar{s}_h^{(1)}|^2. (15)"

    Lemma 3.4 is the load-bearing step that turns one-step output variance into trajectory-level exploration variance, but its conclusion V(T(2)) > V(T(1)) is, by Definition 3.3, the sum over h of exactly the per-step trajectory variances appearing in Eq. (15). Eq. (15) simply asserts that per-step inequality at every h, without deriving it from V(g(2)) > V(g(1)); Eq. (16) then only sums the asserted inequality. The recursion itself is also ill-defined: Eq. (14) writes s_h as p(s_0) times a product of g evaluations, which is not the sequential map s_{h+1} = g(s_h). Theorems 3.5 and 3.6 therefore inherit trajectory-variance dominance as an assumption, not as a proved consequence of Lemma 3.2.

full rationale

Lemma 3.2 is an independent one-step SGD calculation: adding zero-mean label noise increases the trace of the output covariance by the explicit positive term in Eq. (11). That part is not circular. The collapse occurs in Lemma 3.4, where the per-step trajectory variance inequality that constitutes the target conclusion is restated as Eq. (15) after merely assuming V(g(2)) > V(g(1)); summing Eq. (15) yields the lemma, so the advertised exploration guarantee reduces to an assertion of the very inequality it should prove. The self-citations to Ma et al. appear only as contextual reward-shaping references and are not load-bearing, so they do not contribute to the score. The experiments are external benchmark evidence and do not make the theoretical derivation self-contained. A further inconsistency, noted but not counted as circularity, is that Algorithm 1 clips negative noise samples to zero via max{0, epsilon_t - epsilon_t t/(lambda T)}, so the implemented perturbation is positively biased rather than the zero-mean noise assumed in Eq. (1) and Lemma 3.2. On balance, the central theoretical claim of expanded exploration is partially circular because its key trajectory-variance theorem assumes its own per-step conclusion, though the underlying SGD variance computation and the empirical evaluation retain independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim leans on two tuned hyperparameters, sigma_0 and lambda, and on unproven domain assumptions about variance transfer and deterministic transitions. No new physical or mathematical entities are introduced.

free parameters (3)
  • initial reward noise scale sigma_0 (sigma_max) = sigma_0^2 = 1.0 (default)
    Controls perturbation magnitude; ablation tests 0.5, 1.0, and 1.5. The central claim depends on this choice.
  • noise decay period lambda = 0.3 (default)
    Controls the exploration-exploitation balance; ablation tests 0.1, 0.3, and 0.5.
  • noise decay horizon T = not reported per task
    Required in Eq. (2) for the annealing schedule; unclear whether it is tuned per environment.
assumptions (4)
  • standard math First-order Taylor expansion of network outputs around theta_0, ignoring higher-order terms
    Used in Lemma 3.2 Eq. (7); only justified for very small learning rate steps, which is not guaranteed in deep RL.
  • domain assumption Variance increase of network outputs on a fixed dataset under one SGD step transfers to Q-function, value-function, and policy output variances during RL training
    Theorems 3.5 and 3.6 invoke Lemma 3.2 for value or Q networks and then assume V(f) implies V(policy) and V(trajectory) without proof.
  • domain assumption MDP transitions can be treated as deterministic functions g(s) = pi(a|s) T(s'|s,a) for trajectory variance analysis
    Lemma 3.4 requires deterministic maps S to S, but T is a conditional distribution; Eq. (14) is not a valid recursive state update.
  • domain assumption Annealing noise to zero recovers the original optimal policy and preserves zero-mean noise
    Assumed throughout, but Algorithm 1's annealing formula max(0, epsilon - epsilon t / lambda T) truncates negative noise and is not zero-mean, so this premise fails for the implemented SAC variant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploration by Random Reward Perturbation." pith.science (2026). https://pith.science/paper/2N2DMZRL

@misc{pith2026250608737,
  author       = {Pith},
  title        = {Pith review of: Exploration by Random Reward Perturbation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2N2DMZRL}},
  note         = {Machine review of arXiv:2506.08737}
}
abstract

We introduce Random Reward Perturbation (RRP), a novel exploration strategy for reinforcement learning (RL). Our theoretical analyses demonstrate that adding zero-mean noise to environmental rewards effectively enhances policy diversity during training, thereby expanding the range of exploration. RRP is fully compatible with the action-perturbation-based exploration strategies, such as $\epsilon$-greedy, stochastic policies, and entropy regularization, providing additive improvements to exploration effects. It is general, lightweight, and can be integrated into existing RL algorithms with minimal implementation effort and negligible computational overhead. RRP establishes a theoretical connection between reward shaping and noise-driven exploration, highlighting their complementary potential. Experiments show that RRP significantly boosts the performance of Proximal Policy Optimization and Soft Actor-Critic, achieving higher sample efficiency and escaping local optima across various tasks, under both sparse and dense reward scenarios.

Figures

Figures reproduced from arXiv: 2506.08737 by the authors.

Figure 1
Figure 1. How RRP works in exploration. In Figure (a), by “chasing" a perturbed target, the policy’s optimization is guided toward random directions, increasing the diversity of learned policies. Combined with the effects of stochastic policies, RRP further expands the range of sampled actions, thereby improving exploration. In Figure (b), in a grid-maze environment with sparse rewards, where only the grid in the bottom-right… view at source ↗
Figure 2
Figure 2. Continuous-control tasks in experiments with dense and sparse reward structures. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of RRP-SAC and RRP-PPO with several baselines on both dense-reward and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Density of the car’s positions visited by the agents during training in the [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Ablation study on the noise scale σ 2 0 for RRP-SAC and RRP-PPO. 0 200 400 600 800 1000 0 50 100 150 RRP-SAC Episode returns AntFar 0 200 400 600 800 1000 0 25 50 75 100 125 HumanStand 0 200 400 600 800 1000 0 50 100 150 CheetahFar 0 200 400 600 800 1000 0 20 40 60 80 …
Figure 6
Figure 6. Figure 6: Ablation study on the noise decay period [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: 23 [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 7
Figure 7. Figure 7: Network structures for the policy, value function, and Q-function in RRP-SAC and RRP-PPO. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

    cs.CV 2025-08 conditional novelty 6.0 of 10

    AD-FM combines multi-stage reasoning with localization-aware rewards to fine-tune MLLMs for anomaly detection, improving average accuracy by about 22 percentage points over the base model.

Reference graph

Works this paper leans on

65 extracted references · 55 canonical work pages · cited by 1 Pith paper

  1. [1]

    and Goyal, N

    Agrawal, S. and Goyal, N. (2012). Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39--1. JMLR Workshop and Conference Proceedings

    2012

  2. [2]

    P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al

    Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al. (2020). Never give up: Learning directed exploration strategies. In International Conference on Learning Representations

    work page 2020

  3. [3]

    Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems , 29

    work page 2016

  4. [4]

    Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. (2019). Large-scale study of curiosity-driven learning. In International Conference on Learning Representations

    work page 2019

  5. [5]

    Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2018). Exploration by random network distillation. In International Conference on Learning Representations

    work page 2018

  6. [6]

    Cesa-Bianchi, N., Gentile, C., Lugosi, G., and Neu, G. (2017). Boltzmann exploration done right. Advances in neural information processing systems , 30

    work page 2017

  7. [7]

    J., Lee, S

    de Lazcano, R., Andreas, K., Tai, J. J., Lee, S. R., and Terry, J. (2024). Gymnasium robotics

    work page 2024

  8. [8]

    Devidze, R., Kamalaruban, P., and Singla, A. (2022). Exploration-guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems , 35:5829--5842

    work page 2022

Show all 65 references
  1. [9]

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019). Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations

    2019

  2. [10]

    Fang, Z., Yang, K., Tao, J., Lyu, J., Li, L., Shen, L., and Li, X. (2025). Exploration by random distribution distillation. arXiv preprint arXiv:2505.11044

    2025 arXiv

  3. [11]

    G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S

    Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S. (2018). Noisy networks for exploration. In International Conference on Learning Representations

    2018

  4. [12]

    Fox, L., Choshen, L., and Loewenstein, Y. (2018). Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations

    2018

  5. [13]

    Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning , pages 1587--1596. PMLR

    2018

  6. [14]

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. (2017). Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352--1361. PMLR

    2017

  7. [15]

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861--1870. PMLR

    2018

  8. [16]

    Hong, Z.-W., Shann, T.-Y., Su, S.-Y., Chang, Y.-H., Fu, T.-J., and Lee, C.-Y. (2018). Diversity-driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems , 31

    2018

  9. [17]

    Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. Advances in neural information processing systems , 29

    2016

  10. [18]

    Hsu, H.-L., Wang, W., Pajic, M., and Xu, P. (2024). Randomized exploration in cooperative multi-agent reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  11. [19]

    H., Hegde, N., and Schmidt, M

    Hu, B., Zhang, T. H., Hegde, N., and Schmidt, M. (2023). Optimistic thompson sampling-based algorithms for episodic reinforcement learning. In Uncertainty in Artificial Intelligence , pages 890--899. PMLR

    2023

  12. [20]

    Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Ara \ A s jo, J. G. (2022). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1--18

    2022

  13. [21]

    Ibrahim, S., Mostafa, M., Jnadi, A., Salloum, H., and Osinenko, P. (2024). Comprehensive overview of reward engineering and shaping in advancing reinforcement learning applications. IEEE Access

    2024

  14. [22]

    Ishfaq, H., Cui, Q., Nguyen, V., Ayoub, A., Yang, Z., Wang, Z., Precup, D., and Yang, L. (2021). Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , pages 4607--4616. PMLR

    2021

  15. [23]

    Kveton, B., Szepesvari, C., Ghavamzadeh, M., and Boutilier, C. (2019a). Perturbed-history exploration in stochastic linear bandits. arXiv preprint arXiv:1903.09132

    2019 arXiv

  16. [24]

    Kveton, B., Szepesvari, C., Ghavamzadeh, M., and Boutilier, C. (2019b). Perturbed-history exploration in stochastic multi-armed bandits. arXiv preprint arXiv:1902.10089

    2019 arXiv

  17. [25]

    Ladosz, P., Weng, L., Kim, M., and Oh, H. (2022). Exploration in deep reinforcement learning: A survey. Information Fusion , 85:1--22

    2022

  18. [26]

    D., and Russell, S

    Lidayan, A., Dennis, M. D., and Russell, S. (2025). Bamdp shaping: a unified framework for intrinsic motivation and reward shaping. In The Thirteenth International Conference on Learning Representations

    2025

  19. [27]

    P., Hunt, J

    Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

    2015 arXiv

  20. [28]

    Lobel, S., Bagaria, A., and Konidaris, G. (2023). Flipping coins to estimate pseudocounts for exploration in reinforcement learning. In International Conference on Machine Learning , pages 22594--22613. PMLR

    2023

  21. [29]

    Luo, Z., Ma, H., Shi, D., and Gan, W.-S. (2024). Gfanc-rl: Reinforcement learning-based generative fixed-filter active noise control. Neural Networks , page 106687

    2024

  22. [30]

    Y., Luo, Z., Vo, T

    Ma, H., Li, F., Lim, J. Y., Luo, Z., Vo, T. V., and Leong, T.-Y. (2025a). Catching two birds with one stone: Reward shaping with dual random networks for balancing exploration and exploitation. In Forty-second International Conference on Machine Learning . PMLR

    2025

  23. [31]

    V., Sima, K., and Leong, T.-Y

    Ma, H., Luo, Z., Vo, T. V., Sima, K., and Leong, T.-Y. (2024a). Knowledge sharing and transfer via centralized reward agent for multi-task reinforcement learning. arXiv preprint arXiv:2408.10858

    2024

  24. [32]

    V., Sima, K., and Leong, T.-Y

    Ma, H., Luo, Z., Vo, T. V., Sima, K., and Leong, T.-Y. (2025b). Highly efficient self-adaptive reward shaping for reinforcement learning. In Thirteenth International Conference on Learning Representations

    2025

  25. [33]

    V., Fu, D., and Leong, T.-Y

    Ma, H., Sima, K., Vo, T. V., Fu, D., and Leong, T.-Y. (2024b). Reward shaping for reinforcement learning with an assistant reward agent. In Forty-first International Conference on Machine Learning , volume 235, pages 33925--33939. PMLR

    2024

  26. [34]

    V., and Leong, T.-Y

    Ma, H., Vo, T. V., and Leong, T.-Y. (2023). Hierarchical reinforcement learning with human-ai collaborative sub-goals optimization. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems , pages 2310--2312

    2023

  27. [35]

    V., and Leong, T.-Y

    Ma, H., Vo, T. V., and Leong, T.-Y. (2024c). Mixed-initiative bayesian sub-goal optimization in hierarchical reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , pages 1328--1336

    2024

  28. [36]

    C., Bellemare, M

    Machado, M. C., Bellemare, M. G., and Bowling, M. (2020). Count-based exploration with the successor representation. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5125--5133

    2020

  29. [37]

    S., Everitt, T., and Hutter, M

    Martin, J., Narayanan, S. S., Everitt, T., and Hutter, M. (2017). Count-based exploration in feature space for reinforcement learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence , pages 2471--2478

    2017

  30. [38]

    Mavor-Parker, A., Young, K., Barry, C., and Griffin, L. (2022). How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In International Conference on Machine Learning , pages 15220--15240. PMLR

    2022

  31. [39]

    Melnik, A., Lach, L., Plappert, M., Korthals, T., Haschke, R., and Ritter, H. (2021). Using tactile sensing to improve the sample efficiency and performance of deep deterministic policy gradients for simulated in-hand manipulation tasks. Frontiers in Robotics and AI , 8:538773

    2021

  32. [40]

    Mguni, D., Jafferjee, T., Wang, J., Perez-Nieves, N., Song, W., Tong, F., Taylor, M., Yang, T., Dai, Z., Chen, H., et al. (2023). Learning to shape rewards using a game of two partners. In AAAI Conference on Artificial Intelligence , pages 11604--11612

    2023

  33. [41]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pages 1928--1937. PMLR

    2016

  34. [42]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature , 518(7540):529--533

    2015

  35. [43]

    Moore, A. W. (1990). Efficient memory-based learning for robot control. Technical report, University of Cambridge, Computer Laboratory

    1990

  36. [44]

    Nachum, O., Norouzi, M., Xu, K., and Schuurmans, D. (2017). Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems , 30

    2017

  37. [45]

    Osband, I., Aslanides, J., and Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems , 31

    2018

  38. [46]

    G., Oord, A., and Munos, R

    Ostrovski, G., Bellemare, M. G., Oord, A., and Munos, R. (2017). Count-based exploration with neural density models. In International Conference on Machine Learning , pages 2721--2730. PMLR

    2017

  39. [47]

    Painter, M., Baioumy, M., Hawes, N., and Lacerda, B. (2024). Monte carlo tree search with boltzmann exploration. Advances in Neural Information Processing Systems , 36

    2024

  40. [48]

    A., and Darrell, T

    Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning , pages 2778--2787. PMLR

    2017

  41. [49]

    Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al. (2018a). Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464

    2018 arXiv

  42. [50]

    Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M

    Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. (2018b). Parameter space noise for exploration. In International Conference on Learning Representations

    2018

  43. [51]

    and Rockt \"a schel, T

    Raileanu, R. and Rockt \"a schel, T. (2020). Ride: Rewarding impact-driven exploration for procedurally-generated environments. In International Conference on Learning Representations

    2020

  44. [52]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  45. [53]

    Shani, L., Efroni, Y., and Mannor, S. (2019). Exploration conscious reinforcement learning revisited. In International conference on machine learning , pages 5680--5689. PMLR

    2019

  46. [54]

    and Sakashita, Y

    Shibata, K. and Sakashita, Y. (2015). Reinforcement learning with internal-dynamics-based exploration using a chaotic neural network. In 2015 International Joint Conference on Neural Networks (IJCNN) , pages 1--8. IEEE

    2015

  47. [55]

    Strehl, A. L. and Littman, M. L. (2008). An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309--1331

    2008

  48. [56]

    Sun, C., Wang, R., Li, Q., and Hu, X. (2021). Reward space noise for exploration in deep reinforcement learning. International Journal of Pattern Recognition and Artificial Intelligence , 35(10):2152013

    2021

  49. [57]

    Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press

    2018

  50. [58]

    S., McAllester, D., Singh, S., and Mansour, Y

    Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. (1999). Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12

    1999

  51. [59]

    Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. (2017). \# exploration: A study of count-based exploration for deep reinforcement learning. Advances in Neural Information Processing Systems , 30

    2017

  52. [60]

    Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285--294

    1933

  53. [61]

    Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026--5033. IEEE

    2012

  54. [62]

    U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al

    Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al. (2024). Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032

    2024 arXiv

  55. [63]

    Wang, J., Liu, Y., and Li, B. (2020). Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6202--6209

    2020

  56. [64]

    Yang, K., Tao, J., Lyu, J., and Li, X. (2024). Exploration and anti-exploration with distributional random network distillation. In Forty-first International Conference on Machine Learning . PMLR

    2024

  57. [65]

    E., and Tian, Y

    Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J. E., and Tian, Y. (2021). Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems , 34:25217--25230

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.