Pith. sign in

REVIEW 3 major objections 4 minor 37 references

Offline Reinforcement Learning with Penalized Action Noise Injection

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Penalized action-noise injection turns offline RL into a modified MDP—the Noisy Action MDP—that suppresses out-of-distribution overestimation.

desk verdict PANI is a simple, well-tested regularizer that lifts offline RL scores, but the advertised NAMDP theory doesn't yet cover the algorithms that produce those scores. read the letter →

arxiv 2507.02356 v1 pith:PLKEF4JU submitted 2025-07-03 cs.LG cs.AI

JunHyeok Oh , Byung-Jun Lee This is my paper
classification cs.LGcs.AI MSC 68T0590C40
keywords offlinereinforcementlearningPenalizedActionNoiseInjection(PANI)NoisyMDPout-of-distributionoverestimationD4RLbenchmarkhybriddistributionQ-learningaction-spacecoverage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PANI claims that a few lines of code—drawing a noisy action from each dataset action and regressing Q against a target reduced by the squared perturbation distance—can give offline RL the same out-of-distribution protection that diffusion-based methods buy with heavy generative models. The paper proves that this recipe is not ad hoc: the PANI objective is exactly Q-learning in a modified process it calls the Noisy Action MDP (NAMDP), whose reward and transition kernels are the dataset dynamics re-weighted by the noise kernel. Because the noise covers the whole action space, Q updates reach low-density regions, and the penalty term pushes the NAMDP's optimal policy back toward dataset actions. On D4RL, adding PANI to TD3 and IQL improves average scores by roughly 11 and 7 points respectively, with only 3–9% extra training time, and it also helps the diffusion-based algorithm QGPO. The broader claim is that sophisticated generative modeling is not necessary for strong offline RL performance.

What carries the argument

The NAMDP is the central object: an MDP $(\mathcal{S}, \mathcal{A}, R_\sigma, P_\sigma, \gamma)$ whose reward and transition are the dataset reward and dynamics averaged against the noise-weight function $p_D(a'|s,a,\sigma)$. The load-bearing derivation uses the Euler equation for functionals to show that the minimizer of the PANI squared-loss objective satisfies the NAMDP Bellman equation, so minimizing the PANI objective is equivalent to learning the NAMDP's Q-function. The practical engine is the hybrid noise distribution—a mixture of a uniform component with exponentially scaled Gaussian scales—which the paper shows is more robust across noise levels than plain Gaussian or Laplace noise.

What would settle it

Run the reported IQL-AN and TD3-AN critic updates in a small tabular MDP with known dynamics, compute the NAMDP Q-function exactly from the paper's Definition 5.2, and compare the learned Q to it; if the bootstrapped-trained Q does not converge to the NAMDP fixed point, the theoretical foundation does not cover the empirical method.

Watch

Extended reading notes

Core claim

The central discovery is that the penalized noise-injection update is not just a trick but defines a new MDP. Given a noise distribution $q_\sigma$ and a dataset distribution $p_D$, the Noisy Action MDP (NAMDP) has transition kernel $P_\sigma(s'|s,a') = \int p_D(s'|s,a) p_D(a'|s,a,\sigma) da$ and reward $R_\sigma(s,a') = \int p_D(a'|s,a,\sigma)(R(s,a) - \|a-a'\|_2^2) da$, where $p_D(a'|s,a,\sigma)$ is the noise distribution normalized against the dataset action distribution. Theorem 5.3 shows that the minimizer of $\mathbb{E}_{a\sim p_D(\cdot|s), a'\sim q_\sigma(\cdot|a)} \|Q(s,a') - \bar y(s,a,a')\|_2^2$ with target $\bar y = \mathbb{E}_{s'\sim p_D(\cdot|s,a), \bar a\sim\pi(\cdot|s')}[R(s,a) - \|a-a'\|_2^2 + \gamma Q^\pi(s', \bar a)]$ is exactly the $\pi$ Q-function of the NAMDP. Around this fixed point, the paper adds an error bound between NAMDP and true MDP returns, and a small-noise analysis showing the NAMDP's optimal policy selects actions within dataset support. Empirically, the method manifests as TD3-AN, IQL-AN, and QGPO-AN, with Table 1 reporting gains over tuned baselines across Gym-MuJoCo and AntMaze.

Load-bearing premise

The proof that PANI solves the noisy action MDP assumes updates are chased toward the true value of the policy, but the implemented algorithms chase their own bootstrapped estimates, and the paper does not prove those two processes converge to the same answer.

Editorial extensions

If this is right

  • PANI can be dropped into any off-policy Q-learning update: only the sampling line and the target arithmetic change, so TD3, IQL, and even the diffusion-based QGPO all improve with minimal code changes.
  • OOD overestimation falls: the measured probability that a uniformly sampled action outscores a dataset action drops from roughly 0.1–0.5 to below 0.04 on the tested Gym-MuJoCo datasets.
  • The small-noise analysis implies a safety property: as noise tends to zero, the NAMDP's optimal policy avoids actions outside the dataset support, which is exactly the failure mode offline RL needs to prevent.
  • A hybrid noise distribution—mixing a uniform component with exponentially scaled Gaussian scales—makes performance stable across noise levels, reducing the need for careful per-task $\sigma$ tuning.
  • Diffusion models are not required for the gains: lightweight IQL-AN and TD3-AN match or beat several diffusion-policy baselines on the medium and AntMaze suites (e.g., average medium score 90.2 versus 88.7 for the best diffusion-free baseline shown, and AntMaze average 77.7 versus 78.3 for QGPO).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NAMDP equivalence is extended from the population fixed point to the actual bootstrapped target networks used in training, PANI would supply a clean theoretical template for why simple penalties can replace generative policies; a tabular MDP experiment comparing learned Q with the exact NAMDP Q would test this directly.
  • The paper's empirical noise-scale guideline—high action diversity wants low noise, low diversity wants high noise—suggests a practical automatic rule, for instance setting $\sigma$ from nearest-neighbor action distances or dataset density estimates, which the paper does not develop.
  • The same noise-injection-plus-penalty idea could be applied to other value-based and model-based offline methods, and to high-dimensional action spaces where Gaussian noise becomes sample-inefficient; the squared-distance penalty might then need normalization to the reward scale.
  • Because the noise kernel re-weights both reward and transition, the NAMDP view connects PANI to robust MDP and anti-exploration ideas, suggesting that noise scale and penalty weight are dual regularization knobs that could be tuned jointly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Penalized Action Noise Injection (PANI), a modification of Q-learning for offline RL in which dataset actions are perturbed by noise and the target is penalized by the squared distance between the original and noisy action. The authors introduce a 'Noisy Action MDP' (NAMDP) and claim that minimizing the PANI objective solves this modified MDP, thereby providing a theoretical foundation for the method. They also design a hybrid noise distribution intended to be robust to noise-scale selection. Empirically, the paper applies PANI to TD3, IQL, and QGPO, reporting improved D4RL scores on many Gym-MuJoCo and AntMaze tasks compared with IQL and TD3+BC, together with ablations on noise distributions, noise scales, wall-clock time, and a direct measurement of OOD overestimation.

Significance. If the empirical results hold, PANI is an appealingly simple and computationally cheap alternative to diffusion-based offline RL methods, and the extensive D4RL evaluation plus detailed hyperparameter tables and ablations are valuable. The paper also ships code and gives concrete implementation details, which strengthens reproducibility. However, the theoretical contribution as stated does not currently cover the implemented algorithms: the main theorem assumes an oracle target using the true Q^pi, while the reported IQL-AN and TD3-AN use learned, bootstrapped targets, and no convergence argument bridges this gap. The NAMDP equivalence is also largely a fixed-point identity. The significance of the paper therefore rests mainly on the empirical claims, which are substantial but would need to be presented without the overreach in the theoretical framing.

major comments (3)
  1. [Theorem B.3 and Appendix C, Eq. (66), (69)] Theorem B.3 (and its main-text version Theorem 5.3) states that minimizing the PANI objective yields the Q-function of pi in the NAMDP, but its target y-bar contains gamma Q^pi(s', a-bar) with the true Q^pi. The implemented IQL-AN objective in Eq. (66) uses gamma V_psi(s') with V_psi trained by expectile regression, and the TD3-AN objective in Eq. (69) uses gamma min_i Q_{theta'_i}(s', a-tilde) with target networks and clipped policy noise. The paper provides no contraction, convergence, or fixed-point argument that relates these iterative, bootstrapped targets to the NAMDP fixed point of Theorem B.3. Consequently, the claimed theoretical foundation does not apply to the algorithms whose D4RL scores are reported; the empirical gains may be attributable to the squared-distance penalty acting as a plain regularizer. I request either a proof connecting the implemented target updates to the NAMDP solution or an explicit reframing of Theorem B.3 as a population-level fixed-point identity that motivates, rather than establishes, the algorithm.
  2. [Theorem B.3 proof, Eq. (8)-(21)] The theorem is close to a fixed-point identity. Because the target is defined using Q^pi, the derivation shows that any minimizer Q* satisfies a Bellman equation whose solution is already Q^pi; the conclusion then follows from uniqueness of the Bellman equation for a fixed policy. The statement would be more informative if it identified exactly which assumption is being verified, for example that the PANI regression fixed point coincides with the NAMDP Bellman fixed point, rather than presenting the Euler-equation argument as an independent derivation of the NAMDP equivalence.
  3. [Theorem B.11 proof, Eq. (59)] The proof of Theorem B.11 contains an incorrect expectation. In Eq. (59) the proof bounds max_{a in C(a', p_D(·|s))} Q*_sigma(s,a) below by E_{a ~ p_D(·|s)}[Q*_sigma(s,a)], but Lemma B.7 and the subsequent algebra require the expectation over p_C(·|a',s), the restriction of the dataset action distribution to the closest-point set C. As written, the inequality is false because p_D(·|s) can place mass outside C with larger Q-values. Replacing p_D by p_C in that line repairs the argument, but the theorem's proof currently contains a technical error at a load-bearing step.
minor comments (4)
  1. [Definition 5.1] Definition 5.1 says 'with support supp(q_sigma) such that the action space A is a subset of its support'; since q_sigma is a family of distributions parameterized by a, the support condition should be stated per conditional distribution q_sigma(·|a), and the current phrasing is ambiguous.
  2. [Section 7.2] The text states that PANI with the hybrid distribution outperforms baselines 'even with all hyperparameters fixed,' but Table 5 reports per-environment optimal log sigma and alpha values selected from grids; please clarify that only the non-noise hyperparameters are fixed, or provide an explicit no-tuning experiment.
  3. [Table 6a] The OOD overestimation metric P(Q(s,a') > Q(s,a)) with a' drawn uniformly is mechanically lowered by the squared-distance penalty, since distant actions receive penalized targets; please report an additional metric that separates the penalty from genuine value calibration, such as error against true returns in an environment with known dynamics.
  4. [Figures 7-9] The training curves are labeled with log sigma settings that differ in range between Gaussian/Laplace (0 to -2) and hybrid (-20 to -1); please make the legend keys consistent and legible so that the comparison across noise distributions is not visually misleading.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the NAMDP equivalence is a genuine mathematical derivation, and the main self-citation is a standard lemma that is not load-bearing.

full rationale

The central theoretical claim, Theorem B.3, is a direct Euler-Lagrange derivation showing that the minimizer of the PANI population objective satisfies the Bellman equation of the NAMDP. Although the target in the theorem uses the true Q^pi, that is an explicit oracle assumption, not a hidden reuse of the conclusion: the proof derives Q = Q^pi from the first-order condition. The NAMDP reward and transition are defined from the same noise and penalty terms, so the equivalence is a formal identity, but it is presented as a mathematical grounding rather than as an empirical prediction. The implemented algorithms (IQL-AN Eq. 66, TD3-AN Eq. 69) replace the oracle target with bootstrapped targets, so the theory does not directly cover the implementations; this is a rigor gap between theory and practice, but it is not circular because the reported scores are external D4RL results and are not used as evidence inside the theorem. The only self-citation is Lemma A.1 from Lee et al. 2020 used in Theorem B.4's error bound; this is a standard simulation lemma and does not constitute the paper's central premise. The Limitations section honestly states that selecting the optimal noise scale remains challenging, which further supports that the authors are not overclaiming. Overall, the derivation is self-contained in the sense that no fitted parameter is renamed as a prediction and no load-bearing argument reduces to a self-citation chain.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The central method depends on two fitted hyperparameters (noise scale and IQL τ/α), on a theorem whose target is defined using the solution function, and on finite-state assumptions that do not match the continuous benchmarks. No new physical or causal entity is introduced.

free parameters (2)
  • Noise scale log σ (per environment and algorithm) = Grid values: -20, -10, -5, -1 (and -0.5, -30 for QGPO-AN)
    Selected per D4RL task by grid search (Table 5); central PANI performance is sensitive to it, and Figure 4a reports results at the 'Optimal' scale per distribution.
  • IQL expectile τ and AntMaze policy weight α = τ ∈ {0.7, 0.9, 0.99}; α ∈ {0.3, 0.5, 1.0}
    Tuned per environment as part of the PANI pipeline; inherited from IQL but still part of the reported configuration.
assumptions (5)
  • domain assumption Assumption B.5: noise kernel ratios decay to 0/1 in σ→0 limits
    Used to derive the σ→0 behavior of NAMDP (Lemma B.6, B.7); holds for Gaussian and Laplace but not proven for the hybrid distribution actually used.
  • ad hoc to paper The PANI target in Theorem B.3 is evaluated with the policy's true Qπ
    This makes the theorem a fixed-point characterization; implemented targets use learned Vψ or target critics, so the theorem does not directly cover the algorithms in Appendix C.
  • domain assumption Finite state and action spaces for the small-noise OOD guarantees
    Corollaries B.8-B.11 assume finite S,A; D4RL experiments use continuous state/action spaces, so the guarantees do not directly transfer.
  • standard math Lemma A.1 from Lee et al. (2020) bounding occupancy differences
    Imported without proof for the NAMDP error bound (Theorem B.4); authored by the present second author, but plausibly a standard simulation lemma.
  • domain assumption Q-network is updated on noisy actions that may lie outside the bounded action space
    Section 6 flags out-of-range Gaussian samples as invalid, but Algorithms 1 and 2 do not specify clipping or masking for a'.
invented entities (1)
  • Noisy Action MDP (NAMDP)
    purpose: Formal MDP whose Q-function is claimed to equal PANI's penalized noise-injected update.
    A paper-defined mathematical object; its equivalence to PANI relies on the oracle-Qπ target assumption and finite-dataset definitions, not on external falsifiable predictions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Offline Reinforcement Learning with Penalized Action Noise Injection." pith.science (2026). https://pith.science/paper/PLKEF4JU

@misc{pith2026250702356,
  author       = {Pith},
  title        = {Pith review of: Offline Reinforcement Learning with Penalized Action Noise Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PLKEF4JU}},
  note         = {Machine review of arXiv:2507.02356}
}
read the original abstract

Offline reinforcement learning (RL) optimizes a policy using only a fixed dataset, making it a practical approach in scenarios where interaction with the environment is costly. Due to this limitation, generalization ability is key to improving the performance of offline RL algorithms, as demonstrated by recent successes of offline RL with diffusion models. However, it remains questionable whether such diffusion models are necessary for highly performing offline RL algorithms, given their significant computational requirements during inference. In this paper, we propose Penalized Action Noise Injection (PANI), a method that simply enhances offline learning by utilizing noise-injected actions to cover the entire action space, while penalizing according to the amount of noise injected. This approach is inspired by how diffusion models have worked in offline RL algorithms. We provide a theoretical foundation for this method, showing that offline RL algorithms with such noise-injected actions solve a modified Markov Decision Process (MDP), which we call the noisy action MDP. PANI is compatible with a wide range of existing off-policy and offline RL algorithms, and despite its simplicity, it demonstrates significant performance improvements across various benchmarks.

Figures

Figures reproduced from arXiv: 2507.02356 by the authors.

Figure 1
Figure 1. Visualization of learned Q-values on toy datasets. Each pair compares models trained with and without PANI on Rings (left) and Pinwheel (right). Background shows Q-values (red: high, blue: low); circles represent dataset actions, colored by their rewards. This formulation can also be viewed from the perspective of DSM. Defining qσt (·|x) ∼ N (αtx, σ2 t I): ∥ϵθ(zt, t) − ϵ∥ 2 2 = σ 2 t ∥sθ(zt, t) − ∇ log qσt (zt|x)∥ 2… view at source ↗
Figure 2
Figure 2. Left: Noise distributions and the re￾sulting noised distribution. Right: Q-value predic￾tions under the NAMDP, with the background color representing pD(a ′ | a1, σ). Note that a1 = −1, a2 = 1, with rewards r(a1) = 0, r(a2) = 1. The green curve shows the groundtruth Q-values. Given a distribution p, the probability of gen￾erating a sample a ′ given a, incorporating the noise distribution qσ, can be expressed as: p(a… view at source ↗
Figure 3
Figure 3. Comparison of noise distributions (Gaussian and Laplace) and their impact on NAMDP. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Comparison of noise distribution and noise scale effects in Gym-MuJoCo. High, Medium, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Learned Q-value landscapes in Gym-MuJoCo datasets. Colors represent Q-value magni￾tudes (red: high, blue: low), and the white star indicates the dataset action. These plots illustrate how the learned Q-functions evaluate dataset and surrounding actions across different…
Figure 6
Figure 6. Figure 6: Empirical evaluation of OOD overestimation reduction and performance improvement. (a) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Training curves of TD3-AN with Gaussian Noise Distribution in Gym-MuJoCo. [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: Training curves of TD3-AN with Laplace Noise Distribution in Gym-MuJoCo. [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Training curves of TD3-AN with hybrid noise distribution in Gym-MuJoCo. [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Training curves of TD3-AN with hybrid noise distribution in AntMaze. [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: Training curves of IQL-AN with hybrid noise distribution in Gym-MuJoCo [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Training curves of IQL-AN with hybrid noise distribution in AntMaze. [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 14 canonical work pages

  1. [1]

    Uncertainty-based offline reinforcement learning with diversified q-ensemble

    Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34: 0 7436--7447, 2021

    work page 2021

  2. [2]

    Layer normalization

    Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

  3. [3]

    Offline reinforcement learning via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548, 2022

    arXiv 2022

  4. [4]

    Score regularized policy optimization through diffusion behavior

    Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023

    arXiv 2023

  5. [5]

    Diffusion policies creating a trust region for offline reinforcement learning

    Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. arXiv preprint arXiv:2405.19690, 2024

    arXiv 2024

  6. [6]

    Heavy-tailed denoising score matching

    Jacob Deasy, Nikola Simidjievski, and Pietro Li \`o . Heavy-tailed denoising score matching. arXiv preprint arXiv:2112.09788, 2021

    work page Pith review arXiv 2021

  7. [7]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020

    arXiv 2004

  8. [8]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 0 20132--20145, 2021

    2021

Show all 37 references
  1. [9]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587--1596. PMLR, 2018

    2018

  2. [10]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052--2062. PMLR, 2019

    2019

  3. [11]

    Calculus of variations

    Izrail Moiseevitch Gelfand, Richard A Silverman, et al. Calculus of variations. Courier Corporation, 2000

    2000

  4. [12]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  5. [13]

    Estimation of non-normalized statistical models by score matching

    Aapo Hyv \"a rinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6 0 (4), 2005

    2005

  6. [14]

    Understanding diffusion objectives as the elbo with simple data augmentation

    Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  8. [16]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  9. [17]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019

    2019

  10. [18]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 1179--1191, 2020

    2020

  11. [19]

    Reinforcement learning with augmented data

    Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 0 19884--19895, 2020

    2020

  12. [20]

    Batch reinforcement learning with hyperparameter gradients

    Byungjun Lee, Jongmin Lee, Peter Vrancx, Dongho Kim, and Kee-Eung Kim. Batch reinforcement learning with hyperparameter gradients. In International Conference on Machine Learning, pages 5725--5735. PMLR, 2020

    2020

  13. [21]

    Learning energy-based models in high-dimensional spaces with multiscale denoising-score matching

    Zengyi Li, Yubei Chen, and Friedrich T Sommer. Learning energy-based models in high-dimensional spaces with multiscale denoising-score matching. Entropy, 25 0 (10): 0 1367, 2023

    2023

  14. [22]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825--22855. PMLR, 2023

    2023

  15. [23]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  16. [24]

    Anti-exploration by random network distillation

    Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pages 26228--26244. PMLR, 2023

    2023

  17. [25]

    Heavy-tailed diffusion models

    Kushagra Pandey, Jaideep Pathak, Yilun Xu, Stephan Mandt, Michael Pritchard, Arash Vahdat, and Morteza Mardani. Heavy-tailed diffusion models. arXiv preprint arXiv:2410.14171, 2024

    2024 arXiv

  18. [26]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022

    2022 arXiv

  19. [27]

    Efficient differentiable simulation of articulated bodies

    Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning, pages 8661--8671. PMLR, 2021

    2021

  20. [28]

    Offline reinforcement learning as anti-exploration

    Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, L \'e onard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8106--8114, 2022

    2022

  21. [29]

    S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics

    Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning, pages 907--917. PMLR, 2022

    2022

  22. [30]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019

    2019

  23. [31]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  24. [32]

    Revisiting the minimalist approach to offline reinforcement learning

    Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    A connection between score matching and denoising autoencoders

    Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 0 (7): 0 1661--1674, 2011

    2011

  26. [34]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022

    2022 arXiv

  27. [35]

    On scale mixtures of normal distributions

    Mike West. On scale mixtures of normal distributions. Biometrika, 74 0 (3): 0 646--648, 1987

    1987

  28. [36]

    Exploration and anti-exploration with distributional random network distillation

    Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li. Exploration and anti-exploration with distributional random network distillation. arXiv preprint arXiv:2401.09750, 2024

    2024 arXiv

  29. [37]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35: 0 23851--23866, 2022

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.