Pith. sign in

REVIEW 5 major objections 5 minor 65 references

Expert Behavior Prior Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Online RL can generate expert-like policy priors on the fly, without any offline demonstration data.

desk verdict Genuinely new mechanism and a big empirical sweep, but the theory is broken, the headline comparison is partly confounded, and the code is missing. read the letter →

arxiv 2607.21302 v2 pith:UPPGNIWX submitted 2026-07-23 cs.AI

Gong Gao , Weidong Zhao , Xianhui Liu , Ning Jia This is my paper
classification cs.AI
keywords expertbehaviorpriorreinforcementlearningconditionalvariationalautoencoderQ-guidedgenerativemodelsampleefficiencycontinuouscontrolTD3policygradientcorrection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that reinforcement learning agents can obtain expert-quality behavior priors without any offline demonstration data. Its proposed EBP algorithm trains a conditional variational autoencoder on the agent's own replay buffer, using the Q-function as an additional loss to push generated actions toward high value. The generated actions are scored by the critics and the best one is used as a supervised anchor for the actor, with an adaptive weight that rebalances Q-guidance and expert supervision. Across 20 continuous-control tasks in Gym, PyBullet, and DMControl, with a single fixed hyperparameter set, EBP reports higher sample efficiency and more stable convergence than TD3 and several prior-guided baselines.

What carries the argument

The Q-CVAE (Q-guided conditional variational autoencoder) is a CVAE trained on the replay buffer with a reconstruction term and a Q-maximization term, which generates diverse high-value candidate actions. The expert policy guidance (EPG) mechanism selects the highest min-Q action from a support set of H sampled actions to serve as the supervised anchor for the actor. The policy gradient correction (PGC) module computes an adaptive weight from the cosine similarity gap between Q-guidance and expert-supervision gradients, with a margin m, so the expert term corrects Q-gradients when they diverge and is softly clipped when they align.

What would settle it

Intermittently execute the Q-CVAE's selected 'expert' actions directly in the environment (without using them for updates) and compare their true returns with the actor's actions throughout training. If the selected expert actions frequently yield lower true returns than the actor's actions, especially in the early phase, the mechanism's premise fails.

Watch

Extended reading notes

Core claim

The central claim is that a generative policy prior can be produced online — from the replay buffer itself — rather than from static offline expert trajectories. The Q-CVAE learns to reconstruct observed actions while also maximizing Q-values through the double critic; the expert policy guidance step then draws H candidate actions, picks the one with the highest min-Q-value, and uses it as the target in a supervised actor loss. The policy gradient correction module weights this supervised loss by a ReLU function of the cosine similarity gap between the Q-gradient and the supervised gradient, so that supervision is strengthened when the two gradients disagree and weakened when they align. EBP

Load-bearing premise

The load-bearing premise is that the Q-guided CVAE, trained with the current Q-function on the online replay buffer, generates actions that are genuinely high-value in the environment; if Q-estimation is inaccurate early in training, the 'expert' anchors can be worse than the actor's own actions, and the supervision would then reinforce suboptimal behavior.

Editorial extensions

If this is right

  • Online RL can obtain expert-like priors without offline data, removing the data-quality bottleneck of behavior prior RL.
  • A single fixed hyperparameter set transfers across Gym, PyBullet, and DMControl, suggesting the method does not rely on environment-specific tuning.
  • The PGC weighting mechanism provides a way to combine Q-guidance and supervised anchors, potentially reducing policy oscillation.
  • The approach improves both TD3 and DDPG bases, and gains are largest in the early stage (200K timesteps), indicating sample-efficiency gains.
  • EBP's performance drop under reward noise is smaller than TD3's on the Ant task, suggesting robustness to reward perturbation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Q-function is overoptimistic early in training, the Q-CVAE's 'expert' anchors could be systematically biased toward actions that exploit Q error; the paper's ablation measures Q-estimates of generated actions, not their true returns, so it cannot rule this out.
  • The support-set mechanism that samples H candidates and picks the max-Q one is a form of optimism on a learned proposal distribution; viewing EBP this way suggests connections to other 'optimistic exploration' tricks.
  • A natural testable extension is to apply EBP to stochastic actor-critic algorithms or vision-based control, where the replay buffer is more diverse and the Q-function noisier.
  • The paper's own limitation note that moderate reward noise unexpectedly helped HalfCheetah suggests Q-uncertainty may have a dual role; a controlled study of injected stochasticity could reveal when noise aids exploration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Expert Behavior Prior (EBP), an online RL algorithm built on TD3 (and a DDPG variant) that trains a Q-guided conditional variational autoencoder (Q-CVAE) on the current replay buffer, generates H candidate actions per state, selects the action with highest minimum target-Q as an 'expert' anchor, and adds a supervised imitation loss to the actor update. A Policy Gradient Correction (PGC) module reweights the supervised loss using the cosine similarity between Q-gradient and supervision-gradient. The authors compare EBP against TD3, DDPG, SAC, BAC, ALH, and NNPG on eight Gym, four PyBullet, and eight DMControl tasks, report multi-seed results, ablations, reward-noise robustness, RLiable meta-analysis, and runtime/memory overhead. The central claim is that, with fixed hyperparameters, EBP significantly outperforms TD3 and other behavior-prior methods.

Significance. If the empirical claims hold, EBP would be a practically useful and simple plug-in for improving TD3/DDPG in continuous control without offline data. The paper has notable strengths: a broad benchmark suite (20 tasks), ten seeds per task, ablations for the main components (α, H, µ, decay rate), an RLiable aggregate analysis, and explicit runtime/memory measurements. However, the headline comparison against TD3 is confounded by an un-matched policy update frequency: EBP uses d=1 while TD3 is run with its default d=2, and the paper's own PyBullet experiments show that TD3(d=1) alone can outperform TD3(d=2). The theoretical convergence section is also substantially overclaimed. These issues affect the central claim and require correction before the results can be accepted as stated.

major comments (5)
  1. [Section IV-D, Eqs. (15)–(22)] The EBP-vs-TD3 comparison is confounded by policy update frequency. Algorithm 1 updates the actor every environment step (d=1), and Table IV assigns d=2 to TD3 while d=1 to EBP. Section V-C and Fig. 4 explicitly report that TD3(d=1) outperforms TD3(d=2) on three of the four PyBullet tasks (HalfCheetahBullet, AntBullet, Walker2DBullet). Thus the advantage of EBP over TD3 in the Gym (Table I, Fig. 3) and DMControl (Fig. 5) results may be substantially attributable to the synchronized update schedule rather than to the Q-CVAE, EPG, or PGC modules. Since the central claim in Section VII is that EBP 'with a fixed set of hyperparameters' outperforms TD3, this is not a fixed-hyperparameter comparison. Please add TD3(d=1) controls for Gym and DMControl, or rerun with matched update frequencies, and qualify the claim accordingly.
  2. [Section V-E, Fig. 12] The 'convergence analysis' is not a convergence analysis. Equation (21) only bounds the norm of a single gradient step via triangle inequality and Cauchy–Schwarz; no policy improvement, descent, fixed-point, or convergence argument is provided. Moreover, Eq. (15) defines g as a function of the gradients; when differentiating µg·J_Sup in Eq. (18), the derivative of g with respect to ϕ is ignored. The derivation also drops the H×W average defined in Eq. (14), and the key assumption λ2 ≪ λ1 (Eq. 20) is asserted without evidence. Please either provide a genuine convergence or policy-improvement result, or relabel this section as a norm-bounding remark and remove the word 'convergence'.
  3. [Table I, Section VII, Fig. 13] The claim that the Q-CVAE generates 'high-value' actions is validated only against the same Q-function used to train the model (Eq. 7) and to select anchors (Eq. 10). This is circular with respect to the mechanism: early in training, when Q estimates are inaccurate, the 'expert' anchor may exploit Q-estimation error rather than yield high true returns. The ablation in Fig. 12 therefore does not establish that the generated actions are actually high-value in the environment. Please provide rollout-based validation of the prior's actions, or compare against a behavior-cloned prior evaluated by true environment returns.
  4. [Section V-E, Fig. 10; Section VI] The claim of 'significantly outperforms' is not uniform across the reported results. At 200K timesteps, EBP is below TD3 on Ant (1070 vs 1308) and BipedalWalker (93 vs 118); at 2M timesteps, EBP is below NNPG on Hopper (3279 vs 3467). The RLiable aggregate (Fig. 13) is a useful summary, but it does not by itself support the universal wording in the abstract and Section VII. Please either narrow the claim to aggregated or final-performance improvement, or report per-task statistical significance with appropriate multiple-comparison control.
  5. [Eq. (18), Fig. 5 caption, Section V-E] The paper's Limitation section (Section VI) identifies only the reward-noise phenomenon as an open question. Given the update-frequency confound and the circular validation of the Q-CVAE's 'expert' quality, the limitations should also acknowledge these issues, as they directly affect the interpretation of the empirical results.
minor comments (5)
  1. [Section V-E (Fig. 6 discussion)] The caption states 'four DMControl continuous control tasks' but eight DMControl environments are listed and eight panels are shown. Correct to 'eight'.
  2. [Section V-E (noise robustness)] The notation 'EBP(w/ CV AE)' is confusing because 'w/' conventionally means 'with'. Since this variant is the one without the Q-guided loss, use 'EBP(w/o Q-CVAE)' or 'EBP(w/ CVAE-only)'.
  3. [Eq. (18)] The sentence 'We attribute this phenomenon to the smoothing effect introduced in TD3's Q-value estimation process under noisy rewards...' appears twice verbatim in the noise-robustness discussion. Remove the duplicate.
  4. [Section V-E, Eq. (14)] The derivation in Eq. (18) is notational inconsistent: after substituting for µg, the expectation E_s[2(π−ã)·∇π] already equals ∇J_Sup, but the line then multiplies by ∇J_Sup again. Also, the H×W average from Eq. (14) is silently dropped. Please clean up the algebra and align the notation with Eq. (14).
  5. [Section V-E, Eq. (14)] The parameter m is described as 'Gradient similarity margin' in Table IV and set to 0.05·action_dim. Please clarify whether m is task-dependent and how the value was chosen; the fixed-hyperparameter claim is weakened if m scales with the action dimension while other hyperparameters are fixed.

Circularity Check

1 steps flagged · score 3.0 of 10

Q-CVAE 'high-value expert' validation is self-referential (trained, selected, and evaluated with the same Q-function); main benchmark comparisons are external and not circular.

  1. fitted input called prediction [Section V-E 'Value Visualization of the Behavioral Policy', Fig. 12; Eq. (7) and Eq. (10)]
    "The Q-value guided loss can be formulated as HQ(ω) = −0.5 P n=1,2 Qθn(s, qω(s,z)). ... ˜a = argmax ah ( min n=1,2 (Qθ′ n (ah))), ah ∈ Support(Gω(·|s)). ... we collect the maximum and mean Q-values over 10 actions sampled from the Q-CV AE ... the maximum Q-values of actions generated by the policy prior network are consistently higher than those produced by the actor network, while the mean Q-values ... tend to be lower ..."

    The 'expert' action is defined by Eq. (10) as the argmax of the critic Qθ′ over samples from Gω, and Gω is trained in Eq. (7) to maximize the same critic Qθ. The evidence that the Q-CVAE 'can generate high-value actions' (Fig. 12) then measures the Q-values of those same samples with that same critic. The ordering (max over H samples > actor value) is substantially forced by the training objective and by taking a maximum; it is not a check against true environment return. Thus the 'expert prior' validation reduces to comparing Q with itself, and the supervised term JSup in Eq. (15) is a Q-derived target rather than an independent correction signal.

full rationale

The central empirical claims (Tables I and Fig. 3-5, 13) are evaluated by environment returns and are therefore independent of the paper's own equations and self-citations; the claimed superiority over TD3 does not reduce to the Q-network's self-evaluation. The convergence analysis (Eqs. 18-22) is a trivial bound that restates the definition of the PGC weight and the bounded-gradient assumptions, but it makes no falsifiable prediction and is not a circular derivation in the sense prohibited here. The only genuine circular step is the internal validation of the Q-CVAE: the generator is trained to maximize Q, the 'expert' action is selected by maximizing Q, and the claim that the CVAE yields 'high-value' priors is then evidenced by the same Q-values. This supports one module's motivation but does not by itself force the main benchmark result. The d=1 vs d=2 actor-update-frequency confound noted in Section V-C/Table IV is a benchmarking fairness concern, not a circularity, and is not counted in the score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on standard RL assumptions plus several domain-specific premises: the expressivity of the CVAE, the validity of the Q-guided loss, and the unverified gradient-norm ordering λ2 ≪ λ1 in the convergence analysis. The hyperparameters α, H, µ, decay rate, and m are hand-tuned on the same benchmark tasks where the method is evaluated, which is a common but non-negligible source of selection bias.

free parameters (5)
  • Q-guided coefficient α = 0.11
    Chosen via ablation on HalfCheetah and Ant (Fig. 6); applied to all tasks.
  • Number of policy priors H = 10
    Selected by ablation on HalfCheetah and Ant (Fig. 7); balances performance and stability.
  • Expert policy guidance coefficient µ = 1.0
    Tuned via ablation (Fig. 8); µ=1.0 gives best overall performance.
  • EPG exponential decay rate = 0.97
    Chosen by sensitivity analysis (Fig. 9); controls how fast expert guidance fades.
  • Gradient similarity margin m = 0.05·action dim
    Set heuristically per action dimension; no ablation reported.
assumptions (4)
  • domain assumption Gradients of Q-guidance and supervised losses are individually bounded (Eq. 19)
    Assumed to derive the gradient norm bound; not verified for any of the trained networks.
  • ad hoc to paper Supervised gradient norm is much smaller than Q-gradient norm (λ2 ≪ λ1, Eq. 20)
    Asserted without evidence, used to claim stability. No measurement of these norms is provided.
  • domain assumption Q-CVAE trained with Eq. (8) yields high-value actions
    Core premise of the method; validation is only via the same Q-function used for training (Fig. 12), not against true returns.
  • standard math Markov Decision Process framework with deterministic transitions and rewards
    Standard RL setup, used throughout the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expert Behavior Prior Reinforcement Learning." pith.science (2026). https://pith.science/paper/UPPGNIWX

@misc{pith2026260721302,
  author       = {Pith},
  title        = {Pith review of: Expert Behavior Prior Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UPPGNIWX}},
  note         = {Machine review of arXiv:2607.21302}
}
read the original abstract

Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pre-training methods and propose an Expert Behavior Prior (EBP) algorithm. Specifically, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on pre-collected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.

Figures

Figures reproduced from arXiv: 2607.21302 by the authors.

Figure 1
Figure 1. Detailed description of the Toy environment and a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed Expert Behavior Prior (EBP) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Learning curves on eight Gym continuous control tasks. The shaded region represents half a standard deviation of the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Learning curves on four PyBullet continuous control tasks. The shaded region represents half a standard deviation of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Learning curves on four DMControl continuous control tasks. The shaded region represents half a standard deviation [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the average returns obtained by the Q [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 9
Figure 9. Figure 9: Ablation experiments of our algorithm with PGC coef [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 8
Figure 8. Figure 8: Ablation experiments of our algorithm with different [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 10
Figure 10. Figure 10: Average return over 10 random seeds under varying [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Comparison of learning curves obtained by using MSE and KL divergence as optimization objectives across eight [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Q-value evolution curves for actions sampled from the [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: RLiable [65] analysis allowing us to aggregate results across three continuous control environments and show the [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Images for Gym, PyBullet, and DMControl environments used in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

65 extracted references · 13 linked inside Pith

  1. [1]

    Diverse imitation learning via self-organizing generative models,

    A. Vahabpour, T. Wang, Q. Lu, O. Pooladzandi, and V . Roychowdhury, “Diverse imitation learning via self-organizing generative models,”IEEE Trans. Neural Networks Learn. Sys., 2024

    2024

  2. [2]

    Markov balance satisfaction improves performance in strictly batch offline imitation learning,

    R. Agrawal, N. Dahlin, R. Jain, and A. Nayyar, “Markov balance satisfaction improves performance in strictly batch offline imitation learning,” inProc. AAAI Conf. Artif. Intell., AAAI, vol. 39, 2025, pp. 15 311–15 319

    2025

  3. [3]

    X-il: Exploring the design space of imitation learning policies,

    X. Jia, A. Donat, X. Huang, X. Zhao, D. Blessing, H. Zhou, H. A. Wang, H. Zhang, Q. Wang, R. Lioutikovet al., “X-il: Exploring the design space of imitation learning policies,”arXiv preprint arXiv:2502.12330, 2025

    arXiv 2025

  4. [4]

    Augmenting decision with hypothesis in reinforcement learning,

    N. M. Quang and H. W. Lauw, “Augmenting decision with hypothesis in reinforcement learning,” inProc. Mach. Learn. Res., 2024

    2024

  5. [5]

    Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters,

    K. Ghasemipour, S. S. Gu, and O. Nachum, “Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters,”Adv. Neural Inform. Process. Syst., vol. 35, pp. 18 267–18 281, 2022

    2022

  6. [6]

    Epistemic bellman operators,

    P. R. Van der Vaart, M. T. Spaan, and N. Yorke-Smith, “Epistemic bellman operators,” inProc. AAAI Conf. Artif. Intell., AAAI, vol. 39, 2025, pp. 20 973–20 981

    2025

  7. [7]

    Behavior priors for efficient reinforcement learning,

    D. Tirumala, A. Galashov, H. Noh, L. Hasenclever, R. Pascanu, J. Schwarz, G. Desjardins, W. M. Czarnecki, A. Ahuja, Y . W. Teh, and N. Heess, “Behavior priors for efficient reinforcement learning,”J. Mach. Learn. Res., vol. 23, no. 221, pp. 1–68, 2022

    2022

  8. [8]

    Pre-training goal-based models for sample-efficient reinforcement learning,

    H. Yuan, Z. Mu, F. Xie, and Z. Lu, “Pre-training goal-based models for sample-efficient reinforcement learning,” inInt. Conf. Learn. Represent., 2024

    2024

Show all 65 references
  1. [9]

    Blend: Behavior-guided neural population dynamics modeling via privileged knowledge distillation,

    Z. Guo, F. Zhou, W. Wu, Q. Sun, L. Feng, J. Wang, and H. Chen, “Blend: Behavior-guided neural population dynamics modeling via privileged knowledge distillation,”arXiv preprint arXiv:2410.13872, 2024

    2024 arXiv

  2. [10]

    Jump-start reinforcement learning,

    I. Uchendu, T. Xiao, Y . Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiaoet al., “Jump-start reinforcement learning,” in Proc. Mach. Learn. Res., 2023, pp. 34 556–34 583

    2023

  3. [11]

    Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers,

    L. Wang, X. Chen, J. Zhao, and K. He, “Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers,”Adv. Neural Inform. Process. Syst., vol. 37, pp. 124 420–124 450, 2024

    2024

  4. [12]

    Learn to supervise: Deep reinforcement learning-based prototype refinement for few-shot motor fault diagnosis,

    P. Xia, Y . Huang, C. Liu, and J. Liu, “Learn to supervise: Deep reinforcement learning-based prototype refinement for few-shot motor fault diagnosis,”IEEE Trans. Neural Networks Learn. Sys., 2024

    2024

  5. [13]

    Efficient online reinforcement learning with offline data,

    P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient online reinforcement learning with offline data,” inProc. Mach. Learn. Res., 2023, pp. 1577–1594

    2023

  6. [14]

    Leveraging offline data in online reinforcement learning,

    A. Wagenmaker and A. Pacchiano, “Leveraging offline data in online reinforcement learning,” inProc. Mach. Learn. Res., 2023, pp. 35 300– 35 338

    2023

  7. [15]

    Enhanc- ing reinforcement learning agents with local guides,

    P. Daoudi, B. Robu, C. Prieur, L. D. Santos, and M. Barlier, “Enhanc- ing reinforcement learning agents with local guides,”arXiv preprint arXiv:2402.13930, 2024

    2024 arXiv

  8. [16]

    Leveraging demonstrations to improve online learning: Quality matters,

    B. Hao, R. Jain, T. Lattimore, B. Van Roy, and Z. Wen, “Leveraging demonstrations to improve online learning: Quality matters,” inProc. Mach. Learn. Res., 2023, pp. 12 527–12 545

    2023

  9. [17]

    Iterative regularized policy optimization with imperfect demonstrations,

    G. Xudong, F. Dawei, K. Xu, Y . Zhai, C. Yao, W. Wang, B. Ding, and H. Wang, “Iterative regularized policy optimization with imperfect demonstrations,” inProc. Mach. Learn. Res., 2024

    2024

  10. [18]

    Constraint- adaptive policy switching for offline safe reinforcement learning,

    Y . Chemingui, A. Deshwal, H. Wei, A. Fern, and J. Doppa, “Constraint- adaptive policy switching for offline safe reinforcement learning,” in Proc. AAAI Conf. Artif. Intell., AAAI, vol. 39, 2025, pp. 15 722–15 730

    2025

  11. [19]

    Residual skill policies: Learning an adaptable skill-based action space for rein- forcement learning for robotics,

    K. Rana, M. Xu, B. Tidd, M. Milford, and N. S ¨underhauf, “Residual skill policies: Learning an adaptable skill-based action space for rein- forcement learning for robotics,” inProc. Conf. Robot Learning., 2023, pp. 2095–2104

    2023

  12. [20]

    Leveraging skills from unlabeled prior data for efficient online exploration,

    M. Wilcoxson, Q. Li, K. Frans, and S. Levine, “Leveraging skills from unlabeled prior data for efficient online exploration,”arXiv preprint arXiv:2410.18076, 2024

    2024 arXiv

  13. [21]

    Policy regularization with dataset constraint for offline reinforcement learning,

    Y . Ran, Y .-C. Li, F. Zhang, Z. Zhang, and Y . Yu, “Policy regularization with dataset constraint for offline reinforcement learning,” inProc. Mach. Learn. Res., 2023, pp. 28 701–28 717

    2023

  14. [22]

    Accelerating exploration with unlabeled prior data,

    Q. Li, J. Zhang, D. Ghosh, A. Zhang, and S. Levine, “Accelerating exploration with unlabeled prior data,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 67 434–67 458, 2023

    2023

  15. [23]

    Cross-domain offline policy adaptation with optimal transport and dataset constraint,

    J. Lyu, M. Yan, Z. Qiao, R. Liu, X. Ma, D. Ye, J.-W. Yang, Z. Lu, and X. Li, “Cross-domain offline policy adaptation with optimal transport and dataset constraint,” inInt. Conf. Learn. Represent., 2025

    2025

  16. [24]

    Policy gradient for rectangular robust markov decision processes,

    N. Kumar, E. Derman, M. Geist, K. Y . Levy, and S. Mannor, “Policy gradient for rectangular robust markov decision processes,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 59 477–59 501, 2023

    2023

  17. [25]

    Reinforcement learning: An introduction,

    R. Sutton and A. Barto, “Reinforcement learning: An introduction,” IEEE Trans Neural Netw., vol. 9, no. 5, pp. 1054–1054, 1998

    1998

  18. [26]

    Is q-learning provably efficient?

    C. Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan, “Is q-learning provably efficient?”Adv. Neural Inform. Process. Syst., vol. 31, 2018

    2018

  19. [27]

    Actor-critic alignment for offline-to-online re- inforcement learning,

    Z. Yu and X. Zhang, “Actor-critic alignment for offline-to-online re- inforcement learning,” inProc. Mach. Learn. Res., 2023, pp. 40 452– 40 474

    2023

  20. [28]

    Adaptive policy learning for offline-to-online reinforcement learning,

    H. Zheng, X. Luo, P. Wei, X. Song, D. Li, and J. Jiang, “Adaptive policy learning for offline-to-online reinforcement learning,” inProc. AAAI Conf. Artif. Intell., AAAI, vol. 37, 2023, pp. 11 372–11 380

    2023

  21. [29]

    Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online rl,

    Q.-W. Luo, M.-K. Xie, Y . Wang, and S.-J. Huang, “Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online rl,”Adv. Neural Inform. Process. Syst., vol. 37, pp. 108 167–108 207, 2025

    2025

  22. [30]

    Tree-based batch mode rein- forcement learning,

    D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode rein- forcement learning,”J. Mach. Learn. Res., vol. 6, no. 18, pp. 503–556, 2005

    2005

  23. [31]

    Mildly conservative q-learning for offline reinforcement learning,

    J. Lyu, X. Ma, X. Li, and Z. Lu, “Mildly conservative q-learning for offline reinforcement learning,”Adv. Neural Inform. Process. Syst., vol. 35, pp. 1711–1724, 2022

    2022

  24. [32]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,”arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  25. [33]

    De-pessimism offline reinforcement learning via value compensation,

    Z. Huang, J. Zhao, and S. Sun, “De-pessimism offline reinforcement learning via value compensation,”IEEE Trans. Neural Networks Learn. Sys., 2024

    2024

  26. [34]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning,

    M. Nakamoto, S. Zhai, A. Singh, M. Sobol Mark, Y . Ma, C. Finn, A. Kumar, and S. Levine, “Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning,”Adv. Neural Inform. Process. Syst., vol. 36, pp. 62 244–62 269, 2023

    2023

  27. [35]

    Awac: Accelerating online reinforcement learning with offline datasets,

    A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerating online reinforcement learning with offline datasets,”arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  28. [36]

    Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning,

    H. R. Walke, J. H. Yang, A. Yu, A. Kumar, J. Orbik, A. Singh, and S. Levine, “Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning,” inProc. Conf. Robot Learning., 2023, pp. 1652–1662

    2023

  29. [37]

    Behavior prior representation learning for offline reinforce- ment learning,

    H. Zang, X. Li, J. Yu, C. Liu, R. Islam, R. T. des Combes, and R. Laroche, “Behavior prior representation learning for offline reinforce- ment learning,” inInt. Conf. Learn. Represent., 2023

    2023

  30. [38]

    One act play: Single demonstration behavior cloning with action chunking transformers,

    A. George and A. B. Farimani, “One act play: Single demonstration behavior cloning with action chunking transformers,”arXiv preprint arXiv:2309.10175, 2023

    2023 arXiv

  31. [39]

    Policy optimization with demonstrations,

    B. Kang, Z. Jie, and J. Feng, “Policy optimization with demonstrations,” inProc. Mach. Learn. Res., 2018, pp. 2469–2478

    2018

  32. [40]

    Reinforcement learning with sparse rewards using guidance from offline demonstration,

    D. Rengarajan, G. Vaidya, A. Sarvesh, D. Kalathil, and S. Shakkottai, “Reinforcement learning with sparse rewards using guidance from offline demonstration,”arXiv preprint arXiv:2202.04628, 2022

    2022 arXiv

  33. [41]

    Goal-conditioned on-policy reinforcement learning,

    G. Xudong, F. Dawei, K. Xu, B. Ding, and H. Wang, “Goal-conditioned on-policy reinforcement learning,” inAdv. Neural Inform. Process. Syst., 2024

    2024

  34. [42]

    Recurrent experience replay in distributed reinforcement learning,

    S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney, “Recurrent experience replay in distributed reinforcement learning,” in Int. Conf. Learn. Represent., 2018

    2018

  35. [43]

    Learning sparse con- trol tasks from pixels by latent nearest-neighbor-guided explorations,

    R. Zhao, U. Topcu, S. Chinchali, and M. Phielipp, “Learning sparse con- trol tasks from pixels by latent nearest-neighbor-guided explorations,” arXiv preprint arXiv:2302.14242, 2023

    2023 arXiv

  36. [44]

    Theoretically principled deep rl acceleration via nearest neighbor function approximation,

    J. Shen and L. F. Yang, “Theoretically principled deep rl acceleration via nearest neighbor function approximation,” inProc. AAAI Conf. Artif. Intell., AAAI, vol. 35, 2021, pp. 9558–9566

    2021

  37. [45]

    A review of recurrent neural net- works: Lstm cells and network architectures,

    Y . Yu, X. Si, C. Hu, and J. Zhang, “A review of recurrent neural net- works: Lstm cells and network architectures,”Neural Comput., vol. 31, no. 7, pp. 1235–1270, 2019

    2019

  38. [46]

    Frustratingly easy regularization on representation can boost deep reinforcement learning,

    Q. He, H. Su, J. Zhang, and X. Hou, “Frustratingly easy regularization on representation can boost deep reinforcement learning,” inIEEE Conf. Comput. Vis. Pattern Recog., 2023, pp. 20 215–20 225

    2023

  39. [47]

    Q-learning with nearest neighbors,

    D. Shah and Q. Xie, “Q-learning with nearest neighbors,”Adv. Neural Inform. Process. Syst., vol. 31, 2018

    2018

  40. [48]

    Improving policy exploitation in online reinforcement learning with instant retrospect action,

    G. Gao, W. Zhao, X. Liu, and N. Jia, “Improving policy exploitation in online reinforcement learning with instant retrospect action,”Neural Netw., p. 108667, 2026

    2026

  41. [49]

    Seizing serendipity: exploiting the value of past success in off-policy actor-critic,

    T. Ji, Y . Luo, F. Sun, X. Zhan, J. Zhang, and H. Xu, “Seizing serendipity: exploiting the value of past success in off-policy actor-critic,” inProc. Mach. Learn. Res., 2024, pp. 21 672–21 718

    2024

  42. [50]

    Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl,

    Y . Luo, T. Ji, F. Sun, J. Zhang, H. Xu, and X. Zhan, “Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl,” inProc. Mach. Learn. Res.PMLR, 2024, pp. 33 411– 33 431

    2024

  43. [51]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” inProc. Mach. Learn. Res., 2019, pp. 2052–2062

    2019

  44. [52]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,”arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  45. [53]

    Reparameterized policy learning for multimodal trajectory optimization,

    Z. Huang, L. Liang, Z. Ling, X. Li, C. Gan, and H. Su, “Reparameterized policy learning for multimodal trajectory optimization,” inProc. Mach. Learn. Res.PMLR, 2023, pp. 13 957–13 975

    2023

  46. [54]

    Didi: diffusion-guided diver- sity for offline behavioral generation,

    J. Liu, X. Guo, Z. Zhuang, and D. Wang, “Didi: diffusion-guided diver- sity for offline behavioral generation,”arXiv preprint arXiv:2405.14790, 2024

    2024 arXiv

  47. [55]

    Addressing function approxima- tion error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxima- tion error in actor-critic methods,” inProc. Mach. Learn. Res., 2018, pp. 1587–1596

    2018

  48. [56]

    R. A. Horn and C. R. Johnson,Matrix Analysis, 2nd ed. Cambridge, U.K.: Cambridge University Press, 2012

    2012

  49. [57]

    Continuous control with deep reinforcement learning,

    T. Lillicrap, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  50. [58]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inProc. Mach. Learn. Res.Stockholmsm ¨assan, Stockholm Sweden: PMLR, 2018, pp. 1861–1870

    2018

  51. [59]

    E. Deza, M. M. Deza, M. M. Deza, and E. Deza,Encyclopedia of distances. Springer, 2009

    2009

  52. [60]

    Reinforcement learning with stochastic reward machines,

    J. Corazza, I. Gavran, and D. Neider, “Reinforcement learning with stochastic reward machines,” inProc. AAAI Conf. Artif. Intell., AAAI, vol. 36, no. 6, 2022, pp. 6429–6436

    2022

  53. [61]

    Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors

    J. Duan, Y . Guan, S. Li, Y . Ren, Q. Sun, and B. Cheng, “Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors.”IEEE Trans. Neural Networks Learn. Sys., vol. 33, no. 11, pp. 6584–6598, 2022

    2022

  54. [62]

    Softmax deep double deterministic policy gradients,

    L. Pan, Q. Cai, and L. Huang, “Softmax deep double deterministic policy gradients,”Adv. Neural Inform. Process. Syst., vol. 33, pp. 11 767– 11 777, 2020

    2020

  55. [63]

    Logit standardization in knowledge distillation,

    S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, “Logit standardization in knowledge distillation,” inIEEE Conf. Comput. Vis. Pattern Recog., 2024, pp. 15 731–15 740

    2024

  56. [64]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  57. [65]

    Deep reinforcement learning at the edge of the statistical precipice,

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Belle- mare, “Deep reinforcement learning at the edge of the statistical precipice,”Adv. Neural Inform. Process. Syst., vol. 34, pp. 29 304– 29 320, 2021. Gong Gaoreceived the M.S. degree in computer science from ...

    2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.