Pith. sign in

REVIEW 4 major objections 3 minor 41 references

Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance

T0 review · 4 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Executing a time-varying blend of expert and RL actions—a few lines of code on top of any actor-critic method—accelerates early learning by over 160% on average while preserving convergence.

desk verdict A simple, plausible acceleration wrapper whose theory is asserted, experiments lack the control to prove the claim, and early gains are likely just expert return. read the letter →

arxiv 2504.18766 v1 pith:GHBXT2PR submitted 2025-04-26 cs.LG cs.AI

Wenjun Cao This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningsampleefficiencyactioninterpolationexpertguidanceactor-criticcontinuouscontrolstatevisitationdistributionTD3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a single mechanism—interpolating the executed action between an expert's action and the reinforcement learning policy's action with a time-varying weight that anneals from 0 to 1—can accelerate any actor-critic algorithm without new losses, auxiliary networks, or architectural changes. The proposed Dynamic Action Interpolation (DAI) is claimed to reshape the state-visitation distribution toward high-value states early in training, speeding value-function learning, while the convergence of the base algorithm is preserved once the weight reaches 1. Empirically, on four MuJoCo continuous-control tasks, DAI reports average early-stage reward improvements over vanilla TD3 of over 160% and final improvements over 50%, with the largest gains on Humanoid. If correct, this means a few lines of code could substitute for much of the intricate machinery currently used to inject expert knowledge into reinforcement learning.

What carries the argument

The load-bearing object is the time-varying interpolation weight $\alpha(t) = \phi(t/T_{\mathrm{change}})$, where $\phi$ is any monotone function from 0 to 1; the implementation uses the linear schedule $\alpha(t) = \min(\max(t/T_{\mathrm{change}}, 0), 1)$. This weight defines the executed action $a_{\mathrm{mix}}(s)$ and thereby the data distribution the RL algorithm trains on. The theoretical work is carried by an unproved approximation, Eq. (8), which replaces the true state distribution of the blended controller by the convex combination of the expert and RL state distributions under a local Lipschitz assumption, together with Eq. (9), which asserts that the expert guides the agent toward high-value states. These feed the value-learning acceleration hypothesis, Eq. (10): the critic's mean-squared error under the blended distribution is smaller than under the initial policy's distribution. Convergence is handled separately by the observation that $\alpha(t)\to 1$ forces the executed action to converge to $a_{\mathrm{RL}}(s)$.

What would settle it

On a MuJoCo task, replace the expert action $a_E(s)$ with an 'anti-expert' that at each state chooses the action minimizing the current critic's value estimate, while keeping the same schedule $\alpha(t)$. If early rewards still improve, the acceleration is not caused by expert-guided state visitation; if they do not, the mechanism is supported. Alternatively, estimate the empirical state-visitation distribution $d_a(s)$ of the blended controller during training and compare it with $(1-\alpha(t)) d_{\pi_E}(s) + \alpha(t) d_{\pi_\theta}(s)$; a large mismatch would indicate the key approximation is false.

Watch

Extended reading notes

Core claim

DAI's central claim is that executing the blended action $a_{\mathrm{mix}}(s) = (1-\alpha(t)) a_E(s) + \alpha(t) a_{\mathrm{RL}}(s)$, with $\alpha(t)$ increasing monotonically from 0 to 1, makes a generic actor-critic learner collect data from more valuable regions of the state space and therefore learn its value estimates faster. The paper argues that the state distribution induced by the blended actions is approximately $(1-\alpha(t)) d_{\pi_E}(s) + \alpha(t) d_{\pi_\theta}(s)$ under local Lipschitz transition dynamics, and that early in training the expert component raises the probability of visiting high-value states. From this it derives the inequality that the critic's squared approximation error under the blended distribution is smaller than under the initial policy distribution. It then shows that because $\lim_{t\to\infty} \alpha(t) = 1$, the executed actions converge to those of the RL policy, so any stationary policy the base algorithm would converge to remains the long-run behavior. The empirical result is that TD3 with DAI outperforms both vanilla TD3 and a behavior-cloned expert on HalfCheetah, Ant, Walker2d, and Humanoid, with the greatest early advantage in the high-dimensional Humanoid environment.

Load-bearing premise

The paper's acceleration proof rests on the assumption that the states visited by the mixed controller are approximately the weighted average of the states the expert would visit and the states the RL policy would visit, an approximation asserted under a smoothness condition but never proved; if this mixture picture is wrong, the claimed faster value learning does not follow.

Editorial extensions

If this is right

  • Any actor-critic algorithm, on-policy or off-policy, can inherit DAI's early-training speedup by adding one weighted action blend to its environment-interaction step; no auxiliary loss or network is required.
  • Because $\alpha(t)$ anneals to 1, DAI does not lock the agent into expert behavior: the final policy is the base algorithm's own stationary policy, so it can surpass the guiding expert.
  • The reported gains are largest where exploration is hardest, which suggests DAI would be most valuable in high-dimensional or sparse-reward control tasks and least valuable when the expert is weak.
  • DAI extends to discrete action spaces by interpolating logits or probability distributions, so the same execution-level mechanism applies beyond the continuous-control benchmarks tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the proposed mechanism would measure the critic's squared value error under DAI versus vanilla TD3; the paper predicts lower error from the blended distribution but reports only reward curves.
  • The expert in the experiments is itself a behavior-cloned SAC policy trained on the same tasks, so DAI's advantage may depend on the expert's quality and coverage; the paper does not investigate what happens with a poor or biased expert.
  • If the state-distribution-shaping story is correct, the speedup should be reproducible or removable by directly reweighting the replay buffer's state distribution; comparing DAI to such a reweighting would isolate the contribution of action interpolation from the mere effect of visiting better states.
  • Adaptive schedules, such as letting $\alpha(t)$ depend on the critic's uncertainty or the TD-error, are a natural extension the paper leaves open; its linear schedule is only a special case.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes Dynamic Action Interpolation (DAI), a method that executes the action a_mix(s) = (1-alpha(t)) a_E(s) + alpha(t) a_RL(s) with a monotonically increasing alpha(t) from 0 to 1, and claims that this simple execution-level intervention accelerates any actor-critic RL algorithm. The theoretical section attempts to show that DAI reshapes the state visitation distribution toward high-value states, thereby reducing value-estimation error, while preserving asymptotic convergence. Experiments compare TD3-DAI with vanilla TD3 and a behavior-cloned expert on four MuJoCo continuous-control tasks, reporting large early and final performance gains. The central claims are that DAI improves early-stage performance by over 160%, final performance by over 50%, and preserves convergence guarantees without extra losses or networks.

Significance. If the claims were established, DAI would be a practically valuable contribution: a few lines of code that accelerate a broad family of actor-critic methods with no architectural overhead. The empirical numbers are striking and the simplicity of the intervention is appealing. However, the paper's significance depends entirely on whether the acceleration is a genuine learning-speed effect rather than a trivial consequence of executing expert actions early. The theoretical framework as written does not establish the mechanism, and the experiments do not isolate it. The contribution is therefore not yet substantiated at the level required for a strong journal publication.

major comments (4)
  1. [Section 4, Eq. (8)] The central approximation d_a(s) ≈ (1-alpha(t)) d_piE(s) + alpha(t) d_pitheta(s) is asserted under 'the assumption of local Lipschitz transition dynamics,' but no derivation is given. Lipschitz continuity of the transition dynamics does not imply linearity or a convex decomposition of state visitation distributions; for nonlinear dynamics, the state distribution under the interpolated action is generally not a convex combination of the two component distributions. Since Eqs. (9) and (10) both rest on Eq. (8), the paper's theoretical claim that DAI accelerates value learning by shaping the state distribution is unproven.
  2. [Section 4, Eqs. (9) and (10)] Eq. (9) is introduced with 'we can posit,' and Eq. (10) is explicitly labeled a 'hypothesis.' Neither is derived from the preceding assumptions. In particular, Eq. (10) is precisely the claim that DAI reduces value-estimation error relative to the initial policy, which is the paper's main theoretical assertion; labeling it a hypothesis means the theoretical section does not actually provide the promised 'theoretical analysis' or 'guarantees.' The acceleration mechanism is therefore an assumption, not a result.
  3. [Section 4, Eq. (12) and surrounding text] The asymptotic convergence argument is essentially a tautology. Given the stated assumption that the base actor-critic method converges to a stationary policy pi_theta,final and that alpha(t) -> 1, the executed action a(s,t) converges to a_RL(s) by Eq. (5), and the expected return converges to that of pi_theta,final. This restates the schedule rather than proving that DAI preserves convergence of the learning algorithm; the policy update rule is unchanged, so no substantive guarantee is established. The phrase 'DAI preserves the long-term performance of any actor-critic algorithm' overstates what Eq. (12) shows.
  4. [Section 5, Tables 1-2 and Figure 1] The experiments lack a control condition that separates 'the agent learns faster' from 'the agent initially acts like the expert.' Because alpha(0)=0 in Eq. (2), the early executed actions are identical to or very close to expert actions, so early reported rewards largely reflect the expert's performance, not the RL learner's improved value estimates. A hard-switch baseline (execute the expert until time T_change and then run the base RL algorithm) would isolate the effect of the gradual schedule. The value of T_change, which defines the schedule in Eq. (4), is never reported, and no sensitivity analysis is provided. Without such a control, the reported early gains do not support the claim that DAI accelerates learning rather than merely collecting expert reward early.
minor comments (3)
  1. [Section 5.1] The expert is a behavior-cloned policy trained from 20 episodes of a separately trained SAC reference model, but the paper does not report demonstration coverage, the SAC reference performance, or BC training details. This makes it difficult to assess whether the expert is a meaningful baseline or whether the 'surpassing the expert' claim depends on a weak expert.
  2. [Figure 1 and Tables 1-2] The text states that curves show median performance across six runs with 95% bootstrapped confidence intervals, while the tables report mean ± standard deviation. The paper should clarify which statistic is primary and whether the reported percentage improvements are computed from means or medians.
  3. [Section 3, Eq. (4)] The schedule parameter T_change is central to the method but its value is never given in the experimental section. The paper should report T_change for each environment and ideally include a sensitivity analysis over this parameter.

Circularity Check

3 steps flagged · score 6.0 of 10

Theoretical acceleration and convergence claims reduce to assumed hypotheses and the α(t)→1 schedule; early gains partly reflect expert return, though empirical comparisons are external and no self-citation loop exists.

  1. self definitional [Section 4, 'Theoretical Analysis of Asymptotic Performance', Eq. (12)]
    "Under this assumption, and given that the action interpolation coefficient α(t) satisfies lim_{t→∞} α(t) = 1, the executed action a(s,t) deterministically converges to a_RL(s) as α(t) → 1. As a result, the long-term behavior of the DAI-enhanced agent converges to that of the base policy π_{θ,final}, and we have: lim_{t→∞} E[G_0|a(s,t)] = E[G_0|π_{θ,final}] (12)"

    This 'preserves convergence guarantees' result is a restatement of the defining properties of α(t): Eq. (3) requires lim_{x→∞} φ(x)=1, and the preceding paragraph assumes the base actor-critic converges to a stationary policy. Once the schedule reaches α=1, the executed action equals the RL action, so the asymptotic return is the base policy's return. The guarantee is built into the definition of the schedule and the convergence assumption; it is not an independent consequence of the interpolation mechanism.

  2. other [Section 4, 'Value Learning Acceleration Hypothesis', Eq. (10)]
    "Value Learning Acceleration Hypothesis. ... E_{s∼d_a}[(V^{πθ}(s) − ˆV(s))^2] < E_{s∼d_{πθ,initial}}[(V^{πθ}(s) − ˆV(s))^2] (10) ... This hypothesis formalizes the intuition that by guiding exploration toward high-value regions, DAI improves the quality of samples used for critic training, resulting in faster convergence of value estimates."

    The paper's central theoretical claim—that DAI accelerates value-function learning—is introduced as a labeled hypothesis and is not derived from Eqs. (8)–(9). Eq. (9) is itself only 'we can posit', and Eq. (8) is an unproved approximation. Thus the inequality in Eq. (10) is exactly the desired conclusion assumed as an input; the subsequent explanation of faster convergence simply restates the hypothesis rather than proving it from the interpolation rule.

1 more flagged steps
  1. other [Section 4, 'State Distribution Shaping', Eq. (8)]
    "While the exact relationship between d_a(s) and the component policies is complex due to the nonlinear dynamics of MDPs, we can characterize it with the following approximation under the assumption of local Lipschitz transition dynamics: d_a(s)≈ (1−α(t))·d_{πE}(s) +α(t)·d_{πθ}(s) (8)"

    The paper advertises a theoretical framework showing how DAI 'reshapes state visitation distributions'. That reshaping claim is Eq. (8) itself, stipulated rather than derived: no argument shows that local Lipschitz continuity of the transition kernel implies the discounted occupancy measure of the interpolated actions is a convex combination of the expert and RL occupancy measures, which is generally false for nonlinear dynamics. The distribution-shaping premise is therefore an assumption that already contains the desired conclusion, not a consequence of Eq. (2).

full rationale

The paper is a single-author preprint with no self-citations, and the empirical comparisons against TD3 and expert baselines are external measurements, so there is no fitted-parameter or self-citation circularity. However, the theoretical derivation chain is partially circular. The claimed 'reshaping of state visitation distributions' is stipulated in Eq. (8) under an unproved 'local Lipschitz' assumption; the 'acceleration of value learning' is explicitly labeled a hypothesis in Eq. (10) and relies on Eq. (9), which is only posited; and the 'convergence guarantee' in Eq. (12) is a direct consequence of the defining condition lim α(t)=1 plus the assumed convergence of the base algorithm. These steps make the paper's theoretical conclusions equivalent to its assumptions rather than derived from the interpolation rule. The empirical early-training gains also conflate 'acting like the expert early' with 'learning faster': with α(0)=0, early executed actions are essentially expert actions, and no hard-switch baseline or reported T_change isolates the learning acceleration. Still, the empirical results themselves are not fitted to the theory, and the interpolation mechanism is simple and transparent; hence partial circularity rather than full circularity or fabrication.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unproven state-distribution mixture (Eq. 8), a posited high-value-state inequality (Eq. 9), a hypothesis about value-error reduction (Eq. 10), and an assumption that the base actor-critic converges. The schedule parameter T_change is a key free parameter never reported.

free parameters (2)
  • T_change
    Annealing timescale in Eq. (4); determines how quickly the expert is phased out; not reported anywhere in the experiments, so the empirical claim cannot be reproduced or checked for sensitivity.
  • V_threshold
    Threshold defining high-value states in Eq. (9); never specified or estimated, so Eq. (9) cannot be evaluated or falsified.
assumptions (4)
  • domain assumption State visitation under interpolated actions is an alpha-weighted mixture of expert and RL state distributions (Eq. 8)
    Stated under 'local Lipschitz transition dynamics' but no proof or error bound is given; the entire value-acceleration argument rests on it.
  • ad hoc to paper Expert-guided actions visit high-value states more often than the initial RL policy (Eq. 9)
    Posited inequality; depends on the quality of the behavior-cloned expert and is not measured or guaranteed.
  • ad hoc to paper Interpolated data reduces value-estimation error (Eq. 10)
    Explicitly called a hypothesis in Section 4; no derivation from Eq. 8 and Eq. 9 is supplied.
  • domain assumption The base actor-critic algorithm converges to a stationary policy
    Used for Eq. (12); standard for many algorithms but not established for finite-sample TD3 on these benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance." pith.science (2026). https://pith.science/paper/GHBXT2PR

@misc{pith2026250418766,
  author       = {Pith},
  title        = {Pith review of: Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GHBXT2PR}},
  note         = {Machine review of arXiv:2504.18766}
}
abstract

Reinforcement learning (RL) suffers from severe sample inefficiency, especially during early training, requiring extensive environmental interactions to perform competently. Existing methods tend to solve this by incorporating prior knowledge, but introduce significant architectural and implementation complexity. We propose Dynamic Action Interpolation (DAI), a universal yet straightforward framework that interpolates expert and RL actions via a time-varying weight $\alpha(t)$, integrating into any Actor-Critic algorithm with just a few lines of code and without auxiliary networks or additional losses. Our theoretical analysis shows that DAI reshapes state visitation distributions to accelerate value function learning while preserving convergence guarantees. Empirical evaluations across MuJoCo continuous control tasks demonstrate that DAI improves early-stage performance by over 160\% on average and final performance by more than 50\%, with the Humanoid task showing a 4$\times$ improvement early on and a 2$\times$ gain at convergence. These results challenge the assumption that complex architectural modifications are necessary for sample-efficient reinforcement learning.

Figures

Figures reproduced from arXiv: 2504.18766 by the authors.

Figure 1
Figure 1. Performance comparison of TD3-DAI against TD3 and expert policies across four MuJoCo [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages

  1. [1]

    Behavior priors for efficient reinforcement learning

    Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. Journal of Machine Learning Research, 23(221):1–68, 2022

    work page 2022

  2. [2]

    Deep q-learning from demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  3. [3]

    Policy optimization with demonstrations

    Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2469–2478. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/kang18a.html

    work page 2018

  4. [4]

    Overcoming exploration in reinforcement learning with demonstrations, 2018

    Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations, 2018. URL https: //arxiv.org/abs/1709.10089. 8

    arXiv 2018

  5. [5]

    Making efficient use of demonstrations to solve hard exploration problems

    Tom Le Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making efficient use of demonstrations to solve hard exploration problems. arXiv preprint arXiv:1909.01387, 2019

    arXiv 1909

  6. [6]

    Shaping rewards for reinforcement learn- ing with imperfect demonstrations using generative models

    Yuchen Wu, Melissa Mozifian, and Florian Shkurti. Shaping rewards for reinforcement learn- ing with imperfect demonstrations using generative models. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , page 6628–6634. IEEE Press, 2021. doi: 10.1109/ICRA48506.2021.9561333. URL https://doi.org/10.1109/ICRA48506.2021. 9561333

    arXiv 2021

  7. [7]

    Awac: Accelerating online reinforcement learning with offline datasets

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020

    arXiv 2006

  8. [8]

    Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning

    Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 62244–62269. C...

    work page 2023

Show all 41 references
  1. [9]

    Residual reinforcement learning for robot control, 2018

    Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control, 2018. URL https://arxiv.org/abs/1812.03201

    2018 arXiv

  2. [10]

    Stevens, Matthew R

    Xuefeng Liu, Takuma Yoneda, Rick L. Stevens, Matthew R. Walter, and Yuxin Chen. Blending imitation and reinforcement learning for robust policy improvement, 2023. URL https: //arxiv.org/abs/2310.01737

    2023 arXiv

  3. [11]

    Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning

    Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behav- ior cloning regularization for stable offline-to-online reinforcement learning. arXiv preprint arXiv:2210.13846, 2022

    2022 arXiv

  4. [12]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018

    Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018. URL https://arxiv.org/abs/1709.10087

    2018 arXiv

  5. [13]

    Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018

    Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018. URL https: //arxiv...

    2018 arXiv

  6. [14]

    Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble, 2021

    Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble, 2021. URL https://arxiv.org/abs/2107.00591

    2021 arXiv

  7. [15]

    Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning, 2022

    Alex Beeson and Giovanni Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning, 2022. URL https://arxiv.org/abs/2211.11802

    2022 arXiv

  8. [16]

    Online decision transformer

    Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pages 27042–27059. PMLR, 2022

    2022

  9. [17]

    Cog: Connecting new skills to past experience with offline reinforcement learning

    Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020

    2010 arXiv

  10. [18]

    Smart: Self-supervised multi-task pretraining with control transformers

    Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. Smart: Self-supervised multi-task pretraining with control transformers. arXiv preprint arXiv:2301.09816, 2023

    2023 arXiv

  11. [19]

    Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun

    Yuda Song, Yifei Zhou, Ayush Sekhari, J. Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient, 2023. URL https://arxiv.org/abs/2210.06718. 9

    2023 arXiv

  12. [20]

    Residual reinforcement learning from demonstrations, 2021

    Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, and Cordelia Schmid. Residual reinforcement learning from demonstrations, 2021. URL https://arxiv.org/abs/ 2106.08050

    2021 arXiv

  13. [21]

    Residual learning from demonstration: Adapting dmps for contact- rich manipulation

    Todor Davchev, Kevin Sebastian Luck, Michael Burke, Franziska Meier, Stefan Schaal, and Subramanian Ramamoorthy. Residual learning from demonstration: Adapting dmps for contact- rich manipulation. IEEE Robotics and Automation Letters, 7(2):4488–4495, April 2022. ISSN 2377-3774...

    2022

  14. [22]

    How to guide your learner: Imitation learning with active adaptive expert involvement, 2023

    Xu-Hui Liu, Feng Xu, Xinyu Zhang, Tianyuan Liu, Shengyi Jiang, Ruifeng Chen, Zongzhang Zhang, and Yang Yu. How to guide your learner: Imitation learning with active adaptive expert involvement, 2023. URL https://arxiv.org/abs/2303.02073

    2023 arXiv

  15. [23]

    Sheelabhadra Dey, Sumedh Pendurkar, Guni Sharon, and Josiah P. Hanna. A joint imitation- reinforcement learning framework for reduced baseline regret, 2022. URL https://arxiv. org/abs/2209.09446

    2022 arXiv

  16. [24]

    Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, and Razvan Pascanu

    Wojciech Marian Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, and Razvan Pascanu. Mix&match - agent curricula for reinforcement learning, 2018. URL https://arxiv.org/abs/1806.01780

    2018 arXiv

  17. [25]

    Curriculum offline imitating learning

    Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, and Tie-Yan Liu. Curriculum offline imitating learning. In M. Ranzato, A. Beygelz- imer, Y . Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems , vo...

    2021

  18. [26]

    Efficient reductions for imitation learning

    Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, ...

    2010

  19. [27]

    Andrew Bagnell, and Byron Boots

    Wen Sun, J. Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning, 2018. URL https://arxiv.org/abs/1805. 11240

    2018

  20. [28]

    Andrew Bagnell, Zhiwei Steven Wu, Jiantao Jiao, and Kannan Ramchandran

    Gokul Swamy, Nived Rajaraman, Matthew Peng, Sanjiban Choudhury, J. Andrew Bagnell, Zhiwei Steven Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation, 2023. URL https://arxiv.org/abs/2205.15397

    2023 arXiv

  21. [29]

    Andrew Bagnell, and Sanjiban Choudhury

    Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J. Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning, 2024. URL https://arxiv.org/abs/2402.08848

    2024 arXiv

  22. [30]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Confer- ence on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intellig...

    2018

  23. [31]

    Turner, Zoubin Ghahramani, and Sergey Levine

    George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning, 2018. URL https://arxiv.org/abs/1802.10031

    2018 arXiv

  24. [32]

    Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo, 2020. URL https://arxiv.org/abs/2005.12729

    2020 arXiv

  25. [33]

    What matters in on-policy reinforcement learning? a large-scale empirical study, 2020

    Marcin Andrychowicz, Anton Raichuk, Piotr Sta´nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? a large-scale empirical...

    2020 arXiv

  26. [34]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  27. [35]

    A minimalist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  28. [36]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  29. [37]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems , 33: 1179–1191, 2020

    2020

  30. [38]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Rese...

    2018

  31. [39]

    A framework for behavioural cloning

    Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, Intelligent Agents [St. Catherine’s College, Oxford, July 1995], page 103–129, GBR,

    1995

  32. [40]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv.org/abs/1801.01290. 11

    2018 arXiv

  33. [1999]

    ISBN 0198538677

    Oxford University. ISBN 0198538677

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.