Pith. sign in

REVIEW 3 major objections 7 minor 48 references

When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a two-stage state-policy-then-imitation pipeline should be preferred over visual reinforcement learning on hard control tasks, while visual RL remains competitive on easy tasks.

desk verdict A solid, honest empirical comparison of two-stage imitation vs visual RL; the hard/easy split is real but rests on a post-hoc threshold and a single, thinly validated baseline. read the letter →

arxiv 2412.13662 v1 pith:4WM2H6SE submitted 2024-12-18 cs.CV cs.AIcs.LGcs.RO

Tongzhou Mu , Zhaoyang Li , Stanis{l}aw Wiktor Strzelecki , Xiu Yuan , Yunchao Yao , Litian Liang , Hao Su This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords visualreinforcementlearningimitationDAggertwo-stagepolicyasymmetricactorcriticstate-to-visualtransfermanipulationbenchmarkswall-clockefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks when a two-stage 'State-to-Visual DAgger' pipeline—first train a low-dimensional state policy by reinforcement learning, then train a pixel-based policy by online imitation of that teacher—should be preferred over training a policy directly from pixels with visual reinforcement learning. Based on 16 tasks across ManiSkill, DMControl, and Adroit, it claims the two-stage method is markedly better on hard tasks, comparable or slightly worse on easy tasks, and usually more consistent and stable once converged. It also reports that the method does not reliably improve sample efficiency when both approaches can solve a task, but it often cuts wall-clock training time because the expensive image encoder is trained only in the second stage. The paper's practical payoff is a concrete decision rule: use the two-stage pipeline for challenging tasks, for wall-clock savings, or when a state policy already exists; use visual RL when no low-dimensional state is available or when minimal training intervention is wanted.

What carries the argument

The load-bearing mechanism is the two-stage State-to-Visual DAgger procedure itself. Stage 1 trains a state-policy teacher $\pi_S$ with Soft Actor-Critic on low-dimensional observations that may include privileged simulator information; Stage 2 trains the visual policy $\pi_V$ by DAgger—an online imitation algorithm that collects trajectories under the current visual policy, labels them with teacher actions $\pi_S(o^S_t)$, and minimizes the mean squared error $\|\pi_V(o^V_t) - \pi_S(o^S_t)\|^2$—using an off-policy replay buffer and early stopping at an imitation-loss threshold. The comparison baseline is Asymmetric Actor Critic, which lets the critic see the state while the actor sees pixels, so both paradigms use privileged information and the difference is isolated to how the visual policy is learned. This design lets the paper attribute performance differences to the staged decomposition of perception and control rather than to access to privileged data.

What would settle it

Run a modern visual RL method (e.g., DrQ-v2 or an augmented-data SAC) on the paper's hard tasks—PegInsertion, MoveBucket, PickClutterYCB, Adroit Relocate—under identical environment-step and wall-clock budgets, and observe whether it reaches or exceeds the two-stage pipeline's final success rates; if it does on most of these tasks, the paper's central claim that State-to-Visual DAgger is preferred for hard tasks is falsified.

Watch

Extended reading notes

Core claim

The central empirical claim is that State-to-Visual DAgger does not uniformly dominate visual RL, and the pattern of when it wins is governed by task difficulty as measured by state-based RL convergence. On the paper's 'hard' tasks—those where a state-based SAC teacher needs more than four million environment steps—the DAgger-trained visual policy finishes far ahead (for example, PegInsertion and Adroit Relocate move from near-zero success for visual RL to roughly 71% and 49%), while on 'easy' tasks the two methods are similar or visual RL is slightly ahead, as in Walker-Run and Swimmer-6. The paper further claims the two-stage pipeline gives tighter confidence intervals and smoother learning curves, meaning more predictable converged behavior and easier deployment checkpoint selection. It explains the hard-task gap by noting that state-based RL learns the control problem well while visual RL is held back by noisy gradients in visual exploration, and that DAgger's online data collection mitigates covariate shift in the second stage. In terms of efficiency, the paper concludes the method's apparent sample-efficiency edge on hard tasks mainly follows from its higher asymptotic performance, not from needing fewer environment steps, whereas its wall-clock advantage holds broadly because Stage 1 runs on cheap state observations.

Load-bearing premise

The paper's recommendations assume that Asymmetric Actor Critic with SAC represents visual RL as a whole; the paper checks this against DrQ-v2 on only four DMControl tasks using that method's official learning curves rather than re-running it under the same protocol, so a stronger visual RL method on hard tasks could overturn the main conclusion.

Editorial extensions

If this is right

  • On hard visual control tasks, practitioners should expect State-to-Visual DAgger to reach substantially higher final performance than visual RL, with the largest gaps on tasks visual RL essentially fails to solve.
  • On easy tasks, visual RL is sufficient and sometimes slightly better, so the extra stage of State-to-Visual DAgger buys little beyond faster wall-clock training.
  • Wall-clock time is where the two-stage method helps most broadly: even when sample efficiency is comparable, avoiding pixel-based RL in Stage 1 reduces training time.
  • The more consistent converged performance of State-to-Visual DAgger simplifies checkpoint selection and makes reported results less sensitive to random seeds.
  • If low-dimensional state observations are unavailable or cannot be simulated, visual RL is the only viable option among the two.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Implication not tested here: replacing the paper's hard/easy threshold (state RL converging within four million steps) with a visual-RL-specific measure would likely preserve the ordering, because the state teacher is the stronger learner on those tasks.
  • Implication not tested here: the DMControl results suggest that when the underlying dynamics are easy, visual RL can outperform the state teacher itself, indicating a limit case where the two-stage pipeline should not be used.
  • Testable extension: the wall-clock advantage of the two-stage pipeline should grow as image resolution, camera count, or encoder size increase, since those costs are paid only in Stage 2.
  • Testable extension: ablating Stage 2 to offline behavioral cloning would isolate whether DAgger's online correction or the state teacher's control quality causes the hard-task gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper asks when a two-stage State-to-Visual DAgger (S2V) pipeline, in which a state-based SAC teacher is trained and then a visual policy is learned by online imitation, should be preferred over end-to-end visual RL. The authors compare their implementation of S2V against an Asymmetric Actor-Critic SAC baseline on 16 tasks from ManiSkill, DMControl, and Adroit, measuring asymptotic performance, sample efficiency, and wall-clock time. The main reported findings are that S2V markedly outperforms visual RL on tasks classified as hard, performs comparably or slightly worse on easy tasks, is usually more stable at convergence, does not offer a clear sample-efficiency advantage when both methods solve the task, and often saves wall-clock time. They also document implementation choices for S2V and provide practical recommendations.

Significance. If the comparison were fully convincing, the paper would provide useful, actionable guidance for practitioners choosing between two-stage privileged-imitation pipelines and direct visual RL: use S2V for hard tasks and wall-clock-sensitive settings, and visual RL for easy tasks or when state is unavailable. The scope is substantial (16 tasks, three benchmarks), the implementation details and hyperparameter tuning are transparent, the code is released, and the authors state their limitations explicitly. The main reason the significance is conditional is that the entire visual RL side of the comparison is represented by one baseline whose external validation is thin and partly mis-cited, and the easy/hard split that drives the headline is acknowledged to be ad hoc. With additional baseline validation on the hard tasks and robustness checks on the difficulty split, the paper would be a valuable reference.

major comments (3)
  1. [Sec. 3, Appendix D, Fig. 6] The selection of Asymmetric Actor Critic (AAC) as the representative of visual RL is not validated on the tasks that drive the main conclusion. Appendix D compares AAC against DrQ-v2 on only four DMControl tasks using learning curves downloaded from DrQ-v2's official repository rather than curves re-run under the paper's protocol; three of these tasks are classified as easy under the paper's own 4M-step threshold, and the remaining hard task (Humanoid-Walk) appears to be the one where AAC does not surpass DrQ-v2. Moreover, the caption cites DrQ-v2 to Laskin et al. 2020, which is the RAD paper, not the DrQ-v2 paper. Since the headline advantage of S2V is concentrated in hard ManiSkill and Adroit tasks (Fig. 3: normalized score 66 vs 21), a stronger visual RL method such as DrQ-v2, DreamerV3, or TD-MPC2 could plausibly reduce or reverse the reported advantage on those tasks. Please either validate AAC against a strong visual RL baseline on the hard tasks that carry the central claim, or restrict the conclusion to the tested AAC-SAC implementation.
  2. [Sec. 5, Performance Comparison; Sec. 7] The easy/hard task split is defined by whether state-based RL converges within 4M environment steps, and the authors themselves concede that this classification is not rigorous. Because this split is then used to formulate the paper's central recommendation that S2V markedly surpasses visual RL in hard tasks, the threshold is load-bearing rather than cosmetic. The paper should define convergence operationally, justify the 4M cutoff, and show that the hard-task advantage is robust across alternative thresholds or a continuous difficulty measure. Without such robustness analysis, the headline result is partly a function of the chosen threshold, especially since the same split is used to define the comparison groups in Fig. 3 and Fig. 4.
  3. [Sec. 4, Evaluation Metrics; Sec. 5, Performance Comparison] The asymptotic performance metric averages the last 3% of total environment steps, but the paper does not verify that all methods have reached a plateau within the allotted budgets. For example, in Fig. 4 the Adroit Relocate S2V curve appears to still be increasing at the end of training, and several visual RL curves in ManiSkill tasks never leave near-zero performance, so the final-window average may conflate slow progress with asymptotic level. Please report plateau diagnostics or use a fixed-steps-after-convergence criterion; otherwise the asymptotic-performance claims in Sec. 5 are only statements about final-window performance.
minor comments (7)
  1. [Appendix D, Fig. 6] The reference for DrQ-v2 is misattributed to Laskin et al. 2020; DrQ-v2 is by Yarats et al. and is distinct from RAD. Please correct the citation and verify that the downloaded curves are indeed from DrQ-v2.
  2. [Sec. 4, Appendix B, Table 7] Task names are inconsistent: OpenCabinetDrawer and PegInsertionSide in Appendix B and C appear as OpenDrawer and PegInsertion in Table 7 and Fig. 4. Please unify the naming.
  3. [Appendix A] The bullet claim that the first stage handles learning how to observe and the second stage handles learning how to act reverses the roles described in Sec. 3: Stage 1 learns an action policy from state observations, and Stage 2 learns the visual observation mapping.
  4. [Fig. 3] The normalized scores combine success rates and returns divided by 1000 across benchmarks, which implicitly assumes these quantities are comparable. Consider plotting benchmarks separately or using a common standardization.
  5. [Sec. 4, Evaluation Metrics] The asymptotic window of 3% of total environment steps leads to different absolute window lengths for S2V and visual RL if their total step budgets differ; please clarify whether the window is computed per method or per task.
  6. [Table 7] The DMControl Humanoid-Walk visual RL 95% confidence interval includes a negative lower bound for a return that is nonnegative by construction; please report raw seed values or use a bootstrap interval.
  7. [Sec. 7, Appendix F] Appendix F describes real-world applicability through sim-to-real transfer, but this is prospective and not supported by experiments in the paper; consider presenting it explicitly as future outlook rather than as a validated property.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical comparison, not a derivation, and the potentially self-referential hard/easy split is disclosed and not logically forced.

full rationale

This is an empirical study comparing two methods; there is no equation-level derivation whose output is equivalent to its input. The one candidate for circularity—defining 'easy' tasks as those where state-based RL converges within 4M steps and then reporting S2V's advantage on the complementary 'hard' split (Sec. 5, Performance Comparison)—does not reduce by construction: the category is defined by the state-teacher's convergence, not by S2V's advantage over visual RL, and the paper explicitly calls the classification 'not rigorous' (Sec. 5; Sec. 7). The headline claim that S2V is preferable for hard tasks is an empirical observation, not a logical consequence of the definition. The validation of the visual RL baseline against DrQ-v2 (Appendix D, Fig. 6) uses downloaded learning curves and covers only four DMControl tasks; this is a representativeness/external-validity weakness (and the citation 'DrQ-v2 (Laskin et al. 2020)' appears to point to RAD rather than DrQ-v2), but it does not make the comparison circular because the baseline's performance is not fitted to produce the paper's conclusions. Self-citations (ManiSkill2, ManiSkill) are to public benchmarks with externally defined tasks, not load-bearing theoretical claims. The paper's stated limitations (Sec. 7) about the difficulty threshold and checkpoint selection are acknowledged, not hidden. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper does not introduce new theoretical entities or a fitted physical model. Its central claims rest on standard RL/IL algorithms, benchmark environments, and several hand-tuned hyperparameters and a post-hoc difficulty threshold, all of which are disclosed in the paper.

free parameters (6)
  • Task difficulty threshold = 4M environment steps
    Defines easy vs hard tasks in Sec. 5; authors call the categorization not rigorous in Sec. 7.
  • Asymptotic performance window = 3% of total steps
    Used to compute final performance in Sec. 4; window choice affects reported asymptotic numbers.
  • BC loss threshold (ManiSkill) = 0.01
    Early-stopping threshold for Stage 2 imitation; tuned per benchmark, Appendix C Table 1.
  • BC loss threshold (DMControl) = 0.025
    Early-stopping threshold for Stage 2 imitation; tuned per benchmark, Appendix C Table 2.
  • BC loss threshold (Adroit) = 0.1
    Early-stopping threshold for Stage 2 imitation; tuned per benchmark, Appendix C Table 3.
  • Visual RL discount factors = 0.8 (ManiSkill), 0.99 (DMControl), 0.95 (Adroit)
    Tuned per benchmark; Appendix E identifies discount factor as the most critical parameter for visual RL.
assumptions (5)
  • standard math SAC's convergence to a good state-based teacher policy is assumed from prior RL literature.
    Stage 1 relies on SAC (Haarnoja et al. 2018) as a standard algorithm; no proof of optimality is given.
  • domain assumption The environments' low-dimensional state observations include privileged information, and the visual observations contain enough information to imitate the teacher's actions.
    S2V DAgger's Stage 2 requires paired state/visual observations; if visual inputs are insufficient, imitation cannot recover the teacher policy (Sec. 3).
  • domain assumption Asymmetric Actor Critic with SAC is a representative visual RL baseline comparable to state-of-the-art visual RL methods.
    The comparison against DrQ-v2 covers only four DMControl tasks using downloaded curves (Appendix D), yet the paper generalizes findings to visual RL.
  • domain assumption Three random seeds are sufficient to estimate performance and confidence intervals.
    All learning curves and 95% CIs are computed over three seeds (Sec. 5), which is a small sample for stable RL comparisons.
  • ad hoc to paper The 4M-step threshold for task difficulty is meaningful for grouping tasks.
    The threshold is introduced post hoc in Sec. 5 and admitted to be not rigorous in Sec. 7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?." pith.science (2026). https://pith.science/paper/4WM2H6SE

@misc{pith2026241213662,
  author       = {Pith},
  title        = {Pith review of: When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WM2H6SE}},
  note         = {Machine review of arXiv:2412.13662}
}
read the original abstract

Learning policies from high-dimensional visual inputs, such as pixels and point clouds, is crucial in various applications. Visual reinforcement learning is a promising approach that directly trains policies from visual observations, although it faces challenges in sample efficiency and computational costs. This study conducts an empirical comparison of State-to-Visual DAgger, a two-stage framework that initially trains a state policy before adopting online imitation to learn a visual policy, and Visual RL across a diverse set of tasks. We evaluate both methods across 16 tasks from three benchmarks, focusing on their asymptotic performance, sample efficiency, and computational costs. Surprisingly, our findings reveal that State-to-Visual DAgger does not universally outperform Visual RL but shows significant advantages in challenging tasks, offering more consistent performance. In contrast, its benefits in sample efficiency are less pronounced, although it often reduces the overall wall-clock time required for training. Based on our findings, we provide recommendations for practitioners and hope that our results contribute valuable perspectives for future research in visual policy learning.

Figures

Figures reproduced from arXiv: 2412.13662 by the authors.

Figure 1
Figure 1. Overview of Methods. While Visual RL directly trains a visual policy using RL, State-to-Visual DAgger adopts a two-stage process: initially training a teacher policy with low-dimensional state observations, followed by teaching a visual policy via online imitation learning. on this are elaborated in Sec. 3. Our evaluation revealed that State-to-Visual DAgger does not consistently outperform vi￾sual RL, with key find… view at source ↗
Figure 2
Figure 2. Examples of Tasks. We consider control tasks spanning 3 benchmarks. The first row contains tasks from ManiSkill (stationary and mobile robot arm manipulation, dual-arm coordination). The first five tasks in the second row are from DM￾Control (various robot morphologies for locomotion and classical control tasks), and the remaining three tasks in the second row are from Adroit (dexterous hand manipulation). (Lillicra… view at source ↗
Figure 3
Figure 3. Performance Overview. The figure features histograms comparing average performance across different dimensions. On the left, three histograms present performance by benchmark (success rates for ManiSkill and Adroit, and returns for DMControl). In the center, two histograms categorize performance by task difficulty, utilizing normalized scores (success rate for ManiSkill and Adroit, return divided by 1000 for DMContr… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Learning curves against environment steps. Success rate (ManiSkill and Adroit) and return (DMControl) in each task. Tasks are categorized as easy if state-based RL converges within 4M steps, while the others are considered hard. State-to￾Visual DAgger (Stage 2) compari…
Figure 5
Figure 5. Figure 5: Wall-clock Time. Similar to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Asymmetric Actor Critic (Pinto et al. 2017) and DrQ-v2 (Laskin et al. 2020). We selected the Asymmetric Actor Critic as our primary visual RL method due to its capability to utilize low-dimensional state observations. To validate the Asymmetric Actor Crit…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 24 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Akkaya, I.; Andrychowicz, M.; Chociej, M.; Litwin, M.; McGrew, B.; Petron, A.; Paino, A.; Plappert, M.; Powell, G.; Ribas, R.; et al. 2019. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113

    arXiv 2019

  4. [4]

    M.; Baker, B.; Chociej, M.; Jozefowicz, R.; McGrew, B.; Pachocki, J.; Petron, A.; Plappert, M.; Powell, G.; Ray, A.; et al

    Andrychowicz, O. M.; Baker, B.; Chociej, M.; Jozefowicz, R.; McGrew, B.; Pachocki, J.; Petron, A.; Plappert, M.; Powell, G.; Ray, A.; et al. 2020. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1): 3--20

    2020

  5. [5]

    Calli, B.; Singh, A.; Walsman, A.; Srinivasa, S.; Abbeel, P.; and Dollar, A. M. 2015. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), 510--517. IEEE

    work page 2015

  6. [6]

    a henb \

    Chen, D.; Zhou, B.; Koltun, V.; and Kr \"a henb \"u hl, P. 2020. Learning by cheating. In Conference on Robot Learning, 66--75. PMLR

    work page 2020

  7. [7]

    Chen, T.; Tippur, M.; Wu, S.; Kumar, V.; Adelson, E.; and Agrawal, P. 2023. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): eadc9244

    work page 2023

  8. [8]

    Chen, T.; Xu, J.; and Agrawal, P. 2022. A system for general in-hand object re-orientation. In Conference on Robot Learning, 297--307. PMLR

    work page 2022

Show all 48 references
  1. [9]

    Driess, D.; Schubert, I.; Florence, P.; Li, Y.; and Toussaint, M. 2022. Reinforcement learning with neural radiance fields. Advances in Neural Information Processing Systems, 35: 16931--16945

    2022

  2. [10]

    Espeholt, L.; Soyer, H.; Munos, R.; Simonyan, K.; Mnih, V.; Ward, T.; Doron, Y.; Firoiu, V.; Harley, T.; Dunning, I.; et al. 2018. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, 1407--...

    2018

  3. [11]

    S.; Su, H.; and Malik, J

    Gu, J.; Chaplot, D. S.; Su, H.; and Malik, J. 2022. Multi-skill mobile manipulation for object rearrangement. arXiv preprint arXiv:2209.02778

    2022 arXiv

  4. [12]

    Gu, J.; Xiang, F.; Li, X.; Ling, Z.; Liu, X.; Mu, T.; Tang, Y.; Tao, S.; Wei, X.; Yao, Y.; Yuan, X.; Xie, P.; Huang, Z.; Chen, R.; and Su, H. 2023. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. In International Conference on Learning Representations

    2023

  5. [13]

    Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 1861--1870. PMLR

    2018

  6. [14]

    Hafner, D.; Lillicrap, T.; Ba, J.; and Norouzi, M. 2019 a . Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603

    2019 arXiv

  7. [15]

    Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019 b . Learning latent dynamics for planning from pixels. In International conference on machine learning, 2555--2565. PMLR

    2019

  8. [16]

    Hafner, D.; Lillicrap, T.; Fischer, I.; Villegas, R.; Ha, D.; Lee, H.; and Davidson, J. 2019 c . Learning latent dynamics for planning from pixels. In International conference on machine learning, 2555--2565. PMLR

    2019

  9. [17]

    Hafner, D.; Lillicrap, T.; Norouzi, M.; and Ba, J. 2020. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193

    2020 arXiv

  10. [18]

    Hansen, N.; Wang, X.; and Su, H. 2022. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955

    2022 arXiv

  11. [19]

    Hossain, J. 2023. Autonomous Driving with Deep Reinforcement Learning in CARLA Simulation. arXiv preprint arXiv:2306.11217

    2023 arXiv

  12. [20]

    Kalashnikov, D.; Irpan, A.; Pastor, P.; Ibarz, J.; Herzog, A.; Jang, E.; Quillen, D.; Holly, E.; Kalakrishnan, M.; Vanhoucke, V.; et al. 2018. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, 651--673. PMLR

    2018

  13. [21]

    Kaufmann, E.; Bauersfeld, L.; Loquercio, A.; M \"u ller, M.; Koltun, V.; and Scaramuzza, D. 2023. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976): 982--987

    2023

  14. [22]

    Klimov, P. D. A. R. O. 2017. John Schulman, Filip Wolski. Proximal policy optimization algorithms. arXiv, abs/1707.06347

    2017 arXiv

  15. [23]

    Kostrikov, I.; Yarats, D.; and Fergus, R. 2020. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649

    2020 arXiv

  16. [24]

    D.; Gupta, A.; Ionescu, C.; Borgeaud, S.; Reynolds, M.; Zisserman, A.; and Mnih, V

    Kulkarni, T. D.; Gupta, A.; Ionescu, C.; Borgeaud, S.; Reynolds, M.; Zisserman, A.; and Mnih, V. 2019. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32

    2019

  17. [25]

    ???? Rma: Rapid motor adaptation for legged robots

    Kumar, A.; Fu, Z.; Pathak, D.; and Malik, J. ???? Rma: Rapid motor adaptation for legged robots. arXiv 2021. arXiv preprint arXiv:2107.04034

    2021 arXiv

  18. [26]

    Laskin, M.; Lee, K.; Stooke, A.; Pinto, L.; Abbeel, P.; and Srinivas, A. 2020. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19884--19895

    2020

  19. [27]

    Laskin, M.; Srinivas, A.; and Abbeel, P. 2020. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, 5639--5650. PMLR

    2020

  20. [28]

    Lee, J.; Hwangbo, J.; Wellhausen, L.; Koltun, V.; and Hutter, M. 2020. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47): eabc5986

    2020

  21. [29]

    P.; Hunt, J

    Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

    2015 arXiv

  22. [30]

    Loquercio, A.; Kaufmann, E.; Ranftl, R.; M \"u ller, M.; Koltun, V.; and Scaramuzza, D. 2021. Learning high-speed flight in the wild. Science Robotics, 6(59): eabg5810

    2021

  23. [31]

    B.; Chen, T.; Paigwar, K.; Fu, X.; Kim, D.; Kim, S.; and Agrawal, P

    Margolis, G. B.; Chen, T.; Paigwar, K.; Fu, X.; Kim, D.; Kim, S.; and Agrawal, P. 2021. Learning to jump from pixels. arXiv preprint arXiv:2110.15344

    2021 arXiv

  24. [32]

    Miki, T.; Lee, J.; Hwangbo, J.; Wellhausen, L.; Koltun, V.; and Hutter, M. 2022. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62): eabk2822

    2022

  25. [33]

    A.; Veness, J.; Bellemare, M

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529--533

    2015

  26. [34]

    Mu, T.; Ling, Z.; Xiang, F.; Yang, D.; Li, X.; Tao, S.; Huang, Z.; Jia, Z.; and Su, H. 2021. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107.14483

    2021 arXiv

  27. [35]

    V.; Pong, V.; Dalal, M.; Bahl, S.; Lin, S.; and Levine, S

    Nair, A. V.; Pong, V.; Dalal, M.; Bahl, S.; Lin, S.; and Levine, S. 2018. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31

    2018

  28. [36]

    Parisi, S.; Rajeswaran, A.; Purushwalkam, S.; and Gupta, A. 2022. The unsurprising effectiveness of pre-trained vision models for control. In international conference on machine learning, 17359--17371. PMLR

    2022

  29. [37]

    Pinto, L.; Andrychowicz, M.; Welinder, P.; Zaremba, W.; and Abbeel, P. 2017. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542

    2017 arXiv

  30. [38]

    Rajeswaran, A.; Kumar, V.; Gupta, A.; Vezzani, G.; Schulman, J.; Todorov, E.; and Levine, S. 2017. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087

    2017 arXiv

  31. [39]

    Ross, S.; Gordon, G.; and Bagnell, D. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, 627--635. JMLR Workshop and Conference Proceedings

    2011

  32. [40]

    Shah, R.; and Kumar, V. 2021. Rrl: Resnet as representation for reinforcement learning. arXiv preprint arXiv:2107.03380

    2021 arXiv

  33. [41]

    Shang, W.; Wang, X.; Srinivas, A.; Rajeswaran, A.; Gao, Y.; Abbeel, P.; and Laskin, M. 2021. Reinforcement learning with latent flow. Advances in Neural Information Processing Systems, 34: 22171--22183

    2021

  34. [42]

    Silver, D.; Wierstra, A. G. I. A. D.; and Riedmiller, M. 2013. Playing atari with deep reinforcement learning. DeepMind Lab. arXiv, 1312

    2013

  35. [43]

    Tassa, Y.; Doron, Y.; Muldal, A.; Erez, T.; and Li, Y. 2018. Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint arXiv:1801.00690, 1

    2018 arXiv

  36. [44]

    Xu, Y.; Wan, W.; Zhang, J.; Liu, H.; Shan, Z.; Shen, H.; Wang, R.; Geng, H.; Weng, Y.; Chen, J.; et al. 2023. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Co...

    2023

  37. [45]

    Yarats, D.; Kostrikov, I.; and Fergus, R. 2020. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations

    2020

  38. [46]

    Ze, Y.; Liu, Y.; Shi, R.; Qin, J.; Yuan, Z.; Wang, J.; and Xu, H. 2024. H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation. Advances in Neural Information Processing Systems, 36

    2024

  39. [47]

    a henb \

    Zhou, B.; Kr \"a henb \"u hl, P.; and Koltun, V. 2019. Does computer vision matter for action? Science Robotics, 4(30): eaaw6661

    2019

  40. [48]

    Zhuang, Z.; Fu, Z.; Wang, J.; Atkeson, C.; Schwertfeger, S.; Finn, C.; and Zhao, H. 2023. Robot parkour learning. arXiv preprint arXiv:2309.05665

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.