pith. sign in

arxiv: 2310.02635 · v5 · submitted 2023-10-04 · 💻 cs.RO · cs.AI· cs.LG

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own

Weirui Ye , Yunsheng Zhang , Haoyang Weng , Xianfan Gu , Shengjie Wang , Tong Zhang , Mengchen Wang , Pieter Abbeel
show 1 more author
Yang Gao
This is my paper

Pith reviewed 2026-05-24 06:38 UTC · model grok-4.3

classification 💻 cs.RO cs.AIcs.LG
keywords reinforcement learningfoundation modelsrobotic manipulationactor-criticautomatic rewardembodied agentssample efficiencydexterous tasks
0
0 comments X

The pith

Foundation models can supply automatic rewards and guidance that let reinforcement learning agents master dexterous tasks in under an hour of real time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Reinforcement Learning with Foundation Priors to address two core barriers in applying RL to real robots: the need for millions of environment interactions and the manual design of reward functions. It proposes the RLFP framework that pulls policy, value, and success-reward signals directly from foundation models and feeds them into an actor-critic learner. The resulting Foundation-guided Actor-Critic algorithm produces automatic rewards and exploration guidance without task-specific fine-tuning of the models. A sympathetic reader would care because this combination promises to let embodied agents learn new manipulation skills on physical hardware with far less human engineering and far fewer trials than standard RL.

Core claim

The Foundation-guided Actor-Critic algorithm inserts priors from foundation models for policy, value, and success reward into the actor-critic loop, yielding automatic reward functions that enable embodied agents to explore more efficiently. Across five dexterous tasks on real robots the method reaches an average success rate of 86 percent after one hour of real-time learning; across eight Meta-world tasks it reaches 100 percent success in seven of the eight under less than 100k frames while outperforming baselines that use manually designed rewards trained for 1M frames. The framework is stated to be agnostic to the specific form of the foundation models and robust to noisy priors.

What carries the argument

The Foundation-guided Actor-Critic (FAC) algorithm, which inserts guidance and feedback from policy, value, and success-reward foundation models directly into the actor-critic training loop to generate automatic rewards.

If this is right

  • Robotic manipulation tasks become solvable with roughly one hour of real-time interaction instead of millions of samples.
  • Reward engineering effort drops to near zero because success-reward signals come from the foundation model rather than manual design.
  • The same FAC structure works across both real hardware and simulation without changes to the foundation-model components.
  • Performance remains high even when the foundation-model priors contain noise, provided they are used as guidance inside the actor-critic update.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same prior-injection pattern could be tested on non-manipulation RL problems such as navigation or locomotion if comparable foundation models exist for those domains.
  • Combining FAC with larger or more recent foundation models might further reduce the required interaction count, an extension the paper does not measure.
  • If the priors prove stable across many tasks, the approach could support continual learning on a single robot without repeated manual reward redesign.

Load-bearing premise

Foundation models trained on non-robotic data continue to supply useful and stable priors when placed inside the actor-critic loop without any task-specific fine-tuning or manual re-weighting.

What would settle it

Running the FAC algorithm on a new dexterous manipulation task and observing success rates no higher than those of standard RL with hand-crafted rewards after the same number of real-robot trials would falsify the central claim.

Figures

Figures reproduced from arXiv: 2310.02635 by Haoyang Weng, Mengchen Wang, Pieter Abbeel, Shengjie Wang, Tong Zhang, Weirui Ye, Xianfan Gu, Yang Gao, Yunsheng Zhang.

Figure 1
Figure 1. Figure 1: An example of how human solves tasks under the policy, value, and success-reward prior knowl [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overview of Foundation-guided Actor-Critic. In FAC, rewards are derived from foundation success rewards and value shaping. Besides policy gradient updates, the actor is trained using prior policy regularization and success trajectory imitation. Guided by Reward-shaping from Value Prior. Noisy policy prior can mislead agents to undesirable states, so we propose using the value model MV to guide ex￾plora… view at source ↗
Figure 3
Figure 3. Figure 3: Five tasks on real robots, demonstrating the efficiency and accuracy of FAC in real. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: During training, the agent progres￾sively favors actions from the actor, reducing reliance on the prior policy. Efficient and Safe Exploration on Real Robots. To achieve higher sample efficiency on real robots, we choose high update-to-data (UTD) ratios with layer nor￾malization in all MLP layers [64]. To achieve safer explo￾ration, we introduce two key modifications. Firstly, we warm up the learning by ta… view at source ↗
Figure 5
Figure 5. Figure 5: Prior policy attempts to open the door without a [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Success Rate Curves for the 8 Tasks in Meta-World. Our method consistently achieves 100% success rates across all tasks, under the constrained performance of the policy prior model. It significantly outperforms the baselines with manual-designed rewards. 0.25 0.50 0.75 1.00 Frames 1e6 0.0 0.2 0.4 0.6 0.8 1.0 Success rate Bin Picking 0.25 0.50 0.75 1.00 Frames 1e5 Button Press Topdown 0.25 0.50 0.75 1.00 Fr… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison to the DrQ-v2 (warmup), which warm up the actor by 10 collected success demos from [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Results of Ablation Study (a) Three Foundation Priors and the Success Buffer. (b) The Quality of Policy Prior. (c) The Quality of the success-reward Prior. bin-picking and door-unlocking, which take more frames to reach 100% success. Adding noise further reduces performance, but even with 50% noise, FAC still achieves 100% success in many environments. We also tested robustness with systematically wrong po… view at source ↗
Figure 9
Figure 9. Figure 9: Training loss curves of bin-picking and door-open. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example of success identification by GPT-4V in task Watering Plants. Given the question. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Initial observation image of the task Un [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The generated videos from the diffusion model Seer given the initial images as well as task descrip [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Here ‘*’ in DrQ-v2, ALIX, and TACO means only the 0-1 success reward is provided from the [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
read the original abstract

Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks. However, it is challenging to apply the RL algorithms directly in the real world. For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. For another, it is necessary to make heavy engineering efforts to design reward functions manually. To address these issues, we leverage foundation models in this paper. We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks. Visualizations and code are available at https://yewr.github.io/rlfp.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper proposes the Reinforcement Learning with Foundation Priors (RLFP) framework and the Foundation-guided Actor-Critic (FAC) algorithm, which incorporates outputs from policy, value, and success-reward foundation models into actor-critic RL updates. This is claimed to yield sample-efficient learning, minimal manual reward engineering, and robustness to noisy priors without task-specific fine-tuning. Empirical results include an 86% average success rate across 5 real-robot dexterous tasks after one hour of interaction and 100% success in 7/8 Meta-world tasks under 100k frames, outperforming baselines that use manual rewards over 1M frames.

Significance. If the integration of non-robotic foundation-model priors can be shown to be stable and task-agnostic as asserted, the work would meaningfully lower the barrier to real-world robotic RL by replacing hand-crafted rewards and improving exploration efficiency. The real-robot results are practically relevant; however, the absence of fusion details and controls leaves the attribution of gains to the claimed mechanism unverified.

major comments (3)
  1. [Methods (FAC algorithm)] Methods (FAC algorithm): the description of how foundation-model outputs (especially the success-reward signal) are fused into the actor-critic loss lacks any specification of scaling coefficients, normalization, or noise-filtering thresholds. This detail is load-bearing for the abstract's claim of being 'agnostic to foundation model forms and robust to noisy priors' and for the 'minimal engineering' guarantee.
  2. [Experiments] Experiments: no ablation studies isolate the contribution of the policy prior, value prior, or success-reward prior, nor test sensitivity to any weighting or prompting choices. Without these, it is impossible to confirm that the reported 86% and 100% success rates do not rely on per-task engineering that would contradict the central 'minimal and effective reward engineering' claim.
  3. [Results] Results: the headline success rates (86% real-robot average, 100% on 7/8 Meta-world tasks) are given without error bars, number of random seeds, or statistical significance tests against the manual-reward baselines. This prevents assessment of whether the gains are reliable or attributable to the foundation priors rather than implementation details.
minor comments (1)
  1. [Abstract] The abstract states that FAC 'outperforms baseline methods with manual-designed rewards in 1M frames' but does not name the baselines or report the exact frame count used by FAC for the comparison.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our RLFP framework and FAC algorithm. We address each major comment below and commit to revisions that strengthen the manuscript's clarity, rigor, and support for its central claims.

read point-by-point responses

  1. Referee: the description of how foundation-model outputs (especially the success-reward signal) are fused into the actor-critic loss lacks any specification of scaling coefficients, normalization, or noise-filtering thresholds. This detail is load-bearing for the abstract's claim of being 'agnostic to foundation model forms and robust to noisy priors' and for the 'minimal engineering' guarantee.

    Authors: We agree that the current manuscript provides only a high-level overview of the fusion process without explicit implementation details. In the revised version, we will expand the Methods section (specifically the FAC algorithm description) to include the exact loss formulation, scaling coefficients (lambda values for policy, value, and reward priors), normalization steps applied to each foundation model output, and any noise-filtering thresholds used. These additions will directly support the robustness and minimal-engineering claims. revision: yes

  2. Referee: no ablation studies isolate the contribution of the policy prior, value prior, or success-reward prior, nor test sensitivity to any weighting or prompting choices. Without these, it is impossible to confirm that the reported 86% and 100% success rates do not rely on per-task engineering that would contradict the central 'minimal and effective reward engineering' claim.

    Authors: The manuscript does not contain ablation studies isolating each prior or testing sensitivity to weights and prompts. We acknowledge this gap weakens attribution of results to the framework. We will add a new subsection with ablations (removing one prior at a time, varying weights, and testing alternative prompts) on both simulation and real-robot tasks to demonstrate that performance does not depend on per-task tuning. revision: yes

  3. Referee: the headline success rates (86% real-robot average, 100% on 7/8 Meta-world tasks) are given without error bars, number of random seeds, or statistical significance tests against the manual-reward baselines. This prevents assessment of whether the gains are reliable or attributable to the foundation priors rather than implementation details.

    Authors: We agree that statistical reporting is essential. The experiments used multiple seeds (5 for Meta-world, 3 for real robots), but these details and variance measures were omitted from the main text. In revision, we will update the Results section with error bars (standard deviation), explicit seed counts, and statistical significance tests (paired t-tests) against the manual-reward baselines to allow proper evaluation of reliability. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical validation stands independently

full rationale

The paper introduces the RLFP framework and FAC algorithm to leverage foundation model priors for RL in robotics. The central claims are supported by reported success rates on real dexterous tasks (86% average) and Meta-world tasks (100% in 7/8 under 100k frames). No equations or derivations are presented that reduce by construction to inputs or self-citations. The method is described as agnostic and robust, but validation relies on external performance metrics rather than internal self-definition or fitted predictions. This is a standard empirical contribution without load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the transferability of general-purpose foundation models to robotic control signals without additional adaptation; this is a domain assumption rather than a derived result.

axioms (1)
  • domain assumption Foundation models trained outside robotics supply useful policy, value, and success signals for manipulation tasks
    Invoked when the abstract states that the method is agnostic to foundation-model forms and robust to noisy priors.

pith-pipeline@v0.9.0 · 5866 in / 1296 out tokens · 29761 ms · 2026-05-24T06:38:26.469601+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations

    cs.RO 2026-05 unverdicted novelty 6.0

    A framework learns invariant symbolic reward functions from few demonstrations that generalize zero-shot to variations in robotic manipulation tasks.

  2. SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning

    cs.LG 2026-02 unverdicted novelty 5.0

    SLOPE improves MBRL in sparse reward settings by using optimistic distributional regression to build informative potential landscapes that provide better exploration gradients, outperforming baselines across 30+ tasks...

  3. Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agent

    cs.AI 2026-02 unverdicted novelty 2.0

    A survey provides a task-based formalization of meta-learning and meta-RL while chronicling algorithms that lead to DeepMind's Adaptive Agent.

Reference graph

Works this paper leans on

70 extracted references · 70 canonical work pages · cited by 3 Pith papers · 20 internal anchors

  1. [1]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    work page 2020

  2. [2]

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y . Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476–25488, 2021

    work page 2021

  3. [3]

    Arulkumaran, A

    K. Arulkumaran, A. Cully, and J. Togelius. Alphastar: An evolutionary computation perspec- tive. In Proceedings of the genetic and evolutionary computation conference companion, pages 314–315, 2019

    work page 2019

  4. [5]

    A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics , (5): 834–846, 1983

    work page 1983

  5. [6]

    Mahadevan and J

    S. Mahadevan and J. Connell. Automatic programming of behavior-based robots using rein- forcement learning. Artificial intelligence, 55(2-3):311–365, 1992

    work page 1992

  6. [7]

    P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning, pages 2226–2240. PMLR, 2023

    work page 2023

  7. [8]

    J. Yang, M. S. Mark, B. Vu, A. Sharma, J. Bohg, and C. Finn. Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning. arXiv preprint arXiv:2310.15145, 2023

    work page arXiv 2023

  8. [9]

    Haldar, J

    S. Haldar, J. Pari, A. Rai, and L. Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497, 2023

    work page arXiv 2023

  9. [10]

    Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. arXiv preprint arXiv:1712.01815, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  10. [11]

    Yarats, R

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021

    work page arXiv 2021

  11. [13]

    TD-MPC2: Scalable, Robust World Models for Continuous Control

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  12. [14]

    A. N. Meltzoff. Infant imitation after a 1-week delay: long-term memory for novel acts and multiple stimuli. Developmental psychology, 24(4):470, 1988

    work page 1988

  13. [15]

    A. N. Meltzoff. Understanding the intentions of others: re-enactment of intended acts by 18-month-old children. Developmental psychology, 31(5):838, 1995. 9

    work page 1995

  14. [16]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017

    work page 2017

  15. [17]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  16. [18]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    work page 1901

  17. [19]

    GPT-4 Technical Report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi:10.48550/arXiv.2303. 08774. URL https://doi.org/10.48550/arXiv.2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303 2023

  18. [20]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  19. [21]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021

    work page 2021

  20. [22]

    Hierarchical Text-Conditional Image Generation with CLIP Latents

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [23]

    Segment Anything

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, et al. Segment anything.arXiv preprint arXiv:2304.02643, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [24]

    Y . J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V . Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  23. [25]

    Y . Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation.arXiv preprint arXiv:2302.00111, 2023

    work page arXiv 2023

  24. [26]

    X. Gu, C. Wen, J. Song, and Y . Gao. Seer: Language instructed video prediction with latent diffusion models. arXiv preprint arXiv:2303.14897, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  25. [27]

    RT-1: Robotics Transformer for Real-World Control at Scale

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  26. [28]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [29]

    S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y . Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [30]

    T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302.11550, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [31]

    Chebotar, K

    Y . Chebotar, K. Hausman, F. Xia, Y . Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog, K. Pertsch, K. Gopalakrishnan, et al. Q-transformer: Scalable offline reinforcement learning via autore- gressive q-functions. In 7th Annual Conference on Robot Learning, 2023. 10

    work page 2023

  30. [32]

    Di Palo, A

    N. Di Palo, A. Byravan, L. Hasenclever, M. Wulfmeier, N. Heess, and M. Riedmiller. To- wards a unified agent with foundation models. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023

    work page 2023

  31. [33]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  32. [34]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [35]

    PaLM-E: An Embodied Multimodal Language Model

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  34. [36]

    J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658, 2023

    work page arXiv 2023

  35. [37]

    Singh, V

    I. Singh, V . Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 11523– 11530. IEEE, 2023

    work page 2023

  36. [38]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, pages 894–906. PMLR, 2022

    work page 2022

  37. [39]

    Y . Hu, F. Lin, T. Zhang, L. Yi, and Y . Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311.17842, 2023

    work page arXiv 2023

  38. [40]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    work page 2023

  39. [41]

    VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  40. [42]

    J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, and T. Salimans. Imagen video: High definition video generation with diffusion models, 2022

    work page 2022

  41. [43]

    Karamcheti, S

    S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics. arXiv preprint arXiv:2302.12766, 2023

    work page arXiv 2023

  42. [44]

    Shah and V

    R. Shah and V . Kumar. Rrl: Resnet as representation for reinforcement learning.arXiv preprint arXiv:2107.03380, 2021

    work page arXiv 2021

  43. [45]

    Majumdar, K

    A. Majumdar, K. Yadav, S. Arnaud, Y . J. Ma, C. Chen, S. Silwal, A. Jain, V .-P. Berges, P. Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18240, 2023

    work page arXiv 2023

  44. [46]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  45. [47]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023. 11

    work page 2023

  46. [49]

    Jiang, A

    Y . Jiang, A. Gupta, Z. Zhang, G. Wang, Y . Dou, Y . Chen, L. Fei-Fei, A. Anandkumar, Y . Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094, 2022

    work page arXiv 2022

  47. [50]

    L. Fan, G. Wang, Y . Jiang, A. Mandlekar, Y . Yang, H. Zhu, A. Tang, D.-A. Huang, Y . Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08853, 2022

    work page arXiv 2022

  48. [51]

    S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303–1315. PMLR, 2022

    work page 2022

  49. [52]

    Mahmoudieh, D

    P. Mahmoudieh, D. Pathak, and T. Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning , pages 14743–14752. PMLR, 2022

    work page 2022

  50. [53]

    Eteke, D

    C. Eteke, D. Kebüde, and B. Akgün. Reward learning from very few demonstrations. IEEE Transactions on Robotics, 37(3):893–904, 2020

    work page 2020

  51. [54]

    B. Wu, F. Xu, Z. He, A. Gupta, and P. K. Allen. Squirl: Robust and efficient learning from video demonstration of long-horizon robotic manipulation tasks. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9720–9727. IEEE, 2020

    work page 2020

  52. [55]

    Xiong, R

    H. Xiong, R. Mendonca, K. Shaw, and D. Pathak. Adaptive mobile manipulation for articulated objects in the open world. arXiv preprint arXiv:2401.14403, 2024

    work page arXiv 2024

  53. [56]

    Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

    I. Popov, N. Heess, T. Lillicrap, R. Hafner, G. Barth-Maron, M. Vecerik, T. Lampe, Y . Tassa, T. Erez, and M. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipu- lation. arXiv preprint arXiv:1704.03073, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  54. [57]

    W. Ye, Y . Zhang, P. Abbeel, and Y . Gao. Become a proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations, 2022

    work page 2022

  55. [58]

    S. Wang, S. Liu, W. Ye, J. You, and Y . Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprint arXiv:2403.00564, 2024

    work page arXiv 2024

  56. [59]

    Mastering Diverse Domains through World Models

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  57. [60]

    Haldar, V

    S. Haldar, V . Mathur, D. Yarats, and L. Pinto. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32–43. PMLR, 2023

    work page 2023

  58. [61]

    Lancaster, N

    P. Lancaster, N. Hansen, A. Rajeswaran, and V . Kumar. Modem-v2: Visuo-motor world models for real-world robot manipulation. arXiv preprint arXiv:2309.14236, 2023

    work page arXiv 2023

  59. [62]

    A. Y . Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer, 1999

    work page 1999

  60. [63]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    work page 2018

  61. [64]

    P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023. 12

    work page arXiv 2023

  62. [65]

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020

    work page 2020

  63. [66]

    D’Oro, M

    P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022

    work page 2022

  64. [67]

    Dorigo and M

    M. Dorigo and M. Colombetti. Robot shaping: an experiment in behavior engineering . MIT press, 1998

    work page 1998

  65. [68]

    M. J. Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, pages 181–189. Elsevier, 1994

    work page 1994

  66. [69]

    Randløv and P

    J. Randløv and P. Alstrøm. Learning to drive a bicycle using reinforcement learning and shap- ing. In ICML, volume 98, pages 463–471, 1998

    work page 1998

  67. [70]

    Cetin, P

    E. Cetin, P. J. Ball, S. Roberts, and O. Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. arXiv preprint arXiv:2207.00986, 2022

    work page arXiv 2022

  68. [71]

    Zheng, X

    R. Zheng, X. Wang, Y . Sun, S. Ma, J. Zhao, H. Xu, H. Daumé III, and F. Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning.arXiv preprint arXiv:2306.13229, 2023

    work page arXiv 2023

  69. [72]

    Cheng, A

    R. Cheng, A. Verma, G. Orosz, S. Chaudhuri, Y . Yue, and J. Burdick. Control regularization for reduced variance reinforcement learning. In International Conference on Machine Learning, pages 1141–1150. PMLR, 2019. 13 A Appendix Appendix Table of Contents • Appendix A.1: Reward Shaping in FAC • Appendix A.2: Experimental Details of FAC • Appendix A.3: Det...

    work page 2019

  70. [73]

    move_to x1 y1 z1

    The total frames of the 8 tasks are 100k, except for the task bin-picking, which is set to 1M. Notably, we set the same camera view of all the tasks for consistency. On real robots, we set the 14 0.0 0.5 1.0 1.5 2.0 1e5 250 200 150 100 50 0 50 Success rate actor_loss bin-picking-v2 0.0 0.5 1.0 1.5 2.0 1e5 0 100 200 300 400Success rate critic_loss bin-pick...

    work page