Pith. sign in

REVIEW 22 cited by

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.11956 v1 pith:2ET5WOF3 submitted 2019-10-25 cs.LG cs.ROstat.ML

Abhishek Gupta , Vikash Kumar , Corey Lynch , Sergey Levine , Karol Hausman This is my paper · ORCID
classification cs.LGcs.ROstat.ML
keywords learninglong-horizontasksimitationpolicyreinforcementmethodpolicies
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present relay policy learning, a method for imitation and reinforcement learning that can solve multi-stage, long-horizon robotic tasks. This general and universally-applicable, two-phase approach consists of an imitation learning stage that produces goal-conditioned hierarchical policies, and a reinforcement learning phase that finetunes these policies for task performance. Our method, while not necessarily perfect at imitation learning, is very amenable to further improvement via environment interaction, allowing it to scale to challenging long-horizon tasks. We simplify the long-horizon policy learning problem by using a novel data-relabeling algorithm for learning goal-conditioned hierarchical policies, where the low-level only acts for a fixed number of steps, regardless of the goal achieved. While we rely on demonstration data to bootstrap policy learning, we do not assume access to demonstrations of every specific tasks that is being solved, and instead leverage unstructured and unsegmented demonstrations of semantically meaningful behaviors that are not only less burdensome to provide, but also can greatly facilitate further improvement using reinforcement learning. We demonstrate the effectiveness of our method on a number of multi-stage, long-horizon manipulation tasks in a challenging kitchen simulation environment. Videos are available at https://relay-policy-learning.github.io/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills

    cs.RO 2026-08 conditional novelty 6.0 of 10

    A taxonomy of robot learning on a weights-versus-skills axis, with a five-rung self-improvement ladder whose top cell (feedback plus memory plus search) holds only a few recent systems.

  2. Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Closed-loop agentic probing plus minimality/sufficiency masking recovers compact task-sufficient world-model latents that improve sample-efficient policy learning and cross-task generalization.

  3. DASIP: Dynamic Test-Time Compute Scaling for Robot Control with Stochastic Interpolant Policies

    cs.RO 2025-11 reject novelty 6.0 of 10

    A difficulty classifier adaptively selects step count, solver, and ODE/SDE mode for stochastic-interpolant robot policies, reporting 2.6–4.4x compute savings with roughly unchanged success rates.

  4. Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach

    stat.ML 2025-10 conditional novelty 6.0 of 10

    A two-stage RL framework learns value-function envelopes from offline data and uses them to shape online exploration, yielding regret bounds that improve as offline data grows.

  5. D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning

    cs.RO 2025-08 conditional novelty 6.0 of 10

    D3P learns to adaptively allocate denoising steps per robot action, matching fixed-step diffusion policy performance at 2.2x lower inference cost.

  6. Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs

    cs.AI 2025-07 reject novelty 6.0 of 10

    A variational option-critic algorithm with latent option embeddings and an implicit chain-of-thought cold-start is presented; the central optimality-preservation proof has a gap and some reported benchmark wins are in...

  7. FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making

    cs.RO 2025-07 conditional novelty 6.0 of 10

    FOUNDER maps foundation-model embeddings of text or video prompts into world-model goal states and rewards policies by predicted temporal distance to those goals, improving reward-free multi-task offline control.

  8. Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A decentralized diffusion policy for two robot arms that aligns a learned consensus embedding across agents and uses theory-of-mind prediction to keep that embedding informative.

  9. Evolution and The Knightian Blindspot of Machine Learning

    cs.AI 2025-01 conditional novelty 6.0 of 10

    ML's formalisms, particularly RL's, exclude Knightian uncertainty, and evolution's diversify-and-filter mechanisms point toward a direct remedy.

  10. GenPlan: Generative Sequence Models as Adaptive Planners

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A discrete-flow sequence model with energy and entropy guidance enables adaptive planning that beats LEAP and Decision Transformer in BabyAI adaptive benchmarks.

  11. Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Warm-start RL (WSRL) fine-tunes offline-pretrained RL agents online with no offline data retention, using 5,000 warm-up rollouts from the frozen pre-trained policy followed by standard high-UTD SAC.

  12. CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction

    cs.RO 2024-12 conditional novelty 6.0 of 10

    A coarse-to-fine autoregressive policy with multi-scale action tokenization matches or beats diffusion policies on robot manipulation benchmarks at roughly 10x lower inference cost.

  13. VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A two-stage robot manipulation policy that pre-trains a video diffusion transformer on Open X-Embodiment videos and then adapts it with a layer-wise self-attention adapter for action prediction, outperforming GR-1 on CALVIN.

  14. Relative Value Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A critic that learns antisymmetric value differences ∆(s_i,s_j)=V(s_i)−V(s_j) has a provably contracting Bellman operator and an unbiased advantage estimator, and PPO with this critic matches standard PPO on Atari.

  15. Dual-Process Atomic Skill Learning: Decoupling Semantic Reasoning and Real-Time Control

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Asynchronous dual-frequency hierarchical imitation learning with VQ skills and training-only latent diffusion improves compositional language-conditioned robot control and reduces skill codebook collapse.

  16. VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizers

    cs.RO 2025-07 conditional novelty 5.0 of 10

    A convolutional residual VQ-VAE action tokenizer trained on over 100x more data than prior work improves OpenVLA success rates and inference speed on several manipulation tasks.

  17. Generative AI in Embodied Systems: System-Level Analysis of Performance, Efficiency and Scalability

    cs.RO 2025-04 conditional novelty 5.0 of 10

    A workload characterization of 14 embodied LLM agent systems showing that planning and communication dominate latency, communication is often redundant, and multi-agent systems scale poorly.

  18. Improving Vision-Language-Action Model with Online Reinforcement Learning

    cs.RO 2025-01 conditional novelty 5.0 of 10

    Alternating online RL on a frozen vision-language backbone with supervised fine-tuning on collected successes improves a VLA policy's task success and generalization.

  19. GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents

    cs.AI 2024-12 conditional novelty 5.0 of 10

    A weakly supervised latent-variable agent improves multimodal instruction following by combining VAE self-imitating on unlabeled data with a likelihood-based alignment of labeled and video latents.

  20. Steering Robots with Inference-Time Interactions

    cs.RO 2025-06 conditional novelty 4.0 of 10

    Frozen imitation policies can be steered at inference time via user interactions, with a diffusion-sampling method and a constraint-enforcing framework that provides formal task guarantees.

  21. Robustifying Long-term Human-Robot Collaboration through a Multimodal and Hierarchical Framework

    cs.RO 2024-11 conditional novelty 4.0 of 10

    A vision-plus-speech, hierarchy-based robot assistant reduces assembly time by about 16% and reaches a 92% success rate in a 10-person long-term toy-car co-assembly study, though the reported gains lack statistical tests.

  22. Data Pyramid for Embodied Manipulation: A Survey

    cs.RO 2026-07 conditional novelty 3.0 of 10

    Embodied training data form a five-layer pyramid—real-robot, UMI, ego/exo, simulation, general V–L—ordered by the trade-off between scale and robot alignment, and model capabilities track how those layers are mixed.

Pith tools