Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

The paper claims that constrained model-based state sampling, not behavior cloning, is the key to training universal contact-rich manipulation policies.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Guiding goal-conditioned reinforcement learning with samples from a constrained feasible-state manifold lets a simulated double-sphere and a Panda-arm policy succeed far more often than RL with random resets.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection The conceptual move is real and the experiments are mostly honest, but the 'universal' claim is tied to a solver-generated dataset that is both the training reset distribution and the evaluation distribution; the technique survives, but the headline numbers need to be reframed. the 3 major comments →

arxiv 2602.08557 v2 pith:2KZQNQPF submitted 2026-02-09 cs.RO

Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation

Marc Toussaint , Cornelius V. Braun , Armand Jordana , Sayantan Auddy , Eckart Cobo-Briesewitz , Denis Shcherba , Tilman Burghoff , Justin Carpentier This is my paper
classification cs.RO
keywords goal-conditioned RLconstrained samplingcontact-rich manipulationuniversal manipulation policystate visitation biasbehavior cloningaugmented Lagrangiannon-prehensile manipulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that injecting first-principles physics constraints into reinforcement learning, by shaping the start and goal state distribution, is what makes universal contact-rich manipulation policies learnable, not the usual behavior-cloning imitation of demonstrated actions. To do this, it formalizes a Constrained Goal-conditioned MDP in which feasible start and goal states are defined by differentiable collision, contact, and force equilibrium constraints, and sampled with a nonlinear constrained solver. The paper reports that guiding RL with these sampled states, especially through projected interpolation and a schedule that starts episodes near goals, achieves an 83.1% success rate on a double-sphere manipulation task and 5.1% (versus a 0.1% baseline) on a harder Panda-arm whole-body-contact task. A sympathetic reader would take the central insight to be that demonstrating relevant intermediate states matters more than demonstrating actions.

Core claim

The paper introduces Sample-Guided RL, which uses a model-based solver to sample configurations that satisfy collision, contact, and static-equilibrium constraints, then uses those samples to bias the initial state distribution of a goal-conditioned reinforcement learning agent. The central discovery is empirical: biasing state visitation with constraint samples — via trajectory-derived starts or via projected interpolation that snaps random convex blends of feasible states to the nearest feasible state in feature space — is what drives learning, while adding a behavior cloning loss has only minor impact. On the double-sphere domain, the best method reaches 83.1% success on random feasible s

What carries the argument

The central object is the Constrained Goal-conditioned MDP (CG-MDP), which augments a goal-conditioned MDP with differentiable constraints g_c(s)≤0, h_c(s)=0 that define the set S_c of feasible states (collision-free, contact-consistent, static-equilibrium). The machinery that carries the argument is a proximal Augmented-Lagrangian sampler (Eq. 10) that projects uniform box samples onto S_c, plus the projected-interpolation sampler (Eq. 19) that maps random convex interpolations between two constrained states to the nearest feasible state in a feature embedding, and an optional behavior-cloning regularization (Eq. 14). These components together shape the state-visitation distribution of the

Load-bearing premise

The load-bearing premise is that the solver's feasible samples (58.7% and 17.9% per-sample success in the two domains) are representative of the full constrained manifold; without a coverage or ergodicity guarantee, the measured success rates may apply only to the solver-reachable subset.

What would settle it

Retrain using a provably uniform sampler over S_c (e.g., rejection sampling or hit-and-run) with sufficient compute, then evaluate on the same start/goal distribution; if success rates drop to baseline levels, the reported gains are an artifact of solver bias. Alternatively, compute the success rate on states that are feasible but never visited by the solver, and check whether it falls sharply.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • State-visitation bias from model-based constraint samples can replace expert demonstrations as a practical way to guide goal-conditioned RL in contact-rich manipulation.
  • Behavior-cloning regularization adds little beyond the state bias, suggesting that costly trajectory optimization may be unnecessary in simpler domains.
  • The CG-MDP formulation provides a principled method to inject physics first principles — collision, contact, friction, static equilibrium — into an otherwise black-box RL problem.
  • The approach scales to whole-body contact manipulation, achieving 5.1% success versus a 0.1% baseline, indicating applicability to non-prehensile and dynamic tasks.
  • Scheduling starts near the goal and gradually moving them away acts as an effective curriculum, improving final policy performance and reducing training variance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The scheduled interpolation method can be viewed as an automatic curriculum that trains on easy near-goal starts first; this predicts that any goal-conditioned RL algorithm would benefit from the same schedule regardless of the base learner.
  • The near-perfect success after removing sphere-on-sphere balancing goals suggests the limiting factor is controller bandwidth for unstable equilibria, not state-space discovery; a higher-frequency control loop or a dedicated balancing sub-policy could close this gap.
  • The approach transfers to any black-box dynamical system for which a differentiable constraint manifold can be defined, such as in-hand manipulation, object pivoting, or legged locomotion — provided a sampler with better coverage than the reported 58.7% and 17.9% per-sample feasibility rates.
  • The paper's comparison of trajectory-based and interpolation-based guidance mirrors a bias-variance tradeoff; one testable consequence is that interpolation-based guidance should outperform trajectory-based guidance on simpler manifolds and underperform on manifolds where solver reachability is highly non-uniform.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Sample-Guided RL, a framework for goal-conditioned manipulation policies in which a model-based constrained sampler provides the start/goal distribution and guides RL through reset-state initialization. The authors define a Constrained Goal-conditioned MDP (CG-MDP) with p0 uniform over a feasible state set Sc, generate a constrained state dataset D_s via Augmented Lagrangian projection (Eq. 10), generate an open-loop trajectory dataset D_u via CMA-ES (Eq. 12), and then use options (16)-(21) to sample start/goal pairs during TD7 training, optionally adding behavior cloning (Eq. 14). Experiments on a double-sphere domain and a Panda-arm sphere domain report success rates of 83.1% (99.0% after excluding balance goals) on the former and 5.1% vs. a 0.1% baseline on the latter, with 61.2% on the trajectory-state distribution.

Significance. If the results hold, the paper makes a useful contribution to contact-rich manipulation RL: it gives a crisp formulation of how model-based constraint information can be injected into RL through reset-state distributions, without relying on expert demonstrations. The comparison of six guidance variants is systematic, and the reporting of per-sample feasibility, compute time, and variance over five runs is a strength. The main weakness is that the central 'universal policy' claim is evaluated on the same solver-generated D_s used for training, with no independent coverage analysis, so the reported numbers are best understood as in-distribution performance on a finite, possibly biased sample rather than estimates with respect to the stated p0 = U(Sc x Sc).

major comments (3)
  1. [§V-B, Fig. 3(a), Eq. (10)] The sampler used to realize p0 is the Augmented Lagrangian projection from box-uniform restarts with randomly sampled contact modes, but the paper provides no coverage, completeness, or ergodicity guarantee. With per-sample feasibility rates of only 58.7% (double sphere) and 17.9% (panda), the retained D_s may be concentrated on solver-friendly regions of Sc. Since the same D_s is used both for training resets and for the final 'uni' evaluation in Fig. 6, the headline success rates (83.1% and 5.1%) are in-distribution estimates on a finite solver-generated set, not estimates over U(Sc x Sc). I request an independent coverage check or a held-out test set generated by a different mechanism than Eq. (10), e.g., a different initializer or contact-mode sampler, with a comparison of achieved success rates on that set.
  2. [§V-D(iv), Fig. 6] The double-sphere 'w/o balance' column removes 'sphere-on-sphere balance goals' post hoc. These goals are part of Sc as defined in Eq. (1), and the objective (3) is over all of Sc x Sc. If such goals are unreachable or uncontrollable under the MDP, that should be established a priori and incorporated into the problem definition; otherwise the 99.0% figure is an evaluation on a modified distribution and cannot be presented as success on the stated objective. The paper should either redefine Sc to exclude such states with a principled reachability/controllability argument or report the 83.1% number as the primary result with the exclusion clearly labeled as a limitation.
  3. [§V-D(iv), Fig. 6, panda row] The 61.2% panda success rate is evaluated on start/goal pairs taken from trajectory states (consistent with the traj distribution), not from U(Sc x Sc). This number is therefore not evidence for a universal policy; it is an in-distribution measure on the same trajectory-state distribution used for training. The 'uni' rate of 5.1% is the only number directly relevant to the stated objective, and while it is much better than the 0.1% baseline, the absolute performance is low. The paper should clearly separate the trajectory-state results from universal results and, ideally, add an evaluation on a broader set of start/goal pairs that is independent of D_u.
minor comments (4)
  1. [Fig. 6] The caption contains a typo ('panda shpere') and 'w/o balance' is used without defining the balance-goal criterion precisely in the figure caption.
  2. [Eq. (10)] The constraint notation is inconsistent: h_c(x)=0 appears in Eq. (10) while the surrounding text uses h_c(s)=0. Please fix.
  3. [§IV-C, Eq. (14)] The symbol D is reused for both the BC batch in Eq. (14) and the datasets D_s/D_u. This is confusing; use a different symbol for the batch.
  4. [Abstract and §V-B] The abstract describes model-based constrained state sampling as 'highly efficient,' but Fig. 3(a) reports per-sample feasibility rates of 58.7%/17.9% and about one hour to generate 10,000 Panda samples. Consider softening the wording or framing efficiency relative to alternative resampling strategies.

Circularity Check

0 steps flagged

No circularity: the sampler-to-RL evaluation chain is self-contained; D_s coverage caveats affect external validity, not the derivation.

full rationale

The paper's chain is explicit: constraints (5)-(9) define S_c; the Augmented Lagrangian projection (10) generates D_s; optional CMA-ES trajectory optimization (12)-(13) generates D_u; these feed start/goal reset strategies (16)-(21); TD7 trains a goal-conditioned policy; Fig. 6 evaluates it against objective (3). No link takes the measured success rate as an input. D_s is produced by an NLP solver from box-uniform restarts and random contact modes, and the policy is trained by TD7 against black-box simulator rewards, so the reported success is an outcome of the dynamics, not a restatement of the sampler. The closest concern is that Eq. (2) defines p0 as U(S_c x S_c), while the 'uni' evaluation samples (s,g) from the finite solver-generated D_s (Sec. V-A and Fig. 6). The paper itself admits 'the resulting NLP might be infeasible or the solver be stuck in a local optimum' and reports feasibility rates of only 58.7%/17.9% (Fig. 3(a)). This means the 'universal' success rates are estimates on a solver-biased finite set with no coverage proof; but that is a validity/coverage limitation, not a circular reduction, because the evaluation distribution is not used to construct the policy and the baseline is evaluated identically. The 'w/o balance' and 'traj' columns are explicitly labeled distribution restrictions, not hidden predictions. The self-citations are incidental related-work or technical references and are not load-bearing; no uniqueness theorem or imported ansatz is used. Hence no specific circular step can be exhibited.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities; its contributions are algorithmic. The main assumptions are the validity of the static-equilibrium constraints as the definition of feasible states and the unbiasedness of the constrained sampler, which are not formally guaranteed.

free parameters (6)
  • feature weights in ϕ(x) = (2p, 1q, 0.1ṗ, 0.1q̇, 0.1c)
    Hand-chosen weights in the state feature embedding define the trajectory optimization objective (12) and the nearest-neighbor metric in projected interpolation (19); different weights would change which solutions CMA-ES finds and which states are considered near.
  • max active support contacts = 3
    Randomly imposing too many contacts makes the NLP infeasible; capping at 3 biases the goal distribution toward low-contact configurations.
  • schedule parameters = Tsched=1M, Tblock=100k, α:0.1→1
    Curriculum parameters control how close starts are to goals over time; they are chosen by hand and affect training curves and final performance.
  • time limits = 2s (double sphere), 4s (panda)
    Episode truncation limits the reachable distance from the start; longer limits improve coverage but increase training cost.
  • reward threshold = 1 cm object distance
    Sparse reward success criterion; a tighter threshold would lower success rates, a looser one would raise them.
  • B-spline control knots = 4 knots, 2nd order
    Low-dimensional action parameterization for zero-order trajectory optimization; this restricts the set of open-loop trajectories CMA-ES can find.
axioms (5)
  • domain assumption The constraint set S_c = {s: g_c(s)≤0, h_c(s)=0} with collision, contact, friction-cone and static Newton-Euler equations (Eqs. 5–9) is a good proxy for 'statically feasible states' in the simulated domains.
    If these constraints are too restrictive or too loose, the defined objective p0 = U(S_c × S_c) does not match the actual feasible configurations of the robot-object system.
  • ad hoc to paper The Augmented Lagrangian solver (Eq. 10) with random contact modes and box-uniform initialization produces samples representative of S_c.
    No completeness or unbiasedness guarantee; feasibility rates are 58.7% and 17.9%. All downstream data (D_s, D_u) and evaluations depend on this sampler's output.
  • ad hoc to paper The state feature embedding ϕ (weighted concatenation of p, q, ṗ, q̇, c) is a meaningful metric for interpolating between start and goal states in projected interpolation (Eq. 19).
    Nearest-neighbor search in ϕ-space is used to project linear interpolations onto feasible states; if ϕ ignores important degrees of freedom, the curriculum guides RL toward states that are not actually intermediate.
  • domain assumption TD7 with the listed hyperparameter changes is a reasonable off-the-shelf goal-conditioned RL algorithm; no comparison to other baselines is provided.
    Standard RL algorithm assumption in the paper's evaluation.
  • domain assumption MuJoCo's deterministic simulation at 1 kHz for rollout and 20 Hz for MDP actions is a faithful black-box dynamics model.
    Used throughout for trajectory optimization and policy training.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation." pith.science (2026). https://pith.science/paper/2KZQNQPF

@misc{pith2026260208557,
  author       = {Pith},
  title        = {Pith review of: Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KZQNQPF}},
  note         = {Machine review of arXiv:2602.08557}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Training non-prehensile manipulation policies in contact-rich settings is a core challenge in robotics. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies. To address this, we combine two basic ideas: First, designing appropriate reset strategies (the start state distribution of episodes) has shown promise in improving RL exploration and effectiveness. Second, while model-based approaches to finding trajectories through manipulation are hard, recent work showed that model-based approaches to sampling states on constrained manifolds can be highly efficient. Based on these observations, we propose a novel state sampler that boosts the performance of goal-conditioned RL in complex contact-rich manipulation tasks. Our sampler explicitly takes into account the structure of contact in order to provide a rich covering of diverse contact modes. By combining constrained sampling resets with projected interpolation and curriculum learning, our novel approach outperforms RL without constrained sampling and alternative reset methods, and effectively trains universal, non-prehensile, and dynamic manipulation policies in contact-rich settings. See https://www.user.tu-berlin.de/mtoussai/26-CSRL/ for supplementary material.

Figures

Figures reproduced from arXiv: 2602.08557 by Armand Jordana, Cornelius V. Braun, Denis Shcherba, Eckart Cobo-Briesewitz, Justin Carpentier, Marc Toussaint, Sayantan Auddy, Tilman Burghoff.

Figure 1
Figure 1. Figure 1: Random samples from a model-based constrained [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Random samples from the double sphere domain. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Optimization runs with median (shading: 20/80% [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Final Policy Performance: The success rate of [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Manifold Sampling via Entropy Maximization

    cs.LG 2026-05 conditional novelty 7.0

    MASEM samples constrained manifolds with unknown disconnected components via entropy-maximizing k-NN resampling, achieving exponential mean-field KL reduction and order-of-magnitude Sinkhorn improvement on benchmarks.

Reference graph

Works this paper leans on

62 extracted references · 8 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Solving rubik’s cube with a robot hand.arXiv:1910.07113, 2019

    Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik’s cube with a robot hand.arXiv:1910.07113, 2019

    Pith/arXiv arXiv 1910

  2. [2]

    Hindsight experience replay.Ad- vances in neural information processing systems, 30, 2017

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Mc- Grew, Josh Tobin, OpenAI Pieter Abbeel, and Woj- ciech Zaremba. Hindsight experience replay.Ad- vances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Consensus complementarity control for multi-contact mpc.IEEE Transactions on Robotics, 2024

    Alp Aydinoglu, Adam Wei, Wei-Cheng Huang, and Michael Posa. Consensus complementarity control for multi-contact mpc.IEEE Transactions on Robotics, 2024

    2024

  4. [4]

    Guided goal generation for hindsight multi- goal reinforcement learning.Neurocomputing, 359: 353–367, 2019

    Chenjia Bai, Peng Liu, Wei Zhao, and Xianglong Tang. Guided goal generation for hindsight multi- goal reinforcement learning.Neurocomputing, 359: 353–367, 2019

    2019

  5. [5]

    Efficient online reinforcement learn- ing with offline data

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. InInternational Conference on Machine Learning, pages 1577–1594. PMLR, 2023

    2023

  6. [6]

    Cambridge university press, 2004

    Stephen Boyd and Lieven Vandenberghe.Convex optimization. Cambridge university press, 2004

    2004

  7. [7]

    Trajectory first: A curriculum for discovering diverse policies.arXiv:2506.01568, 2025

    Cornelius V Braun, Sayantan Auddy, and Marc Tou- ssaint. Trajectory first: A curriculum for discovering diverse policies.arXiv:2506.01568, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Tenenbaum, Tim Rockt¨aschel, and Edward Grefenstette

    Andres Campero, Roberta Raileanu, Heinrich K ¨uttler, Joshua B. Tenenbaum, Tim Rockt¨aschel, and Edward Grefenstette. Learning with amigo: Adver- sarially motivated intrinsic goals. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021

    2021

  9. [9]

    Goal-conditioned reinforcement learning with imagined subgoals

    Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. InProceedings of the In- ternational Conference on Machine Learning, 2021

    2021

  10. [10]

    Whole-body motion planning with centroidal dynamics and full kinematics

    Hongkai Dai, Andr ´es Valenzuela, and Russ Tedrake. Whole-body motion planning with centroidal dynamics and full kinematics. In2014 IEEE-RAS International Conference on Humanoid Robots, pages 295–302. IEEE, 2014

    2014

  11. [11]

    Imitating Task and Motion Planning with Visuomotor Transformers

    Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Dieter Fox. Imitating Task and Motion Planning with Visuomotor Transformers. InConference on Robot Learning (CoRL). PMLR, 2023. doi: 10.48550/arXiv. 2305.16309

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.16309 2023

  12. [12]

    Footstep planning on uneven terrain with mixed-integer convex opti- mization

    Robin Deits and Russ Tedrake. Footstep planning on uneven terrain with mixed-integer convex opti- mization. In2014 IEEE-RAS international conference on humanoid robots, pages 279–286. IEEE, 2014

    2014

  13. [13]

    Curriculum-guided hindsight experience replay.Advances in neural information processing systems, 32, 2019

    Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay.Advances in neural information processing systems, 32, 2019

    2019

  14. [14]

    Reverse cur- riculum generation for reinforcement learning

    Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on Robot Learning, pages 482–495. PMLR, 2017

    2017

  15. [15]

    A mini- malist approach to offline reinforcement learning

    Scott Fujimoto and Shixiang Shane Gu. A mini- malist approach to offline reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 34, pages 20132–20145, 2021

    2021

  16. [16]

    Addressing function approximation error in actor- critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. InInternational Conference on Machine Learning, pages 1587–1596. PMLR, 2018

    2018

  17. [17]

    For sale: State-action representation learning for deep reinforcement learning.Advances in neural information processing systems, 36:61573–61624, 2023

    Scott Fujimoto, Wei-Di Chang, Edward Smith, Shix- iang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning.Advances in neural information processing systems, 36:61573–61624, 2023

    2023

  18. [18]

    Goal-conditioned on-policy reinforcement learning

    Xudong Gong, Dawei Feng, Kele Xu, Bo Ding, and Huaimin Wang. Goal-conditioned on-policy reinforcement learning. InAdvances in Neural In- formation Processing Systems, 2024

    2024

  19. [19]

    Rosati Papini, Patrick M

    Gianluigi Grandesso, Elisa Alboni, Gastone P . Rosati Papini, Patrick M. Wensing, and Andrea Del Prete. Cacto: Continuous actor-critic with trajectory optimization—towards global optimality. IEEE Robotics and Automation Letters, 8(6):3318–3325, 2023

    2023

  20. [20]

    Hopkins, Georg Wiedebach, Jared Bishop, Steven Pickles, David M ¨uller, and Moritz B ¨acher

    Ruben Grandia, Espen Knoop, Michael A. Hopkins, Georg Wiedebach, Jared Bishop, Steven Pickles, David M ¨uller, and Moritz B ¨acher. Design and Con- trol of a Bipedal Robotic Character. InProceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024

    2024

  21. [21]

    Relay policy learning: Solving long-horizon tasks via imitation 9 and reinforcement learning

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation 9 and reinforcement learning. InConference on Robot Learning, pages 1025–1037. PMLR, 2020

    2020

  22. [22]

    Com- pletely derandomized self-adaptation in evolution strategies.Evolutionary computation, 9(2):159–195, 2001

    Nikolaus Hansen and Andreas Ostermeier. Com- pletely derandomized self-adaptation in evolution strategies.Evolutionary computation, 9(2):159–195, 2001

    2001

  23. [23]

    Adaptive curriculum generation from demon- strations for sim-to-real visuomotor control

    Lukas Hermann, Max Argus, Andreas Eitel, Artemij Amiranashvili, Wolfram Burgard, and Thomas Brox. Adaptive curriculum generation from demon- strations for sim-to-real visuomotor control. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6498–6505. IEEE, 2020

    2020

  24. [24]

    Imitation bootstrapped reinforcement learn- ing

    Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learn- ing. InRobotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024, 2024

    2024

  25. [25]

    MRHER: model-based relay hindsight experience replay for sequential object manipulation tasks with sparse rewards

    Yuming Huang, Bin Ren, Ziming Xu, and Lianghong Wu. MRHER: model-based relay hindsight experience replay for sequential object manipulation tasks with sparse rewards. In International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024

    2024

  26. [26]

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Ad- nan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas Godden, Ivan Goryachev, Lachy Groom, Hunter Hancock, Karol Hausman, Gashon Hussein, Brian Ichter, Sz...

    Pith/arXiv arXiv 2025

  27. [27]

    SEIL: Simulation-augmented Equivariant Imitation Learning

    Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. SEIL: Simulation-augmented Equivariant Imitation Learning. InIEEE International Conference on Robotics and Automation (ICRA), pages 1845–1851. IEEE, 2023

    2023

  28. [28]

    Reinforcement learning from imperfect demon- strations under soft expert guidance

    Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demon- strations under soft expert guidance. InProceedings of the AAAI conference on artificial intelligence, vol- ume 34, pages 5109–5116, 2020

    2020

  29. [29]

    An Introduction to Zero-Order Optimization Techniques for Robotics

    Armand Jordana, Jianghan Zhang, Joseph Amigo, and Ludovic Righetti. An Introduction to Zero-Order Optimization Techniques for Robotics. arXiv:2506.22087, October 2025

    arXiv 2025

  30. [30]

    Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine ma- nipulation

    Liyiming Ke, Jingqiang Wang, Tapomayukh Bhat- tacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine ma- nipulation. InIEEE International Conference on Robotics and Automation (ICRA), pages 6185–6191. IEEE, 2021

    2021

  31. [31]

    CCIL: Continuity-based Data Augmentation for Correc- tive Imitation Learning.arXiv:2310.12972, June 2024

    Liyiming Ke, Yunchu Zhang, Abhay Deshpande, Siddhartha Srinivasa, and Abhishek Gupta. CCIL: Continuity-based Data Augmentation for Correc- tive Imitation Learning.arXiv:2310.12972, June 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Prehensile manipulation planning: Modeling, algorithms and implementation.IEEE Transactions on Robotics, 38 (4):2370–2388, 2021

    Florent Lamiraux and Joseph Mirabel. Prehensile manipulation planning: Modeling, algorithms and implementation.IEEE Transactions on Robotics, 38 (4):2370–2388, 2021

    2021

  33. [33]

    LaValle.Planning Algorithms

    Steven M. LaValle.Planning Algorithms. Cambridge university press, 2006

    2006

  34. [34]

    Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems.Nonlinear Analysis: Hybrid Systems, 52:101468, 2024

    Quentin Le Lidec, Fabian Schramm, Louis Montaut, Cordelia Schmid, Ivan Laptev, and Justin Carpen- tier. Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems.Nonlinear Analysis: Hybrid Systems, 52:101468, 2024

    2024

  35. [35]

    Hier- archical actor-critic.arXiv:1712.00948, 2017

    Andrew Levy, Robert Platt, and Kate Saenko. Hier- archical actor-critic.arXiv:1712.00948, 2017

    Pith/arXiv arXiv 2017

  36. [36]

    Diversity progress for goal selection in discriminability-motivated RL.arXiv:2411.01521, 2024

    Erik M Lintunen, Nadia M Ady, and Christian Guckelsberger. Diversity progress for goal selection in discriminability-motivated RL.arXiv:2411.01521, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Opt2skill: Imi- tating dynamically-feasible whole-body trajectories for versatile humanoid loco-manipulation.IEEE Robotics and Automation Letters, 2025

    Fukang Liu, Zhaoyuan Gu, Yilin Cai, Ziyi Zhou, Hyunyoung Jung, Jaehwi Jang, Shijie Zhao, Sehoon Ha, Yue Chen, Danfei Xu, et al. Opt2skill: Imi- tating dynamically-feasible whole-body trajectories for versatile humanoid loco-manipulation.IEEE Robotics and Automation Letters, 2025

    2025

  38. [38]

    Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learn- ing

    Jinxin Liu, Donglin Wang, Qiangxing Tian, and Zhengyu Chen. Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learn- ing. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 7558–7566, 2022

    2022

  39. [39]

    A simple motion-planning algorithm for general robot manipulators.IEEE Journal on Robotics and Automation, 3(3):224–238, 2003

    Tomas Lozano-Perez. A simple motion-planning algorithm for general robot manipulators.IEEE Journal on Robotics and Automation, 3(3):224–238, 2003

    2003

  40. [40]

    Crocoddyl: An efficient and versatile framework for multi-contact optimal control

    Carlos Mastalli, Rohan Budhiraja, Wolfgang Merkt, Guilhem Saurel, Bilal Hammoud, Maximilien Naveau, Justin Carpentier, Ludovic Righetti, Sethu Vijayakumar, and Nicolas Mansard. Crocoddyl: An efficient and versatile framework for multi-contact optimal control. In2020 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 2536–

  41. [41]

    Discovery of complex behaviors through contact-invariant optimization.ACM Transactions on Graphics (ToG), 31(4):1–8, 2012

    Igor Mordatch, Emanuel Todorov, and Zoran Popovi´c. Discovery of complex behaviors through contact-invariant optimization.ACM Transactions on Graphics (ToG), 31(4):1–8, 2012. 10

    2012

  42. [42]

    Data-efficient hierarchical reinforce- ment learning

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. InAdvances in Neural Information Processing Systems, 2018

    2018

  43. [43]

    Visual rein- forcement learning with imagined goals

    Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual rein- forcement learning with imagined goals. InAd- vances in Neural Information Processing Systems, 2018

    2018

  44. [44]

    Deep exploration via bootstrapped DQN.Advances in neural information processing systems, 29, 2016

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN.Advances in neural information processing systems, 29, 2016

    2016

  45. [45]

    Deep black-box reinforcement learning with movement primitives

    Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Zi- esche, Vien Anh Ngo, and Gerhard Neumann. Deep black-box reinforcement learning with movement primitives. InConference on Robot Learning, pages 1244–1265. PMLR, 2023

    2023

  46. [46]

    Learning from trajectories via subgoal discovery.Advances in Neural Information Processing Systems, 32, 2019

    Sujoy Paul, Jeroen Vanbaar, and Amit Roy- Chowdhury. Learning from trajectories via subgoal discovery.Advances in Neural Information Processing Systems, 32, 2019

    2019

  47. [47]

    Deepmimic: Example- guided deep reinforcement learning of physics- based character skills.ACM Transactions On Graphics (TOG), 37(4):1–14, 2018

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics- based character skills.ACM Transactions On Graphics (TOG), 37(4):1–14, 2018

    2018

  48. [48]

    Amp: Adversarial motion priors for stylized physics-based character control

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1–20, 2021

    2021

  49. [49]

    A direct method for trajectory optimization of rigid bodies through contact.The International Journal of Robotics Research, 33(1):69–81, 2014

    Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization of rigid bodies through contact.The International Journal of Robotics Research, 33(1):69–81, 2014

    2014

  50. [50]

    Universal value function approxima- tors

    Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. InInternational Conference on Machine Learning, pages 1312–1320. PMLR, 2015

    2015

  51. [51]

    Manipulation planning with probabilistic roadmaps.The International Journal of Robotics Research, 23(7-8):729–746, 2004

    Thierry Sim ´eon, Jean-Paul Laumond, Juan Cort ´es, and Anis Sahbani. Manipulation planning with probabilistic roadmaps.The International Journal of Robotics Research, 23(7-8):729–746, 2004

    2004

  52. [52]

    Hybrid RL: Using both offline and online data can make RL efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bag- nell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  53. [53]

    Bundled gradients through contact via randomized smoothing.IEEE Robotics and Automation Letters, 7 (2):4000–4007, 2022

    Hyung Ju Terry Suh, Tao Pang, and Russ Tedrake. Bundled gradients through contact via randomized smoothing.IEEE Robotics and Automation Letters, 7 (2):4000–4007, 2022

    2022

  54. [54]

    Synthesis and stabilization of complex behaviors through online trajectory optimization

    Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4906–4913. IEEE, 2012

    2012

  55. [55]

    Mu- joco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mu- joco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  56. [56]

    De- scribing physics for physical reasoning: Force-based sequential manipulation planning.IEEE Robotics and Automation Letters, 5(4):6209–6216, 2020

    Marc Toussaint, Jung-Su Ha, and Danny Driess. De- scribing physics for physical reasoning: Force-based sequential manipulation planning.IEEE Robotics and Automation Letters, 5(4):6209–6216, 2020

    2020

  57. [57]

    Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards.Advances in Neural Information Processing Systems, 32, 2019

    Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards.Advances in Neural Information Processing Systems, 32, 2019

    2019

  58. [58]

    Jump-start reinforcement learning

    Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Jos ´ephine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. InInternational Conference on Machine Learning, pages 34556–34583. PMLR, 2023

    2023

  59. [59]

    Epistemically-guided forward-backward exploration.arxiv:2507.05477, July 2025

    N ´uria Armengol Urp ´ı, Marin Vlastelica, Georg Martius, and Stelian Coros. Epistemically-guided forward-backward exploration.arxiv:2507.05477, July 2025

    Pith/arXiv arXiv 2025

  60. [60]

    Kinodynamic motion planning for a team of multirotors transporting a cable-suspended payload in cluttered environ- ments

    Khaled Wahba, Joaquim Ortiz-Haro, Marc Tous- saint, and Wolfgang H ¨onig. Kinodynamic motion planning for a team of multirotors transporting a cable-suspended payload in cluttered environ- ments. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 12750– 12757. IEEE, 2024

    2024

  61. [61]

    Numerical optimization.Springer Science, 35(67-68):7, 1999

    Stephen Wright, Jorge Nocedal, et al. Numerical optimization.Springer Science, 35(67-68):7, 1999

    1999

  62. [62]

    Uni- versal manipulation policy network for articulated objects.IEEE robotics and automation letters, 7(2): 2447–2454, 2022

    Zhenjia Xu, Zhanpeng He, and Shuran Song. Uni- versal manipulation policy network for articulated objects.IEEE robotics and automation letters, 7(2): 2447–2454, 2022. 11

    2022

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.