REVIEW 2 major objections 4 minor 25 references
Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator
T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A fully simulation-trained policy solves real-world legged pick-and-place in 78% of episodes, beating human teleoperation.
desk verdict A well-executed sim-to-real systems paper with an honest empirical core; the absolute ~78% success claim holds up, but the claim of beating human teleoperation is within sampling noise and should be toned down. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Progressive Policy Expansion: the teacher network is not one network but a growing set of per-subtask policies, one per stage of the task, with a new network added when a new stage is encountered and gated by a privileged subtask id, so previously learned skills are preserved in their own parameters and exploration capacity is reset for each new stage. The other load-bearing mechanism is distillation-guided RL: the student soft actor-critic agent replaces the entropy term with a KL divergence to a fixed-dispersion version of the teacher, encouraging imitation of the teacher's action mode while still allowing exploration—this is what lets pixel-and-language input learn the long-horizon policy
What would settle it
Deploy the same SLIM policy in a scene where the target cube has the same color as the floor under strong shadows and record the full-task success rate. If the segmentation/depth estimates fail and the success rate drops well below the reported 78.3%, with failures concentrated in search and approach stages, the bottleneck is the load-bearing fragility.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a fully simulation-trained high-level visuomotor policy can solve the entire multi-stage mobile pick-and-place task in the real world with 78.3% ± 5.8% success, the highest among all compared methods, including a human teleoperator at 75.0% ± 5.0%, while completing episodes in 43.8 seconds on average versus 65.5 seconds for teleoperation. The policy relies only on ego-centric RGB from a wrist-mounted camera, proprioceptive state, and a natural-language instruction; it is the student in a teacher–student pipeline. The teacher, trained with privileged object states, is structured as a set of per-subtask policies activated progressively to avoid c
Load-bearing premise
The visual information bottleneck—segmentation and depth maps estimated from the wrist camera—must be reliable enough in diverse real-world scenes to preserve task-relevant information for the downstream policy, even though the vision module is trained only in simulation.
Editorial extensions
If this is right
- Long-horizon mobile manipulation with only onboard sensing can be trained without real-world demonstrations or fine-tuning.
- Progressive policy expansion is a workable answer to catastrophic forgetting and lost plasticity in long-horizon RL, and could be reused whenever a task has identifiable stage boundaries.
- The identified sim-to-real techniques—object, arm, and mount perturbation, color and visual augmentation, and the segmentation/depth bottleneck—transfer to novel scenes zero-shot.
- On this task, an autonomous visuomotor policy can outperform a human teleoperator who shares the same camera view, in both success and speed.
- Emergent re-grasping and task chaining indicate that the learned components compose into robust, repeated full-task behavior under disturbance.
Reading between the lines
- Because the success claim passes through a segmentation-and-depth bottleneck trained only in simulation, the approach should be stress-tested on scene distributions that attack that module specifically—low-contrast objects, strong shadows, or unseen colors. If the vision module degrades while the downstream policy is unchanged, the bottleneck, not the policy, is the transfer limit.
- The progressive expansion idea should extend to other long-horizon manipulation or navigation tasks with clear milestones, but likely needs modification for tasks where stages are not sequentially ordered or overlap in time.
- The comparison to human teleoperation is specific to this setup: a single wrist camera, one hour of practice, and joystick control. It does not imply general superiority over teleoperation, only that on this task the learned policy exploits the available view more consistently.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SLIM, a system for training a visuo-motor policy for legged mobile manipulation entirely in simulation and deploying it zero-shot in the real world. The task is a multi-stage mobile pick-and-place: search for a colored cube, approach, grasp, search for the correct basket, transport, and drop the cube, using only a wrist-mounted camera and a language instruction. The method has three training stages: a frozen low-level locomotion policy, a teacher policy trained with privileged state information and a progressive policy-expansion scheme that allocates a separate policy per subtask, and a student policy trained by distillation-guided RL from ego-centric images, proprioception, and language. Real-world experiments cover 400 episodes across multiple scenes and ablations, reporting a full-task success rate of 78.3% ± 5.8% for SLIM, which is higher than the reported baselines, and also faster than a human teleoperation baseline. The paper additionally reports generalization to novel scenes, novel object shapes, re-grasping, and task chaining, and provides an honest failure-mode analysis.
Significance. If the results are taken at face value, this is a strong systems contribution to long-horizon mobile manipulation with legged robots. The main strengths are the scale and honesty of the real-world evaluation (400 episodes, multiple seeds, ablations, reported failure modes), the use of onboard sensing without global cameras or external APIs, and the identification of practical sim-to-real ingredients (perturbations, visual augmentation, arm-retract shaping, distillation-guided RL). The progressive policy expansion idea is a sensible mechanism for addressing capacity loss and catastrophic forgetting in long-horizon RL. However, the paper contains a statistically unsupported comparative claim against human teleoperation, and an internal inconsistency about whether the deployed student policy consumes raw RGB images or segmentation/depth maps. These issues affect the precision of the central claims and need to be resolved before the paper is fully convincing.
major comments (2)
- [Section V-A, Table I] The statement that SLIM 'achieves the highest subtask and full task success rates among all the methods including Human Teleop' is not statistically supported. From Table I, the full-task success rates are 78.3% ± 5.8% (SLIM) and 75.0% ± 5.0% (Human Teleop), each based on 60 episodes (3 seeds × 20). Treating these as binomial proportions, a two-proportion z-test gives z ≈ 0.43, p ≈ 0.67, and the 95% Wilson intervals substantially overlap. The observed 3.3-point gap is well within sampling noise. The absolute success rate of ~78% is credible, but the comparative superiority over human teleoperation should either be supported with a proper significance test or the claim should be softened to 'comparable to human teleoperation.'
- [Section IV-B, Eq. (2) and Section IV-C.2] There is an inconsistency in the description of the student policy input. In Section IV-B, Eq. (2) defines s_stu = [o, s_r, l] with o denoting 'the (stacked) ego-centric RGB images' and the text states the policy perceives 'the single RGB stream.' However, Section IV-C.2 lists as the first visual-gap reduction technique 'a visual information bottleneck (a pair of segmentation and depth maps) between the perception module and the policy.' These two descriptions cannot both be true of the same policy. If the deployed policy actually consumes segmentation/depth estimates rather than raw RGB, the term 'RGB images' is misleading and the method should be described accordingly; if it consumes raw RGB, the role of the bottleneck in the deployed architecture is unclear. This is essential for reproducibility and for understanding the sim-to-real transfer mechanism.
minor comments (4)
- [Section V, Baselines] The 'No Distillation' baseline is excluded with the assertion that it cannot learn the task at all in simulation. This is plausible, but the claim would be more convincing if supported by at least a simulation training curve or a short description of the observed failure (e.g., no improvement over random). As written, an entire ablation is omitted without direct evidence.
- [Section IV-A.2] Typo: 'Seciton' should be 'Section.' Also in Section V, 'we used used 4 GPUs' contains a duplicated word.
- [Section IV-B, Eq. (2)] The teacher policy π_tea is described as a set of policies gated by subtask id k, but Eq. (2) writes π_tea(a_hi | s_r, s_p) without making the gating explicit. Clarify how s_p (including the subtask id) selects the active teacher policy, and how the fixed modal dispersion σ is used to form the teacher action distribution for the KL term.
- [Section V-C, Fig. 8] The generalization results in Fig. 8 are based on only 10 episodes per scene. Reporting success rates without confidence intervals makes it hard to judge whether the ~78% performance in new scenes is meaningfully different from the main result. Consider adding Wilson intervals or additional trials for at least one novel scene.
Circularity Check
No circular derivation; self-citations are not load-bearing.
full rationale
The paper's central claim—a sim-trained visuo-motor policy achieving 78.3% real-world full-task success—is an empirically measured outcome, not a quantity derived from a fitted parameter or an input definition. The teacher-student pipeline, progressive PEX, distillation-guided RL, and sim-to-real augmentations are all implemented and evaluated in-paper: Table I reports real deployments across seeds, Fig. 6 compares standard vs progressive PEX in simulation, and Fig. 9 compares sim/real success across ablations. The authors cite their own prior PEX [20] and entropy-reward [24] works, but these citations supply technique names and design choices, not the measured success numbers; the load-bearing evidence is the new experiments. No equation in the paper defines the reported success in terms of an input fit, no predicted quantity is equivalent by construction to a training signal, and no uniqueness theorem is imported from the authors' prior work. The only notable concern is statistical: the claimed superiority over Human Teleop (78.3% vs 75.0%, 60 episodes each) has overlapping confidence intervals, but that is a power/uncertainty issue, not circular reasoning. Therefore no significant circularity.
Assumptions & free parameters
free parameters (3)
- Subtask decomposition (K=7) =
7
- Distillation weight alpha
- Teacher action dispersion sigma
assumptions (3)
- domain assumption The low-level locomotion policy, trained following [16], reliably tracks high-level velocity commands in the real world when carrying the arm and object.
- domain assumption The simulation environment is an adequate proxy for the real robot and objects, given the domain randomization techniques used.
- domain assumption The teacher-student distillation framework with privileged state can produce a student policy that retains teacher performance from images and language alone.
Cite this review
Pith. "Pith review of Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator." pith.science (2026). https://pith.science/paper/XNN2TJRD
@misc{pith2026250903859,
author = {Pith},
title = {Pith review of: Learning Multi-Stage Pick-and-Place with a Legged Mobile Manipulator},
year = {2026},
howpublished = {\url{https://pith.science/paper/XNN2TJRD}},
note = {Machine review of arXiv:2509.03859}
}
read the original abstract
Quadruped-based mobile manipulation presents significant challenges in robotics due to the diversity of required skills, the extended task horizon, and partial observability. After presenting a multi-stage pick-and-place task as a succinct yet sufficiently rich setup that captures key desiderata for quadruped-based mobile manipulation, we propose an approach that can train a visuo-motor policy entirely in simulation, and achieve nearly 80\% success in the real world. The policy efficiently performs search, approach, grasp, transport, and drop into actions, with emerged behaviors such as re-grasping and task chaining. We conduct an extensive set of real-world experiments with ablation studies highlighting key techniques for efficient training and effective sim-to-real transfer. Additional experiments demonstrate deployment across a variety of indoor and outdoor environments. Demo videos and additional resources are available on the project page: https://horizonrobotics.github.io/gail/SLIM.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Deep whole-body control: Learning a unified policy for manipulation and locomotion,
Z. Fu, X. Cheng, and D. Pathak, “Deep whole-body control: Learning a unified policy for manipulation and locomotion,” in Conference on Robot Learning , 2022
work page 2022
-
[2]
ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation,
N. Yokoyama, A. W. Clegg, J. Truong, E. Undersander, J. Yang, S. Arnaud, S. Ha, D. Batra, and A. Rai, “ASC: Adaptive Skill Coordination for Robotic Mobile Manipulation,” IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 779–786, 2024
work page 2024
-
[3]
RoboDuet: Learning a cooperative policy for whole-body legged loco- manipulation,
G. Pan, Q. Ben, Z. Yuan, G. Jiang, Y . Ji, J. Pang, H. Liu, and H. Xu, “RoboDuet: Learning a cooperative policy for whole-body legged loco- manipulation,” IEEE Robotics and Automation Letters , vol. 10, no. 5, pp. 4564–4571, 2025
work page 2025
-
[4]
Visual whole-body control for legged loco-manipulation,
M. Liu, Z. Chen, X. Cheng, Y . Ji, R. Yang, and X. Wang, “Visual whole-body control for legged loco-manipulation,” in Conference on Robot Learning , 2024
work page 2024
-
[5]
Learning generalizable feature fields for mobile manipulation,
R.-Z. Qiu, Y . Hu, G. Yang, Y . Song, Y . Fu, J. Ye, J. Mu, R. Yang, N. Atanasov, S. Scherer, and X. Wang, “Learning generalizable feature fields for mobile manipulation,” CoRR, vol. arXiv:2403.07563, 2024
arXiv 2024
-
[6]
GAMMA: Graspability-aware mobile manipulation policy learning based on online grasping pose fusion,
J. Zhang, N. Gireesh, J. Wang, X. Fang, C. Xu, and W. Chen, “GAMMA: Graspability-aware mobile manipulation policy learning based on online grasping pose fusion,” in IEEE International Conference on Robotics and Automation , 2024
work page 2024
-
[7]
Legged robots for object manipulation: A review,
Y . Gong, G. Sun, A. Nair, A. Bidwai, R. CS, J. Grezmak, G. Sartoretti, and K. A. Daltorio, “Legged robots for object manipulation: A review,” Frontiers in Mechanical Engineering , vol. V olume 9 - 2023, 2023
work page 2023
-
[8]
Next generation legged robot locomotion: A review on control techniques,
S. S. K. et al., “Next generation legged robot locomotion: A review on control techniques,” Heliyon, vol. 10, no. 18, p. e37237, 2024
work page 2024
Show all 25 references
-
[9]
Versatile multicontact planning and control for legged loco-manipulation,
J.-P. Sleiman, F. Farshidian, and M. Hutter, “Versatile multicontact planning and control for legged loco-manipulation,” Science Robotics , vol. 8, no. 81, p. eadg5014, 2023
2023
-
[10]
Combining learning-based locomotion policy with model-based manipulation for legged mobile manipulators,
Y . Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter, “Combining learning-based locomotion policy with model-based manipulation for legged mobile manipulators,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 2377–2384, 2022
2022
-
[11]
Manipulator as a tail: Promoting dynamic stability for legged locomotion,
H. Huang, A. Loquercio, A. Kumar, N. Thakkar, K. Goldberg, and J. Malik, “Manipulator as a tail: Promoting dynamic stability for legged locomotion,” in IEEE International Conference on Robotics and Automation, 2024
2024
-
[12]
Learning whole-body manipulation for quadrupedal robot,
S. Jeon, M. Jung, S. Choi, B. Kim, and J. Hwangbo, “Learning whole-body manipulation for quadrupedal robot,” IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 699–706, 2024
2024
-
[13]
Learning to open and traverse doors with a legged manipulator,
M. Zhang, Y . Ma, T. Miki, and M. Hutter, “Learning to open and traverse doors with a legged manipulator,” in Conference on Robot Learning , 2024
2024
-
[14]
Helpful DoggyBot: Open-world object fetching using legged robots and vision-language models,
Q. Wu, Z. Fu, X. Cheng, X. Wang, and C. Finn, “Helpful DoggyBot: Open-world object fetching using legged robots and vision-language models,” CoRR, vol. arXiv:2410.00231, 2024
2024 arXiv
-
[15]
Benchmarking cluttered robot pick-and- place manipulation with the box and blocks test,
A. S. Morgan, K. Hang, W. G. Bircher, F. M. Alladkani, A. Gandhi, B. Calli, and A. M. Dollar, “Benchmarking cluttered robot pick-and- place manipulation with the box and blocks test,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 454–461, 2020
2020
-
[16]
Walk these ways: Tuning robot control for generalization with multiplicity of behavior,
G. B. Margolis and P. Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” in Conference on Robot Learning, 2022
2022
-
[17]
Secant: Self-expert cloning for zero-shot generalization of visual policies,
L. Fan, G. Wang, D.-A. Huang, Z. Yu, L. Fei-Fei, Y . Zhu, and A. Anandkumar, “Secant: Self-expert cloning for zero-shot generalization of visual policies,” in International Conference on Machine Learning , 2021
2021
-
[18]
Understanding and preventing capacity loss in reinforcement learning,
C. Lyle, M. Rowland, and W. Dabney, “Understanding and preventing capacity loss in reinforcement learning,” in International Conference on Learning Representations, 2022
2022
-
[19]
Understanding plasticity in neural networks,
C. Lyle, Z. Zheng, E. Nikishin, B. Avila Pires, R. Pascanu, and W. Dabney, “Understanding plasticity in neural networks,” in International Conference on Machine Learning , 2023
2023
-
[20]
Policy expansion for bridging offline-to- online reinforcement learning,
H. Zhang, W. Xu, and H. Yu, “Policy expansion for bridging offline-to- online reinforcement learning,” in International Conference on Learning Representations, 2023
2023
-
[21]
Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning , 2018
2018
-
[22]
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in Conference on Robot Learning , 2019
2019
-
[23]
PaCo: Parameter- compositional multi-task reinforcement learning,
L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “PaCo: Parameter- compositional multi-task reinforcement learning,” in Advances in Neural Information Processing Systems , 2022
2022
-
[24]
Do you need the entropy reward (in practice)?
H. Yu, H. Zhang, and W. Xu, “Do you need the entropy reward (in practice)?” CoRR, vol. arXiv:2201.12434, 2022
2022 arXiv
-
[25]
Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,
D. Yarats, I. Kostrikov, and R. Fergus, “Image augmentation is all you need: Regularizing deep reinforcement learning from pixels,” in International Conference on Learning Representations , 2021
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.