REVIEW 4 major objections 5 minor 1 cited by
Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A zero-training high-level policy that optimizes the low-level value function outperforms end-to-end RL on rough terrain.
desk verdict The HLP objective as written has V(s_t) independent of p, so the value function plays no role in the optimization — the gains are from the directional term. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the low-level policy's goal-conditioned value function $V(s_t, p)$, which estimates the expected discounted sum of future rewards for the current state $s_t$ and a candidate eight-dimensional vector $p$ of next footstep targets. The high-level policy's objective is $R_{HLP} := V(s_t) + \kappa_{HD}H(p)$, where $H$ is a directional heading term that promotes progress along a desired angle, and the weight $\kappa_{HD}$ balances safety against forward motion. The optimization is solved by grid-search-initialized gradient ascent: an 8-dimensional grid over $[-B,B]^8$ is scored by $R_{HLP}$, and the best grid point seeds $N$ gradient ascent steps. The value function is obtained once during low-level training with an on-policy actor-critic method, so the high level requires no additional learning or environment samples.
What would settle it
Take a held-out set of footstep targets sampled both inside and outside the training distribution, roll out the low-level policy to each target, and record the actual discounted return; if the value function's ranking of targets does not track the actual ranking, then the grid-search and gradient ascent optimization in Algorithm 1 would select poor or unsafe foot placements despite the paper's good average results.
Extended reading notes
Core claim
The central claim is that a high-level policy that selects footstep targets by optimizing the low-level policy's learned value function, without additional training or environment samples, yields higher normalized rewards and fewer collisions than an end-to-end reinforcement learning baseline on challenging terrain, including terrain harder than the training distribution. In head-to-head runs, the proposed method beats the end-to-end baseline on 10 of 12 test terrains by normalized reward, exceeds its own training-time reward on 8 of 12 terrains, and avoids the near-constant collision counts that the baseline suffers in two gap-heavy terrains. The method travels shorter distances in most environments, which the authors attribute to its preference for high-value, safe footstep targets over aggressive forward progress.
Load-bearing premise
The high-level policy's choices are only as reliable as the low-level value function's predictions for candidate footstep targets, including targets far outside the training distribution, and the paper does not validate that accuracy.
Editorial extensions
If this is right
- A trained goal-conditioned locomotion policy becomes directly reusable for planning: the high-level layer adds a planner without any extra neural network training.
- Value-function maximization acts as a safety prior: it avoids terrain gaps and other low-value placements, which is consistent with the collision reduction on gap terrains.
- The framework is modular: the low-level policy could be swapped for a model-based controller or a different gait without changing the high-level optimization.
- Because the high-level policy is training-free, the added computational cost is only the grid search and gradient ascent, making it a lightweight addition to an existing policy at deployment.
Reading between the lines
- If the value function is well calibrated for out-of-distribution goals, the same 'optimize the critic' recipe could transfer to other goal-conditioned skills such as manipulation or navigation, where candidate goal sets are easy to enumerate.
- A decisive check on robustness would be to compare predicted value with actual rollout return for random footstep targets inside and outside the training box; if the rank correlation is weak, the grid search could still pick unsafe targets even though the paper reports strong averages.
- The directional heading weight $\kappa_{HD}$ and the grid bounds $B$ together encode a prior about how far and how fast the robot should move; per-terrain tuning of these two scalars could trade speed against safety beyond the fixed settings used here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hierarchical reinforcement learning framework for quadruped locomotion. A goal-conditioned low-level policy (LLP) is trained with PPO to hit randomized footstep targets using a PMTG action space, and a high-level policy (HLP) selects footstep target locations online by optimizing the LLP's learned value function plus a directional heading term, without additional training or environment samples (Section III, Algorithm 1). The authors compare the method with an end-to-end RL baseline on simulated terrains with varying infill and height variation, reporting higher normalized rewards in 10/12 terrains, lower collision counts in several environments, and better extrapolation to terrains harder than the training distribution (Section IV).
Significance. If the central mechanism is validated, the contribution is genuinely useful: a modular hierarchy in which the high-level planner is obtained without separate training and does not require extra environment samples is an attractive design. The paper also ships code and video resources, which supports reproducibility. However, the current manuscript does not establish that the reported gains come from value-based goal optimization: the objective in Equation 12 is written in a way that makes the optimization independent of the learned value function, the reward coefficients are never reported, and no ablation separates the value term from the hand-tuned directional term. These issues bear directly on the paper's central claim, so the significance cannot be assessed from the present text.
major comments (4)
- [Section III-B, Eq. (12) and Algorithm 1] The central claim is that the HLP improves performance by optimizing the low-level value function over candidate footstep targets, but Equation 12 defines RHLP := V(s_t) + κ_HD H(p), with V evaluated at s_t only, and Algorithm 1 lines 8 and 17 repeat this same expression. If the equations are implemented literally, the optimization over p reduces to maximizing the directional term H(p) alone, because V(s_t) is constant with respect to p. The paper must either correct the notation to V(s_t, p), with the appropriate dependence on the candidate goal p in Eq. (12) and in Algorithm 1, or explicitly acknowledge that the reported improvements arise from the heuristic heading term rather than from value-based goal selection. As written, the experiments in Section IV do not demonstrate the claimed value-optimization mechanism.
- [Section II-D and Section IV-C] The numerical values of the reward weights κ_FT, κ_VT, κ_S, κ_SL, κ_FS, and κ_C are never given, while κ_HD is reported as 50.0. Without these values, the relative scale of V(s_t) and κ_HD H(p) in Eq. (12) is unknown, so it is possible that the directional term dominates the objective and the value function is effectively ignored. The authors should report all reward coefficients and include a sensitivity analysis or at least a comparison of the two terms' typical magnitudes, since this is required to interpret the optimization in Section III-C.
- [Section IV-C, Figures 4-6] Each bar in Figures 4, 5, and 6 represents the average of only five rollouts, with no error bars, standard deviations, or significance tests. Given that the end-to-end baseline appears to get stuck in holes in some environments, a single outlier rollout can dominate the reported averages. The paper should add more rollouts, report per-trial variability, and, if possible, perform a statistical test or at least show individual trial outcomes before claiming robust improvements on 10/12 terrains.
- [Section III-C and Section IV-C] There is no ablation that isolates the contribution of the learned value function from the directional term H. The logical experiment is to compare four variants on the same test terrains: (i) the full objective V + κ_HD H, (ii) κ_HD H alone, (iii) V alone, and (iv) a baseline using random footstep targets. Without such an ablation, even a corrected Equation 12 would not establish that the value function, rather than the forward bias, is responsible for the observed reward and collision improvements. Additionally, the paper should validate that V(s_t, p) is accurate for candidate goals p that are far outside the training distribution, since Algorithm 1's grid search and gradient ascent explore such goals.
minor comments (5)
- [Section III-C, Lemma III.1] The lemma's statement and proof are not connected to the paper's algorithm: the inequality is essentially a consequence of the definition of 'best over G', and the phrase 'the rest of the proof is trivial' is not a rigorous argument. Either remove the lemma or replace it with a precise statement that is actually used in the analysis of Algorithm 1.
- [Section IV-C and Figure 5] The proposed method travels a shorter distance than the end-to-end policy in all but two environments, which is explained as conservatism. This is a notable discrepancy with the reward improvements and deserves more discussion, for example by reporting success rates for traversing the full terrain, not only per-timestep reward.
- [Section II-D, Eq. (4)] The explanation of the factor that 'triples' the per-foot reward is confusing: the multiplicative factor (2.0 ∏ h_i,t + 1) equals 3 only when both indicators are 1, but the surrounding text reads as if the factor is always applied. Please clarify that the tripling occurs only when both feet hit simultaneously.
- [Section IV-C, paragraph 1] The definition of normalized reward is vague: it says rewards are normalized by the maximum reward achieved during training, but it is not stated whether this is the maximum per-timestep reward, the maximum discounted return, or a rolling training maximum. Please specify the normalization exactly.
- [Throughout] There are several typos and minor wording issues, such as 'comparision' in the introduction and the inconsistent use of 'P' versus 'p' in Section III-C. A careful proofreading pass would improve readability.
Circularity Check
No significant circularity: Eq. 12's apparent p-independence dissolves because p is part of the goal-conditioned observation, and the headline reward is measured in fresh rollouts against an external end-to-end baseline.
full rationale
The central claim — that a training-free HLP selecting footstep goals via the LLP's learned value function improves reward and reduces collisions relative to an end-to-end baseline — is tested with fresh rollouts on held-out terrains against an external comparator, so it is not circular. The strongest candidate for a by-construction reduction is the form of Eq. 12, RHLP := V(s_t) + κ_HD H(p), which read literally seems to make V constant in the optimization variable p. That reading fails under the paper's own definitions: the footstep-target vector p is part of the observation (Section II-C: "p = {p1,x, p1,y, ..., p4,y} ∈ R^8 gives the x and y distances from each foot to the next ... footstep targets"), the HLP action space is exactly this p (Section III-A: "A_HLP := p ⊂ O"), and both actor and critic take "the same input consisting of goal footstep target locations and robot observations" (Section II). Hence V(s_t) varies with the candidate p through the goal-conditioned observation, and the grid-search/gradient-ascent in Algorithm 1 is not vacuous. The headline metric (Section IV-C1) is the Section II-D per-timestep reward measured in new rollouts, normalized by the training maximum; maximizing a learned, imperfect estimate of the discounted version of that reward is the standard goal-conditioned RL evaluation loop, not an identity, and the end-to-end baseline uses a different reward function, so the comparison is confounded but not forced. No fitted parameter is renamed as a prediction (the reward coefficients κ_FT, κ_VT, κ_S, κ_SL, κ_FS, κ_C are never reported, which weakens reproducibility but is not circularity), and no load-bearing self-citations appear (PMTG [7], terrain [8], ALLSTEPS [16], QT-Opt [18], and offline HRL [19] are all external). Two non-circular weaknesses should be weighed separately: (i) Lemma III.1 is near-tautological and its proof is omitted ("the rest of the proof is trivial"); as stated it also requires the grid to cover the random-initialization support, so it is a correctness concern rather than circularity; (ii) the value function's accuracy on out-of-distribution candidate goals is never validated and no ablation isolates the κ_HD H term, so the value function's causal contribution to the reported gains is undemonstrated. These are evidence gaps, not circular reductions.
Assumptions & free parameters
free parameters (4)
- Reward weights kappa_FT, kappa_VT, kappa_S, kappa_SL, kappa_FS, kappa_C =
unreported
- kappa_HD (directional weight) =
50.0
- Optimizer hyperparameters (B, R, N, eta) =
B=15 cm, R=5, N=5, eta=1e-4
- Footstep hit threshold d_hit and force threshold =
7.5 cm, 5 N
assumptions (4)
- domain assumption PPO converges to a near-optimal policy for the LLP
- domain assumption The learned value function generalizes to out-of-distribution goal queries
- domain assumption Isaac Gym simulation faithfully models the Aliengo robot and terrain
- ad hoc to paper The lemma's inequality holds for the objective f
Cite this review
Pith. "Pith review of Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion." pith.science (2026). https://pith.science/paper/WU537KON
@misc{pith2026250620036,
author = {Pith},
title = {Pith review of: Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion},
year = {2026},
howpublished = {\url{https://pith.science/paper/WU537KON}},
note = {Machine review of arXiv:2506.20036}
}
read the original abstract
We propose a novel hierarchical reinforcement learning framework for quadruped locomotion over challenging terrain. Our approach incorporates a two-layer hierarchy in which a high-level policy (HLP) selects optimal goals for a low-level policy (LLP). The LLP is trained using an on-policy actor-critic RL algorithm and is given footstep placements as goals. We propose an HLP that does not require any additional training or environment samples and instead operates via an online optimization process over the learned value function of the LLP. We demonstrate the benefits of this framework by comparing it with an end-to-end reinforcement learning (RL) approach. We observe improvements in its ability to achieve higher rewards with fewer collisions across an array of different terrains, including terrains more difficult than any encountered during training.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
PUMA: Perception-driven Unified Foothold Prior for Mobility Augmented Quadruped Parkour
An end-to-end RL quadruped parkour controller that estimates an egocentric polar foothold prior (distance and heading) from depth and proprioception beats Cartesian foothold priors and prior baselines in simulation an...
Reference graph
Works this paper leans on
-
[19]
Hierarchical plan- ning through goal-conditioned offline reinforcement learning,
J. Li, C. Tang, M. Tomizuka, and W. Zhan, “Hierarchical plan- ning through goal-conditioned offline reinforcement learning,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 216–10 223, 2022
2022
-
[1]
Learning to walk via deep reinforcement learning,
T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,”arXiv preprint arXiv:1812.11103, 2018
arXiv 2018
-
[2]
Learning to walk in the real world with minimal human effort,
S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,”arXiv preprint arXiv:2002.08550, 2020
arXiv 2002
-
[3]
Learning to walk in minutes using massively parallel deep reinforcement learning,
N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in Conference on Robot Learning. PMLR, 2022, pp. 91–100
2022
-
[4]
Learning fast adapta- tion with meta strategy optimization,
W. Yu, J. Tan, Y . Bai, E. Coumans, and S. Ha, “Learning fast adapta- tion with meta strategy optimization,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2950–2957, 2020
work page 2020
-
[5]
Legged locomotion in challenging terrains using egocentric vision,
A. Agarwal, A. Kumar, J. Malik, and D. Pathak, “Legged locomotion in challenging terrains using egocentric vision,” inConference on robot learning. PMLR, 2023, pp. 403–415
2023
-
[6]
Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,
R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang, “Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers,”arXiv preprint arXiv:2107.03996, 2021
arXiv 2021
-
[7]
Policies modulating trajectory generators,
A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V . Sindhwani, and V . Vanhoucke, “Policies modulating trajectory generators,” in Conference on Robot Learning. PMLR, 2018, pp. 916–926
work page 2018
Show all 22 references
-
[8]
Learning quadrupedal locomotion over challenging terrain,
J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,”Science robotics, vol. 5, no. 47, p. eabc5986, 2020
2020
-
[9]
Visual-locomotion: Learning to walk on complex terrains with vision,
W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang, “Visual-locomotion: Learning to walk on complex terrains with vision,” in5th Annual Conference on Robot Learning, 2021
2021
-
[10]
Zero-shot terrain generalization for visual locomotion policies,
A. Escontrela, G. Yu, P. Xu, A. Iscen, and J. Tan, “Zero-shot terrain generalization for visual locomotion policies,”arXiv preprint arXiv:2011.05513, 2020
2011 arXiv
-
[11]
Learning agile locomotion skills with a mentor,
A. Iscen, G. Yu, A. Escontrela, D. Jain, J. Tan, and K. Caluwaerts, “Learning agile locomotion skills with a mentor,” in2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 2019–2025
2021
-
[12]
Learning agile robotic locomotion skills by imitating animals,
X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,”arXiv preprint arXiv:2004.00784, 2020
2004 arXiv
-
[13]
Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning,
S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis, “Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning,” in2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021, pp. 5973–5979
2021
-
[14]
Guided constrained policy optimization for dynamic quadrupedal robot locomotion,
S. Gangapurwala, A. Mitchell, and I. Havoutis, “Guided constrained policy optimization for dynamic quadrupedal robot locomotion,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3642–3649, 2020
2020
-
[15]
Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning,
V . Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter, “Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning,”IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3699–3706, 2020
2020
-
[16]
Allsteps: Curriculum-driven learning of stepping stone skills,
Z. Xie, H. Y . Ling, N. H. Kim, and M. van de Panne, “Allsteps: Curriculum-driven learning of stepping stone skills,” inComputer Graphics F orum, vol. 39, no. 8. Wiley Online Library, 2020, pp. 213–224
2020
-
[17]
Learning gen- eralizable locomotion skills with hierarchical reinforcement learning,
T. Li, N. Lambert, R. Calandra, F. Meier, and A. Rai, “Learning gen- eralizable locomotion skills with hierarchical reinforcement learning,” in2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020, pp. 413–419
2020
-
[18]
Scalable deep reinforcement learning for vision-based robotic manipulation,
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhouckeet al., “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Conference on robot learning. PMLR, 2018, pp. 651–673
2018
-
[20]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[21]
High- dimensional continuous control using generalized advantage estima- tion,
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, “High- dimensional continuous control using generalized advantage estima- tion,”arXiv preprint arXiv:1506.02438, 2015
2015 arXiv
-
[22]
Isaac gym: High performance gpu-based physics simulation for robot learning,
V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Mack- lin, D. Hoeller, N. Rudin, A. Allshire, A. Handaet al., “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470, 2021
2021 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.