REVIEW 4 major objections 5 minor 2 references
Evaluating Reinforcement Learning Algorithms for Navigation in Simulated Robotic Quadrupeds: A Comparative Study Inspired by Guide Dog Behaviour
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that PPO, a policy-gradient reinforcement-learning algorithm with clipped updates, outperforms DQN and Q-learning for teaching a simulated quadruped robot to navigate and avoid obstacles.
desk verdict A conscientious master's thesis that reads like a research paper: the PPO-wins claim is not supported by the paper's own near-zero success rates and missing statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is PPO's clipped surrogate objective, $$\hat{L}(\$\theta$)=\mathbb{E}[\min(r(\$\theta$)\hat{A},\ \mathrm{clip}(r(\$\theta$),1-\varepsilon,1+\varepsilon)\hat{A})],$$ with probability ratio $r(\theta)=\pi_\theta(a\mid s)/\pi_{\theta_{\mathrm{old}}}(a\mid s)$. The clipping restrains how far each policy update can move, which the paper associates with stable learning, continued exploration, and fewer collisions. Against this, DQN uses a squared temporal-difference loss with experience replay and a target network, and Q-learning uses the tabular Bellman update $$Q(s,a)\leftarrow Q(s,a)+\$\alpha$(R+\gamma\max_{a'}Q(s',a')-Q(s,a))$$ on a 20-by-20 occupancy grid of the 10-by-10 meter map. The shared experimental machinery is a reward structure that penalizes collisions (-100) and falls (-50), gives graded distance-based penalties (-10 to -1), and rewards goal arrival (+100), with episodes capped near 15,000.
What would settle it
Retrain all three algorithms with the identical state encoding, for example by giving Q-learning a neural-network value function or by feeding PPO and DQN only the same 20-by-20 occupancy grid the tabular agent sees, and compare median steps to goal over the same episode budget; if PPO's margin over DQN vanishes or reverses, the reported advantage is an effect of state representation rather than algorithmic superiority.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that Proximal Policy Optimization outperforms both Deep Q-Networks and tabular Q-learning in every recorded metric for a simulated quadruped navigation task, in both a simple and a dynamic environment. The clearest result is average and median steps to the goal per episode: PPO's values are roughly 50% below DQN's, and Q-learning never reaches the goal in the dynamic environment. Reward medians in the simple environment follow the same ordering, with PPO near -125, DQN near -130, and Q-learning near -300, while the dynamic environment compresses all three rewards near -120 and leaves success rates close to zero because training stopped near 15,000 episodes instead of the intended 20,000. The paper nevertheless treats PPO as the clear winner and as a potential foundation for real-world robotic guide-dog training.
Load-bearing premise
The comparison is only as fair as its state representations: Q-learning sees a coarse 20-by-20 grid while PPO and DQN use richer sensor inputs, so the measured gap may reflect what each algorithm perceives, not which algorithm learns better.
Editorial extensions
If this is right
- PPO is the recommended starting algorithm for a simulated robotic guide-dog navigation stack; the paper states that PPO and DQN can serve as a foundation for real-world robotic guide-dog training.
- Tabular Q-learning, as configured with a 20-by-20 grid, is not competitive in dynamic environments and should be upgraded or hybridized before further use.
- DQN remains a viable alternative, performing close to PPO in the simple environment, so it is not ruled out as a candidate for future experiments.
- Any deployment claim is provisional: at roughly 15,000 episodes success rates are near zero, and the paper recommends at least 50,000 episodes plus larger batch sizes or distributed training.
- The recorded metrics, including reward per episode, steps, collisions, and success rate, give a reusable protocol for comparing navigation learners in controlled simulated environments.
Reading between the lines
- Because only Q-learning is forced through a 20-by-20 discretization while PPO and DQN receive richer sensor features, the comparison conflates algorithmic strength with state-representation quality; a neural-network variant of Q-learning with the same features could plausibly match PPO in the simple environment.
- PPO's margin may widen as obstacles move faster or become more numerous, since on-policy updates can track a non-stationary reward landscape; this can be tested directly by varying obstacle dynamics while holding everything else fixed.
- The paper's guide-dog comparison lacks a quantitative baseline from real guide dogs; adding measured success rates, crossing decisions, and reaction times from trained guide dogs would make the feasibility claim testable.
- For assistive deployment, the reward weighting may need to shift toward collision avoidance, because PPO's collision count, although lowest, still fluctuates above zero throughout training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares PPO, DQN, and tabular Q-learning for a simulated Bioloid quadruped in two custom Webots environments (simple and dynamic), using reward per episode, learning curves, steps per episode, average/median steps to goal, and collision counts. It claims PPO outperforms DQN and Q-learning across all metrics, especially average and median steps to goal, and interprets this as evidence that PPO is the most suitable algorithm for a robotic guide-dog navigation task.
Significance. The topic is timely, and the manuscript is transparent about its limited training budget (about 15,000 episodes) and about the near-zero success rates. The literature review is broad and draws on relevant quadruped navigation and reinforcement learning work. However, the central empirical claim is not supported by the reported evidence: no statistical tests are performed, success rates are near zero in both environments, and the dynamic-environment reward distributions are similar across algorithms. The contribution is therefore preliminary at best; the paper does not meet the standard for a published comparative study.
major comments (4)
- [§3.3, Fig. 7] The claim that PPO 'easily outperformed' DQN and Q in average and median steps to goal is unsupported because the success rates in Fig. 7 are reported as 'very low (close to 0)' in both environments. Average steps to goal can only be computed for episodes in which the goal is reached, and the paper does not report the number of successful episodes per algorithm, the variance of the steps-to-goal distribution, or any error bars. With near-zero success rates, the PPO advantage could rest on a handful of lucky episodes rather than a reliable algorithmic advantage, which makes the abstract's headline metric unquantified.
- [§3.3, Fig. 10] The paper's own dynamic-environment results contradict the conclusion that PPO 'outperformed both DQN and Q in all key areas.' The text around Fig. 10 states that all three algorithms show very similar performance with median rewards around -120 in the dynamic environment, with differences appearing only in outliers. The reward distributions therefore do not support the 'all key metrics' claim, and the conclusion should be substantially qualified or revised.
- [§3.2] The comparison is not shown to be algorithmically fair. Q-learning operates on a 20x20 discrete grid with 0.5 m cells and four discrete actions, while DQN and PPO use neural networks with 272,644 and 274,437 parameters, respectively. The paper does not report the state features used by DQN and PPO, does not report hyperparameter values, and does not describe the tuning process. Without this information, the observed differences could reflect representational or implementational asymmetry rather than inherent algorithmic quality.
- [§1.3 and §3.3] No statistical analysis is reported despite the paper specifying appropriate candidate tests. Section 1.3 says a difference-in-means test 'may be taken' later, and §3.1 lists t-test, pairwise t-test, and Wilcoxon test as possibilities, but §3.3 presents only boxplots and qualitative descriptions. The central comparative claim—that PPO outperforms the other algorithms—is thus not supported by confidence intervals, effect sizes, or significance tests, and the phrase 'strong evidence that PPO is learning effectively' in §3.3 is not justified by the reported analysis.
minor comments (5)
- [§3.3] The text after Fig. 4 says 'As you can see from Figure 7' when discussing reward graphs, but Fig. 7 shows average and median steps to goal; the reward graphs are Fig. 4, and other figure references are inconsistent throughout the section.
- [§3.3, Simple environment] The sentence 'Q-learning appears to have zero success, which was expected because in Fig. 7. (Right), we can see PPO does not reach the goal in the dynamic environment' mixes two different algorithms and two different environments, making the intended observation unclear.
- [§3.3, Dynamic environment] The dynamic-environment subsection begins with 'When evaluating the simple environment,' which appears to be a copy-paste error that should refer to the dynamic environment.
- [§3.2] The paper first states the simulation was capped at 20,000 episodes, then reports in §3.3 that roughly 15,000 episodes were completed; the discrepancy should be resolved for clarity.
- [§4.2] The phrase 'conversion tren ds' appears to be a typo for 'convergence trends,' and the future-work paragraph contains several spacing errors that should be corrected.
Circularity Check
No significant circularity: the paper reports an empirical benchmark of standard externally sourced RL algorithms, with no fitted parameter being relabelled as a prediction and no load-bearing self-citation.
full rationale
The manuscript contains no derivation chain that reduces to its own inputs. The three algorithms (Q-learning, DQN, PPO) are standard methods with equations quoted from the literature (Bellman update, DQN loss, PPO clipped objective), and the implementations are explicitly sourced to external public repositories (MinRL, CleanRL, geekyutao, Shahbazi, Murphy, Gambino). No parameter is fitted to the outcome data and then reported as a prediction: the metrics in Section 3.3 (reward, learning rate, steps, success rate, collisions) are direct measurements of simulation runs, and the abstract's claim that PPO 'outperformed' is a summary of those measurements, not a result forced by construction. The paper's own limitation statements, such as 'All three algorithms show very low success rates (close to 0)' and 'I have tried to run for 20,000 episodes but only ended up with roughly 15,000 episodes,' weaken the empirical support for the headline comparison, but a weak or under-powered experiment is not circular reasoning. The expectation that PPO would win is stated before the results and is not used as evidence; it is a hypothesis, not an input to the measurement. There are no self-citations, no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result. The only arguable issue is statistical robustness (near-zero success rates, no significance test), which is a correctness or validity concern, not a circularity concern under the hard rules.
Assumptions & free parameters
free parameters (5)
- Goal reward =
+100
- Collision penalty =
-100
- Fall penalty =
-50
- Proximity punishment =
-10 to -1
- DQN and PPO hyperparameters (learning rate, gamma, epsilon, batch size) =
not reported
assumptions (5)
- standard math Bellman updates converge to the optimal Q-function in tabular settings when learning rates decay appropriately.
- standard math PPO's clipped surrogate objective provides stable policy improvement.
- domain assumption The Webots simulator accurately models the Bioloid quadruped and its sensors for navigation research.
- domain assumption The hand-designed reward function captures the objective of safe and efficient guide-dog-like navigation.
- domain assumption The three algorithms were implemented and tuned fairly, so performance differences reflect algorithmic merit rather than implementation quality.
Cite this review
Pith. "Pith review of Evaluating Reinforcement Learning Algorithms for Navigation in Simulated Robotic Quadrupeds: A Comparative Study Inspired by Guide Dog Behaviour." pith.science (2026). https://pith.science/paper/AD3NXYXU
@misc{pith2026250713277,
author = {Pith},
title = {Pith review of: Evaluating Reinforcement Learning Algorithms for Navigation in Simulated Robotic Quadrupeds: A Comparative Study Inspired by Guide Dog Behaviour},
year = {2026},
howpublished = {\url{https://pith.science/paper/AD3NXYXU}},
note = {Machine review of arXiv:2507.13277}
}
read the original abstract
Robots are increasingly integrated across industries, particularly in healthcare. However, many valuable applications for quadrupedal robots remain overlooked. This research explores the effectiveness of three reinforcement learning algorithms in training a simulated quadruped robot for autonomous navigation and obstacle avoidance. The goal is to develop a robotic guide dog simulation capable of path following and obstacle avoidance, with long-term potential for real-world assistance to guide dogs and visually impaired individuals. It also seeks to expand research into medical 'pets', including robotic guide and alert dogs. A comparative analysis of thirteen related research papers shaped key evaluation criteria, including collision detection, pathfinding algorithms, sensor usage, robot type, and simulation platforms. The study focuses on sensor inputs, collision frequency, reward signals, and learning progression to determine which algorithm best supports robotic navigation in complex environments. Custom-made environments were used to ensure fair evaluation of all three algorithms under controlled conditions, allowing consistent data collection. Results show that Proximal Policy Optimization (PPO) outperformed Deep Q-Network (DQN) and Q-learning across all metrics, particularly in average and median steps to goal per episode. By analysing these results, this study contributes to robotic navigation, AI and medical robotics, offering insights into the feasibility of AI-driven quadruped mobility and its role in assistive robotics.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
House of Commons Library (2024/2025) UK disability statistics: Prevalence and life experiences. Available at: https://commonslibrary.parliament.uk/research-briefings/cbp-9602/ (Accessed: 27/02/2025). Erin, B., Abiyeva, R. & Ibrahim, D. (2010). Teaching robot navigation in the presence of obstacles using a computer simulation program, Procedia - Social and...
arXiv 2010
-
[2006]
Learning Forward Dynamics Model and Informed Trajectory Sampler for Safe Quadruped Navigation
Available at: https://web.eecs.umich.edu/~honglak/icra06_quadruped.pdf (Accessed: 31/01/2025). Vásconez, J. P., Basoalto, F., Briceño, I. C., Pantoja, J. M., Larenas, R. A., Rios, J. H., & Castro, F. A. (2023). Comparison of path planning methods for robot navigation in simulated agricultural environments, Procedia Computer Science, 220, pp. 898–903. Ava...
work page Pith review arXiv 2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.