Pith. sign in

REVIEW 3 major objections 5 minor 23 references

Vision-based Navigation Using Deep Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding depth-map and segmentation prediction as auxiliary tasks to batched A2C makes target-image navigation in indoor scenes converge in roughly 500,000 frames on AI2-THOR and roughly 3,000,000 frames on SUNCG houses.

desk verdict A useful architecture for target-driven visual navigation with public code, but the headline speedups don't survive scrutiny because the baselines were not trained under matched conditions. read the letter →

arxiv 1908.03627 v2 pith:SRR6IURQ submitted 2019-08-08 cs.RO cs.AIcs.LGstat.ML

Jonáš Kulhánek , Erik Derner , Tim de Bruin , Robert Babuška This is my paper · ORCID
classification cs.ROcs.AIcs.LGstat.ML
keywords visualnavigationdeepreinforcementlearningauxiliarytasksactor-criticA2Ctarget-imageindoorAI2-THOR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an image-driven reinforcement-learning agent can navigate to a pictured target in realistic indoor environments much faster if the network is trained with auxiliary visual tasks alongside the main navigation objective. The proposed A2CAT-VN method extends the batched A2C actor-critic algorithm with the two standard UNREAL auxiliary tasks and adds three navigation-specific ones: predicting the depth map of the current observation, segmenting the observed image, and segmenting the target image. These side tasks make it possible to pre-train most of the network by supervised learning before reinforcement learning, and a curriculum that gradually starts episodes farther from the target is also used. If this is right, the practical cost of learning visual navigation drops from millions of environment frames to roughly hundreds of thousands on the tested simulators, and the same compact architecture can handle several targets and several rooms at once.

What carries the argument

The load-bearing mechanism is the shared network with auxiliary prediction heads: a two-stream convolutional base (one stream for the observation image, one for the target image) feeds an LSTM, and the LSTM features drive the actor, critic, pixel-control head, reward-prediction head, and three visual-navigation heads. The visual-navigation heads reconstruct the depth map, segment the observed scene, and segment the target image through deconvolutional layers attached to the last convolutional layer. Because these tasks are supervised, they permit the bottom of the network to be pre-trained on cheap pre-collected observations, and they keep the target image actively used during training. The LSTM addresses partial observability, which the paper shows outperforms feeding the past four frames.

What would settle it

Run the same SUNCG training with the same pre-training and complexity schedule while deleting only the three visual-navigation auxiliary heads (keeping the UNREAL tasks); if the agent still reaches average episode length 200 in roughly 3,000,000 frames, the claimed speedup is not caused by the new tasks. A second check: retrain the target-driven baseline on the exact same AI2-THOR environments and action set as A2CAT-VN; if it converges comparably fast, the reported order-of-magnitude difference disappears.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that adding visual auxiliary tasks to a batched A2C agent yields a large and fast convergence improvement for target-image navigation. In AI2-THOR, A2CAT-VN reaches a near-optimal policy after roughly 500,000 frames, while the target-driven navigation baseline from the literature does not converge within 7,000,000 steps in the same (larger, harder) environments. In SUNCG houses, the method reaches an average episode length of 200 in about 3,000,000 frames, compared with about 8,000,000 frames when only the UNREAL auxiliary tasks are used. The paper interprets the result as evidence that depth-map, observation-segmentation, and target-segmentation prediction give the shared convolutional layers good features early, and that supervised pre-training plus gradual environment-complexity increase further shortens training.

Load-bearing premise

The load-bearing premise is that the faster convergence comes from the new visual auxiliary tasks, but the current experiments vary pre-training, curriculum, and environment difficulty at the same time, so the credited mechanism is not actually isolated from those other changes.

Editorial extensions

If this is right

  • On the AI2-THOR comparison, A2CAT-VN reaches a near-optimal policy after roughly 500,000 frames, whereas the target-driven navigation baseline has not converged after 7,000,000 frames.
  • On SUNCG houses, the full method reaches an average episode length of 200 after about 3,000,000 frames, versus about 8,000,000 frames with only the UNREAL auxiliary tasks.
  • The three visual-navigation auxiliary tasks make supervised pre-training possible for most of the network, which is what the paper credits for the reduced training time.
  • The environment-complexity curriculum, which starts episodes close to the target and gradually allows farther starts, is presented as a further stabilising factor.
  • The resulting architecture navigates from raw images alone, without hand-crafted features or an explicit map, and can pursue multiple targets in multiple environments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate the contribution of the three new auxiliary tasks from the supervised pre-training and the complexity curriculum; a controlled ablation that varies only the auxiliary-task heads would settle that, and the paper itself lists such an ablation as future work.
  • If the advantage over the target-driven baseline depends on the baseline being run on larger and harder environments than it was designed for, the headline speedup could shrink in an exactly matched comparison; this is an open question, not something the current experiments answer.
  • A practical consequence, if the auxiliary tasks are what matters, is that any simulator offering depth and segmentation can be used to bootstrap a navigation policy before real-world deployment, but real robots without those labels would need a substitute or a transfer step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes A2CAT-VN, a reinforcement learning architecture for vision-based navigation in which the batched A2C algorithm is augmented with auxiliary tasks: depth-map prediction, observation image segmentation, and target image segmentation. The auxiliary tasks are also used for supervised pre-training of the convolutional base. An environment-complexity curriculum that gradually increases the distance between initial and target states is introduced. The method is evaluated in DeepMind Lab, AI2-THOR (including a continuous variant), and House3D with SUNCG. The central empirical claims are that A2CAT-VN converges roughly an order of magnitude faster than the target-driven navigation baseline of Zhu et al. (ICRA 2017) on AI2-THOR, and that the visual-navigation auxiliary tasks reduce the number of training frames by about a factor of two on SUNCG.

Significance. If the reported results hold under controlled comparison, the paper would make a useful contribution to deep RL for navigation: it demonstrates a concrete architecture with three navigation-specific auxiliary tasks, an LSTM-based solution to partial observability, and a simple pre-training strategy, all validated in realistic indoor simulators. The authors release their implementation and framework on GitHub, which strengthens reproducibility. However, the experimental support for the headline claims is weakened by uncontrolled comparisons. The work is potentially publishable after a major revision that isolates the contributions of the auxiliary tasks, pre-training, and curriculum, and that replays the AI2-THOR baseline on matched environments.

major comments (3)
  1. [Section IV-E, Fig. 9] The comparison with the baseline of Zhu et al. [3] is uncontrolled. The authors state that the environments chosen for this experiment "were bigger and more difficult to navigate than those used in [3]" but came from the same AI2-THOR simulator. Because the baseline is evaluated only on scenes outside the regime where it was originally validated, the faster convergence of A2CAT-VN (about 5e5 frames to a solution versus no convergence for [3] in 7e6 steps) cannot be unambiguously attributed to the proposed algorithm. To support the claim "outperforms state-of-the-art," the authors need to run both methods on identical scenes with identical action sets, reward definitions, and evaluation protocols, or otherwise demonstrate that the difficulty mismatch does not account for the gap. As written, the result is consistent with an alternative explanation: the baseline may simply be slower on harder scenes.
  2. [Section IV-G, Fig. 11] The SUNCG ablation conflates at least three independent variables: the three visual-navigation auxiliary tasks, the supervised pre-training of the network, and the environment-complexity curriculum. The text reports that A2CAT-VN was pre-trained on approximately 20,000 images and that the environment complexity was linearly increased from 0.3 at step 5M to 1.0 at step 10M, while the comparison baseline A2C+UNREAL is not described as receiving either. The factor-of-two improvement in frames to reach an average episode length of 200 (3e6 vs. 8e6) may therefore be due partly or entirely to pre-training or the curriculum, rather than to the proposed auxiliary tasks. The paper itself lists an ablation study of the individual auxiliary tasks as future work; given that the central claim of Section IV-G is about the auxiliary tasks, this ablation is necessary for the claim to be supported. A controlled experiment should train all variants with identical pre-training, curriculum, and hyperparameters, and ideally ablate each auxiliary task separately.
  3. [Table I and Section IV-G] There is a numerical inconsistency in the pre-training dataset size. Table I lists the pre-training dataset size as 2e5 images, while Section IV-G states "The number of images we used for pre-training was approximately 20,000." This order-of-magnitude discrepancy directly affects the interpretability of the SUNCG ablation (the contribution of pre-training may be much smaller than implied by the table) and must be corrected for reproducibility. Please state the exact dataset size and reconcile the table with the text.
minor comments (5)
  1. [Section IV-D, Fig. 8] The conclusion that LSTM "clearly outperforms" frame stacking is based on a single training run. Reporting multiple seeds with error bars (e.g., standard deviation or confidence intervals) would make the claim more robust, especially because RL training curves are noisy.
  2. [Section IV-E, Fig. 9] The statement that A2CAT-VN "found the optimal solution after approximately 5e5 frames" is not accompanied by a quantitative success criterion. Please define "optimal" (for example, an episode-length or success-rate threshold) in the text or figure caption.
  3. [Section IV-F, Fig. 10] The continuous AI2-THOR experiment uses a single scene and no comparison baseline. The claim that the agent can "navigate in non-stationary environments" would be strengthened by showing results on more scenes or by comparing to a non-continuous baseline in the same setting.
  4. [Abstract and Section IV-E] The abstract states that the method "outperforms state-of-the-art goal-oriented visual navigation methods from the literature," but the only direct comparison on AI2-THOR is to [3]. Please qualify the claim to refer to the specific baseline and setting evaluated.
  5. [Section IV-C, Table I] The learning-rate schedule is written as 7e-4 * (fmax - f)/fmax without a description of when f reaches fmax or whether the rate is annealed to zero. A brief sentence clarifying the schedule would help reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: A2CAT-VN is an empirical method study whose main comparisons are against external baselines ([3], [7]), and no fitted parameter is relabeled as a prediction or forced by a self-citation chain.

full rationale

The paper does not present a formal derivation in which a claimed prediction reduces by construction to its inputs. The auxiliary-task losses, pre-training procedure, and curriculum are all training-time inputs; the evaluation metrics are episode length and return measured in external simulators, so the results are not definitionally equal to the training objectives. The comparison with [3] on AI2-THOR and with A2C+UNREAL on SUNCG are empirical, externally grounded benchmarks; although the SUNCG ablation co-varies pre-training and environment-complexity curriculum with the novel auxiliary tasks, and the AI2-THOR comparison uses scenes described as harder than those in [3], these are experimental confounds rather than circular reductions. The self-citations ([12], [21]) are used for background or minor design choices and are not load-bearing for the central claim. The minor inconsistency in the reported pre-training dataset size (2e5 in Table I vs. approximately 20,000 in Section IV-G) is a reporting error, not a circularity. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper relies on standard RL and CNN methodology and simulator realism; no new entities are postulated. Several hyperparameters and the curriculum schedule are chosen by hand, and the comparisons depend on assumptions of comparable environments and clean ablations.

free parameters (5)
  • Auxiliary task loss weights = depth 0.1, observation segmentation 0.1, target segmentation 0.1, pixel control 0.05, reward prediction 1.0
    Chosen by hand and experimental tuning; these weights control the balance of learning signals and are central to the claimed benefit of the auxiliary tasks.
  • Environment complexity schedule = tau linear from 0.3 at 5M frames to 1.0 at 10M frames
    Curriculum hyperparameters introduced ad hoc; strongly affect training dynamics and convergence, and are not grounded in theory.
  • Pre-training dataset size = 2e5 in Table I; approximately 20,000 in Section IV-G
    Inconsistent specification across the paper; pre-training is claimed to reduce training steps substantially.
  • Learning rate schedule = initial 7e-4, linear decay with frames
    Tuned experimentally; standard in A2C and UNREAL implementations.
  • Maximum episode length and rollout length = 900 and 20
    Chosen by hand; affect return estimates and training stability.
assumptions (5)
  • domain assumption AI2-THOR and SUNCG/House3D are representative of realistic indoor visual navigation
    The central empirical claims are based entirely on these simulators (Sections IV-A, IV-E, IV-F, IV-G); transfer to physical robots is not demonstrated.
  • domain assumption Sparse reward (+1 at target, 0 otherwise) is an adequate learning signal when combined with auxiliary tasks
    The method's success depends on auxiliary tasks compensating for sparse rewards (Introduction and Section IV-C).
  • domain assumption LSTM memory resolves partial observability better than frame stacking
    Shown experimentally on DeepMind Lab (Fig. 8), but assumed to transfer to other environments; no theoretical guarantee.
  • domain assumption The supervised pre-training distribution matches the RL training distribution
    Pre-training on SUNCG data is used to initialize features for SUNCG training (Section IV-G); transfer to other environments is not tested.
  • standard math Batched A2C with n-step returns and off-policy critic updates converges to good policies
    Standard RL algorithm from [11], [17]; the paper builds on it without proving convergence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision-based Navigation Using Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/SRR6IURQ

@misc{pith2026190803627,
  author       = {Pith},
  title        = {Pith review of: Vision-based Navigation Using Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SRR6IURQ}},
  note         = {Machine review of arXiv:1908.03627}
}
read the original abstract

Deep reinforcement learning (RL) has been successfully applied to a variety of game-like environments. However, the application of deep RL to visual navigation with realistic environments is a challenging task. We propose a novel learning architecture capable of navigating an agent, e.g. a mobile robot, to a target given by an image. To achieve this, we have extended the batched A2C algorithm with auxiliary tasks designed to improve visual navigation performance. We propose three additional auxiliary tasks: predicting the segmentation of the observation image and of the target image and predicting the depth-map. These tasks enable the use of supervised learning to pre-train a large part of the network and to reduce the number of training steps substantially. The training performance has been further improved by increasing the environment complexity gradually over time. An efficient neural network structure is proposed, which is capable of learning for multiple targets in multiple environments. Our method navigates in continuous state spaces and on the AI2-THOR environment simulator outperforms state-of-the-art goal-oriented visual navigation methods from the literature.

Figures

Figures reproduced from arXiv: 1908.03627 by the authors.

Figure 2
Figure 2. Convolutional base. The feature size after applying [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Visual navigation auxiliary task network – observation [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 3
Figure 3. Pixel control network. D. Additional Auxiliary Tasks for Visual Navigation Motivated by [23] and [8], we introduced additional auxil￾iary tasks that are specific to visual navigation. They were de￾signed to enhance the training process as well as to help the network generalize. We train the model to predict the depth￾map, image segmentation of the observation, and image segmentation of the target. For the image segm… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Sample images from DeepMind Lab framework [9]. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Sample images from AI2-THOR framework [10]. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: SUNCG [6] scene images from the House3D [5] [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Comparison of LSTM model (LSTM) with frame [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Comparison of our model (A2CAT-VN) with target [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 11
Figure 11. Figure 11: Comparison of our method (A2CAT-VN) with [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

  1. [3]

    Target-driven visual navigation in indoor scenes using deep reinforcement learning,

    Y . Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in 2017 IEEE International Conference on Robotics and Automation (ICRA) , May 2017, pp. 3357–3364

    work page 2017

  2. [1]

    R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning , 2nd ed. Cambridge, MA, USA: MIT Press, 2017

    work page 2017

  3. [2]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770–778, 2016

    2016

  4. [4]

    One-shot reinforcement learning for robot navigation with interactive replay,

    J. Bruce, N. Suenderhauf, P. Mirowski, R. Hadsell, and M. Milford, “One-shot reinforcement learning for robot navigation with interactive replay,” 2017

    work page 2017

  5. [5]

    Building generalizable agents with a realistic and rich 3d environment,

    Y . Wu, Y . Wu, G. Gkioxari, and Y . Tian, “Building generalizable agents with a realistic and rich 3d environment,” 2018

    work page 2018

  6. [6]

    Semantic scene completion from a single depth image,

    S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser, “Semantic scene completion from a single depth image,” Proceedings of 29th IEEE Conference on Computer Vision and Pattern Recognition , 2017

    work page 2017

  7. [7]

    Reinforcement learning with unsupervised auxiliary tasks,

    M. Jaderberg, V . Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Sil- ver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” 2016

    work page 2016

  8. [8]

    Learning to navigate in complex environments,

    P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, and R. Hadsell, “Learning to navigate in complex environments,” 2016

    work page 2016

Show all 23 references
  1. [9]

    Deepmind lab,

    C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Kttler, A. Lefrancq, S. Green, V . Valds, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen, “Deepmind lab,” 2016

    2016

  2. [10]

    AI2-THOR: An Interactive 3D Environment for Visual AI,

    E. Kolve, R. Mottaghi, D. Gordon, Y . Zhu, A. Gupta, and A. Farhadi, “AI2-THOR: An Interactive 3D Environment for Visual AI,” arXiv, Dec 2017

    2017

  3. [11]

    Scalable trust-region method for deep reinforcement learning using kronecker- factored approximation,

    Y . Wu, E. Mansimov, S. Liao, R. B. Grosse, and J. Ba, “Scalable trust-region method for deep reinforcement learning using kronecker- factored approximation,” CoRR, vol. abs/1708.05144, 2017

    2017 arXiv

  4. [12]

    A survey of actor-critic reinforcement learning: Standard and natural policy gradients,

    I. Grondman, L. Bus ¸oniu, G. Lopes, and R. Babu ˇska, “A survey of actor-critic reinforcement learning: Standard and natural policy gradients,” IEEE Transactions on Systems, Man, and Cybernetics. Part C: Applications and Reviews , vol. 42, no. 6, pp. 1291–1307, 2012

    2012

  5. [13]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning,

    R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning , vol. 8, no. 3, pp. 229–256, May 1992. [Online]. Available: https://doi.org/10.1007/BF00992696

    1992 doi

  6. [14]

    Policy gradient methods for reinforcement learning with function approxima- tion,

    R. S. Sutton, D. A. McAllester, S. P. Singh, and Y . Mansour, “Policy gradient methods for reinforcement learning with function approxima- tion,” in Advances in neural information processing systems , 2000, pp. 1057–1063

    2000

  7. [15]

    Q-learning,

    C. J. C. H. Watkins and P. Dayan, “Q-learning,” in Machine Learning, 1992, pp. 279–292

    1992

  8. [16]

    Incremental learning of evaluation functions for absorbing markov chains: New methods and theorems,

    L. Gurvits, L. Lin, and S. Hanson, “Incremental learning of evaluation functions for absorbing markov chains: New methods and theorems,” preprint, 1994

    1994

  9. [17]

    Asynchronous methods for deep rein- forcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep rein- forcement learning,” in International conference on machine learning , 2016, pp. 1928–1937

    2016

  10. [18]

    Playing atari with deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013

    2013

  11. [19]

    Striving for simplicity: The all convolutional net,

    J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” 2014

    2014

  12. [20]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735

    1997 doi

  13. [21]

    Learning state representation for deep actor-critic control,

    J. Munk, J. Kober, and R. Babuska, “Learning state representation for deep actor-critic control,” 12 2016

    2016

  14. [22]

    Dueling network architectures for deep reinforcement learning,

    Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling network architectures for deep reinforcement learning,” 2015

    2015

  15. [23]

    Deep auto-encoder neural networks in reinforcement learning,

    S. Lange and M. Riedmiller, “Deep auto-encoder neural networks in reinforcement learning,” in The 2010 International Joint Conference on Neural Networks (IJCNN) , July 2010, pp. 1–8

    2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.