Situational Fusion of Visual Representation for Visual Navigation

Bokui Shen; Danfei Xu; Leonidas J. Guibas; Li Fei-Fei; Silvio Savarese; Yuke Zhu

REVIEW 4 major objections 6 minor 41 references

Situational Fusion of Visual Representation for Visual Navigation

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Action-level fusion of diverse visual representations doubles zero-shot navigation success.

desk verdict A clean, plausible paper on action-level fusion of 25 pretrained visual representations for navigation, but the headline gain over majority voting comes almost entirely from an under-tested affinity regularizer and the evaluation lacks error bars. read the letter →

arxiv 1908.09073 v2 pith:2QAHCQCX submitted 2019-08-24 cs.CV

Bokui Shen , Danfei Xu , Yuke Zhu , Leonidas J. Guibas , Li Fei-Fei , Silvio Savarese This is my paper · ORCID

classification cs.CV

keywords visualnavigationrepresentationfusionaction-leveltaskaffinityregularizationzero-shotgeneralizationsemanticindoorfrozenrepresentations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that a navigation agent generalizes to unseen indoor environments when it fuses many task-specific visual representations at the level of actions rather than features. Each frozen representation—depth, semantics, layout, and 22 others—gets its own action predictor, and a small gating network weights their action votes based on the current image. The authors add a regularizer that penalizes giving high weight to pairs of representations judged redundant by a data-driven task-affinity measure. The result is a reported 44.2% average success rate in novel environments, versus 20.9% for a standard pretrained visual network and 41.1% for majority voting across the branches. If correct, this gives a concrete recipe for reusing diverse perception modules in embodied tasks without retraining them.

What carries the argument

The load-bearing mechanism is action-level fusion: each of 25 frozen visual representations is paired with its own action-prediction module, so every branch emits a full action candidate $\tilde{a}^i_t$; a situational fusion network $f(o_t)$ outputs softmax weights $g_t$, and the final action is $a_t = \sum_i g^i_t \tilde{a}^i_t$. Because every branch is trained to produce a valid action, no representation is undertrained, and the weighted combination behaves as an ensemble. The second mechanism is the inter-task affinity regularizer $L_g(g_t) = g_t^T F g_t$, where $F_{ij}$ is the pairwise affinity of tasks $i$ and $j$; minimizing this bilinear form penalizes giving high weight to two high-affinity, redundant representations.

What would settle it

Re-run the same training with the task-affinity matrix shuffled (random permutation of its entries) and again with it set to zero. If success rates stay at roughly 44% in both altered runs, the regularizer's benefit does not come from the affinity information. Conversely, a test set of environments that require two high-affinity representations in every episode should cause the regularizer to hurt; observing that would also count against the paper's generalization claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that a navigation policy generalizes to unseen environments when it fuses many frozen, task-specific visual representations at the action level and regularizes the fusion weights with a data-driven task-affinity matrix. The authors show that each representation can be trained to emit a plausible action, and that the weighted ensemble of those action candidates substantially outperforms raw-pixel learning, feature concatenation, and feature-level gating. The best model reports 44.2% average success in environments never seen during training, versus 20.9% for a standard pretrained network and 41.1% for majority voting among branches. The regularizer shifts gating toward complementary skills, and the learned weights are situational—geometric representations are trusted in narrow spaces, semantic ones near object clusters.

Load-bearing premise

The load-bearing premise is that the paper's pairwise task-affinity scores, measured by transfer between static image tasks, correctly measure how redundant two representations are when both feed a dynamic navigation policy.

Editorial extensions

If this is right

In unseen environments, action-level fusion with the affinity regularizer reaches 44.2% average success, about double the 20.9% of a standard pretrained network.
Feature-level fusion and simple concatenation barely beat the raw-pixel baseline, so where fusion happens is the decisive design choice.
The affinity regularizer outperforms a load-balancing loss that only encourages uniform weights, showing that modeling redundancy between representations helps.
Action-level fusion keeps working when some representations are dropped or corrupted, because each branch already outputs a usable action.
Learned gating is situational: geometric representations dominate in corridors, semantic representations near object clusters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The action-level fusion recipe should transfer to other embodied tasks with frozen perception banks, such as active perception or manipulation, where a wrong feature is costlier; this is an untested extrapolation.
If the regularizer is truly capturing redundancy, an online estimate of representation redundancy during deployment could outperform the static affinity matrix; the paper does not explore this.
Because individual branch performance ranges widely, pruning the weakest branches and re-measuring would reveal whether the full 25-representation bank or the regularizer drives the gain.
The robustness results imply the architecture could be deployed in real systems where perception modules fail or are attacked, since per-branch action outputs allow graceful degradation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: the paper is not really about beating ResNet by 2x, even though that is the headline number. The more honest comparison is against the majority-voting baseline, and the proposed method beats it by about 3 percentage points (44.2 vs 41.1). The ablation suggests that gain comes almost entirely from the T.Aff regularizer, not from the action-level fusion scheme per se. That is the part you should read carefully before trusting the "situational fusion" story.

What is genuinely new and good: the action-level fusion design, where each representation gets its own action predictor and a gating network combines the candidate actions, is a sensible alternative to feature concatenation and is not something I have seen done with 25 Taskonomy representations. The paper also includes a nice robustness analysis for dropped or noisy representations, which is a practical consideration most navigation papers ignore. The writing is clear, and the authors do report the majority-voting baseline and the individual branch performances, so they are not hiding the obvious comparison.

Soft spots, in proportion: first, there are no error bars or multiple seeds anywhere. Table 1 is a single run per model, so the 3-point gap over majority voting could be within run-to-run variance. That is a real problem for a paper whose central claim is a specific number. Second, the T.Aff regularizer uses Taskonomy's affinity matrix, computed from static-image transfer performance, as a proxy for redundancy in a dynamic navigation policy. The paper never tests that proxy directly. The qualitative weight visualizations in Section 4.3 are interesting but they do not show that high-affinity pairs are redundant in navigation, only that the regularizer changes which representations are used. Third, the action-level fusion without T.Aff is not better than majority voting, so the mechanism that makes the method work is the regularizer, and we do not know whether a generic diversity penalty would do as well. The authors did compare against LBL, which is evidence, but it is not a full control.

The central idea is still plausible, and the paper's own discussion is honest about the size of the T.Aff gain. The circularity concern with using the authors' own Taskonomy representations is minor; the affinity matrix is fixed and not fit to navigation rewards.

Who this is for: anyone working on representation reuse, modular policies, or embodied generalization. It is a useful recipe and a reasonable baseline for future work. I would not take the 44.2% number as established until there are multi-seed results and a control with a shuffled affinity matrix, but the paper deserves a real refereeing process.

Recommendation: send it to peer review. Ask for error bars over at least 3-5 seeds, code or detailed training setup, and a control where the affinity matrix is shuffled or replaced with a random matrix. If those come back and the 3-point gap holds, this becomes a solid contribution.

Referee Report

4 major / 6 minor

Summary. The paper proposes a visual navigation policy that fuses 25 Taskonomy visual representations at the action level via a gating network, with a bilinear regularizer (Eq. 1) that penalizes co-selection of representation pairs with high Taskonomy affinity. The method is evaluated on four semantic navigation tasks in Gibson, and Table 1 reports 44.2% average success in unseen environments versus 20.9% for an ImageNet-pretrained ResNet baseline and 41.1% for majority voting among the 25 action branches. The central claims are that action-level fusion generalizes better than feature-level fusion or end-to-end residual networks, and that the task-affinity regularizer further improves zero-shot generalization by reducing redundancy among fused representations.

Significance. If the results are robust, the paper offers a practical recipe for reusing diverse frozen perception modules in interactive navigation and demonstrates a useful interaction between an action-level mixture-of-experts style architecture and a data-driven redundancy penalty. The paper includes a plausible experimental setup, multiple baselines, ablations of the proposed components, and a qualitative analysis of the learned fusion weights. The main strengths are the breadth of the representation set, the clean action-level fusion formulation, and the use of a standard simulation benchmark. However, the central quantitative contribution over the majority-voting baseline is small (3.1 percentage points), and the paper provides no variance estimates; the specific role of the Taskonomy affinity prior is not directly validated.

major comments (4)

[Table 1, Avg row] The proposed method (action-level fusion with T.Aff) reaches 44.2% average success, while the majority-voting baseline is 41.1% and action-level fusion without the regularizer is 40.3%. The reported advantage over majority voting is therefore 3.1 percentage points, and the advantage of the full method over its own ablation is 3.9 points. Since every number in Table 1 appears to be a point estimate from a single training run with no error bars, seed variance, or confidence intervals over the fixed 1024 test episodes, the paper currently provides no way to assess whether these differences are statistically significant. This is a load-bearing issue because the paper's claims about both the situational-fusion scheme and the affinity regularizer rest on these margins.
[Sec. 3.3.2, Eq. (1)] The regularizer Lg(g_t) = g_t^T F g_t assumes that the Taskonomy pairwise affinity matrix, estimated from transfer performance on static image tasks, is a valid proxy for redundancy among representations in a dynamic navigation policy. The paper does not provide a direct test of this assumption. The qualitative weight visualizations in Sec. 4.3 show that weights change with scene geometry, but they do not distinguish the effect of the affinity structure from a generic diversity or balance penalty. A controlled ablation with a scrambled or shuffled affinity matrix (or with an identity matrix) is needed to establish that the specific Taskonomy affinity values, rather than any quadratic penalty on the weights, are responsible for the improvement from 40.3% to 44.2%.
[Sec. 4.2, Table 1] The action-level fusion without regularization achieves 40.3%, which is slightly worse than uniform majority voting at 41.1%. This means the situational gating mechanism alone does not outperform a non-situational ensemble; the reported benefit of the proposed method over majority voting comes entirely from adding the T.Aff regularizer. The paper's narrative that situational fusion drives the gain is therefore only supported when the regularizer is present. To substantiate the situational-fusion story, the paper should additionally compare against a fixed-weight action-level fusion (for example, a single learned softmax vector applied at all time steps) and against majority voting over a subset selected by the regularizer. Without such a comparison, the contribution of the situational weighting itself is unclear.
[Sec. 4.3, Fig. 5] The claim that the fusion network exhibits situational behavior (e.g., 3D representations receiving higher weights in corridors and semantic representations in object-cluttered spaces) is supported only by qualitative heatmaps and a bar chart that appears to lack error bars or statistical tests. A quantitative analysis with confidence intervals, or a direct correlation between the openness measure and the gating weights across environments, would strengthen this part of the paper. As it stands, the situational claim is plausible but not firmly demonstrated.

minor comments (6)

[Table 1] The table caption is missing from the text, and the column structure is not fully described. In particular, the meaning of the 'Both' columns (for feature-level and action-level fusion) should be stated explicitly in the main text or a caption.
[Sec. 4.1] The sentence 'we use on average 28 environments for training and 14 for testing' is imprecise; please specify the exact train/test split for each of the four tasks or explain why the number varies.
[Fig. 7] The robustness curves report performance for a single model run; adding error bars or multiple seeds would make the robustness comparison between action-level and feature-level fusion more convincing.
[Sec. 4.1, Baselines] Calling ResNet 'state-of-the-art pretrained ResNet model' is misleading, as ResNet is a standard backbone rather than a navigation-specific state-of-the-art method; please adjust the wording.
[Sec. 3.2] Please clarify that the Taskonomy representations are frozen and are not fine-tuned during navigation policy training, since this is important for understanding the fixed-prior aspect of the method.
[Sec. 3.3.2, Eq. (1)] Please specify the range or normalization of the affinity values aff(i,j) and whether the matrix F is symmetric; this information is useful for interpreting the magnitude of the regularizer.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Taskonomy representations and affinity matrix are fixed external inputs, and none of the navigation success numbers are fed back into them.

full rationale

The paper's derivation chain is self-contained with respect to its central claim. The 25 visual representations and the pairwise affinity matrix come from Taskonomy [38], a prior published and code-released system, and the paper treats them as fixed external inputs: the navigation policy is trained on Gibson environments and evaluated on unseen environments, while the representations and affinities are not refit to navigation success. The T.Aff regularizer in Eq. 1 uses the Taskonomy affinities as a surrogate for redundancy, but this is an explicit modeling assumption rather than a circular reduction, since the affinities were estimated from static-image transfer experiments and do not incorporate the navigation policy's outputs. The closest thing to a concern is that action-level fusion without regularization (40.3%) is slightly below uniform majority voting (41.1%), so the reported gain over the non-situational ensemble comes from adding T.Aff; however, that regularizer is still an independently supplied prior, not a fitted parameter renamed as a prediction. The self-citation to Taskonomy is real external evidence under the stated rules, and no equation or evaluation metric is defined in terms of the quantity it is claimed to predict. Correctness risks about the suitability of static-image affinity as a proxy for dynamic navigation redundancy belong to the validity domain, not to circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on external priors from Taskonomy (representations and affinity matrix), a behavioral-cloning supervision choice, and the assumption that held-out Gibson scenes measure real generalization. None of these are derived in the paper, and no code or data are released to check them.

free parameters (4)

Taskonomy affinity matrix F (25x25) = fixed values from Taskonomy paper
Used in Eq. 1 as the regularization kernel; values are fitted to static-image transfer tasks, not to navigation, and the best model (T.Aff) depends on them.
Number of visual representations n = 25
Set to all Taskonomy tasks; no ablation varies n in the fusion model, so the method's success at 25 is not shown to be robust to this choice.
Training setup (batch size, iterations, LR schedule) = 256, 16K iterations, 10x LR decay every 5K
Chosen by hand; no sensitivity analysis, so reported success rates are conditional on this setup.
Learned parameters of gating network and 25 action predictors = trained on navigation data
All policy parameters are fit to the training environments; the paper reports no seed variance, so the fitted parameters are not shown to be stable.

assumptions (4)

domain assumption Taskonomy representations pretrained on static image tasks transfer their information to navigation decision-making.
Section 3.2 treats the 25 representations as given and does not fine-tune them; the entire method relies on this transfer being useful.
domain assumption Taskonomy pairwise affinity is a valid surrogate for redundancy between representations in a navigation policy.
Section 3.3.2 defines the regularizer from aff(i,j) and gives no navigation-specific validation.
domain assumption Shortest-path action labels from ground-truth object annotations are sufficient supervision for learning a navigable policy.
Section 4.1 trains all models by supervised imitation on optimal actions; no RL or interactive correction is used.
domain assumption Held-out Matterport3D scenes are a valid measure of zero-shot generalization.
Section 4.1 evaluates on unseen environments from the same dataset; this is not sim-to-real or novel object generalization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Situational Fusion of Visual Representation for Visual Navigation." pith.science (2026). https://pith.science/paper/2QAHCQCX

@misc{pith2026190809073,
  author       = {Pith},
  title        = {Pith review of: Situational Fusion of Visual Representation for Visual Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QAHCQCX}},
  note         = {Machine review of arXiv:1908.09073}
}

read the original abstract

A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the agent might need to identify a chair in a living room using semantics, follow along a hallway using vanishing point cues, and avoid obstacles using depth. Therefore, utilizing the appropriate visual perception abilities based on a situational understanding of the visual environment can empower these navigation models in unseen visual environments. We propose to train an agent to fuse a large set of visual representations that correspond to diverse visual perception abilities. To fully utilize each representation, we develop an action-level representation fusion scheme, which predicts an action candidate from each representation and adaptively consolidate these action candidates into the final action. Furthermore, we employ a data-driven inter-task affinity regularization to reduce redundancies and improve generalization. Our approach leads to a significantly improved performance in novel environments over ImageNet-pretrained baseline and other fusion methods.

Figures

Figures reproduced from arXiv: 1908.09073 by the authors.

**Figure 1.** (Top) The visual navigation task requires an agent to handle large diversity in real-world environments. (Bottom) Instead of using a black-box model, we propose to adaptively fuse visual representations from a diverse set of vision tasks to better generalize to new environments. cally unlimited synthetic training data have warranted the wide applicability of these methods. However, obviating structures and prior kn… view at source ↗

**Figure 2.** Three schemes of learning neural network policies from visual perception to action, including end-to-end learning with blackbox neural networks, and representation fusion methods at both feature and action levels. we propose to combine the representations at the action level (Sec. 3.3.1), which ensures that each representation is individually trained to make meaningful prediction for the overall task. We also intro… view at source ↗

**Figure 3.** The agent receives an RGB image and its corresponding visual representations. We use a situational-fusion network to adaptively weigh and combine the representations at the action level based on the current observation. The fusion weight is regularized by inter-task affinity to encourage a more balanced selection of the representations for better generalization. tions [5, 32] is to use fused representations as joint… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Training and testing performance comparison. Testing scenes are previously unseen environments. Although models achieve similar levels of performance on training scenes, our action-level fusion model generalize significantly better to unseen test environments than ba…

**Figure 5.** Figure 5: (Left) fusion weight heat-map of different representation domains on two example environments, and a quantitative bar chart of distribution over all testing scenes. 3D representations receive higher weights in narrow space like corridor, while semantic representations …

**Figure 7.** Figure 7: Robustness Analysis: The left figure shows how performance varies when a number of representations are dropped. The right figure shows how performance varies when a number of representations are set to zero. Horizontal axis shows how many representations remain unaffe…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 29 canonical work pages

[1]

Vi- sual navigation for mobile robots: A survey

Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Vi- sual navigation for mobile robots: A survey. Journal of in- telligent and robotic systems, 53(3):263–296, 2008. 2

work page 2008
[2]

A robust layered control system for a mobile robot

Rodney Brooks. A robust layered control system for a mobile robot. IEEE journal on robotics and automation , 2(1):14–23, 1986. 4

work page 1986
[3]

Matterport3D: Learning from RGB- D data in indoor environments

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB- D data in indoor environments. International Conference on 3D Vision (3DV), 2017. 5

work page 2017
[4]

Embodied question answer- ing

Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. arXiv preprint arXiv:1711.11543, 2017. 2

arXiv 2017
[5]

Language modeling with gated convolutional net- works

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional net- works. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 933–941. JMLR. org, 2017. 4

work page 2017
[6]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009. 1, 2, 5

work page 2009
[7]

Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsu- pervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV) ,

work page
[8]

Fast r-cnn

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter- national conference on computer vision , pages 1440–1448,

Show all 41 references

[9]

Iqa: Visual question answering in interactive environments

Daniel Gordon, Aniruddha Kembhavi, Mohammad Raste- gari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR). IEEE, 2018. 2

2018
[10]

Cognitive mapping and plan- ning for visual navigation

Saurabh Gupta, James Davidson, Sergey Levine, Rahul Suk- thankar, and Jitendra Malik. Cognitive mapping and plan- ning for visual navigation. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2616–2625,
[11]

Efﬁcient quadratic reg- ularization for expression arrays

Trevor Hastie and Robert Tibshirani. Efﬁcient quadratic reg- ularization for expression arrays. Biostatistics, 5(3):329– 340, 2004. 5

2004
[12]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5, 6

2016
[13]

Grounded language learning in a simulated 3d world

Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wo- jtek Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017. 2

2017 arXiv
[14]

Virtual-to-real: Learning to control in visual semantic segmentation

Zhang-Wei Hong, Chen Yu-Ming, Shih-Yang Su, Tzu-Yun Shann, Yi-Hsiang Chang, Hsuan-Kung Yang, Brian Hsi-Lin Ho, Chih-Chieh Tu, Yueh-Chuan Chang, Tsu-Ching Hsiao, et al. Virtual-to-real: Learning to control in visual semantic segmentation. arXiv preprint arXiv:1802.00285, 2018. 2

2018 arXiv
[15]

What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016

Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016. 2

2016 arXiv
[16]

Learning visual features from large weakly supervised data

Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference on Com- puter Vision, pages 67–84, 2016. 2

2016
[17]

Generalization through simulation: In- tegrating simulated and real data into deep reinforcement learning for vision-based autonomous ﬂight

Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, and Sergey Levine. Generalization through simulation: In- tegrating simulated and real data into deep reinforcement learning for vision-based autonomous ﬂight. arXiv preprint arXiv:1902.03701, 2019. 2

1902 arXiv
[18]

Adam: A method for stochastic optimization

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

arXiv
[19]

Shamma, Michael S

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

2016
[20]

Imagenet classiﬁcation with deep convolutional neural net- works

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolutional neural net- works. In Advances in neural information processing sys- tems, pages 1097–1105, 2012. 1

2012
[21]

End-to-end training of deep visuomotor policies

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334– 1373, 2016. 1

2016
[22]

Understanding deep image representations by inverting them

Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 2

2015
[23]

Adversarially robust policy learning: Active construction of physically-plausible perturbations

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausible perturbations. In Intel- ligent Robots and Systems (IROS), 2017 IEEE/RSJ Interna- tional Conference on, pages 3932–3939. I...

2017
[24]

Neural autonomous navigation with riemannian motion pol- icy

Xiangyun Meng, Nathan Ratliff, Yu Xiang, and Dieter Fox. Neural autonomous navigation with riemannian motion pol- icy. In IEEE International Conference on Robotics and Au- tomation. IEEE, 2019. 2

2019
[25]

Learning to navigate in cities without a map

Piotr Mirowski, Matthew Koichi Grimes, Mateusz Ma- linowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning to navigate in cities without a map. arXiv preprint arXiv:1804.00168, 2018. 2

2018 arXiv
[26]

Learn- ing to navigate in complex environments

Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learn- ing to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016. 2

2016 arXiv
[27]

Driving policy transfer via modularity and abstraction

Matthias M ¨uller et al. Driving policy transfer via modularity and abstraction. arXiv preprint arXiv:1804.09364, 2018. 2

2018 arXiv
[28]

Neural map: Structured memory for deep reinforcement learning

Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint arXiv:1702.08360, 2017. 2

2017 arXiv
[29]

Context encoders: Feature learn- ing by inpainting

Deepak Pathak, Philipp Kr ¨ahenb¨uhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learn- ing by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 2

2016
[30]

Epopt: Learning robust neural network policies using model ensembles

Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravin- dran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016. 2

2016 arXiv
[31]

Universal value function approximators

Tom Schaul, Daniel Horgan, Karol Gregor, and David Sil- ver. Universal value function approximators. In Interna- tional Conference on Machine Learning , pages 1312–1320,
[32]

Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint arXiv:1701.06538, 2017. 4, 5, 6

2017 arXiv
[33]

Very deep convo- lutional networks for large-scale image recognition

Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 1

2014 arXiv
[34]

Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Pre- cup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Age...

2011
[35]

Proba- bilistic Robotics

Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Proba- bilistic Robotics. MIT Press, 2005. 2

2005
[36]

Domain randomization for transferring deep neural networks from simulation to the real world

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 2...

2017
[37]

Gibson env: Real-world percep- tion for embodied agents

Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world percep- tion for embodied agents. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 3, 5

2018
[38]

Zamir, Alexander Sax, William B

Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,
[39]

Visualizing and under- standing convolutional networks

Matthew D Zeiler and Rob Fergus. Visualizing and under- standing convolutional networks. InEuropean conference on computer vision, pages 818–833. Springer, 2014. 2

2014
[40]

Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks

Fangyi Zhang, J ¨urgen Leitner, Michael Milford, and Pe- ter Corke. Sim-to-real transfer of visuo-motor policies for reaching in clutter: Domain randomization and adaptation with modular networks. world, 7:8, 2017. 2

2017
[41]

Target-driven vi- sual navigation in indoor scenes using deep reinforcement learning

Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Ab- hinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven vi- sual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation, pages 3357–3364. IEEE, 2017. 1, 2, 3

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Vi- sual navigation for mobile robots: A survey

Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Vi- sual navigation for mobile robots: A survey. Journal of in- telligent and robotic systems, 53(3):263–296, 2008. 2

work page 2008

[2] [2]

A robust layered control system for a mobile robot

Rodney Brooks. A robust layered control system for a mobile robot. IEEE journal on robotics and automation , 2(1):14–23, 1986. 4

work page 1986

[3] [3]

Matterport3D: Learning from RGB- D data in indoor environments

Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Hal- ber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB- D data in indoor environments. International Conference on 3D Vision (3DV), 2017. 5

work page 2017

[4] [4]

Embodied question answer- ing

Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answer- ing. arXiv preprint arXiv:1711.11543, 2017. 2

arXiv 2017

[5] [5]

Language modeling with gated convolutional net- works

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional net- works. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 933–941. JMLR. org, 2017. 4

work page 2017

[6] [6]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009. 1, 2, 5

work page 2009

[7] [7]

Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsu- pervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV) ,

work page

[8] [8]

Fast r-cnn

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter- national conference on computer vision , pages 1440–1448,