REVIEW 3 major objections 5 minor 50 references
A training-free navigation agent can follow instructions in unseen environments more successfully with a structured, agent-authored memory of places, objects, and failures than with its full raw visual history—while using about two-thirds f
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 03:47 UTC pith:WHKDVF3U
load-bearing objection Solid memory design and clean ablations, but the headline numbers compare a 100-episode subset against full-split baselines with the supporting appendix missing—conditional acceptance at best. the 3 major comments →
HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a hierarchical agentic memory—comprising a bounded working-memory window, an instruction-progress record, and a persistent depth-grounded world graph of places, objects, and topology—allows a multimodal LLM planner to navigate zero-shot more successfully than if it were given the full raw visual history. The paper reports that when the same model call that selects the action also writes room semantics, object observations, progress updates, and failure notes into the graph, and when older experience re-enters context only through a retrieval scored by relevance, recency, and salience plus one-hop topological expansion, success rate on R2R-CE reaches 61.0%, on RxR-CE
What carries the argument
The central object is the depth-grounded world graph, a typed place–object graph built incrementally from RGB-D observations: place nodes aggregate spatial anchors and agent-authored semantics, object nodes carry depth-grounded category and open-vocabulary descriptions, place–place edges encode navigational connectivity, and place–object edges encode semantic containment. The mechanism is decision-coupled write-back: the large multimodal model call that chooses the action also emits a structured record that updates the instruction-progress record and graph, so memory maintenance costs no extra model calls. Retrieval is subgoal-conditioned: a local sentence encoder scores places by relevance,
Load-bearing premise
The load-bearing premise is that the 100-episode subsets of the val-unseen splits used for evaluation faithfully represent the full splits; the paper supports this only by citing a prior work's appendix that is not included, so if the subsets are unrepresentative the headline success rates and baseline comparisons do not transfer to the actual benchmarks.
What would settle it
Run the same configuration on the full val-unseen splits of R2R-CE, RxR-CE, and HM3D-v2 ObjectNav. If the success rate deviates from the subset-reported 61.0%, 52.7%, and 79.7% by more than about 1.1 percentage points, or if it no longer beats the strongest zero-shot baseline on every metric, the central claim is falsified. A lighter check is to obtain the cited prior work's appendix and verify that the subset-to-full deviations are actually at most 1.1 points.
If this is right
- If the central claim holds, zero-shot vision-and-language navigation does not need longer context as a remedy for long trajectories; structured memory beats raw history at lower cost.
- Context per decision drops to about 17.9k tokens and per episode to about 244.9k tokens, which would make long-horizon deployment of API-based navigators markedly cheaper.
- Failure notes (reflection memory) matter most for path efficiency: removing them cuts success-weighted path length by 11.2 points, suggesting that recording and re-reading dead ends makes exploration more economical.
- The same configuration, without benchmark-specific retuning, improves all four reported metrics on both R2R-CE and RxR-CE, implying the memory scheme transfers across instruction formats and lengths.
- The graph representation scales with discovered places and objects rather than mapped area, which would help navigation in large environments.
Where Pith is reading between the lines
- The decision-coupled write-back suggests a general recipe for embodied agents: let the planning call also perform memory curation. This could extend to manipulation or search tasks where the same object appears under different viewpoints and failure notes are reusable.
- Because retrieval weights are fixed across benchmarks, a cheap per-benchmark or per-subgoal tuning of these weights might yield further gains; the paper does not explore this.
- Reflection memory is reset each episode. Keeping failure evidence across episodes in the same environment could enable within-environment learning without any training, effectively a lifelong version of the mechanism.
- The paper's headline numbers rest on 100-episode subsets whose representativeness is supported only by a cited prior work's appendix, which is not included in this manuscript; verifying on full splits is the natural next check.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. HAM-VLN proposes a zero-shot vision-and-language navigation agent whose main novelty is a hierarchical agentic memory: a bounded working memory of recent frames, a depth-grounded place–object world graph, and episodic, semantic, and reflection memory views that select what enters the planner context. The same System-2 MLLM call that chooses an action also writes structured memory records; older history re-enters context only through subgoal-conditioned retrieval scored by relevance, recency, and salience, plus one-hop graph expansion. The paper reports zero-shot results on VLN-CE R2R, VLN-CE RxR, and HM3D-v2 ObjectNav (61.0%, 52.7%, and 79.7% SR, respectively), controlled comparisons against raw-history baselines, token-cost reductions, and ablations of the memory components.
Significance. If the results hold, the paper makes a useful contribution to training-free navigation by showing that agent-authored, decision-coupled memory can be more token-efficient than raw history while improving navigation quality. The internal evaluation is a genuine strength: Table 4 is a controlled ablation with three-seed means and standard deviations, Table 3 holds the planner fixed while varying only the memory/context representation, and the claimed context-length reduction is quantified in tokens. These comparisons support the value of the memory design in a way that does not depend on the external baselines. The main obstacle is external validity: the headline comparisons to prior state-of-the-art methods rest on subset evaluations whose representativeness is asserted but not demonstrated in the manuscript.
major comments (3)
- [§4.1, Tables 1–2] The central 'outperforms prior methods' claim compares HAM-VLN's 100-episode val-unseen subset scores against published full-split results of prior methods. Section 4.1 states that the subsets are those of Ding et al. (2026b), that Ding et al. reported subset-to-full deviations of no more than 1.1 pp, and that 'the appendix summarizes these comparisons.' However, no appendix is present in the manuscript. This is load-bearing: a 100-episode subset of R2R val-unseen is not by itself guaranteed to reproduce full-split difficulty, and the missing support means the 22.7-point SR gain over the best zero-shot baseline in Table 1 and the ObjectNav gains in Table 2 could be subset artifacts. The authors should either supply the promised appendix, evaluate on the full val-unseen splits, or include subset-matched baseline results for every compared method.
- [§3.3, Eq. (5)] The retrieval score uses manually fixed parameters (α, β, γ) = (1.0, 0.3, 0.3), ρ = 0.85, K = 1, Ks = 3, and Kp = 5, with no sensitivity analysis and no statement of whether these values were chosen on the evaluation subsets. The paper emphasizes that the same configuration is used across benchmarks, which is good, but the 'zero-shot / training-free' framing requires clarity that no hyperparameters were selected using the val-unseen subsets. Please report a sensitivity sweep for α, β, γ, and ρ, or at least clarify the selection procedure.
- [§3.1, Algorithm 1, Abstract] The abstract claims that the design 'requires no additional LLM calls beyond the per-waypoint decision.' In the architecture, System 2 (φ2) is invoked once per waypoint for reasoning and memory writes, but System 1 (φ1) is a separate grounding MLLM invoked at line 17 of Algorithm 1 to produce a bounding box for the chosen direction. The sentence is therefore inaccurate unless it is explicitly scoped to 'no additional calls for memory maintenance.' Table 3 also reports only System-2 tokens, so the 65% context-reduction claim should clarify whether it accounts for the grounding-model tokens.
minor comments (5)
- [§4.2] The phrase 'the strongest previously reported training-free baseline across all metrics' is misleading because the strongest baseline differs by metric (e.g., SR vs. NE in Table 1). Please state that the comparison takes the best reported value per metric.
- [§3.2] The incremental single-link clustering used to form place states is described only verbally; the overlap threshold for merging place states is not specified. For reproducibility, please give the threshold or state that it is a fixed constant and report its value.
- [§4.5] The subsection titled 'Quantitative Analysis' is mostly a qualitative walkthrough of one episode; the only quantitative content is the backtracking frequency in the first sentence. Consider renaming it or moving the statistics into Section 4.4.
- [§4.1] The reference to Ding et al. (2026b) is used as the source of the evaluation subsets, but the evaluation section does not state which episodes are included or how a reader can obtain the same subsets. If the subsets are available, a URL or appendix pointer should be provided.
- [Eq. (5)] The terms rel(q_t, p), rec_t(p), and sal(p) in Eq. (5) are not formally defined as functions. In particular, 'landmark agreement' and 'object richness' in sal need precise definitions so that the scoring is reproducible.
Circularity Check
Subset representativeness rests on a missing-appendix self-citation, but the method itself is empirically self-contained.
specific steps
-
self citation load bearing
[Section 4.1 (Benchmarks and metrics), Tables 1-2, Section 4.2 (Main Results)]
"using the same 100-episode subsets of the val-unseen splits as Ding et al. (2026b). For these three benchmarks, Ding et al. (2026b) reported that the SR values reproduced on these subsets deviate from the corresponding full-split results by no more than 1.1 percentage points; the appendix summarizes these previously reported subset-to-full comparisons."
The headline claim that HAM-VLN 'outperforms the strongest previously reported training-free baseline across all metrics' is established by comparing HAM-VLN's 100-episode subset numbers (Tables 1-2) against full-split published numbers of prior methods. The only support for treating the subset as representative is a self-cited companion paper by the same first author, and the appendix that would document the comparison is absent from this manuscript. Thus the central comparison to prior work is not self-contained: it stands or falls on an unverifiable self-citation rather than on full-split measurements or independent evidence presented here. This does not make the navigation outcomes circular, but it makes the headline comparison with external baselines conditional on the authors' own pr
full rationale
The manuscript is not built on a closed-form derivation; it is an empirical system paper. The world-graph memory, retrieval scoring, and backtracking are evaluated on simulator metrics, and the ablations in Table 4 compare configurations under identical conditions, so those internal comparisons are self-contained. The token-count and context-length reductions come from direct instrumentation and are not construction-level circular. The only load-bearing self-citation is in the evaluation protocol: all benchmark numbers are produced on 100-episode subsets of val-unseen splits defined in Ding et al. (2026b), a prior paper whose first author is also an author here, and the only evidence that these subsets are representative is Ding et al.'s cited claim of at most 1.1 pp deviation, with the summarizing appendix not present in this manuscript. Since Tables 1 and 2 compare HAM-VLN's subset numbers against full-split published baselines, the headline 'outperforms the strongest previously reported training-free baseline' is conditional on that self-cited, unverified representativeness claim. This is a real self-citation that is load-bearing for comparability, but it does not make the method itself circular; the navigation outcomes are still externally measured on the chosen subset. Score 4 reflects the load-bearing self-citation and missing appendix, while recognizing that the central method has independent empirical content.
Axiom & Free-Parameter Ledger
free parameters (5)
- Retrieval weights (α, β, γ) =
1.0, 0.3, 0.3
- Recency decay ρ =
0.85
- Working memory window size K =
1
- Retrieval budgets K_s and K_p =
3, 5
- Place clustering overlap threshold =
unspecified
axioms (4)
- domain assumption Depth and odometry are accurate enough for projecting pixels to a stable world frame and for incremental place clustering.
- domain assumption The 100-episode subset of each val-unseen split is representative of the full split, so results can be compared with full-split numbers of prior work.
- domain assumption Pretrained encoders and MLLMs (Gemini-3.1-Pro, Qwen3.6-35B, Grounding DINO, SAM, bge-large-en-v1.5) behave as described and are available at the reported configuration.
- ad hoc to paper A linear combination of relevance, recency, and salience with fixed weights is a sufficient retrieval ranking for navigation memory.
read the original abstract
Vision-and-language navigation (VLN) enables robots to follow instructions in previously unseen environments. Recently, a training-free paradigm has emerged: the robot queries a multimodal LLM to understand its observations and plan the next action. However, long-horizon navigation based on either image streams or dense map inevitably introduces a growing memory and reasoning bottleneck. We present HAM-VLN, a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph. In the same model call used to select the next action, HAM-VLN also records semantic and reflective information---including room type, objects, navigation progress, and failure notes. Recent waypoints remain verbatim within a bounded window, while older history re-enters the context only through retrieval scored by relevance, recency, and salience, together with one-hop topological expansion. This design requires no additional LLM calls beyond the per-waypoint decision. Compared to previous methods, HAM-VLN not only improves various navigation metrics but also reduces the context length by more than 65%. Specifically, HAM-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.
Figures
Reference graph
Works this paper leans on
-
[1]
Shi, Lucy Xiaoyang and Ichter, Brian and Equi, Michael Robert and Ke, Liyiming and Pertsch, Karl and Vuong, Quan and Tanner, James and Walling, Anna and Wang, Haohuan and Fusai, Niccolo and Li-Bell, Adrian and Driess, Danny and Groom, Lachy and Levine, Sergey and Finn, Chelsea , booktitle =
-
[2]
The Fourteenth International Conference on Learning Representations , year =
Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation , author =. The Fourteenth International Conference on Learning Representations , year =
-
[3]
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
-
[4]
European Conference on Computer Vision , pages =
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments , author =. European Conference on Computer Vision , pages =
-
[5]
Ku, Alexander and Anderson, Peter and Patel, Roma and Ie, Eugene and Baldridge, Jason , booktitle =
-
[6]
and Savva, Manolis and Zhao, Yili and Batra, Dhruv , booktitle =
Ramakrishnan, Santhosh Kumar and Gokaslan, Aaron and Wijmans, Erik and Maksymets, Oleksandr and Clegg, Alexander and Turner, John and Undersander, Eric and Galuba, Wojciech and Westbury, Andrew and Chang, Angel X. and Savva, Manolis and Zhao, Yili and Batra, Dhruv , booktitle =. Habitat-Matterport
-
[7]
Advances in Neural Information Processing Systems , year =
Object Goal Navigation Using Goal-Oriented Semantic Exploration , author =. Advances in Neural Information Processing Systems , year =
-
[8]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =
Segment Anything , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =
-
[9]
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , pages =
Generative Agents: Interactive Simulacra of Human Behavior , author =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , pages =. 2023 , publisher =
2023
-
[10]
Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighoff, Niklas and Lian, Defu and Nie, Jian-Yun , booktitle=
-
[11]
Anderson, Peter and Chang, Angel and Chaplot, Devendra Singh and Dosovitskiy, Alexey and Gupta, Saurabh and Koltun, Vladlen and Kosecka, Jana and Malik, Jitendra and Mottaghi, Roozbeh and Savva, Manolis and Zamir, Amir R. , year=. 1807.06757 , archivePrefix=
-
[12]
NeurIPS Workshop on Visually Grounded Interaction and Language , year =
General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping , author =. NeurIPS Workshop on Visually Grounded Interaction and Language , year =
-
[13]
Zhou, Gengze and Hong, Yicong and Wu, Qi , journal=
-
[14]
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages =
Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions , author =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics , pages =
-
[15]
2025 , publisher =
Shi, Xiangyu and Li, Zerui and Lyu, Wenqi and Xia, Jiatong and Dayoub, Feras and Qiao, Yanyuan and Wu, Qi , booktitle =. 2025 , publisher =
2025
-
[16]
IEEE Transactions on Pattern Analysis and Machine Intelligence , publisher=
Constraint-aware zero-shot vision-language navigation in continuous environments , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , publisher=
-
[17]
Qiao, Yanyuan and Lyu, Wenqi and Wang, Hui and Wang, Zixu and Li, Zerui and Zhang, Yuan and Tan, Mingkui and Wu, Qi , booktitle =
-
[18]
Long, Yuxing and Cai, Wenzhe and Wang, Hongcheng and Zhan, Guanqi and Dong, Hao , booktitle =
-
[19]
Yin, Hang and Wei, Haoyu and Xu, Xiuwei and Guo, Wenxuan and Zhou, Jie and Lu, Jiwen , booktitle =
-
[20]
Zhang, Jiazhao and Wang, Kunyu and Xu, Rongtao and Zhou, Gengze and Hong, Yicong and Fang, Xiaomeng and Wu, Qi and Zhang, Zhizheng and Wang, He , booktitle =
-
[21]
Zhang, Jiazhao and Wang, Kunyu and Wang, Shaoan and Li, Minghan and Liu, Haoran and Wei, Songlin and Wang, Zhongyuan and Zhang, Zhizheng and Wang, He , booktitle=
-
[22]
Cheng, An-Chieh and Ji, Yandong and Yang, Zhaojing and Gongye, Zaitian and Zou, Xueyan and Kautz, Jan and Biyik, Erdem and Yin, Hongxu and Liu, Sifei and Wang, Xiaolong , booktitle =
-
[23]
Wei, Meng and Wan, Chenyang and Yu, Xiqian and Wang, Tai and Yang, Yuqiang and Mao, Xiaohan and Zhu, Chenming and Cai, Wenzhe and Wang, Hanqing and Chen, Yilun and Liu, Xihui and Pang, Jiangmiao , booktitle=
-
[24]
The Fourteenth International Conference on Learning Representations , year=
JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation , author=. The Fourteenth International Conference on Learning Representations , year=
-
[25]
Xue, Xinda and Hu, Junjun and Luo, Minghua and Xie, Shichao and Chen, Jintao and Xie, Zixun and Quan, Kuichen and Guo, Wei and Chu, Zedong and Xu, Mu and Zhu, Zhengzhou , booktitle =
-
[26]
InternVLA-N1: An Open Dual-System Vision-Language Navigation Foundation Model with Learned Latent Plans , author=
-
[27]
Chu, Zedong and Xie, Shichao and Wu, Xiaolong and Shen, Yanfen and Luo, Minghua and Wang, Zhengbo and Liu, Fei and Leng, Xiaoxu and Hu, Junjun and Yin, Mingyang and others , year=. 2602.11598 , archivePrefix=
-
[28]
Liu, Jiahang and Xu, Tianyu and Chen, Jiawei and Yue, Lu and Zhang, Jiazhao and Wang, Zhiyong and Li, Minghan and Zhao, Qisheng and Li, Anqi and Su, Qi and Zhang, Zhizheng and Wang, He , year=. 2603.09163 , archivePrefix=
-
[29]
Lyu, Kailin and Wu, Kangyi and Li, Pengna and Hu, Xiuyu and Si, Qingyi and Miao, Cui and Yang, Ning and Wang, Zihang and Xiao, Long and Hu, Lianyu and Sun, Jingyuan and Hao, Ce , year=. 2603.14807 , archivePrefix=
-
[30]
Zheng, Wanrong and Ge, Yunhao and Itti, Laurent , booktitle =
-
[31]
International Conference on Learning Representations (ICLR) , year =
Erik Wijmans and Abhishek Kadian and Ari Morcos and Stefan Lee and Irfan Essa and Devi Parikh and Manolis Savva and Dhruv Batra , title =. International Conference on Learning Representations (ICLR) , year =
-
[32]
Ramrakhya, Ram and Undersander, Eric and Batra, Dhruv and Das, Abhishek , booktitle=
-
[33]
Ramrakhya, Ram and Batra, Dhruv and Wijmans, Erik and Das, Abhishek , booktitle=
-
[34]
Yadav, Karmesh and Majumdar, Arjun and Ramrakhya, Ram and Yokoyama, Naoki and Baevski, Alexei and Kira, Zsolt and Maksymets, Oleksandr and Batra, Dhruv , year=. 2303.07798 , archivePrefix=
-
[35]
Yokoyama, Naoki and Ha, Sehoon , booktitle=
-
[36]
Yu, Bangguo and Kasaei, Hamidreza and Cao, Ming , booktitle=
-
[37]
Yokoyama, Naoki and Ha, Sehoon and Batra, Dhruv and Wang, Jiuguang and Bucher, Bernadette , booktitle=
-
[38]
Findings of the Association for Computational Linguistics: NAACL 2024 , year =
Yuxuan Kuang and Hai Lin and Meng Jiang , title =. Findings of the Association for Computational Linguistics: NAACL 2024 , year =
2024
-
[39]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Hang Yin and Xiuwei Xu and Zhenyu Wu and Jie Zhou and Jiwen Lu , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[40]
IEEE Robotics and Automation Letters , year=
Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion , author=. IEEE Robotics and Automation Letters , year=
-
[41]
An, Weitao and Liu, Qi and Xu, Chenghao and Chai, Jiayi and Yang, Xu and Wei, Kun and Deng, Cheng , year=. 2601.21409 , archivePrefix=
-
[42]
Wu, Feng and Zuo, Wei and Yang, Wenliang and Xiao, Jun and Liu, Yang and Zeng, Xinhua , year=. 2603.26788 , archivePrefix=
-
[43]
IEEE International Conference on Robotics and Automation (ICRA) , year =
Hongyu Ding and Ziming Xu and Yudong Fang and You Wu and Zixuan Chen and Jieqi Shi and Jing Huo and Yifan Zhang and Yang Gao , title =. IEEE International Conference on Robotics and Automation (ICRA) , year =
-
[44]
Ding, Hongyu and Zhang, Sizhuo and Xu, Ziming and Guo, Jinwen and Liu, Hongxiu and Cheng, Xingzhi and Chen, Zixuan and Qi, Haifei and Wang, Duo and Xu, Hao and others , year=. 2605.27582 , archivePrefix=
-
[45]
Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Jiang, Qing and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei , booktitle=
-
[46]
IEEE International Conference on Robotics and Automation (ICRA) , year=
Visual Language Maps for Robot Navigation , author=. IEEE International Conference on Robotics and Automation (ICRA) , year=
-
[47]
Advances in Neural Information Processing Systems (NeurIPS) , year=
Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=
-
[48]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
-
[49]
Conference on Robot Learning , year=
Rana, Krishan and Haviland, Jesse and Garg, Sourav and Abou-Chakra, Jad and Reid, Ian and S. Conference on Robot Learning , year=
-
[50]
and Stoica, Ion and Gonzalez, Joseph E
Packer, Charles and Wooders, Sarah and Lin, Kevin and Fang, Vivian and Patil, Shishir G. and Stoica, Ion and Gonzalez, Joseph E. , year=. 2310.08560 , archivePrefix=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.