Pith. sign in

REVIEW 4 major objections 6 minor 56 references

This paper argues that a video-based vision-language model can navigate and reason about space far better if it is given two memory systems — a learnable geometric-semantic world embedding that acts as a cognitive map, and an episodic memor

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

VEME, a dual-memory cross-modal alignment framework built on Qwen-2.5-VL, reports modest gains on VLN-CE and VSI-Bench that are contradicted by its own internal numbers.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A plausible dual-memory architecture for video-based embodied models, but the reported numbers contradict each other and the claimed gains are unverified. the 4 major comments →

arxiv 2509.00210 v1 pith:SQYMHGI4 submitted 2025-08-29 cs.CV cs.AI

Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment

Jinzhou Tang , Jusheng zhang , Sidi Liu , Waikit Xiu , Qinhan Lv , Xiying Li This is my paper
classification cs.CV cs.AI
keywords embodied AIvision-language navigationworld modelepisodic memoryspatial reasoningmultimodal alignmentpoint cloudvideo question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VEME is a training-time wrapper for a 7B vision-language model that tries to close the embodied reasoning gap: standard VLMs see 2D frames but have no persistent sense of 3D layout or of where the agent has been. The paper claims that adding a world embedding (spatial semantic memory) and a trajectory-conditioned episodic memory, trained with two contrastive losses, makes the same VLM generalize to unseen scenes. On VLN-CE R2R Val-Unseen it reports SR 57.0 and SPL 51.0, and on VSI-Bench an average of 49.3, ahead of the listed baselines. Ablations suggest episodic memory is the largest single contributor on navigation. If true, the practical upshot is that spatial intelligence for embodied agents can be injected into an off-the-shelf VLM without building an explicit map.

Core claim

The paper's central claim is that the reason embodied VLMs fail is not lack of visual semantics but spatial naivety and statelessness, and both can be fixed by a dual-memory architecture. A learnable world embedding Eworld is grounded to geometry by cross-attending visual features to depth/normal-aware features and pulling matched pairs together with a spatial contrastive loss. A separate episodic query, built by concatenating global point-cloud features and the action trajectory, attends to Eworld to produce a per-episode fingerprint, trained with an episodic contrastive loss. These memory tokens are concatenated into the VLM's input sequence, letting Qwen-2.5-VL reason over instruction, cu

What carries the argument

The load-bearing object is the world embedding Eworld, a learnable matrix meant to serve as a geometric-semantic cognitive map: current-frame visual semantics are bound to image-derived geometry via cross-attention and a spatial contrastive loss, while a query made from the global point cloud and action history attends to Eworld to form an episodic trace under an episodic contrastive loss. All of these token streams are then concatenated into the VLM's input, so the model's own attention decides what to retrieve.

Load-bearing premise

The whole apparatus assumes a clean, complete, static 3D point cloud of the scene is available at decision time; if real maps are noisy or the scene changes, the memory modules' claimed benefits are not shown to transfer.

What would settle it

Run VEME in the Habitat simulator but alter the scene after the agent builds its point cloud — move a chair or open a door mid-episode — and measure SR/SPL; if navigation success does not degrade, the static-scene assumption is not limiting. Separately, corrupt the point cloud (drop 30% of points or add Gaussian noise) and check whether the 15.3-point episodic-memory advantage survives; if it vanishes, the headline gains depend on map quality rather than on the memory design.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Episodic memory, not just scene geometry, is what the ablations identify as the biggest lever: removing it costs 15.3 SPL points, suggesting trajectory history matters more than spatial grounding for navigation success.
  • The same architecture, without task-specific heads, transfers across navigation (VLN-CE) and video spatial QA (VSI-Bench), implying a single world-embedding mechanism can serve both planning and question answering.
  • Because the memory tokens are fed directly into the VLM's context, the approach is compatible with any instruction-tuned VLM, so gains should compound as base models improve.
  • The spatial contrastive loss forces a unique correspondence between 2D semantics and geometric features, which is a direct, testable training signal for object-to-geometry binding.
  • On R2R Val-Unseen the method ties ETPNav's SR (57.0) while improving SPL (51.0 vs 49.0), so the claimed edge on that benchmark is primarily path efficiency rather than raw success.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the inputs include a complete global point cloud at decision time and the appendix states the framework assumes static scenes, the abstract's claim of gains in 'dynamic environments' is untested; the real-world benefit depends on whether noisy, incomplete, or changing SLAM maps can be tolerated.
  • The episodic-memory mechanism could plausibly extend to other spatially grounded VLM tasks — 3D visual grounding, object-state tracking, or manipulation affordance prediction — but the paper only demonstrates navigation and QA, so that transfer is an open question rather than a result.
  • The RxR numbers in Table 1 tell a more modest story than the R2R numbers: VEME trails ETPNav on RxR Val-Unseen (50.7 vs 54.7 SR), so the claimed superiority is specific to the R2R benchmark and to SPL.
  • A direct stress test of the framework's robustness would be to feed it point clouds with simulated sensor noise or occlusion, since the geometric and episodic modules are both built on that input; the ablation table suggests VGGT removal collapses MRA from 55.7 to 19.2, so the system's spatial QA is sharply dependent on geometry-feature quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VEME, a VLM extension for embodied navigation and spatial reasoning. VEME augments Qwen-2.5-VL-7B with (i) a learnable world embedding serving as spatial semantic memory, (ii) a cross-modal alignment module that grounds 2D visual semantics in geometric features via cross-attention and a contrastive loss, and (iii) an episodic memory module that encodes trajectory and point-cloud history into a query vector attended against the world embedding. The model is trained end-to-end with a weighted combination of task loss, spatial loss, and episodic loss, and is evaluated on VLN-CE (R2R, RxR) and VSI-Bench. The paper claims considerable improvements over specialized baselines such as ETPNav, NaVILA, and Spatial-MLLM.

Significance. If the headline results were reliable, VEME would provide a concrete, modular recipe for injecting geometric and episodic priors into VLMs, with a clearly specified architecture and training setup. The work is not circular in the derivation sense: the contrastive losses are learned supervision, not fitted parameters that analytically reduce the benchmark results. The paper also ships a high level of implementation detail (encoders, LoRA, hyperparameters, data recipe). However, the empirical claims are the central contribution, and the reported evidence contains internal numerical inconsistencies that undermine those claims. The static-environment limitation stated in Appendix E also directly contradicts the abstract's dynamic-environment claim.

major comments (4)
  1. [§4.2, Table 1, Table 3] The headline VLN-CE results are internally inconsistent. Section 4.2 states VEME achieves SR 57.0 and SPL 46.7 on R2R Val-Unseen, but Table 1 lists SPL 51.0. The same paragraph says ETPNav scores 42.0/36.5, while Table 1 lists ETPNav as 57.0/49.0. Table 3 reports the Full Model SPL as 65.1 on VLN-CE, 14.1 points above Table 1's 51.0, with no stated difference in split, metric, or checkpoint. These are not baseline-choice disagreements; the paper's own tables do not support a stable comparison.
  2. [Appendix C.4] The reported Wilcoxon signed-rank p-values are impossible for n=5. With five paired observations, the smallest achievable two-sided p-value is 2/2^5 = 0.0625 (and 1/32 = 0.03125 one-sided), assuming no ties. The values p=0.0003, 0.0005, 0.0007, and 0.0011 cannot be obtained from five samples. The statistical significance claims for the ablations are therefore unsupported and the reported numbers cannot be correct.
  3. [Table 2] The VSI-Bench 'Avg' column is not reproducible from the per-task scores. For VEME, the eight listed task accuracies average to 372.9/8 = 46.6, not the reported 49.3; for Spatial-MLLM they average to 375.6/8 = 47.0, not 48.4. No alternative weighting formula is given. As a result, the claim that VEME 'surpassed' Spatial-MLLM on average cannot be verified from the table.
  4. [Abstract and Appendix E] The abstract claims improvements in 'dynamic environments' and 'dynamic, open-set tasks', and the introduction repeats this framing. Appendix E explicitly states that 'our current framework operates under a static environment assumption. It is not designed to handle dynamic scenes with moving objects, interacting agents, or significant changes in layout.' Since the method relies on a complete global point cloud Pt at decision time (Section 3.1) and on geometric/episodic memory built from that static input, the dynamic-environment claim is overstated. This is a scope mismatch, not a minor wording issue.
minor comments (6)
  1. [§3.1] Heading contains a typo: 'Premilinaries' should be 'Preliminaries'.
  2. [References] The reference 'brian ichter et al. 2022' is not properly capitalized; the author name should be formatted consistently with other references.
  3. [A.3 / Table 5] LoRA rank is given as r=8 in the main text and Section A.2, but Table 5 lists LoRA Rank = 16. These need to be reconciled.
  4. [Appendix C.2] The 'Computational Cost and Efficiency' section appears to be empty in the submitted manuscript, immediately followed by C.3. Either content is missing or the heading is stray.
  5. [Table 2] The 'Rank' column is internally inconsistent: Gemini-1.5 Pro and Spatial-MLLM both receive rank 2, while the subsequent rank labels (3, 4, 5, ...) do not match a strict rank ordering. This makes the rank column confusing.
  6. [§4.2] The heading 'Computational Cost and Efficiency' appears placed within the visual navigation results paragraph, and there is no corresponding content there. This is likely a formatting artifact but should be fixed.

Circularity Check

0 steps flagged

No significant circularity: the claimed improvements are empirical benchmark results, not derived by construction from fitted parameters or self-citations.

full rationale

The paper's central claims are empirical: VEME is trained end-to-end on navigation and spatial-reasoning benchmarks, and its reported gains come from external evaluations (VLN-CE, VSI-Bench) against external baselines. The 'world embedding' Eworld is a learnable parameter matrix; the contrastive losses Lspatial and Lepisodic are standard training objectives that shape representations, but no benchmark number is computed from these losses by definition, nor is any 'prediction' equal to a fitted parameter. The method is not justified by a uniqueness theorem or by an ansatz imported from the authors' prior work; the few self-citations (e.g., Zhang et al. 2025, Sheng 2025) appear only in related-work context and do not carry the derivation. The internal numerical inconsistencies in the tables (e.g., differing SPL values for the same configuration) are correctness or reproducibility concerns, not circularity. The derivation chain is therefore self-contained with respect to circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claim rests on learned components (Eworld, Fepisodic, contrastive losses) rather than on parameter-free derivation. The free parameters are standard training choices, but several are inconsistently reported, and the data-overlap assumption is unverified.

free parameters (5)
  • World Embedding size Nw = not specified
    Number of learnable spatial concept tokens in Eworld in R^{Nw x D}; a core learned component of the cognitive map (Section 3.2).
  • Spatial loss weight lambda_s = 0.1
    Manually chosen trade-off between task loss and spatial contrastive loss (Section 4.1, Appendix Table 5).
  • Episodic loss weight lambda_e = 0.1
    Manually chosen trade-off between task loss and episodic contrastive loss (Section 4.1, Appendix Table 5).
  • Contrastive temperature tau = 0.07
    Temperature for both contrastive losses (Section 3.2, Appendix Table 5).
  • LoRA rank r = 8 (Section A.2) vs 16 (Appendix Table 5)
    Low-rank adaptation rank; reported inconsistently in two places, affecting reproducibility.
axioms (5)
  • domain assumption Contrastive alignment between 2D semantic features and 3D geometric features improves spatial reasoning and generalization.
    The central mechanism (Lspatial) is assumed to create useful grounding; no independent evidence is given.
  • domain assumption A learnable 'world embedding' can serve as a reusable cognitive map of spatial concepts.
    The model's capacity to generalize is assumed to come from Eworld, a randomly initialized matrix trained with gradients.
  • domain assumption Episodic memory formed from point clouds and action trajectories is useful for downstream navigation and QA.
    The episodic contrastive loss (Lepisodic) is assumed to yield discriminative memories that help the VLM.
  • domain assumption The training data does not overlap with the evaluation benchmarks.
    The zero-shot / unseen-scene claims depend on no VSI-Bench or VLN-CE val data appearing in the custom video QA dataset (Appendix A.1).
  • standard math Standard Transformer backpropagation and InfoNCE contrastive learning work as expected.
    Underlying the architecture; no formal verification.
invented entities (2)
  • World Embedding (Eworld) no independent evidence
    purpose: Acts as a learned spatial-semantic memory / cognitive map, included in the VLM input.
    A learnable parameter matrix; no external falsifiable prediction outside training.
  • Episodic feature (Fepisodic) no independent evidence
    purpose: Memory trace for a specific navigation episode, derived from point cloud and trajectory features.
    Computed from the model's own encoders; no independent handle beyond the reported clusters.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment." pith.science (2026). https://pith.science/paper/SQYMHGI4

@misc{pith2026250900210,
  author       = {Pith},
  title        = {Pith review of: Beyond Pixels: Introducing Geometric-Semantic World Priors for Video-based Embodied Models via Spatio-temporal Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQYMHGI4}},
  note         = {Machine review of arXiv:2509.00210}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Achieving human-like reasoning in deep learning models for complex tasks in unknown environments remains a critical challenge in embodied intelligence. While advanced vision-language models (VLMs) excel in static scene understanding, their limitations in spatio-temporal reasoning and adaptation to dynamic, open-set tasks like task-oriented navigation and embodied question answering (EQA) persist due to inadequate modeling of fine-grained spatio-temporal cues and physical world comprehension. To address this, we propose VEME, a novel cross-modal alignment method that enhances generalization in unseen scenes by learning an ego-centric, experience-centered world model. Our framework integrates three key components: (1) a cross-modal alignment framework bridging objects, spatial representations, and visual semantics with spatio-temporal cues to enhance VLM in-context learning; (2) a dynamic, implicit cognitive map activated by world embedding to enable task-relevant geometric-semantic memory recall; and (3) an instruction-based navigation and reasoning framework leveraging embodied priors for long-term planning and efficient exploration. By embedding geometry-aware spatio-temporal episodic experiences, our method significantly improves reasoning and planning in dynamic environments. Experimental results on VSI-Bench and VLN-CE demonstrate 1%-3% accuracy and exploration efficiency improvement compared to traditional approaches.

Figures

Figures reproduced from arXiv: 2509.00210 by Jinzhou Tang, Jusheng Zhang, Qinhan Lv, Sidi Liu, Waikit Xiu, Xiying Li.

Figure 1
Figure 1. Figure 1: An illustrative comparison of navigation behaviors. (a) Our agent, equipped with a cognitively inspired dual￾memory framework, constructs an implicit cognitive map from world embedding, observation history, and past trajectory. This allows it to ground the complex spatial instruction (“kitchen behind the sofa”) in the environment, devise an efficient path, and successfully reach the goal. (b) In contrast, … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our Dual-Memory Framework for Spatial Intelligence. Our framework endows a Vision-Language Model with robust spatial intelligence through two synergistic memory systems. The Spatial Semantic Memory learns a general cognitive map by aligning 2D visual features with 3D geometric features via a spatial contrastive loss . Concurrently, the Episodic Memory creates unique memory traces for specific e… view at source ↗
Figure 3
Figure 3. Figure 3: Visualizations of Qualitative Study. Our data for￾mulation includes navigational action planning (i.e., VLN￾CE) and Spatio-temporal Reasoning (i.e., VSI-Bench). The entire architecture is trained end-to-end with a com￾posite objective that reflects our cognitive design: Ltotal = LCE + λsLspatial + λeLepisodic (12) Here, the primary task loss (e.g., cross-entropy for text gen￾eration, LCE) is guided by our … view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results from VLN-CE. We deploy VEME in the simulation environment (i.e., Habitat) for long-horizon navigation task. Given an instruction, the agent moves through different areas of the house and stops at the specified goal. As shown in the figure, its navigation path aligns well with the commands [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: t-SNE and distance distribution visualization of episodic features (Fepisodic). Left: Each point corresponds to a full navigation episode. Points are colored by their environment ID. The clear clustering of points of the same color demon￾strates that our model learns discriminative representations for different spatio-temporal experiences. Right: Blue distribution represents intra-scene episodic feature di… view at source ↗
Figure 6
Figure 6. Figure 6: Failure case illustration with the instruction: ”Walk [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 39 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    An, D.; Wang, H.; Wang, W.; Wang, Z.; Huang, Y.; He, K.; and Wang, L. 2024. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2024

  4. [4]

    Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; Gould, S.; and van den Hengel, A. 2018. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), Spotlight Oral

    work page 2018

  5. [6]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025 b . Qwen2.5-VL Technical Report. ArXiv, abs/2502.13923

    Pith/arXiv arXiv 2025

  6. [7]

    J.; and Wilber, A

    Bermudez-Contreras, E.; Clark, B. J.; and Wilber, A. 2020. The Neuroscience of Spatial Navigation and the Relationship to Artificial Intelligence. Frontiers in Computational Neuroscience, Volume 14 - 2020

    work page 2020

  7. [8]

    brian ichter; Brohan, A.; Chebotar, Y.; Finn, C.; Hausman, K.; Herzog, A.; Ho, D.; Ibarz, J.; Irpan, A.; Jang, E.; Julian, R.; Kalashnikov, D.; Levine, S.; Lu, Y.; Parada, C.; Rao, K.; Sermanet, P.; Toshev, A. T.; Vanhoucke, V.; Xia, F.; Xiao, T.; Xu, P.; Yan, M.; Brown, N.; Ahn, M.; Cortes, O.; Sievers, N.; Tan, C.; Xu, S.; Reyes, D.; Rettinghouse, J.; Q...

    work page 2022

  8. [9]

    Chang, A.; Dai, A.; Funkhouser, T.; Halber, M.; Niessner, M.; Savva, M.; Song, S.; Zeng, A.; and Zhang, Y. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. International Conference on 3D Vision (3DV)

    work page 2017

  9. [10]

    Chen, G.; Pan, L.; Chen, Y.; Xu, P.; Wang, Z.; Wu, P.; Ji, J.; and Chen, X. 2020. Robot Navigation with Map-Based Deep Reinforcement Learning. In Proceedings of the IEEE International Conference on Networking, Sensing and Control (ICNSC), 1--6

    work page 2020

  10. [11]

    Chen, K.; Xiao, J.; Liu, J.; Tong, Q.; Zhang, H.; Liu, R.; Zhang, J.; Ajoudani, A.; and Chen, S. 2025. Semantic visual simultaneous localization and mapping: A survey. IEEE Transactions on Intelligent Transportation Systems

    work page 2025

  11. [12]

    Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR

    work page 2024

  12. [13]

    Cheng, A.-C.; Ji, Y.; Yang, Z.; Gongye, Z.; Zou, X.; Kautz, J.; Bıyık, E.; Yin, H.; Liu, S.; and Wang, X. 2025 a . NaVILA: Legged Robot Vision-Language-Action Model for Navigation. arXiv:2412.04453

    Pith/arXiv arXiv 2025

  13. [14]

    Cheng, A.-C.; Ji, Y.; Yang, Z.; Zou, X.; Kautz, J.; Biyik, E.; Yin, H.; Liu, S.; and Wang, X. 2025 b . NaVILA: Legged Robot Vision-Language-Action Model for Navigation. In RSS

    work page 2025

  14. [15]

    Coppolino, S.; and Migliore, M. 2023. An explainable artificial intelligence approach to spatial navigation based on hippocampal circuitry. Neural Networks, 163: 97--107

    work page 2023

  15. [16]

    X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nie ner, M

    Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nie ner, M. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE

    work page 2017

  16. [17]

    A.; Patai, E

    Epstein, R. A.; Patai, E. Z.; Julian, J. B.; and Spiers, H. J. 2017. The cognitive map in humans: spatial navigation and beyond. Nature Neuroscience, 20: 1504--1513

    work page 2017

  17. [18]

    Hong, Y.; Wang, Z.; Wu, Q.; and Gould, S. 2022. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 15439--15449

    work page 2022

  18. [19]

    Hong, Y.; Wu, Q.; Qi, Y.; Rodriguez‑Opazo, C.; and Gould, S. 2021. VLN‑BERT: A Recurrent Vision‑and‑Language BERT for Navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2021

  19. [20]

    Huang, J.; Yong, S.; Ma, X.; Linghu, X.; Li, P.; Wang, Y.; Li, Q.; Zhu, S.-C.; Jia, B.; and Huang, S. 2024. An Embodied Generalist Agent in 3D World. In Proceedings of the International Conference on Machine Learning (ICML)

    work page 2024

  20. [21]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al

    Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    Pith/arXiv arXiv 2024

  21. [22]

    Jia, Z.; Lin, K.; Zhao, Y.; Gao, Q.; Thattai, G.; and Sukhatme, G. S. 2022. Learning to Act with Affordance-Aware Multimodal Neural SLAM

    work page 2022

  22. [23]

    Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Li, Y.; Liu, Z.; and Li, C. 2024 a . LLaVA-OneVision: Easy Visual Task Transfer. arXiv preprint arXiv:2408.03326

    Pith/arXiv arXiv 2024

  23. [24]

    Li, J.; Zhang, N.; Qu, X.; Lu, K.; Li, G.; Wan, J.; and Wang, J. 2025. RATE-Nav: Region-Aware Termination Enhancement for Zero-shot Object Navigation with Vision-Language Models. In ACL (Findings), 6564--6574. Association for Computational Linguistics

    work page 2025

  24. [25]

    Li, K.; Yu, B.; Zheng, Q.; Zhan, Y.; Zhang, Y.; Zhang, T.; Yang, Y.; Chen, Y.; Sun, L.; Cao, Q.; Shen, L.; Li, L.; Tao, D.; and He, X. 2024 b . MuEP: A Multimodal Benchmark for Embodied Planning with Foundation Models. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI 2024), Main Track, 129--138. Internati...

    work page 2024

  25. [26]

    Liang, J.; Wang, Z.; Cao, Y.; Chiun, J.; Zhang, M.; and Sartoretti, G. A. 2023. Context-Aware Deep Reinforcement Learning for Autonomous Robotic Navigation in Unknown Area. In Tan, J.; Toussaint, M.; and Darvish, K., eds., Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, 1425--1436. PMLR

    work page 2023

  26. [27]

    Lin, J.; Yin, H.; Ping, W.; Lu, Y.; Molchanov, P.; Tao, A.; Mao, H.; Kautz, J.; Shoeybi, M.; and Han, S. 2023. VILA: On Pre-training for Visual Language Models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 26679--26689

    work page 2023

  27. [28]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning

    work page 2023

  28. [29]

    Liu, X.; Armstrong, V.; Nabil, S.; and Muise, C. 2021. Exploring multi-view perspectives on deep reinforcement learning agents for embodied object navigation in virtual home environments. In Proceedings of the 31st Annual International Conference on Computer Science and Software Engineering, CASCON '21, 190–195. USA: IBM Corp

    work page 2021

  29. [30]

    Liu, Y.; Chen, W.; Bai, Y.; Liang, X.; Li, G.; Gao, W.; and Lin, L. 2025. Aligning cyber space with physical world: A comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics

    work page 2025

  30. [31]

    Ma, X.; Yong, S.; Zheng, Z.; Li, Q.; Liang, Y.; Zhu, S.-C.; and Huang, S. 2023. SQA 3D: Situated Question Answering in 3D Scenes. In The Eleventh International Conference on Learning Representations

    work page 2023

  31. [32]

    Moscovitch, M.; Rosenbaum, R.; Gilboa, A.; Addis, D.; Westmacott, R.; Grady, C.; McAndrews, M.; Levine, B.; Black, S.; Winocur, G.; et al. 2005. Functional neuroanatomy of remote episodic, semantic and spatial memory: a unified account based on multiple trace theory. Journal of Anatomy, 207(1): 35--66

    work page 2005

  32. [33]

    Nadel, L.; and Hardt, O. 2011. Update on memory systems and processes. Neuropsychopharmacology, 36(1): 251--273

    work page 2011

  33. [34]

    Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H. V.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; Howes, R.; Huang, P.-Y.; Xu, H.; Sharma, V.; Li, S.-W.; Galuba, W.; Rabbat, M.; Assran, M.; Ballas, N.; Synnaeve, G.; Misra, I.; Jegou, H.; Mairal, J.; Labatut, P.; Joulin, A.; and Bojanowski, P. 2023. DINOv2: Learning Robust...

    2023

  34. [35]

    S.; O'Brien, J.; Cai, C

    Park, J. S.; O'Brien, J.; Cai, C. J.; Morris, M. R.; Liang, P.; and Bernstein, M. S. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, 1--22

    work page 2023

  35. [36]

    Z.; Clark, J.; Dixit, A.; Itkina, M.; Majumdar, A.; and Sadigh, D

    Ren, A. Z.; Clark, J.; Dixit, A.; Itkina, M.; Majumdar, A.; and Sadigh, D. 2024. Explore until Confident: Efficient Exploration for Embodied Question Answering. In arXiv preprint arXiv:2403.15941

    Pith/arXiv arXiv 2024

  36. [37]

    Sarch, G.; Wu, Y.; Tarr, M.; and Fragkiadaki, K. 2023. Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 3468--3500. Singapore: Association for Computational Linguistics

    work page 2023

  37. [38]

    T.; Sunny, M

    Shahria, M. T.; Sunny, M. S. H.; Zarif, M. I. I.; Ghommam, J.; Ahamed, S. I.; and Rahman, M. H. 2022. A comprehensive review of vision-based robotic applications: Current state, components, approaches, barriers, and potential solutions. Robotics, 11(6): 139

    work page 2022

  38. [39]

    Sheng, J. Z. 2025. GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning . https://arxiv.org/abs/2505.23399. ArXiv:2505.23399, arXiv:2505.23399

    Pith/arXiv arXiv 2025

  39. [40]

    I.; Burnell, R.; Bai, L.; Gulati, A.; Tanzer, G.; Vincent, D.; Pan, Z.; Wang, S.; et al

    Team, G.; Georgiev, P.; Lei, V. I.; Burnell, R.; Bai, L.; Gulati, A.; Tanzer, G.; Vincent, D.; Pan, Z.; Wang, S.; et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    Pith/arXiv arXiv 2024

  40. [41]

    Wang, H.; Liang, W.; Van Gool, L.; and Wang, W. 2023. Dreamwalker: Mental planning for continuous vision-language navigation. In Proceedings of the IEEE/CVF international conference on computer vision, 10873--10883

    work page 2023

  41. [42]

    Wang, J.; Chen, M.; Karaev, N.; Vedaldi, A.; Rupprecht, C.; and Novotny, D. 2025. VGGT: Visual Geometry Grounded Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    work page 2025

  42. [43]

    Wu, D.; Liu, F.; Hung, Y.-H.; and Duan, Y. 2025 a . Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence. arXiv preprint arXiv:2505.23747

    Pith/arXiv arXiv 2025

  43. [44]

    Wu, X.; DeTone, D.; Frost, D.; Shen, T.; Xie, C.; Yang, N.; Engel, J.; Newcombe, R.; Zhao, H.; and Straub, J. 2025 b . Sonata: Self-Supervised Learning of Reliable Point Representations. In CVPR

    work page 2025

  44. [45]

    Xue, F.; Chen, Y.; Li, D.; Hu, Q.; Zhu, L.; Li, X.; Fang, Y.; Tang, H.; Yang, S.; Liu, Z.; He, E.; Yin, H.; Molchanov, P.; Kautz, J.; Fan, L.; Zhu, Y.; Lu, Y.; and Han, S. 2024. LongVILA: Scaling Long-Context Visual Language Models for Long Videos. ArXiv, abs/2408.10188

    Pith/arXiv arXiv 2024

  45. [46]

    Yang, J.; Yang, S.; Gupta, A.; Han, R.; Fei-Fei, L.; and Xie, S. 2024. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces . arXiv preprint arXiv:2412.14171

    Pith/arXiv arXiv 2024

  46. [47]

    W.; Han, R.; Fei-Fei, L.; and Xie, S

    Yang, J.; Yang, S.; Gupta, A. W.; Han, R.; Fei-Fei, L.; and Xie, S. 2025. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, 10632--10643

    work page 2025

  47. [48]

    Yin, B.; Wang, Q.; Zhang, P.; Zhang, J.; Wang, K.; Wang, Z.; Zhang, J.; Chandrasegaran, K.; Liu, H.; Krishna, R.; et al. 2025. Spatial Mental Modeling from Limited Views. arXiv preprint arXiv:2506.21458

    arXiv 2025

  48. [49]

    Yu, B.; Kasaei, H.; and Cao, M. 2023. L3MVN: Leveraging Large Language Models for Visual Target Navigation. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3554–3560. IEEE

    work page 2023

  49. [50]

    Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, 11975--11986

    work page 2023

  50. [51]

    Zhang, J.; Huang, Z.; Fan, Y.; Liu, N.; Li, M.; Yang, Z.; Yao, J.; Wang, J.; and Wang, K. 2025. KABB : Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems. In Forty-second International Conference on Machine Learning

    work page 2025

  51. [52]

    Zhang, J.; Wang, K.; Xu, R.; Zhou, G.; Hong, Y.; Fang, X.; Wu, Q.; Zhang, Z.; and Wang, H. 2024 a . Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852

    Pith/arXiv arXiv 2024

  52. [53]

    Zhang, P.; Zhang, K.; Li, B.; Zeng, G.; Yang, J.; Zhang, Y.; Wang, Z.; Tan, H.; Li, C.; and Liu, Z. 2024 b . Long Context Transfer from Language to Vision. ArXiv, abs/2406.16852

    Pith/arXiv arXiv 2024

  53. [54]

    Zhang, Y.; Wu, J.; Li, W.; Li, B.; Ma, Z.; Liu, Z.; and Li, C. 2024 c . Video Instruction Tuning With Synthetic Data. ArXiv, abs/2410.02713

    Pith/arXiv arXiv 2024

  54. [55]

    Zheng, D.; Huang, S.; and Wang, L. 2025. Video-3d llm: Learning position-aware video representation for 3d scene understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, 8995--9006

    work page 2025

  55. [56]

    Zheng, Q.; Liu, D.; Wang, C.; Zhang, J.; Wang, D.; and Tao, D. 2025. Vision-and-Language Navigation with Episodic Scene Memory. International Journal of Computer Vision, 133(1): 254--274

    work page 2025

  56. [57]

    Zou, X.; Song, Y.; Qiu, R.-Z.; Peng, X.; Ye, J.; Liu, S.; and Wang, X. 2025. 3D-SPATIAL MULTIMODAL MEMORY. In The Thirteenth International Conference on Learning Representations

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.