Pith. sign in

REVIEW 5 major objections 7 minor 40 references

VL-KnG claims that a robot can compress an egocentric tour video into a spatiotemporal knowledge graph once, then answer navigation questions from the graph alone in about a second, matching frontier vision-language models on accuracy while

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:51 UTC pith:EEA4VGTI

load-bearing objection A sensible graph-memory pipeline with a useful new benchmark, but the abstract overclaims and the VLM descriptor bottleneck deserves a hard look. the 5 major comments →

arxiv 2510.01483 v3 pith:EEA4VGTI submitted 2025-10-01 cs.RO cs.AI

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

Mohamad Al Mdfaa , Svetlana Lukina , Timur Akhtyamov , Arthur Nigmatzyanov , Dmitrii Nalberskii , Sergey Zagoruyko , Gonzalo Ferrer This is my paper
classification cs.RO cs.AI
keywords spatiotemporal knowledge graphembodied question answeringvisual navigationscene memoryvision-language modelsgraph retrievalobject associationrobotic navigation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that a robot can answer natural-language questions about an environment by first compressing an egocentric tour video into a spatiotemporal knowledge graph, then querying only that graph at runtime. The graph is built chunk-by-chunk with a vision-language model extracting object descriptors and a large language model merging repeated objects into persistent identities, so raw video never needs to be re-processed for a new query. On three embodied question-answering benchmarks, including a new one called WalkieKnowledge with about 200 manually annotated questions across eight indoor and outdoor trajectories, the method reaches accuracy competitive with frontier VLMs while answering in about a second rather than minutes. On a real differential-drive robot, it matches the strongest general-purpose VLM with 77% navigation success and 77% answer accuracy, while keeping query latency stable as observation history grows. The significance is a concrete, updatable scene memory that could replace per-query video reprocessing in embodied agents.

Core claim

VL-KnG's central claim is that a persistent, queryable knowledge graph is a sufficient substrate for embodied question answering: once the graph is constructed from a demonstration tour video, query-time reasoning operates entirely on the graph, and correctness no longer depends on re-examining the raw frames. Construction proceeds in chunks of eight frames; a vision-language model produces object descriptors (bounding box, color, material, size, affordances, spatial relationships) for each chunk, and an LLM-based Spatiotemporal Object Association step merges chunk graphs by semantic similarity, maintaining a single identity for each physical object across time. At query time, a Graph-Enhanc

What carries the argument

The central object is the spatiotemporal knowledge graph G=(V,E), whose nodes are persistent object instances carrying rich textual descriptors—bounding box, id, color, material, size, timestamp, affordances, and spatial relationships—and whose edges encode spatial relations between objects. Two mechanisms carry the argument: Spatiotemporal Object Association (STOA), an LLM-based semantic similarity threshold that fuses chunk-level graphs into one global graph while preserving object identity across appearance changes; and Graph-Enhanced Retrieval (GER), a graph-retrieval pipeline that decomposes a query, retrieves a local subgraph, and reasons over it. The graph serves as persistent scene m

Load-bearing premise

Answers are capped by whatever the per-chunk vision-language model chooses to put in its object descriptors: if the VLM misses a small object, misreads a color, or fails to note a spatial relation in the few frames of a chunk, no later retrieval or reasoning step can recover that fact, and this bottleneck is not validated against the raw frames.

What would settle it

Take a WalkieKnowledge trajectory and a query whose target object is clearly visible in the raw video but absent from the chunk descriptors produced by the vision-language model; if a whole-video VLM with full context answers correctly while VL-KnG does not, the descriptor bottleneck is confirmed. More systematically, compare per-chunk descriptor coverage against a manually annotated object inventory for the benchmark trajectories—if coverage is below answer quality on the objects of interest, the graph's recall ceiling is exposed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Query latency becomes independent of video length, dropping from about two minutes of direct VLM inference to roughly one second, enabling real-time interaction.
  • The same graph supports multiple query types—object search, scene description, action-place association, and spatial relations—without rebuilding the representation.
  • Structured graph memory provides explainable, traceable reasoning grounded in specific objects and relations visible in the tour.
  • Persistent object identity via semantic association is robust to lighting, occlusion, and viewpoint changes that break visual similarity trackers.
  • The graph representation can be updated incrementally and is compatible with vision-only navigation policies, as well as classic metric navigation stacks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the graph's facts all derive from per-chunk VLM descriptors, improving descriptor coverage—prompting for small objects, materials, or richer spatial predicates—would likely lift QA accuracy more than better graph reasoning; this is a testable prediction.
  • The decoupling of query cost from video length suggests the same construction-then-query pattern could apply to other long-horizon video tasks such as video summarization, anomaly detection, or locating rare events, where a one-time indexing pass amortizes over many queries.
  • The WalkieKnowledge benchmark, with its four question types and frame-interval ground truth, could serve as a reusable testbed for comparing structured memory against long-context VLMs as both improve.
  • If the graph is meant to be a persistent scene memory, the unresolved issue of dynamic environments (objects moving, appearing, disappearing) will require a graph-update protocol; the paper's modular architecture suggests this can be added without changing the query pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes VL-KnG, a training-free pipeline that converts an egocentric tour video into a persistent spatiotemporal knowledge graph. Video is processed in chunks; a VLM extracts object descriptors and relations for each chunk; an LLM-based association step (STOA) merges objects across chunks; and a GraphRAG-style query processor retrieves subgraphs and performs reasoning to answer natural-language questions and localize goal frames. The authors introduce a new benchmark, WalkieKnowledge (~193 questions on 8 trajectories), report comparisons against several VLMs and prior navigation-memory baselines, and describe a real-robot deployment. The main claims are that the graph decouples query latency from video length and that answer accuracy is competitive with frontier VLMs.

Significance. Persistent, explainable scene memory for embodied question answering is a timely and useful goal, and the paper has several genuine strengths: a real hardware deployment, external VLM baselines on the new benchmark (which gives some independent grounding), a modular design that separates graph construction from retrieval and reasoning, and the release of a new benchmark with four query types. If the accuracy and latency claims were properly supported, the contribution would be of interest to the embodied-AI and robotics communities. However, the evidence in the current manuscript is thin in statistical terms and, on the paper's own Table II, does not clearly support the headline claim of frontier-VLM-competitive accuracy. The central ideas are plausible, but the manuscript needs substantial additional validation and honest recalibration of its claims before it can be recommended for publication.

major comments (5)
  1. [Abstract; §VI-B, Table II] The central claim that VL-KnG is "accuracy competitive with frontier VLMs" is not supported by the paper's own main benchmark. In Table II, Ours(Full KG) achieves Answer Accuracy 58.1% and Retrieval Acc@1 57.5%, while Gemini Flash achieves 66.3% / 61.7% and Gemini Pro 61.6% / 68.9%. The abstract also states that results on OpenEQA, NaVQA, and WalkieKnowledge are reported, but the body contains only WalkieKnowledge. Either add the missing benchmark results or substantially soften the claim to "competitive with open-weight VLMs" / "better than prior persistent-representation baselines".
  2. [§IV-A, §VI-B] The graph is built exclusively from per-chunk VLM object descriptors. Any object, attribute, or spatial relation not emitted by the descriptor VLM is permanently absent; no subsequent retrieval or LLM step can recover it. The paper never validates descriptor completeness against the raw frames. Table II is consistent with this bottleneck: Ours(R) Answer Acc 50.0 and Ours(F) 58.1 versus Gemini Flash 66.3, indicating information loss relative to direct video access. The authors should test for this by comparing graph contents against raw-frame inspection on a sample of chunks, or by reporting an error analysis of which questions fail and why.
  3. [§VI-B, Table II; §VI-C, Table I] Statistical support is weak. The benchmark contains 193 questions over 8 trajectories, but no error bars, confidence intervals, or significance tests are reported. In the hardware experiment, VL-KnG and Gemini 2.5 Pro are reported as identical at 77.27% success and 76.92% answer accuracy, but the number of trials is not stated; with a small N, identical percentages are not evidence of matched performance. Report trial counts and uncertainty intervals, and use appropriate tests for the claimed equivalences and for the "significantly outperforms" statements about CWR and RoboHop.
  4. [§IV-B, §VI-D] The two main hyperparameters — chunk size b and association threshold τ — are tuned on the same 193-question benchmark, and only b=8 is reported. No sensitivity analysis is given, and τ in Eq. (3) is never specified, despite being load-bearing for object identity. Report the chosen τ, a sensitivity sweep for both parameters, and ideally a separate validation split to guard against overfitting to this small benchmark.
  5. [§IV-D, §VI-C] The latency claim ("~1 s" vs "~120 s" for Gemini 2.5 Pro) is presented without a measurement protocol, hardware/API details, or the number of queries timed. The complexity bound O(|V_sub|+|E_sub|+|Q|) is a trivial property of subgraph retrieval and does not by itself establish the "decoupling from video length" claim. Add an empirical latency-vs-video-length plot (or at least report the measurement conditions and variance) to substantiate the main efficiency advantage.
minor comments (7)
  1. [§IV-A, Eq. (1)] The chunk definition uses k=0,...,B with B=floor(T/b)-1; if T is not divisible by b, the final partial chunk is silently dropped. Specify handling of the tail frames.
  2. [§IV-B, Eq. (2)-(3)] The LLM similarity function Sim is defined informally as mapping two textual descriptors to [0,1], but no prompt is given and no instructions are provided for how the LLM should calibrate scores. Reproducibility would benefit from including the exact prompt and a few worked association examples.
  3. [§IV-C] The method description mentions "visual grounding" as part of Graph-Enhanced Retrieval, but the paper never defines how visual grounding is implemented or how it interacts with subgraph retrieval. Clarify this component.
  4. [Fig. 3] The two-stage prompt template pipeline is referenced in the figure caption but the actual templates are not included in the text or an appendix. Since the whole system depends on these prompts, they should be provided in full.
  5. [Table III] The table mixes percentage values and raw counts without clear notation (e.g., some entries appear to be counts, such as "33"). Add explicit units and avoid ambiguous formatting.
  6. [Table IV] MRR@3 and MRR@5 are identical to MRR@1 for several rows; this is likely due to ties or to the evaluation script, but it should be explained or corrected.
  7. [§V] WalkieKnowledge is built on EgoWalk, but the relationship to EgoWalk's existing annotations and the exact annotation procedure for the 193 questions are not described in enough detail. State how many annotators were used and how inter-annotator agreement was measured.

Circularity Check

0 steps flagged

No derivation reducible to its own inputs; only a minor self-benchmark concern keeps the score slightly above zero.

full rationale

VL-KnG is presented as a pipeline rather than as a derived mathematical result. The graph is constructed from VLM descriptors via Eq. (1) and the STOA association in Eqs. (2)-(3); query answers are produced by subgraph retrieval plus LLM reasoning. No fitted parameter or intermediate output is later renamed as a prediction: the chunk size b is tuned in Sec. VI-D, but no headline accuracy result is defined as b or as the association threshold τ. The main accuracy claims are evaluated against external Gemini and Qwen models on the same WalkieKnowledge questions, so the comparisons do not reduce to the system's own construction. The self-referential element is that WalkieKnowledge is built on EgoWalk [38], whose authors overlap with the current paper; however, the questions are manually annotated and the external VLM baselines provide independent grounding. This is an evaluation-validity note rather than a circular derivation. The information-bottleneck issue — that the graph can only contain what the per-chunk VLM descriptors emit — is an empirical limitation, not a definitional circularity. No specific equation or fitted quantity can be exhibited that makes a claimed result equivalent to its input by construction. The score of 2 reflects the minor self-benchmark/self-citation shadow, not a reducible derivation chain.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central results depend on the quality of VLM and LLM outputs. The only tuned hyperparameters are chunk size and association threshold, but the bigger burden is the domain assumption that text descriptors preserve the spatial and visual information needed to answer queries.

free parameters (2)
  • chunk size b = 8
    Tuned in Section VI-D on the WalkieKnowledge benchmark as the optimal balance; affects which objects are seen per VLM pass and the association problem.
  • association threshold τ = not reported
    Used in Eq. (3) to decide whether objects from different chunks are the same; no value or sensitivity analysis is given.
axioms (3)
  • domain assumption VLM-generated textual object descriptors are accurate and complete enough to support downstream QA
    Invoked throughout Section IV-A; the entire graph is built from these descriptors, so any omission or error bounds all later reasoning.
  • domain assumption LLM-based semantic similarity correctly associates objects across chunks
    Section IV-B; STOA relies on textual descriptions and a threshold to maintain identity, with no quantitative validation of association errors.
  • domain assumption Pose estimates from SLAM accurately localize the goal frame for navigation
    Section VI-C; the goal pose is taken from the tour frame's pose, so SLAM drift would directly reduce navigation success.

pith-pipeline@v1.3.0-alltime-deepseek · 10484 in / 10674 out tokens · 77621 ms · 2026-08-04T12:51:59.902499+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering." pith.science (2026). https://pith.science/paper/EEA4VGTI

@misc{pith2026251001483,
  author       = {Pith},
  title        = {Pith review of: Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EEA4VGTI}},
  note         = {Machine review of arXiv:2510.01483}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query. We present VL-KnG, a training-free framework that constructs spatiotemporal knowledge graphs from monocular egocentric video, bridging fine-grained scene graphs and global topological graphs without 3D reconstruction. VL-KnG processes video in chunks, maintains persistent object identities via large language model (LLM)-based Spatiotemporal Object Association (STOA), and answers questions through Graph-Enhanced Retrieval (GER), which combines subgraph retrieval with visual grounding. Once constructed, the knowledge graph removes the need to revisit raw video at query time, decoupling query latency from video length. Across three embodied question answering benchmarks -- OpenEQA, NaVQA, and our newly introduced WalkieKnowledge -- VL-KnG achieves accuracy competitive with frontier VLMs while answering queries at substantially lower latency and with explainable, graph-grounded reasoning; it further surpasses prior persistent-representation baselines and open-weight VLMs in several settings. Deployment on a real robot demonstrates practical applicability, with query latency remaining stable as observation history grows. As a persistent, queryable scene memory, VL-KnG provides a concrete substrate for memory maintenance and knowledge updating in embodied AI agents.

Figures

Figures reproduced from arXiv: 2510.01483 by Arthur Nigmatzyanov, Dmitrii Nalberskii, Gonzalo Ferrer, Mohamad Al Mdfaa, Sergey Zagoruyko, Svetlana Lukina, Timur Akhtyamov.

Figure 1
Figure 1. Figure 1: Real-world deployment examples of VL-KnG [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: VL-KnG system architecture showing the complete pipeline from video frame input to navigation goal [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: VL-KnG employs a two-stage prompt template pipeline for spatiotemporal knowledge graph construction [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The WalkieKnowledge Benchmark includes ∼ 200 questions across 8 trajectories, with question types distributed according to the environment (in￾door/outdoor). both retrieval and answer metrics. Retrieval Accuracy@k checks whether the correct frames appear among the top￾k results, showing if the system can actually find the right moment in the video. Answer Accuracy is defined for multiple choice questions, … view at source ↗
Figure 5
Figure 5. Figure 5: Examples from the Walkie-Knowledge Dataset, covering diverse indoor and outdoor environments such as [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 11 linked inside Pith

  1. [1]

    What’s in the image? a deep- dive into the vision of vision language models,

    O. Kaduri, S. Bagon, and T. Dekel, “What’s in the image? a deep- dive into the vision of vision language models,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 14 549–14 558

    2025

  2. [2]

    A survey of attacks on large vision–language models: Resources, advances, and future trends,

    D. Liu, M. Yang, X. Qu, P. Zhou, Y . Cheng, and W. Hu, “A survey of attacks on large vision–language models: Resources, advances, and future trends,”IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  3. [3]

    Wmnav: Integrating vision-language models into world models for object goal navigation,

    D. Nie, X. Guo, Y . Duan, R. Zhang, and L. Chen, “Wmnav: Integrating vision-language models into world models for object goal navigation,”arXiv preprint arXiv:2503.02247, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Robohop: Segment-based topological map representation for open-world visual navigation,

    S. Garg, K. Rana, M. Hosseinzadeh, L. Mares, N. S ¨underhauf, F. Dayoub, and I. Reid, “Robohop: Segment-based topological map representation for open-world visual navigation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 4090–4097. TABLE II: Overall Performance Comparison of All Models. All metrics are reported a...

    2024

  5. [5]

    Remembr: Building and reasoning over long-horizon spatio- temporal memory for robot navigation,

    A. Anwar, J. Welsh, J. Biswas, S. Pouya, and Y . Chang, “Remembr: Building and reasoning over long-horizon spatio- temporal memory for robot navigation,” in2025 IEEE Interna- tional Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 2838–2845

    2025

  6. [6]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Knowledge graphs as tools for explainable machine learning: A survey,

    I. Tiddi and S. Schlobach, “Knowledge graphs as tools for explainable machine learning: A survey,”Artificial Intelligence, vol. 302, p. 103627, 2022

    2022

  9. [9]

    An introduction to vision-language modeling,

    F. Bordes, R. Y . Pang, A. Ajay, A. C. Li, A. Bardes, S. Petryk, O. Ma ˜nas, Z. Lin, A. Mahmoud, B. Jayaramanet al., “An introduction to vision-language modeling,”arXiv preprint arXiv:2405.17247, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    A review of multi-modal large language and vision models,

    K. Carolan, L. Fennelly, and A. F. Smeaton, “A review of multi-modal large language and vision models,”ArXiv, vol. abs/2404.01322, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,”IEEE transactions on pattern analysis and machine intelligence, vol. 46, no. 8, pp. 5625–5644, 2024

    2024

  12. [12]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,”arXiv preprint arXiv:2304.10592, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    From local to global: A graph rag approach to query-focused summa- rization,

    D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson, “From local to global: A graph rag approach to query-focused summa- rization,”arXiv preprint arXiv:2404.16130, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Vision-language navigation: a survey and taxonomy,

    W. Wu, T. Chang, X. Li, Q. Yin, and Y . Hu, “Vision-language navigation: a survey and taxonomy,”Neural Computing and Applications, vol. 36, no. 7, pp. 3291–3316, 2024

    2024

  15. [15]

    Visual language maps for robot navigation,

    C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in2023 IEEE International Con- ference on Robotics and Automation (ICRA). London, United Kingdom: IEEE, 2023, pp. 10 608–10 615

    2023

  16. [16]

    Conceptgraphs: Open-vocabulary 3d scene graphs for percep- tion and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappaet al., “Conceptgraphs: Open-vocabulary 3d scene graphs for percep- tion and planning,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5021–5028

    2024

  17. [17]

    Lm-nav: Robotic nav- igation with large pre-trained models of language, vision, and action,

    D. Shah, B. Osi ´nski, S. Levineet al., “Lm-nav: Robotic nav- igation with large pre-trained models of language, vision, and action,” inConference on robot learning. PMLR, 2023, pp. 492–504

    2023

  18. [18]

    Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world,

    K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.- H. Zeng, K. P. Singh, Y . Kim, W. Han, A. Herrastiet al., “Spoc: Imitating shortest paths in simulation enables effective navigation and manipulation in the real world,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 238–16 250

    2024

  19. [19]

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,

    P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. S ¨underhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3674–3683

    2018

  20. [20]

    Speaker-follower models for vision-and-language navigation,

    D. Fried, R. Hu, V . Cirik, A. Rohrbach, J. Andreas, L.- P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell, “Speaker-follower models for vision-and-language navigation,”Advances in neural information processing systems, vol. 31, 2018

    2018

  21. [21]

    Reinforced cross- modal matching and self-supervised imitation learning for vision- language navigation,

    X. Wang, Q. Huang, A. Celikyilmaz, J. Gao, D. Shen, Y .- F. Wang, W. Y . Wang, and L. Zhang, “Reinforced cross- modal matching and self-supervised imitation learning for vision- language navigation,” inProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, 2019, pp. 6629–6638

    2019

  22. [22]

    Cows on pasture: Baselines and benchmarks for language- driven zero-shot object navigation,

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language- driven zero-shot object navigation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2023, pp. 23 171–23 181

    2023

  23. [23]

    Navrag: Generat- ing user demand instructions for embodied navigation through retrieval-augmented llm,

    Z. Wang, Y . Zhu, G. H. Lee, and Y . Fan, “Navrag: Generat- ing user demand instructions for embodied navigation through retrieval-augmented llm,” inFindings of the Association for Computational Linguistics (ACL 2025). Singapore: Association for Computational Linguistics, 2025, pp. 442–456

    2025

  24. [24]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  25. [25]

    Cliprover: Zero-shot vision-language exploration and target discovery by mobile robots,

    Y . Zhang, A. Abdullah, S. J. Koppal, and M. J. Islam, “Cliprover: Zero-shot vision-language exploration and target discovery by mobile robots,”arXiv preprint arXiv:2502.08791, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Mobility vla: Multimodal instruction navigation with long-context vlms and topological graphs,

    H.-T. L. Chiang, Z. Xu, Z. Fu, M. G. Jacob, T. Zhang, T.- W. E. Lee, W. Yu, C. Schenck, D. Rendleman, D. Shah, F. Xia, J. Hsu, J. Hoech, P. Florence, S. Kirmani, S. Singh, V . Sindhwani, C. Parada, C. Finn, P. Xu, S. Levine, and J. Tan, “Mobility vla: Multimodal instruction navigation with long-context vlms and topological graphs,” inConference on Robot L...

    2024

  27. [27]

    Navid: Video-based vlm plans the next step for vision-and-language navigation,

    J. Zhang, K. Wang, R. Xu, G. Zhou, Y . Hong, X. Fang, Q. Wu, Z. Zhang, and H. Wang, “Navid: Video-based vlm plans the next step for vision-and-language navigation,”Robotics: Science and Systems, 2024

    2024

  28. [28]

    Uni-navid: A video-based vision- language-action model for unifying embodied navigation tasks,

    J. Zhang, K. Wang, S. Wang, M. Li, H. Liu, S. Wei, Z. Wang, Z. Zhang, and H. Wang, “Uni-navid: A video-based vision- language-action model for unifying embodied navigation tasks,” Robotics: Science and Systems, 2025

    2025

  29. [29]

    Conceptfusion: Open-set multimodal 3d mapping,

    K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. S. Iyer, S. Saryazdi, N. V . Keetha et al., “Conceptfusion: Open-set multimodal 3d mapping,” in ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023

    2023

  30. [30]

    M3dmap: Object-aware multimodal 3d mapping for dynamic environments,

    D. Yudin, “M3dmap: Object-aware multimodal 3d mapping for dynamic environments,”arXiv preprint arXiv:2508.17044, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    3d scene graph: A structure for unified seman- tics, 3d space, and camera,

    I. Armeni, Z.-Y . He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified seman- tics, 3d space, and camera,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5664– 5673

    2019

  32. [32]

    Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,”arXiv preprint arXiv:2201.13360, 2022

    Pith/arXiv arXiv 2022

  33. [33]

    Clio: Real-time task-driven open-set 3d scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3d scene graphs,”IEEE Robotics and Automation Letters, 2024

    2024

  34. [34]

    Ving: Learning open-world navigation with visual goals,

    D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “Ving: Learning open-world navigation with visual goals,” in 2021 IEEE International Conference on Robotics and Automa- tion (ICRA). IEEE, 2021, pp. 13 215–13 222

    2021

  35. [35]

    Graph database applications and concepts with neo4j,

    J. J. Miller, “Graph database applications and concepts with neo4j,” inProceedings of the southern association for informa- tion systems conference, Atlanta, GA, USA, vol. 2324, no. 36, 2013, pp. 141–147

    2013

  36. [36]

    Vint: A foundation model for visual navigation,

    D. Shah, A. K. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “Vint: A foundation model for visual navigation,” inConference on Robot Learning, 2023

    2023

  37. [37]

    Nomad: Goal masked diffusion policies for navigation and exploration,

    A. Sridhar, D. Shah, C. Glossop, and S. Levine, “Nomad: Goal masked diffusion policies for navigation and exploration,” in 2024 IEEE International Conference on Robotics and Automa- tion (ICRA). IEEE, 2024, pp. 63–70

    2024

  38. [38]

    Egowalk: A multimodal dataset for robot navigation in the wild,

    T. Akhtyamov, M. A. Mdfaa, J. A. Ramirez, S. Bakulin, G. Devchich, D. Fatykhov, A. Mazurov, K. Zipa, M. Mohrat, P. Kolesniket al., “Egowalk: A multimodal dataset for robot navigation in the wild,”arXiv preprint arXiv:2505.21282, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Slam toolbox: Slam for the dynamic world,

    S. Macenski and I. Jambrecic, “Slam toolbox: Slam for the dynamic world,”Journal of Open Source Software, vol. 6, no. 61, p. 2783, 2021

    2021

  40. [40]

    Ros navigation: Concepts and tutorial,

    R. L. Guimar ˜aes, A. S. de Oliveira, J. A. Fabro, T. Becker, and V . A. Brenner, “Ros navigation: Concepts and tutorial,” inRobot Operating System (ROS) The Complete Reference (Volume 1). Springer, 2016, pp. 121–160

    2016