Pith. sign in

REVIEW 2 major objections 6 minor 54 references

SCOUT finds hidden household objects by scoring 3D scene graphs with LLM-distilled room and co-occurrence utilities, matching online LLM planners without their cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 14:21 UTC pith:AORY7K4R

load-bearing objection Solid systems paper: offline-distilled relational utilities on 3DSGs match online LLMs for interactive open-vocab search at a fraction of the cost, with a reusable symbolic benchmark and real-robot transfer. the 2 major comments →

arxiv 2603.05642 v2 pith:AORY7K4R submitted 2026-03-05 cs.RO cs.AI

Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search

Imen Mahdi , Matteo Cassinelli , Fabien Despinoy , Tim Welschehold , Abhinav Valada This is my paper
classification cs.RO cs.AI
keywords interactive object search3D scene graphsopen vocabularyrelational semanticsknowledge distillationrobot explorationSymSearchutility scoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Robots searching homes for arbitrary objects often need to open cabinets and doors, so they need common-sense ideas about which rooms and objects go together. Similarity between vision–language embeddings does not reliably encode those relations, and calling large language models online is too slow and expensive for real-time robots. This paper shows that containment and co-occurrence knowledge can be extracted offline from LLMs into small models, then used to score rooms, frontiers, and objects on a live 3D scene graph. The resulting method, SCOUT, beats embedding baselines, matches LLM-level success while using far less compute, and transfers to a real mobile manipulator. The authors also introduce SymSearch, a symbolic scene-graph benchmark that isolates semantic reasoning without full simulation overhead.

Core claim

Assigning utility scores to 3D scene-graph nodes with two relational heuristics—room–object containment and object–object co-occurrence—estimated by lightweight models distilled offline from LLMs, guides open-vocabulary interactive object search as well as online LLM planners and better than embedding similarity, at a small fraction of the inference cost, including transfer to physical robots under realistic sensing.

What carries the argument

SCOUT’s node utility on a hierarchical 3D scene graph, computed by two small MLPs (containment and co-occurrence) trained on procedurally LLM-generated open-vocabulary datasets built via the hierarchy Household → Rooms → Categories → Objects; high-utility nodes are then mapped to navigation or open actions via node affordances.

Load-bearing premise

The robot’s online 3D scene graph must be accurate enough that scoring rooms and objects reflects the real world; if perception mislabels or duplicates objects, the distilled priors cannot recover the right plan.

What would settle it

On identical SymSearch or simulation episodes with oracle-perfect scene graphs, if SCOUT’s success rate and path efficiency fell to embedding-baseline levels, or if real-robot trials with ground-truth object labels still failed mainly by choosing irrelevant rooms or containers, the claim that relational utility drives the gains would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Onboard open-vocabulary interactive search can run without repeated LLM API calls at every decision.
  • Explicit containment and co-occurrence models separate positive and negative pairs where embedding similarity fails.
  • Scene-graph node selection with learned utilities can replace or reduce online LLM planning for this task.
  • SymSearch lets researchers scale evaluation of semantic reasoning without simulator cost or instability.
  • The same pipeline transfers to a physical mobile manipulator when perception yields a usable scene graph.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The procedural distillation hierarchy is reusable for other settings (schools, hospitals) with different room and object taxonomies, as the paper only sketches.
  • Perception quality is the binding constraint; better open-vocabulary segmentation may yield larger gains than further refining the utility MLPs.
  • Hybrid planners could use SCOUT for cheap routine search and invoke an LLM only when utilities are flat or contradictory.
  • Household-specific online adaptation of the priors, listed as future work, would test whether static common-sense priors suffice for personalized homes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents SCOUT, a method for open-world interactive object search that scores rooms, frontiers, and objects on hierarchical 3D scene graphs using learned room–object containment and object–object co-occurrence utilities. Relational priors are obtained offline via a procedural multi-stage LLM querying pipeline that builds open-vocabulary household object sets and supervised co-occurrence/containment datasets, then distilled into lightweight MLPs over frozen text embeddings for on-robot inference. High-level node selection (with a utility margin and distance tie-break) is grounded to navigation and manipulation policies via node affordances. The authors also introduce SymSearch, a dataset-agnostic symbolic benchmark that rolls out incremental discovery of rooms, frontiers, and nested objects on InteriorGS-derived 3DSGs. Evaluations on SymSearch (200 episodes, many OOD queries), OmniGibson simulation, embedding-distribution analysis, ablations, LLM-variant comparisons, and 36 real HSR trials support the claim that SCOUT outperforms embedding-similarity baselines and approaches LLM planners at far lower cost, with transfer under realistic sensing.

Significance. If the results hold, the work offers a practical systems path for open-vocabulary interactive search that avoids online LLM calls while retaining relational common sense. Strengths include: (i) a clear diagnosis that embedding similarity fails to separate co-occurrence and containment pairs (Fig. 7); (ii) a reusable procedural distillation pipeline rather than ad-hoc prompt engineering; (iii) SymSearch as a scalable, simulator-light testbed for semantic reasoning; (iv) multi-setting evidence (symbolic, simulation, real robot) with ablations on room influence and utility margin; and (v) honest isolation of perception vs. reasoning failures on the HSR. Code release is promised. These contributions are useful to the mobile-manipulation and open-vocabulary navigation communities even if perception remains the dominant real-world bottleneck.

major comments (2)
  1. §VI-I and §VI-J correctly identify that performance depends on a faithful online 3DSG, and Fig. 11 shows perception as the dominant real-robot failure mode (~64% SR over 36 trials). The central comparative claim is still supported by privileged-graph SymSearch and simulation, but the manuscript should more clearly quantify how much of the real-robot gap is attributable to graph errors versus utility scoring (e.g., offline re-scoring of logged graphs with ground-truth labels, or a controlled noise injection study). Without that, the transfer claim remains systems-level rather than a clean validation of the distilled priors under realistic noise.
  2. §IV-B–C and free parameters (room weight w, margin Δ, default unobserved-room score, frontier radius): ablations in §VI-E show room influence and the utility margin matter, but sensitivity ranges and selection criteria for these hyperparameters are not fully reported across environments. A short sensitivity table or fixed default protocol would strengthen reproducibility of the “matches LLM-level performance” claim, especially when comparing to GODHS/MoMa-LLM whose exploration–exploitation balance differs.
minor comments (6)
  1. Fig. 1 and several pipeline figures use placeholder/garbled text in the provided manuscript rendering; ensure final figures are legible and that node-type legends match the 5-layer hierarchy in §III.
  2. §IV-C: state the frozen text encoder (SBert / OpenAI / other), embedding dimension d, MLP architecture, and train/val splits for D_co-occur and D_contain so the distillation is fully reproducible.
  3. Table I footnote and InteriorGS annotation status: clarify how many scenes have full nested-object annotations used in the 200-episode evaluation and whether annotation bias could favor co-occurrence priors.
  4. §VI-H / Table XI: pricing and model names are useful; add a one-line note on temperature / structured-output settings for the GPT-5-mini MoMa-LLM and GODHS baselines to aid replication.
  5. Notation: u_q(n) is defined in Eq. (1) then specialized; ensure consistent use of p_contains vs. f_contain_θ2 and of “regions/frontiers” across §III–V.
  6. Related work: briefly contrast Seek [30] (industrial relational nets) with the household interactive setting and procedural open-vocab generation to sharpen novelty.

Circularity Check

0 steps flagged

No significant circularity: offline LLM distillation into utility models is supervised transfer, evaluated empirically against independent baselines on held-out scenes and real robots.

full rationale

SCOUT defines utility via explicit heuristics (room–object containment p_contains and object–object co-occurrence p_co-occur, Eqs. 2–4) and learns them by offline procedural querying of an LLM to build supervised datasets D_co-occur and D_contain, then trains lightweight MLPs on frozen text embeddings (Eqs. 8–11). The resulting scores guide node selection on 3DSGs; performance is measured by SR/SPL against embedding-similarity agents, online LLM planners (MoMa-LLM, GODHS), random, and ablations on SymSearch (200 episodes, 61 OOD categories), OmniGibson, and 36 real-robot trials. Nothing is forced by construction: the teacher labels are not the evaluation metric, parameters are not fitted to the test episodes, and no uniqueness theorem or self-citation chain is invoked to declare the method the only possible solution. Self-citations (e.g., to MoMa-LLM/MORE for baselines or 3DSG construction) are ordinary related-work references, not load-bearing premises. The paper is an empirical systems contribution whose claims stand or fall on the reported comparisons; the derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central claim rests on engineering assumptions and a few hand-chosen scalars, not on new physical entities. Load-bearing premises are that LLM-elicited co-occurrence/containment are good open-vocab search heuristics, that online 3DSGs from RGB-D are usable, and that high-level node selection plus affordances grounds to real actions. Free parameters (room weight w, margin Δ, default unexplored scores) shape exploration but are ablated in part. No graviton-like unobserved mediators.

free parameters (5)
  • room influence weight w
    In Eq. 4, w ∈ [0,1] mixes parent-room utility into object scores; chosen design parameter that ablations show strongly affects SR.
  • utility selection margin Δ
    Nodes within [Umax−Δ, Umax] compete by distance (Eq. 12); Δ=0 is pure greedy. Ablations report efficiency impact.
  • default exploration score for unobserved rooms/frontiers
    Unclassified doors/frontiers get a fixed score so they remain selectable when visible nodes are uninformative; value is a design choice.
  • frontier aggregation radius
    Frontiers inherit scores from nearby objects within a spatial radius; radius is a hand-set geometric parameter.
  • N_max episode step limit
    SymSearch uses N_max=50; success definition depends on this cutoff.
axioms (5)
  • domain assumption Room–object containment and object–object co-occurrence are sufficiently informative heuristics for interactive household object search.
    Core of §IV-B utility design; motivated by human search literature but not proved optimal among possible relational features.
  • domain assumption Offline LLM answers over procedurally generated household vocabularies are a valid teacher for open-vocabulary containment/co-occurrence.
    §IV-C distillation pipeline; authors manually spot-check subsets but do not independently validate against large real placement statistics.
  • domain assumption A hierarchical 3DSG can be constructed online from RGB-D, pose, segmentation, and Voronoi room segmentation well enough for node-level planning.
    §IV-A and problem formulation; real-robot section shows this is the practical bottleneck.
  • domain assumption High-level node affordances map reliably to low-level navigation and manipulation policies.
    §IV-D; real experiments use pre-recorded N2M2 motions with AR markers, a stronger grounding assumption than fully open manipulation.
  • ad hoc to paper Binary labels suit co-occurrence and continuous scores suit containment for supervised distillation.
    Explicit design choice in §IV-C after experimentation; shapes the learned utilities.
invented entities (3)
  • SCOUT utility-based 3DSG explorer independent evidence
    purpose: Assign learned relational utilities to rooms/frontiers/objects and select next high-level action without online LLMs.
    Methodological system, not a physical entity; evaluated empirically against baselines.
  • Procedural LLM relational distillation pipeline independent evidence
    purpose: Generate open household vocabularies and co-occurrence/containment datasets to train lightweight MLPs.
    New data-generation procedure; evidence is the resulting models' separation vs embeddings and search performance.
  • SymSearch symbolic interactive search benchmark independent evidence
    purpose: Scalable evaluation of semantic exploration on open-vocab 3DSGs without full simulation cost.
    New evaluation artifact built mainly from InteriorGS; independent of SCOUT's online robot stack.

pith-pipeline@v1.1.0-grok45 · 22050 in / 3477 out tokens · 39708 ms · 2026-07-15T14:21:04.933492+00:00 · methodology

0 comments
read the original abstract

Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on vision-language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-robot inference. Furthermore, we present SymSearch, a scalable symbolic benchmark for evaluating semantic reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and navigation constraints.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 17 linked inside Pith

  1. [1]

    Human learning of contextual priors for object search: where does the time go?

    B. Hidalgo-Sotelo, A. Oliva, and A. Torralba, “Human learning of contextual priors for object search: where does the time go?” inProc. of the IEEE Conf. Computer Vision Pattern Recognition, 2005, pp. 86–86

    2005

  2. [2]

    Object co-occurrence serves as a contextual cue to guide and facilitate visual search in a natural viewing environment,

    S. C. Mack and M. P. Eckstein, “Object co-occurrence serves as a contextual cue to guide and facilitate visual search in a natural viewing environment,”Journal of vision, vol. 11, no. 9, pp. 9–9, 2011

    2011

  3. [3]

    Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” inProc. of the IEEE Conf. Computer Vision Pattern Recognition, 2023, pp. 23 171–23 181

    2023

  4. [4]

    Vlfm: Vision- language frontier maps for zero-shot semantic navigation,

    N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher, “Vlfm: Vision- language frontier maps for zero-shot semantic navigation,” inProc. of the IEEE Int. Conf. on Robotics and Automation, 2024, pp. 42–48

    2024

  5. [5]

    Uncertainty- informed active perception for open vocabulary object goal navigation,

    U. Bajpai, J. R ¨uckin, C. Stachniss, and M. Popovi ´c, “Uncertainty- informed active perception for open vocabulary object goal navigation,” arXiv preprint arXiv:2506.13367, 2025

    arXiv 2025

  6. [6]

    Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,

    M. Zhang, Y . Du, C. Wu, J. Zhou, Z. Qi, J. Ma, and B. Zhou, “Apexnav: An adaptive exploration strategy for zero-shot object navigation with target-centric semantic fusion,”arXiv preprint arXiv:2504.14478, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Navigation with large language models: Semantic guesswork as a heuristic for planning,

    D. Shah, M. R. Equi, B. Osi ´nski, F. Xia, B. Ichter, and S. Levine, “Navigation with large language models: Semantic guesswork as a heuristic for planning,” inConference on Robot Learning, 2023

    2023

  8. [8]

    V oronav: V oronoi-based zero-shot object navigation with large language model,

    P. Wu, Y . Mu, B. Wu, Y . Hou, J. Ma, S. Zhang, and C. Liu, “V oronav: V oronoi-based zero-shot object navigation with large language model,” arXiv preprint arXiv:2401.02695, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Open scene graphs for open-world object- goal navigation,

    J. Loo, Z. Wu, and D. Hsu, “Open scene graphs for open-world object- goal navigation,”The International Journal of Robotics Research, 2025

    2025

  10. [10]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation,

    H. Yin, X. Xu, Z. Wu, J. Zhou, and J. Lu, “Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation,”Advances in neural information processing systems, vol. 37, 2024

    2024

  11. [11]

    Commonsense scene graph-based target localization for object search,

    W. Ge, C. Tang, and H. Zhang, “Commonsense scene graph-based target localization for object search,” inProc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2024, pp. 13 318–13 325

    2024

  12. [12]

    Language- enhanced mobile manipulation for efficient object search in indoor environments,

    L. Zhang, Z. Li, K. Cai, Q. Huang, Z. Bing, and A. Knoll, “Language- enhanced mobile manipulation for efficient object search in indoor environments,”arXiv preprint arXiv:2508.20899, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Open-vocabulary and semantic-aware reasoning for search and retrieval of objects in dynamic and concealed spaces,

    R. Menon, Y . Schmiede, M. Bennewitz, and H. Blum, “Open-vocabulary and semantic-aware reasoning for search and retrieval of objects in dynamic and concealed spaces,”IROS Workshop on Perception and Planning for Mobile Manipulation in Changing Environments, 2025

    2025

  14. [14]

    Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning,

    G. Chalvatzaki, A. Younes, D. Nandha, A. T. Le, L. F. Ribeiro, and I. Gurevych, “Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning,” Frontiers in Robotics and AI, vol. 10, p. 1221739, 2023

    2023

  15. [15]

    Delta: Decomposed efficient long-term robot task planning using large language models,

    Y . Liu, L. Palmieri, S. Koch, I. Georgievski, and M. Aiello, “Delta: Decomposed efficient long-term robot task planning using large language models,”arXiv preprint arXiv:2404.03275, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Embodiedrag: Dynamic 3d scene graph retrieval for efficient and scalable robot task planning,

    M. Booker, G. Byrd, B. Kemp, A. Schmidt, and C. Rivera, “Embodiedrag: Dynamic 3d scene graph retrieval for efficient and scalable robot task planning,”arXiv preprint arXiv:2410.23968, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Language-grounded dynamic scene graphs for interactive object search with mobile manipulation,

    D. Honerkamp, M. B ¨uchner, F. Despinoy, T. Welschehold, and A. Valada, “Language-grounded dynamic scene graphs for interactive object search with mobile manipulation,”IEEE Robotics and Automation Letters, 2024

    2024

  18. [18]

    More: Mo- bile manipulation rearrangement through grounded language reasoning,

    M. Mohammadi, D. Honerkamp, M. B ¨uchner, M. Cassinelli, T. Welschehold, F. Despinoy, I. Gilitschenski, and A. Valada, “More: Mo- bile manipulation rearrangement through grounded language reasoning,” inProc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2025

    2025

  19. [19]

    Inference-time computations for llm reasoning and planning: A benchmark and insights,

    S. Parashar, B. Olson, S. Khurana, E. Li, H. Ling, J. Caverlee, and S. Ji, “Inference-time computations for llm reasoning and planning: A benchmark and insights,”arXiv preprint arXiv:2502.12521, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Scaling up and distilling down: Language-guided robot skill acquisition,

    H. Ha, P. Florence, and S. Song, “Scaling up and distilling down: Language-guided robot skill acquisition,” inConference on Robot Learning, 2023, pp. 3766–3777

    2023

  21. [21]

    Distilling on-device language models for robot planning with minimal human intervention,

    Z. Ravichandran, I. Hounie, F. Cladera, A. Ribeiro, G. J. Pappas, and V . Kumar, “Distilling on-device language models for robot planning with minimal human intervention,”arXiv preprint arXiv:2506.17486, 2025

    arXiv 2025

  22. [22]

    3d scene graph: A structure for unified semantics, 3d space, and camera,

    I. Armeni, Z.-Y . He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, “3d scene graph: A structure for unified semantics, 3d space, and camera,” inProc. of the Int. Conf. Comput. Vis., 2019

    2019

  23. [23]

    Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,

    A. Werby, C. Huang, M. B ¨uchner, A. Valada, and W. Burgard, “Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation,” inProc. of the Robotics: Science and Systems, 2024

    2024

  24. [24]

    Collaborative dynamic 3d scene graphs for open-vocabulary urban scene understanding,

    T. Steinke, M. B ¨uchner, N. V ¨odisch, and A. Valada, “Collaborative dynamic 3d scene graphs for open-vocabulary urban scene understanding,” inIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025, pp. 6000–6007

    2025

  25. [25]

    Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,

    K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, and N. Suender- hauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,”arXiv preprint arXiv:2307.06135, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Learning hierarchical interactive multi-object search for mobile manipulation,

    F. Schmalstieg, D. Honerkamp, T. Welschehold, and A. Valada, “Learning hierarchical interactive multi-object search for mobile manipulation,” IEEE Robotics and Automation Letters, vol. 8, no. 12, pp. 8549–8556, 2023

    2023

  27. [27]

    Learning long-horizon robot exploration strategies for multi-object search in continuous action spaces,

    ——, “Learning long-horizon robot exploration strategies for multi-object search in continuous action spaces,” inThe International Symposium of Robotics Research, 2022, pp. 52–66

    2022

  28. [28]

    Perception matters: Enhancing embodied ai with uncertainty- aware semantic segmentation,

    S. Prasanna, D. Honerkamp, K. Sirohi, T. Welschehold, W. Burgard, and A. Valada, “Perception matters: Enhancing embodied ai with uncertainty- aware semantic segmentation,”arXiv preprint arXiv:2408.02297, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Robotic task ambiguity resolution via natural language interaction,

    E. Chisari, J. O. V on Hartz, F. Despinoy, and A. Valada, “Robotic task ambiguity resolution via natural language interaction,” inIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025, pp. 14 821–14 827. 16

    2025

  30. [30]

    Seek: Semantic reasoning for object goal nav- igation in real world inspection tasks,

    M. F. Ginting, S.-K. Kim, D. D. Fan, M. Palieri, M. J. Kochenderfer, and A.-a. Agha-Mohammadi, “Seek: Semantic reasoning for object goal nav- igation in real world inspection tasks,”arXiv preprint arXiv:2405.09822, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Habitat: A platform for embodied ai research,

    M. Savva, A. Kadian, O. Maksymets, Y . Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Maliket al., “Habitat: A platform for embodied ai research,” inProc. of the IEEE Conf. Computer Vision Pattern Recognition, 2019, pp. 9339–9347

    2019

  32. [32]

    Habitat 2.0: Training home assistants to rearrange their habitat,

    A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y . Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymetset al., “Habitat 2.0: Training home assistants to rearrange their habitat,”Advances in neural information processing systems, vol. 34, pp. 251–266, 2021

    2021

  33. [33]

    Habitat-matterport 3d semantics dataset,

    K. Yadav, R. Ramrakhya, S. K. Ramakrishnan, T. Gervet, J. Turner, A. Gokaslan, N. Maestre, A. X. Chang, D. Batra, M. Savvaet al., “Habitat-matterport 3d semantics dataset,” inProc. of the IEEE Conf. Computer Vision Pattern Recognition, 2023, pp. 4927–4936

    2023

  34. [34]

    Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation. in 2024 ieee,

    N. Yokoyama, R. Ramrakhya, A. Das, D. Batra, and S. Ha, “Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation. in 2024 ieee,” inProc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, 2024, pp. 5543–5550

    2024

  35. [35]

    Ai2-thor: An interactive 3d environment for visual ai,

    E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y . Zhuet al., “Ai2-thor: An interactive 3d environment for visual ai,”arXiv preprint arXiv:1712.05474, 2017

    Pith/arXiv arXiv 2017

  36. [36]

    Behavior-1k: A human- centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation,

    C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Mart ´ın-Mart´ın, C. Wang, G. Levine, W. Ai, B. Martinezet al., “Behavior-1k: A human- centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation,”arXiv preprint arXiv:2403.09227, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Taskography: Evaluating robot task planning over large 3d scene graphs,

    C. Agia, K. M. Jatavallabhula, M. Khodeir, O. Miksik, V . Vineet, M. Mukadam, L. Paull, and F. Shkurti, “Taskography: Evaluating robot task planning over large 3d scene graphs,” inConference on Robot Learning, 2022, pp. 46–58

    2022

  38. [38]

    Kimera: From slam to spatial perception with 3d dynamic scene graphs,

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone, “Kimera: From slam to spatial perception with 3d dynamic scene graphs,”The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1510–1546, 2021

    2021

  39. [39]

    Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,

    N. Hughes, Y . Chang, and L. Carlone, “Hydra: A real-time spatial perception system for 3d scene graph construction and optimization,” arXiv preprint arXiv:2201.13360, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agar- wal, C. Rivera, W. Paul, K. Ellis, R. Chellappaet al., “Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning,” inProc. of the IEEE Int. Conf. on Robotics and Automation, 2024, pp. 5021–5028

    2024

  41. [41]

    Clio: Real-time task-driven open-set 3d scene graphs,

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone, “Clio: Real-time task-driven open-set 3d scene graphs,”IEEE Robotics and Automation Letters, 2024

    2024

  42. [42]

    Frontier based exploration for autonomous robot,

    A. Topiwala, P. Inani, and A. Kathpal, “Frontier based exploration for autonomous robot,”arXiv preprint arXiv:1806.03581, 2018

    Pith/arXiv arXiv 2018

  43. [43]

    A formal basis for the heuristic determination of minimum cost paths,

    P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,”IEEE transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100–107, 1968

    1968

  44. [44]

    Interiorgs: A 3d gaussian splatting dataset of semantically labeled indoor scenes,

    M. T. I. SpatialVerse Research Team, “Interiorgs: A 3d gaussian splatting dataset of semantically labeled indoor scenes,” https://huggingface.co/ datasets/spatialverse/InteriorGS, 2025

    2025

  45. [45]

    A note on two problems in connexion with graphs,

    E. W. Dijkstra, “A note on two problems in connexion with graphs,” in Edsger Wybe Dijkstra: his life, work, and legacy, 2022, pp. 287–290

    2022

  46. [46]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,”arXiv preprint arXiv:1908.10084, 2019

    Pith/arXiv arXiv 1908

  47. [47]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInt. Conf. on Machine Learning, 2021, pp. 8748–8763

    2021

  48. [48]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Biet al., “Deepseek-r1 incentivizes reasoning in llms through reinforcement learning,”Nature, vol. 645, no. 8081, pp. 633–638, 2025

    2025

  52. [52]

    The claude 3 model family: Opus, sonnet, haiku,

    A. Anthropic, “The claude 3 model family: Opus, sonnet, haiku,”Claude- 3 Model Card, vol. 1, no. 1, p. 4, 2024

    2024

  53. [53]

    Yolo- world: Real-time open-vocabulary object detection,

    T. Cheng, L. Song, Y . Ge, W. Liu, X. Wang, and Y . Shan, “Yolo- world: Real-time open-vocabulary object detection,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 16 901–16 911

    2024

  54. [54]

    N2m2: Learning navigation for arbitrary mobile manipulation motions in unseen and dynamic environments,

    D. Honerkamp, T. Welschehold, and A. Valada, “N2m2: Learning navigation for arbitrary mobile manipulation motions in unseen and dynamic environments,”IEEE Trans. on Robot., vol. 39, no. 5, pp. 3601–3619, 2023

    2023