Pith. sign in

REVIEW 4 major objections 5 minor 38 references

This paper claims that building complete 3D scene graphs before a task is known degrades robot reasoning; instead, a just-in-time graph that activates only task-relevant objects on demand keeps perception accurate, fast, and memory-bounded.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

JITOMA keeps 3D scene graph nodes dormant until a natural-language query activates only task-relevant objects, bounding active graph size and captioning latency while preserving grounding accuracy.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection JITOMA's efficiency is structurally real, but its accuracy case rests on a gating stage that can permanently lose the very objects the task needs — and the paper never measures that recall. the 4 major comments →

arxiv 2607.13245 v1 pith:LYQQOVIZ submitted 2026-07-14 cs.CV cs.RO

Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics

Yue Chang , Rufeng Chen , Yifan Tian , Dazhi Huang , Zhaofan Zhang , Yi Chen , Wenze Zhang , Li Chen
show 2 more authors
Hui Xiong Sihong Xie
This is my paper
classification cs.CV cs.RO
keywords 3D scene graphsjust-in-time growthperceptual saturationlong-horizon roboticstask-driven perceptiondormant anchorson-demand memory activationLLM robot planning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard 'build-everything-then-filter' approach to 3D scene graphs is wrong for long-horizon robotics: exhaustively instantiating dense structure before a task is known floods the reasoning engine with irrelevant detail, a 'perceptual saturation' the authors demonstrate when a full 18-object graph makes an LLM planner drop the final step of 'pack my eyeglasses for travel,' while a 3-object task-relevant subgraph yields a complete plan. The proposed fix is JITOMA, a closed-loop framework where task intent gates perception from the frontend: a task heatmap filters streaming observations, storing only lightweight 'dormant anchors' (coarse geometry, a crop, a CLIP key); when a command arrives, an LLM parses primary and latent objects, activates a few anchors, and spends dense captioning and functional subgraph growth only in that local subgraph, which then collapses and distills after execution. On the new JITOMA-Bench, this keeps active graph size bounded (peak 4–12 nodes) at a stable 0.56–0.63 s per frame across task switches, while matching or exceeding full-graph baselines in grounding accuracy. The upshot is that scene-graph quality concerns not only what is represented, but when and where it is instantiated.

Core claim

JITOMA's central claim is that a robot should not remember everything in full detail; it should instantiate only what the task immediately executes. The paper operationalizes this as a two-tier memory of dormant anchors and ephemeral hypotheses, with a top-down task heatmap gating observations before they are committed to memory. Only on a cognitive query do the parsed concepts retrieve candidate anchors (via CLIP similarity plus accumulated heatmap evidence), trigger dense captioning and functional inference on at most a few candidates, and grow a task-conditioned subgraph; after execution the subgraph collapses, with reusable captions and functional parts distilled back into anchors. This

What carries the argument

The named central objects are the 'dormant anchor' (a lightweight memory node storing only existence, coarse geometry, a representative crop, and a CLIP visual key) and the 'task heatmap' (a top-down attention map from a vision-language model that gates observations via TopK selection). The activation equation s_ret = cos(CLIP_text(c), z_j) + λ ĥ_c_j retrieves three candidates per concept; DAM captioning and an LLM reranker select one anchor to 'awaken.' Dense captioning and VLM-based functional part grounding are then restricted to the activated subgraph, and Eq. (10) distills reusable captions and parts back into the anchors. Together these mechanisms decouple accumulated memory from activ

Load-bearing premise

The frontend admits into memory only observations that survive the parser's concept set and the vision-language heatmap's TopK, so the paper's accuracy results presuppose that the gating stage never drops an object a future task will need — a recall that the paper does not measure.

What would settle it

Measure the recall of the gating stage: run a scene replay, take every Tier-2/Tier-3 target, and check whether its track ever appears in the TopK-admitted set before the query is issued. If a task-relevant object with a weak heatmap score is found to be dropped, JITOMA will fail to retrieve it — an observable failure that an AOT pipeline would not have, and one that would cap the true accuracy of the just-in-time paradigm.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If JITOMA is right, active memory and latency no longer grow with exploration length; a robot that has seen a building for hours and one that just entered it process the current task at the same cost.
  • Long-horizon task switching stops accumulating overhead: each command activates a fresh local subgraph, and previously expanded structure hibernates with distilled reusable attributes, keeping per-frame time stable from Tier 1 to Tier 2.
  • Complex instructions that imply latent objects (containers, tools, sources) become tractable because the reasoning context never sees irrelevant anchors; the Tier-3 results support this by beating even parser-augmented baselines.
  • Scene-graph evaluation should include process-level efficiency metrics (active node count, peak, time per frame), not just final graph statistics; JITOMA-Bench provides a protocol for that.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The design implies that the recall of the frontend gating stage is the system's critical vulnerability: an object that never becomes a dormant anchor can never be woken by a query, so the measured retrieval accuracy presumes the parser-heatmap combination saw the object in the first place.
  • The same just-in-time principle could transfer to other lifelong memory systems for embodied agents — semantic SLAM, navigation maps, object-code databases — wherever full annotation cost is prohibitive and tasks arrive unevenly in time.
  • A testable extension would sweep the TopK size / heatmap threshold and observe the saturation-vs-recall trade-off directly; JITOMA's reported frontier is only one operating point.
  • Distillation of reusable captions and parts suggests a memory-consolidation property: repeated similar tasks should get cheaper over time; replaying the same scene with progressive tasks and measuring TPF would reveal if this actually holds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces JITOMA, a just-in-time 3D scene graph construction framework for long-horizon robotics. A front-end task heatmap filters continuous RGB-D observations so that only task-salient tracks become dormant anchors; at query time an LLM parses the command, retrieves a small set of anchors, and performs expensive dense captioning and functional subgraph growth only within the activated local subgraph. The authors also introduce JITOMA-Bench, a three-tier benchmark built on Clio's scenes, with new long-horizon and complex-instruction annotations. They report experiments against ConceptGraphs, ReasoningGraph, and Clio, and claim that JITOMA bounds active graph size and latency while preserving or improving grounding accuracy.

Significance. The core idea is genuinely useful and timely: deferring heavy semantic expansion until a task arrives, and gating perception by task intent, is a clear architectural direction for edge robotics. The efficiency claims are credible: the bounded active graph size follows directly from the design (Top-3 retrieval and one anchor per parsed concept), and the reported TPF numbers are consistent with that structural bound. The motivation experiment in Section 3 is a nice illustration of the perceptual saturation effect. However, the accuracy evidence is currently incomplete in a load-bearing way. The front-end gating stage is irreversible and its recall is never measured, the closest prior JIT-style baseline (FOUND-IT) is omitted, and the results lack statistical support. The introduced benchmark also has a potential circularity risk because its primary/latent annotation format matches the output of JITOMA's own parser.

major comments (4)
  1. [Section 4.2, Eq. (3); Section 4.4, Eqs. (4)-(6); Section 6.2, Table 1] The end-to-end accuracy claim is incomplete because the front-end gating stage is irreversible and its recall is never measured. Eq. (3) admits only the TopK tracks by task-heatmap score; unadmitted observations never enter V_eph and therefore cannot be promoted to V_anc. Eqs. (4)-(6) then retrieve from V_anc only, with a hard Top-3 candidate cutoff before the LLM reranker. A task-relevant object missed at the gating stage (e.g., the sponge in "clean the cup") is permanently unavailable at query time. The IoU/mR@K numbers in Table 1 are computed for objects that already exist as anchors; they do not reflect gating failures. Please report (i) the fraction of ground-truth objects that ever reach V_anc, (ii) end-to-end accuracy with missed objects counted as failures, and (iii) sensitivity to K in Eq. (3) and to the retriever hyperparameters.
  2. [Section 2, Section 6.1, Table 1] FOUND-IT [14] is described in Related Work as the closest prior attempt at just-in-time adaptation, but it is not included in the experiments. Since the paper's central contribution is the JIT paradigm, the comparison table otherwise covering only AOT/task-driven methods makes the claimed consistent advantage incomplete. Please add FOUND-IT to Table 1 using the same tiers, or, if that is impossible (e.g., unavailable code or benchmark), provide a quantitative or detailed qualitative comparison and explain how the evaluation protocols differ.
  3. [Section 6.1, Table 1] No error bars, confidence intervals, or significance tests are reported for any accuracy metric. Several differences are small (e.g., Cubicle Tier-3 IoU 21.6 vs. 21.1 for augmented Clio; Office Tier-1 mR@3 59.0 vs. 56.0 for Clio), so it is unclear whether the reported improvements are stable. Please report standard deviations across repeated runs, or bootstrap confidence intervals over tasks/queries, and state whether the observed differences are statistically significant.
  4. [Section 5.1 (Tier 3), Section 6.1] The Tier-3 benchmark is authored by the same group using GPT-5.4 with author-assigned primary/latent object sets, and this format matches the output of JITOMA's own parser. The augmented-baseline comparison in Table 1 controls for parser use at test time but does not independently validate the latent-object ground truth. To reduce the risk of benchmark circularity, please (i) provide the full annotation instructions, (ii) report inter-annotator agreement statistics, and (iii) confirm that the latent-object set was created without reference to JITOMA's C(q) predictions.
minor comments (5)
  1. [Section 5, Section 6] The benchmark is called JITOMA-Bench but the text says it is built on Clio-Bench; please clarify the relationship between the two names throughout.
  2. [Table 1] The table is hard to read because the three tiers and multiple metrics are compressed into one row per method. Consider separate tables per tier or clearer subheaders.
  3. [Section 4.4, Eq. (4)] The heatmap-cosine blend weight lambda is introduced but its value is never specified. This matters for reproducibility; please report the setting or state that it was tuned.
  4. [Section 3.1] The term "perceptual saturation effect" is presented as a finding, but it is not operationalized into a measurable quantity. Please either formalize it or explicitly label it as a qualitative phenomenon.
  5. [Appendix A.1/A.2] The prompt templates are useful, but the coordinate format in the appendix should be visually aligned with the examples in the main text and Figure 2 to avoid confusion.

Circularity Check

1 steps flagged

The bounded-active-graph efficiency claim is a design restatement of Eqs. (4)-(6); Tier 1-3 accuracy is benchmarked against external Clio labels and is not forced, though frontend gating recall is unmeasured.

specific steps
  1. self definitional [Sec. 4.4, Eqs. (4)-(6); Sec. 6.2 (Q2) and Table 1]
    "JITOMA first ranks dormant anchors with lightweight keys and retrieves three candidates: ... B(c) = Top3_{v_j ∈ V^anc} s^ret_j(c). ... The reranker activates one anchor for each concept; all unselected candidates return to the dormant state. ... Across Tier 1 and Tier 2, JITOMA maintains only one or two objects at query time and an active peak of at most four."

    The headline efficiency bound is not an independently derived prediction: Eq. (4) caps candidates at Top-3 per parsed concept and Eq. (6) selects exactly one anchor per concept, so the number of simultaneously active anchors is bounded by 3|C(q)| by construction. The paper presents the resulting Peak/Objs values as an experimental demonstration ('These results show that JITOMA scales expensive graph computation with current cognitive demand'), but this is a restatement of the activation rule, not a consequence that could have failed. The accuracy results remain external; the circularity is confined to the efficiency claim.

full rationale

No circularity is present in the accuracy derivation: Tier 1-3 IoU/mR@K scores are measured against Clio-derived geometric ground-truth labels, and the augmented-baseline protocol gives the baselines JITOMA's parser output, so the comparison does not smuggle the method's own outputs into the ground truth. The benchmark's primary/latent annotation format resembles the parser's C_pri/C_lat output, but the labels are human-reviewed and independent of the parser, so this is self-reference rather than a forced reduction. The one true restatement is the efficiency headline: the bounded active graph size is a direct consequence of the Top-3 candidate limit and one-anchor-per-concept selection in Eqs. (4)-(6), making the Peak and Objs 'results' partly architectural tautology. A separate, non-circular validity risk is that Eq. (3) gates which observations ever become anchors, while Eq. (4) retrieves only from V_anc; the paper never reports recall of that gating stage, so end-to-end grounding accuracy is conditional on objects already being admitted. This omission deserves a correctness flag but does not make the reported accuracy numbers equivalent to their inputs. Overall: partial self-definition of the efficiency claim, central accuracy content still independent -- score 4.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claim rests on four hand-set parameters (λ, TopK, Top3, promotion thresholds) and five domain assumptions, including the reliability of the LLM parser that drives the gating stage. The named invented constructs (perceptual saturation effect, two-tier memory, task heatmap) are internal to the paper and lack external evidence.

free parameters (4)
  • λ heatmap-cosine blend weight (Eq. 4) = unreported
    Retrieval score blends CLIP cosine similarity and accumulated task-heatmap evidence with an unreported weight λ; it affects which anchors enter the top-3 candidates and is never specified or swept.
  • TopK frontend selection count (Eq. 3) = unreported K
    Only the top-K observation tracks are admitted into ephemeral memory; K is never given, yet it controls how much information survives the gating stage.
  • Top3 candidate retrieval count (Eq. 4) = 3
    Exactly three anchors are retrieved per concept. This bounds compute by design, but if 3 is too small for cluttered scenes, recall drops.
  • Ephemeral-to-anchor promotion thresholds = unreported
    Sec 4.3 says ephemeral hypotheses are promoted when they pass a 'stability check', but the criteria and thresholds are not specified, even though they determine the composition of V_anc and therefore all later retrieval.
axioms (5)
  • domain assumption CLIP embeddings and DAM captioning provide sufficient semantic fidelity for open-vocabulary retrieval without dense annotation
    Sec 4.3-4.4 rely on pre-trained CLIP, DAM, and segmentation models; the paper does not evaluate their failure modes in the gating or retrieval stages.
  • domain assumption LLM parser reliably infers the complete set of primary and latent objects for a command
    Sec 4.2 and Eq. 3: the frontend gates observations using C(q); a missing latent object is never stored, so it cannot be retrieved later.
  • domain assumption Class-agnostic segmentation tracker produces temporally stable object tracks
    Sec 4.2: every memory entry derives from tracker [29]; tracking failures silently create or destroy anchors.
  • domain assumption Clio's three real-world scenes are representative of long-horizon embodied deployment
    JITOMA-Bench reuses three Clio scenes (Sec 5); generalization beyond these scenes is untested.
  • domain assumption Reviewer filtering of Tier-3 tasks removes ambiguity without imposing selection bias
    Sec 5.1: candidates are removed if two of three reviewers find ambiguity or unnecessary latent objects, which may preferentially keep tasks aligned with the parser's primary/latent output format.
invented entities (3)
  • Perceptual saturation effect no independent evidence
    purpose: Newly named phenomenon where exhaustive AOT scene graphs degrade downstream LLM planning; used as the core motivation for the framework
    Supported only by the n=1 motivation study (Sec 3.1, Fig 1); no quantitative characterization or independent replication is provided.
  • Dormant anchor / ephemeral hypothesis two-tier memory no independent evidence
    purpose: Architecture construct to hold scene knowledge at low cost and expand only on task activation
    Internal design choice; no external falsifiable handle, though the reported metrics (Peak, Objs) measure its effect inside this paper.
  • Task heatmap (top-down observation gate) no independent evidence
    purpose: Filters which observations are written into memory, implementing closed-loop task-conditioned perception
    Described via Eq. 3 and Fig 2; its standalone accuracy is not measured, and it cannot be compared outside this system.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics." pith.science (2026). https://pith.science/paper/LYQQOVIZ

@misc{pith2026260713245,
  author       = {Pith},
  title        = {Pith review of: Just-In-Time Scene Graph Growth: Combating Perceptual Saturation in Long-Horizon Robotics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYQQOVIZ}},
  note         = {Machine review of arXiv:2607.13245}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While 3D Scene Graphs (3DSGs) provide crucial structured representations for embodied agents, conventional Ahead-of-Time, build-everything-then-filter pipelines conflict with the real-time, low-latency demands of edge platforms, inducing a perceptual saturation effect via severe observation redundancy. To resolve this, we present JITOMA (Just-In-Time On-demand Memory Activation), a closed-loop framework that unifies task reasoning, perception, and memory into a just-in-time growth process. Instead of exhaustively mapping the entire environment, JITOMA leverages a top-down task heatmap at the frontend to filter continuous observations, routing minimal streams to maintain a global foundation of low-cost, dormant anchors. Upon a cognitive query, the backend Large Language Model (LLM) parses the robotic intent to dynamically awaken task-relevant anchors, triggering resource-intensive operations -- such as dense node captioning and functional inference -- exclusively within the activated local subgraph. To evaluate these dynamic capabilities and study perceptual saturation trade-offs, we introduce JITOMA-Bench, a comprehensive suite for long-horizon multi-tasking and complex multi-step reasoning. Extensive experiments demonstrate that JITOMA substantially reduces active graph size and captioning latency, while maintaining stable processing time under long-horizon task switching.

Figures

Figures reproduced from arXiv: 2607.13245 by Dazhi Huang, Hui Xiong, Li Chen, Rufeng Chen, Sihong Xie, Wenze Zhang, Yi Chen, Yifan Tian, Yue Chang, Zhaofan Zhang.

Figure 1
Figure 1. Figure 1: Study on the Perceptual Saturation Effect. Given a user command within a cluttered cubicle scene: Left (AOT Full Graph): Providing the LLM planner with an unconditional 3D scene graph containing all 18 environmental entities (including 15 task-irrelevant distractors) induces cog￾nitive myopia, causing the plan to terminate prematurely and miss the final containment step. Right (JIT Subgraph, Ours): By sele… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the JITOMA framework. (a) Intent Parsing: An LLM converts a natural￾language command into explicit primary objects and latent functional targets required for execution. (b) Frontend Just-In-Time Perception: The parsed intent produces top-down task heatmaps that gate continuous RGB-D observations before they are written into memory, admitting only task￾salient tracks as lightweight hypotheses. (… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the three-tier JITOMA-Bench evaluation. The benchmark progressively evaluates explicit single-object grounding (Tier 1), long-horizon task switching within a continuous scene (Tier 2), and complex instruction grounding over multiple primary and latent objects (Tier 3). 5.1 Task Annotations and Complementary Evaluation Tracks Tier 1: Foundational Grounding. As shown in [PITH_FULL_IMAGE:figures/… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 7 linked inside Pith

  1. [1]

    Hughes, Y

    N. Hughes, Y . Chang, and L. Carlone. Hydra: A real-time spatial perception system for 3d scene graph construction and optimization.arXiv preprint arXiv:2201.13360, 2022

    Pith/arXiv arXiv 2022

  2. [2]

    Rosinol, A

    A. Rosinol, A. Violette, M. Abate, N. Hughes, Y . Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs.The International Journal of Robotics Research, 40(12-14):1510–1546, 2021

    2021

  3. [3]

    S.-C. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7515–7525, 2021

    2021

  4. [4]

    Zhang, A

    C. Zhang, A. Delitzas, F. Wang, R. Zhang, X. Ji, M. Pollefeys, and F. Engelmann. Open- vocabulary functional 3d scene graphs for real-world indoor spaces. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19401–19413, 2025

    2025

  5. [5]

    Q. Gu, A. Kuwajerwala, S. Morin, K. M. Jatavallabhula, B. Sen, A. Agarwal, C. Rivera, W. Paul, K. Ellis, R. Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024

    2024

  6. [6]

    K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping.arXiv preprint arXiv:2302.07241, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    S. Koch, N. Vaskevicius, M. Colosi, P. Hermosilla, and T. Ropinski. Open3dsg: Open- vocabulary 3d scene graphs from point clouds with queryable objects and open-set relation- ships. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 14183–14193, 2024

    2024

  8. [8]

    Linok, T

    S. Linok, T. Zemskova, S. Ladanova, R. Titkov, D. Yudin, M. Monastyrny, and A. Valenkov. Beyond bare queries: Open-vocabulary object grounding with 3d scene graph. In2025 IEEE International Conference on Robotics and Automation (ICRA), pages 13582–13589. IEEE, 2025

    2025

  9. [9]

    Maggio, Y

    D. Maggio, Y . Chang, N. Hughes, M. Trang, D. Griffith, C. Dougherty, E. Cristofalo, L. Schmid, and L. Carlone. Clio: Real-time task-driven open-set 3d scene graphs.IEEE Robotics and Automation Letters, 2024

    2024

  10. [10]

    Werby, C

    A. Werby, C. Huang, M. B¨uchner, A. Valada, and W. Burgard. Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024

    2024

  11. [11]

    Yamazaki, T

    K. Yamazaki, T. Hanyu, K. V o, T. Pham, M. Tran, G. Doretto, A. Nguyen, and N. Le. Open- fusion: Real-time open-vocabulary 3d mapping and queryable scene representation. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 9411–9417. IEEE, 2024

    2024

  12. [12]

    Z. Yan, S. Li, Z. Wang, L. Wu, H. Wang, J. Zhu, L. Chen, and J. Liu. Dynamic open-vocabulary 3d scene graphs for long-term language-guided mobile manipulation.IEEE Robotics and Au- tomation Letters, 2025

    2025

  13. [13]

    Chang, R

    Y . Chang, R. Chen, Z. Zhang, Y . Chen, Y . Tian, and S. Xie. Rag-3dsg: Enhancing 3d scene graphs with re-shot guided retrieval-augmented generation.arXiv preprint arXiv:2601.10168, 2026

    arXiv 2026

  14. [14]

    Maggio, N

    D. Maggio, N. Gorlo, and L. Carlone. Found-it: Foundation-model-first task-driven 3d scene graphs with granularity on demand.arXiv preprint arXiv:2605.25371, 2026. 11

    Pith/arXiv arXiv 2026

  15. [15]

    C. Agia, K. M. Jatavallabhula, M. Khodeir, O. Miksik, V . Vineet, M. Mukadam, L. Paull, and F. Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In Conference on Robot Learning, pages 46–58. PMLR, 2022

    2022

  16. [16]

    Maggio and L

    D. Maggio and L. Carlone. Bayesian fields: Task-driven open-set semantic gaussian splatting. arXiv preprint arXiv:2503.05949, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Y . Ju, Y . Liang, Y .-J. Wang, N. Gireesh, Y . Ju, S. Lee, Q. Gu, E. Hsieh, F. Huang, and K. Sreenath. Momagraph: State-aware unified scene graphs with vision-language model for embodied task planning.arXiv preprint arXiv:2512.16909, 2025

    arXiv 2025

  18. [18]

    B ¨uchner, A

    M. B ¨uchner, A. R¨ofer, T. Engelbracht, T. Welschehold, Z. Bauer, H. Blum, M. Pollefeys, and A. Valada. Articulated 3d scene graphs for open-world mobile manipulation.arXiv preprint arXiv:2602.16356, 2026

    arXiv 2026

  19. [19]

    Rotondi, F

    D. Rotondi, F. Scaparro, H. Blum, and K. O. Arras. Fungraph: Functionality aware 3d scene graphs for language-prompted scene interaction. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4083–4090. IEEE, 2025

    2025

  20. [20]

    S. Y . Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Clip on wheels: Zero-shot object navigation as object localization and exploration.arXiv preprint arXiv:2203.10421, 3 (4):7, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    D. Shah, B. Osi ´nski, S. Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. InConference on robot learning, pages 492–504. PMLR, 2023

    2023

  22. [22]

    H. Yin, X. Xu, Z. Wu, J. Zhou, and J. Lu. Sg-nav: Online 3d scene graph prompting for llm- based zero-shot object navigation.Advances in neural information processing systems, 37: 5285–5307, 2024

    2024

  23. [23]

    R. Chen, Y . Chang, X. Tang, H. Chen, and S. Xie. Psg-nav: Probabilistic scene graph naviga- tion via multiverse decision making.arXiv preprint arXiv:2606.01313, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Z. Xia, C. Xiong, L. Wei, X. Hu, and L. Pei. Exploring bottlenecks in vlm-llm navigation: How 3d scene understanding capability impacts zero-shot vln.arXiv preprint arXiv:2605.14801, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. InConference on robot learning, pages 894–906. PMLR, 2022

    2022

  26. [26]

    Rashid, S

    A. Rashid, S. Sharma, C. M. Kim, J. Kerr, L. Y . Chen, A. Kanazawa, and K. Goldberg. Lan- guage embedded radiance fields for zero-shot task-oriented grasping. In7th Annual Conference on Robot Learning, 2023

    2023

  27. [27]

    Honerkamp, M

    D. Honerkamp, M. B ¨uchner, F. Despinoy, T. Welschehold, and A. Valada. Language-grounded dynamic scene graphs for interactive object search with mobile manipulation.IEEE Robotics and Automation Letters, 2024

    2024

  28. [28]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps: //qwen.ai/blog?id=qwen3.5

    2026

  29. [29]

    Gorlo, L

    N. Gorlo, L. Schmid, and L. Carlone. Describe anything anywhere at any moment. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 35002–35013, 2026

    2026

  30. [30]

    L ¨uddecke and A

    T. L ¨uddecke and A. Ecker. Image segmentation using text and image prompts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086–7096, 2022. 12

    2022

  31. [31]

    L. Lian, Y . Ding, Y . Ge, S. Liu, H. Mao, B. Li, M. Pavone, M.-Y . Liu, T. Darrell, A. Yala, et al. Describe anything: Detailed localized image and video captioning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 21766–21777, 2025

    2025

  32. [32]

    Introducing GPT-5.4.https://openai.com/index/introducing-gpt-5-4/, Mar

    OpenAI. Introducing GPT-5.4.https://openai.com/index/introducing-gpt-5-4/, Mar. 2026. Accessed: July 14, 2026

    2026

  33. [33]

    full 001

    A. G. Puigjaner, A. Zacharia, and K. Alexis. Relationship-aware hierarchical 3d scene graph for task reasoning.arXiv preprint arXiv:2602.02456, 2026. 13 A Experiment A.1 Motivation Experiment: Ahead-of-Time (AOT) Full Graph Prompt This prompt template is utilized in our exploratory motivation experiment (Sec. 3) to evaluate down- stream Large Language Mod...

    arXiv 2026

  34. [37]

    rel primary

    Use short imperative actions. Do not explain. A.2 Motivation Experiment: Just-In-Time (JIT) Subgraph Prompt This prompt template corresponds directly to our proposed Just-In-Time (JIT) mapping paradigm operationalized in JITOMA. Instead of compiling an unconstrained global structure, the input scene graph undergoes process-level resource gating. Computati...

  35. [38]

    Mention all objects that are required to complete the task

  36. [39]

    If a destination, container, support, source, or tool is needed and exists in the graph, include it

  37. [40]

    Every mentioned object must include its center coordinate exactly once in that step

  38. [41]

    Do not explain

    Use short imperative actions. Do not explain. 15

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.