pith. sign in

arxiv: 2607.06534 · v1 · pith:FBAD7WPR · submitted 2026-07-07 · cs.CV

CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

He Liang , Chenyang Ma , Yiming Zhang , Sangyun Shin , Andrew Markham , Niki Trigoni , Yuhang He This is my paper

pith:FBAD7WPRreviewed 2026-07-08 02:26 UTCmodel glm-5.2open to challenge →

classification cs.CV
keywords 3D scene understandinglarge language modelsmulti-room reasoninghierarchical attentionscene graphstopology-aware attentionembodied AIspatial reasoning
0
0 comments X

The pith

Aligning attention with room topology boosts 3D-LLM cross-room reasoning

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing 3D scene-language models treat all objects in a scene as a flat, fully-connected list of tokens — fine for a single room but problematic for multi-room households where most object interactions are local to a room and cross-room dependencies are sparse. This paper argues that constraining transformer attention to match the hierarchical structure of a scene — objects attend only within their room, cross-room information flows through learned room tokens, and geometric biases encode spatial priors — is the primary driver of improved multi-room understanding. The authors introduce CAIRN, a 3D-LLM that encodes scenes as two-layer hierarchical scene graphs (object-level relations plus room-level connectivity), tokenizes them into object and room tokens, and applies a hierarchical attention mask with learned geometric bias terms. They also introduce CAIRN-MR, a benchmark of 673 multi-room HM3D scenes with tasks spanning grounding, captioning, and four QA types that progressively require cross-room reasoning. On this benchmark, CAIRN outperforms prior 3D-LLMs across all tasks, with the largest gains on tasks requiring the most room-level reasoning (inter-room comparison +6.7 EM, captioning +14.9 CIDEr). Ablation confirms that removing the hierarchical attention mask causes the largest performance drops across all metrics, supporting the claim that topology-aware attention routing is the central mechanism.

Core claim

The key finding is that when 3D-LLM attention is structurally aligned with scene topology — specifically, by masking object-object attention to within-room pairs and routing cross-room communication exclusively through learned room tokens — the model gains scale with the degree of room-level reasoning a task requires. Tasks confined to recognizing an object within an already-localized room (identification) show minimal change, while tasks demanding cross-room comparison and existence verification show the largest improvements. This graded pattern, confirmed by ablation where removing the hierarchical mask produces the biggest drops, indicates that the attention mask's topology constraint —而非

What carries the argument

The central object is a hierarchical attention mask M_H defined over a two-layer scene graph: object nodes connect to in-room neighbors, room nodes connect to adjacent rooms, and cross-layer edges assign objects to rooms. The mask permits attention only along graph edges (object-object within room, object-room, room-room), while a learned geometric bias B_theta adds spatial priors (relative position, distance, adjacency) to attention logits for the first few layers. Room tokens are computed via cross-attention from learned queries to in-room object tokens, serving as the sole bridge for inter-room information flow.

If this is right

  • If topology-aware attention generalizes beyond indoor scenes, the same principle — constraining attention to match domain-specific graph structure rather than allowing dense all-to-all interactions — could apply to any setting where entities have natural locality, such as multi-floor buildings, outdoor environments, or temporal scene sequences.
  • The graded gains pattern (minimal for within-room tasks, large for cross-room tasks) suggests that flat-attention 3D-LLMs have a specific bottleneck in cross-context reasoning, not in per-object perception — a diagnostic that could guide where structural priors are most needed in other multimodal domains.
  • The block-sparse attention pattern reduces object-object attention complexity from O(N²) to O(Σ_r |O(r)|²), which may make topology-aware approaches more scalable as scene sizes grow, potentially enabling 3D-LLMs to handle building-scale or neighborhood-scale environments.
  • The benchmark design — where target rooms must be inferred from implicit spatial descriptions rather than explicit labels — establishes a methodology for testing whether models truly understand spatial context versus exploiting surface cues, applicable to other embodied AI evaluation settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The co-design of model and benchmark means the reported gaps reflect CAIRN's architectural advantage under evaluation conditions that specifically test the capabilities its topology-aware design targets. If baselines were given equivalent room-level tokens or scene graph inputs, the gap might narrow, particularly on tasks like identification where gains are already minimal.
  • The finding that geometric bias provides smaller gains than the attention mask suggests that routing structure (which tokens can see which) matters more than fine-grained spatial encoding (how strongly they attend) — implying that for structural reasoning, the topology of information flow may be more important than the geometry of interactions.
  • If the hierarchical mask is the dominant contributor, one could test whether simpler topology constraints (e.g., hard room partitioning without learned room tokens) achieve similar gains, which would clarify whether the room token abstraction or the masking itself is the causal factor.

Load-bearing premise

The benchmark CAIRN-MR and the model CAIRN are co-designed by the same authors, and the baselines are re-implemented under the authors' training protocol rather than taken from their original papers. The very low baseline grounding numbers (18.9–20.4 Acc@0.25) suggest possible under-tuning, and if baselines were given the same multi-room scene graph or room-level tokens that CAIRN uses, the performance gap might narrow substantially.

What would settle it

If prior 3D-LLMs, when given the same hierarchical scene graph input and room-level tokens that CAIRN uses (but without topology-aware masking), achieved comparable cross-room reasoning performance, the central claim that attention alignment with topology is the primary driver would be undermined.

Figures

Figures reproduced from arXiv: 2607.06534 by Andrew Markham, Chenyang Ma, He Liang, Niki Trigoni, Sangyun Shin, Yiming Zhang, Yuhang He.

Figure 1
Figure 1. Figure 1: 3D-LLMs for multi-room scene understanding. We introduce CAIRN, a topology-aware 3D-LLM for multi-room scene understanding, along with CAIRN-MR, a multi-room 3D scene understanding benchmark with diverse tasks. By representing each scene as a hierarchical scene graph and aligning attention with scene topology through structured masking and geometric bias, CAIRN achieves substantial gains over prior 3D-LLMs… view at source ↗
Figure 2
Figure 2. Figure 2: CAIRN-MR benchmark for multi-room 3D scene understanding. The benchmark includes grounding, captioning, and question answering tasks covering room localization, intra￾room reasoning, and cross-room comparison. Colored boxes denote parsed room regions. Right: benchmark scale and task composition. description, and then identify the target object within that room. Errors in room localization therefore directl… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of CAIRN. Given a 3D scene, CAIRN constructs a hierarchical scene graph capturing object relations and room adjacencies (bottom), tokenizes it into object and room tokens (middle), and feeds them to an LLM with hierarchical masked attention and geometric bias (top). The mask routes information along scene topology, while learned bias terms inject spatial priors into attention logits, enabling topo… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on a multi-room scene with visually similar living rooms. Green denotes CAIRN; red denotes the best baseline. 5.5 Performance on Single-Room Setting Tab. 3 reports results on five ScanNet-based single-room benchmarks. Since these scenes contain a single room, the hierarchical struc￾ture reduces to a single level, leaving graph to￾kens and geometric bias as the effective compo￾nents. CAI… view at source ↗
Figure 5
Figure 5. Figure 5: Prompt template for querying Qwen3-VL-32B. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template for generating QA-I questions. The target class name is masked in the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
read the original abstract

Existing 3D scene-grounded Large Language Models (3D-LLMs) focus on answering questions grounded in simplified single-room 3D scenes, lacking the ability to reason over real-world household environments containing multiple interconnected rooms and diverse object categories. We introduce CAIRN, a topology-aware 3D-LLM for multi-room 3D scene understanding. CAIRN aligns transformer attention with scene hierarchy, giving the model explicit awareness of object-level relations and room-level connectivity. It enriches object tokens with room-local relational context via a graph neural network, introduces learned room tokens for room-level abstraction, and applies a hierarchical attention mask with geometric bias to route information according to scene topology. CAIRN is developed on CAIRN-MR, a benchmark we introduce on HM3D for multi-room 3D scene understanding, covering grounding, captioning, and four question-answering tasks that progressively evaluate from intra-room perception to cross-room reasoning. Experiments show that CAIRN outperforms prior 3D-LLMs by a large margin across all CAIRN-MR tasks while remaining competitive on five single-room benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. The paper introduces CAIRN, a topology-aware 3D-LLM for multi-room 3D scene understanding, along with CAIRN-MR, a new benchmark built on HM3D. CAIRN encodes scenes as hierarchical scene graphs (object-level and room-level), tokenizes them into object and room tokens, and applies a hierarchical attention mask with geometric bias to route information according to scene topology. The benchmark covers grounding, captioning, and four QA tasks ranging from intra-room perception to cross-room comparison. Experiments show CAIRN outperforms prior 3D-LLMs (Chat-Scene, 3DGraphLLM, Inst3D-LMM) on CAIRN-MR and remains competitive on five single-room ScanNet benchmarks. The ablation (Table 4) isolates the contributions of graph tokens, room tokens, the hierarchical mask, and geometric bias.

Significance. The paper addresses a genuine gap: existing 3D-LLMs are designed for single-room scenes, and the multi-room setting introduces real challenges (room localization, cross-room disambiguation, scalability). The hierarchical masked attention design is a principled and novel contribution for this setting. The benchmark includes human calibration (Table 1) and statistical baselines, which is commendable. The ablation is well-structured and the component-level analysis is informative. The single-room results (Table 3) demonstrate that the multi-room extensions do not degrade fine-grained object-level performance. The work is a solid step toward structured multi-room 3D reasoning.

major comments (3)
  1. §4.2, Appendix C.2: The graph token tG_i is constructed using VL-SAT relation embeddings as edge features in the message-passing GNN. The CAIRN-MR benchmark's referring expressions and QA tasks are constructed from SceneVerse pairwise spatial relations (11 relation types, Table 5, §B.2), which overlap semantically with the VL-SAT relation types (e.g., 'above,' 'next to'). This creates a representational alignment between CAIRN's input encoding and the benchmark's task structure that baselines (Chat-Scene, 3DGraphLLM, Inst3D-LMM) do not share. The flat ablation (19.6 Acc@0.25) matching Chat-Scene (19.6) is reassuring for baseline fairness at the flat level, but the graph token contribution (+1.2 Acc@0.25 grounding, +4.0 CIDEr captioning, +3.5 EM existence) could be inflated by this alignment. The authors should discuss this potential confound and, ideally, provide evidence that the graph-
  2. §5.4, Table 2: The baselines (Chat-Scene, 3DGraphLLM, Inst3D-LMM) are re-implemented under the authors' training protocol with Qwen3-8B, but their grounding Acc@0.25 numbers (18.9–20.4) are very low in absolute terms. The paper does not report whether hyperparameter tuning was performed for baselines on CAIRN-MR, or whether the baselines were given access to room-level information comparable to CAIRN's room tokens. The paper should clarify: (1) whether baselines received any multi-room structural information (e.g., room segmentation, room labels) or were trained on flat object sequences only; (2) what hyperparameter search was conducted for baselines; and (3) whether the low baseline numbers reflect task difficulty or under-tuning. This is load-bearing because the headline gains (+5.4 Acc@0.25, +14.9 CIDEr) are the primary evidence for the contribution.
  3. §5.4, Table 2: The headline gains are measured on a benchmark (CAIRN-MR) that the same authors constructed and evaluated. While the benchmark construction procedure (§B) is described in detail and includes uniqueness verification (Algorithm 1), the paper does not provide an independent evaluation or external validation of the benchmark. The human calibration (Table 1) is a positive step, but it covers only QA tasks and a subset. The paper should discuss the risk of benchmark-model co-design more explicitly, particularly whether the task design (§3.1) systematically favors capabilities that CAIRN's architecture addresses (room localization, cross-room comparison, topology-aware reasoning). A concrete test would be to evaluate CAIRN on an independently constructed multi-room benchmark or on CAIRN-MR tasks with relation compositions not seen during training.
minor comments (6)
  1. §4.3, Eq. (6): The descriptor g_uv for geometric bias is described qualitatively ('relative position and distance for OO pairs, object-to-room location for OR pairs, and relative layout for RR pairs') but its exact formulation is not specified. Providing the explicit form of g_uv for each interaction type would improve reproducibility.
  2. §5.1: The max-grad-norm of 0.01 is described as 'aggressive' and 'empirically chosen.' The paper should note whether this was applied to all models including baselines, or only to CAIRN. If baselines used a different gradient clipping value, this could affect fairness.
  3. Table 2: CAIRN is evaluated with four LLM backbones (Llama-3.2-3B, Llama-3.1-8B, Qwen3-4B, Qwen3-8B), but baselines are only evaluated with Qwen3-8B. Including at least one baseline with a different backbone would strengthen the comparison.
  4. §B.1: The seed-room selection criterion (spatial extent > 30 m²) and the selection of 3–5 nearest rooms may introduce bias toward larger or more connected scenes. The paper should discuss whether this affects benchmark diversity.
  5. Figure 3: The notation for the hierarchical scene graph and tokenization pipeline is dense. A clearer legend or a simplified version of the figure would help readability.
  6. §C.2: The maximum object cap of 300 per scene excludes ~4% of CAIRN-MR task instances. The paper should report whether excluded instances are concentrated in specific scene types or room counts, as this could introduce systematic bias.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The comments identify three important concerns: (1) potential representational alignment between VL-SAT relation embeddings used in CAIRN's graph tokens and the SceneVerse spatial relations used to construct CAIRN-MR, (2) fairness of baseline evaluation including hyperparameter tuning and access to multi-room structural information, and (3) benchmark-model co-design risk. We address each point below and commit to revisions where appropriate.

read point-by-point responses

  1. Referee: §4.2, Appendix C.2: The graph token tG_i is constructed using VL-SAT relation embeddings as edge features in the message-passing GNN. The CAIRN-MR benchmark's referring expressions and QA tasks are constructed from SceneVerse pairwise spatial relations (11 relation types, Table 5, §B.2), which overlap semantically with the VL-SAT relation types (e.g., 'above,' 'next to'). This creates a representational alignment between CAIRN's input encoding and the benchmark's task structure that baselines (Chat-Scene, 3DGraphLLM, Inst3D-LMM) do not share. The flat ablation (19.6 Acc@0.25) matching Chat-Scene (19.6) is reassuring for baseline fairness at the flat level, but the graph token contribution (+1.2 Acc@0.25 grounding, +4.0 CIDEr captioning, +3.5 EM existence) could be inflated by this alignment. The authors should discuss this potential confound and, ideally, provide evidence that the graph-

    Authors: We agree this is a legitimate concern and appreciate the referee identifying it. Let us clarify the precise relationship between VL-SAT and SceneVerse relations, and then address the confound directly. VL-SAT [49] is a pretrained 3D scene graph prediction model whose relation embeddings are learned from ScanNet data. SceneVerse [24] provides its own pairwise spatial relation annotations for HM3D scenes, derived from geometric heuristics (e.g., bounding-box overlap, relative position). While there is semantic overlap in relation types (both include categories like 'above,' 'next to'), the VL-SAT embeddings are continuous learned vectors, not discrete labels, and they are computed from object geometry rather than from the SceneVerse annotation pipeline. So the alignment is at the level of relation semantics, not at the level of shared labels or shared annotation sources. That said, we acknowledge that even semantic-level alignment could confer an advantage, since VL-SAT embeddings may encode spatial patterns that are correlated with the relation types used in CAIRN-MR expressions. We will add an explicit discussion of this potential confound in the revised manuscript. Regarding evidence: we can provide two pieces of evidence that the graph token gains are not solely driven by this alignment. First, 3DGraphLLM [60] also uses a graph neural network with spatial relation features, yet its performance (18.9 Acc@0.25) is comparable to the flat Chat-Scene baseline (19.6), suggesting that graph-based relational encoding alone does not confer a large advantage on CAIRN-MR without the hierarchical structure. Second, the ablation (Table 4) shows that the graph token contribution (+1.2 Acc@0.25) is substantially smaller than the contributions of the hierarchical mask (+4.2 Acc@0.25) revision: partial

  2. Referee: §5.4, Table 2: The baselines (Chat-Scene, 3DGraphLLM, Inst3D-LMM) are re-implemented under the authors' training protocol with Qwen3-8B, but their grounding Acc@0.25 numbers (18.9–20.4) are very low in absolute terms. The paper does not report whether hyperparameter tuning was performed for baselines on CAIRN-MR, or whether the baselines were given access to room-level information comparable to CAIRN's room tokens. The paper should clarify: (1) whether baselines received any multi-room structural information (e.g., room segmentation, room labels) or were trained on flat object sequences only; (2) what hyperparameter search was conducted for baselines; and (3) whether the low baseline numbers reflect task difficulty or under-tuning. This is load-bearing because the headline gains (+5.4 Acc@0.25, +14.9 CIDEr) are the primary evidence for the contribution.

    Authors: We agree that these details are important and should be stated explicitly in the revised manuscript. To address each sub-question: (1) Baselines were trained on flat object token sequences without room-level structural information. This is because none of the three baseline architectures (Chat-Scene, 3DGraphLLM, Inst3D-LMM) were designed with room-level tokens or hierarchical attention. We did not modify their architectures to incorporate room segmentation or room labels, as doing so would constitute a non-trivial architectural change rather than a faithful re-implementation. We acknowledge that this means baselines do not have access to room-level information comparable to CAIRN's room tokens, and we will state this clearly. (2) For hyperparameter tuning, we used the same training protocol for all methods: the same two-stage training schedule, the same Qwen3-8B backbone, the same LoRA configuration (rank 16), and the same learning rate (5e-6). We did not conduct per-method hyperparameter search on CAIRN-MR. We used each baseline's recommended architectural settings from their respective papers (e.g., 3DGraphLLM's graph construction parameters, Inst3D-LMM's instance-aware module configuration). We will add a table in the appendix specifying all baseline configurations. (3) The low absolute numbers reflect task difficulty rather than under-tuning. Several factors make CAIRN-MR grounding substantially harder than ScanRefer: scenes contain ~115 objects across 4.5 rooms on average (vs. ~20-40 in ScanNet single-room), the target is not specified by room, and recurring object categories across rooms create many plausible candidates. The flat ablation of CAIRN (19.6 Acc@0.25) matches Chat-Scene (19.6), confirming that at the flat level the methods are comparable. CAIRN's gains revision: yes

  3. Referee: §5.4, Table 2: The headline gains are measured on a benchmark (CAIRN-MR) that the same authors constructed and evaluated. While the benchmark construction procedure (§B) is described in detail and includes uniqueness verification (Algorithm 1), the paper does not provide an independent evaluation or external validation of the benchmark. The human calibration (Table 1) is a positive step, but it covers only QA tasks and a subset. The paper should discuss the risk of benchmark-model co-design more explicitly, particularly whether the task design (§3.1) systematically favors capabilities that CAIRN's architecture addresses (room localization, cross-room comparison, topology-aware reasoning). A concrete test would be to evaluate CAIRN on an independently constructed multi-room benchmark or on CAIRN-MR tasks with relation compositions not seen during training.

    Authors: This is a fair and important concern. We acknowledge that benchmark-model co-design is a genuine risk when the same authors construct both the benchmark and the model, and we will add an explicit discussion of this risk in the revised manuscript. We can partially address the concern with the following points. First, the task design in CAIRN-MR is motivated by the structural properties of multi-room environments (recurring categories, implicit room localization, cross-room comparison) rather than by CAIRN's specific architectural choices. The tasks—grounding, captioning, counting, existence verification, comparison—are standard 3D scene understanding tasks extended to the multi-room setting. The progression from intra-room to cross-room reasoning reflects the natural structure of the problem, not CAIRN's design. Second, the human calibration (Table 1) demonstrates that the tasks are well-defined and solvable by humans (86.8%–93.5%), which provides evidence that the benchmark is not artificially constructed to favor a particular architecture. We agree that the calibration covers only QA tasks and will extend it to grounding and captioning in the revision. Third, regarding the concrete test of evaluating on relation compositions not seen during training: this is a valuable suggestion. We can construct a held-out evaluation set where the test referring expressions use relation compositions (e.g., specific combinations of relation types) that do not appear in the training split, and evaluate CAIRN and baselines on this set. We commit to including this compositional generalization evaluation in the revised manuscript. We note that the Limitations section already mentions compositional generalization as a direction for future work, but we agree it should be empirically tested. revision: partial

Circularity Check

0 steps flagged

No significant circularity: CAIRN's gains are measured on a co-designed benchmark, but the architecture's contributions are independently ablatable and the benchmark construction does not reduce to the model's outputs by construction.

full rationale

The paper co-designs CAIRN (model) and CAIRN-MR (benchmark), and the benchmark's referring expressions are constructed from SceneVerse spatial relations (Sec. B.2) while CAIRN's graph tokens encode VL-SAT relation embeddings (Sec. C.2) over the same relation types. This creates a representational alignment risk — the spatial relations the model must reason about are pre-encoded in its graph tokens in a similar semantic space. However, this is a benchmark fairness concern, not circularity in the derivation chain. The benchmark expressions are generated from SceneVerse annotations (an external dataset) and verified for uniqueness via Algorithm 1, not derived from CAIRN's outputs. The model's architecture (hierarchical masked attention, geometric bias, room tokens, graph tokens) is defined independently of the benchmark tasks. The ablation (Tab. 4) removes components and measures performance changes — the flat baseline (19.6 Acc@0.25) matches Chat-Scene (19.6), providing evidence that the gains are not tautological. The single-room results (Tab. 3) on five external benchmarks provide independent evaluation where CAIRN's room-level components are inactive. No equation or definition reduces to its own input by construction. The co-design risk is real but falls under correctness/benchmark fairness, not circularity. The one minor concern is that the benchmark tasks are designed to test capabilities CAIRN's architecture addresses (room localization, cross-room comparison), but task design targeting architectural strengths is standard practice, not self-definitional circularity.

Axiom & Free-Parameter Ledger

10 free parameters · 5 axioms · 3 invented entities

The model introduces 10+ free parameters, most empirically chosen without systematic search. The key axioms are domain assumptions about multi-room scene structure that are reasonable but not independently validated. The invented entities (room tokens, geometric bias, hierarchical mask) are architectural components validated through ablation, not new physical postulates. The circularity concern is moderate: the benchmark is author-constructed and the model is designed to exploit the benchmark's structure.

free parameters (10)
  • Kg (graph neighbor count) = not stated explicitly
    Number of nearest in-room neighbors for object-object edges; controls graph density
  • Kr (room queries) = 4
    Number of learned queries per room for cross-attention aggregation; chosen empirically
  • Lg (message passing layers) = 2
    Depth of GNN for graph tokens; chosen to avoid over-smoothing
  • Lb (bias layers) = 4
    Number of transformer layers where geometric bias Bθ is applied; chosen empirically
  • LoRA rank = 16
    Low-rank adaptation dimension; standard choice
  • max-grad-norm = 0.01
    Aggressively low gradient clipping, empirically chosen to stabilize hierarchical components; values in [1.0, 5.0] caused instability
  • learning rate = 5e-6
    Base learning rate for AdamW optimizer
  • max objects per scene = 300
    Cap on object instances; retains >96% of CAIRN-MR instances
  • MLP parameters for b_τ (geometric bias) = not stated
    Type-specific MLPs for OO, OR, RR interaction types; architecture details not fully specified
  • Projection MLP parameters = 2-layer, not fully specified
    MLPs projecting Uni3D, DINOv2, and graph features to LLM hidden dimension
axioms (5)
  • domain assumption Multi-room scenes exhibit primarily intra-room object interactions with sparse, structured cross-room dependencies
    Sec. 1, Sec. 4.3: motivates the hierarchical mask design. This is a reasonable assumption for indoor environments but is not empirically validated beyond the performance gains.
  • domain assumption Room-level semantic units can be obtained via bird's-eye-view occupancy partitioning
    Sec. 4.1: room nodes VR are obtained via occupancy partitioning [50]. The quality of room segmentation directly affects all downstream components.
  • domain assumption Two message-passing layers suffice to propagate spatial neighborhood information without over-smoothing
    Appendix C.2: Lg=2 following GraphSAGE convention. Not validated against deeper alternatives in the ablation.
  • ad hoc to paper Geometric bias need only be applied to the first Lb=4 transformer layers
    Sec. 4.3: Bθ applied to first Lb layers, later layers retain only structural sparsity. The choice of 4 is not justified by ablation or theory.
  • domain assumption SceneVerse spatial relations and HM3D room annotations are accurate and sufficient for multi-room benchmark construction
    Sec. 3.2, Appendix B: benchmark built on SceneVerse [24] annotations over HM3D [44]. Quality of these annotations directly determines benchmark quality.
invented entities (3)
  • Room tokens (learned queries q_{r,k}) independent evidence
    purpose: Room-level abstraction tokens that cross-attend to object tokens within each room, enabling cross-room communication
    These are learned representations, not new physical entities. Their effectiveness is validated through ablation (Tab. 4: removing room tokens drops captioning by 3.4 CIDEr and comparison by 2.1 EM).
  • Geometric bias Bθ independent evidence
    purpose: Learned additive offset to attention logits encoding spatial priors (relative position, distance, room adjacency)
    Validated through ablation (Tab. 4: removing bias drops grounding by 1.3 Acc@0.25). The type-specific MLPs b_τ are standard neural components.
  • Hierarchical attention mask MH independent evidence
    purpose: Structured mask constraining token interactions to scene topology edges (S_OO ∪ S_OR ∪ S_RR)
    Core contribution; ablation shows it is the primary driver of gains (Tab. 4: removing mask causes largest drops across all metrics).

pith-pipeline@v1.1.0-glm · 22914 in / 3815 out tokens · 428463 ms · 2026-07-08T02:26:49.678132+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 74 canonical work pages · 32 internal anchors

  1. [1]

    Scanqa: 3d question answering for spatial scene understanding

    Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022

    work page 2022

  2. [2]

    Longformer: The Long-Document Transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2004

  3. [3]

    SAM 3: Segment Anything with Concepts

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

    Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks.arXiv preprint arXiv:2411.00081, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Scanrefer: 3d object localization in rgb-d scans using natural language

    Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. InEuropean conference on computer vision, pages 202–221. Springer, 2020

    work page 2020

  6. [6]

    Structure-aware transformer for graph representa- tion learning

    Dexiong Chen, Leslie O’Bray, and Karsten Borgwardt. Structure-aware transformer for graph representa- tion learning. InInternational conference on machine learning, pages 3469–3489. PMLR, 2022

    work page 2022

  7. [7]

    Language conditioned spatial relation reasoning for 3d object grounding.Advances in Neural Information Processing Systems, 35:20522–20535, 2022

    Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding.Advances in Neural Information Processing Systems, 35:20522–20535, 2022

    work page 2022

  8. [8]

    LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning. arXiv preprint arXiv:2311.18651, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, YU Gang, Taihao Li, and Tao Chen. V ote2cap-detr++: Decoupling localization and describing for end-to-end 3d dense captioning.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    work page 2024

  10. [10]

    Grounded 3D-LLM with Referent Tokens

    Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Ruiyuan Lyu, Runsen Xu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens.arXiv preprint arXiv:2405.10370, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Scan2cap: Context-aware dense captioning in rgb-d scans

    Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning in rgb-d scans. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3193–3203, 2021

    work page 2021

  12. [12]

    Generating Long Sequences with Sparse Transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  13. [13]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017

    work page 2017

  14. [14]

    Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

    Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning.arXiv preprint arXiv:2403.11401, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024

    work page 2024

  16. [16]

    Inductive representation learning on large graphs

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017

    work page 2017

  17. [17]

    Language- grounded dynamic scene graphs for interactive object search with mobile manipulation.IEEE Robotics and Automation Letters, 9(10):8298–8305, 2024

    Daniel Honerkamp, Martin Büchner, Fabien Despinoy, Tim Welschehold, and Abhinav Valada. Language- grounded dynamic scene graphs for interactive object search with mobile manipulation.IEEE Robotics and Automation Letters, 9(10):8298–8305, 2024. 10

    work page 2024

  18. [18]

    3D-LLM: Injecting the 3D World into Large Language Models

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.arXiv preprint arXiv:2307.12981, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  20. [20]

    Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

    Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.arXiv preprint arXiv:2312.08168, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Chat-scene: Bridging 3d scene and large language models with object identifiers

    Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  22. [22]

    An Embodied Generalist Agent in 3D World

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world.arXiv preprint arXiv:2311.12871, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Hughes, Y

    N. Hughes, Y . Chang, and L. Carlone. Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization. InRobotics: Science and Systems (RSS), 2022

    work page 2022

  24. [24]

    Sceneverse: Scaling 3d vision-language learning for grounded scene understanding

    Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. InEuropean Conference on Computer Vision, pages 289–310. Springer, 2024

    work page 2024

  25. [25]

    Context-aware alignment and mutual masking for 3d-language pre-training

    Zhao Jin, Munawar Hayat, Yuwei Yang, Yulan Guo, and Yinjie Lei. Context-aware alignment and mutual masking for 3d-language pre-training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10984–10994, 2023

    work page 2023

  26. [26]

    Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

    Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Robin3d: Improving 3d large language model via robust instruction tuning.arXiv preprint arXiv:2410.00255, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Chang, and Manolis Savva

    Mukul Khanna*, Yongsen Mao*, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexan- der Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation.arXiv preprint, 2023

    work page 2023

  28. [28]

    LISA: Reasoning Segmentation via Large Language Model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10): 200102, 2025

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10): 200102, 2025

    work page 2025

  30. [30]

    Gradformer: Graph Transformer with Exponential Decay

    Chuang Liu, Zelin Yao, Yibing Zhan, Xueqi Ma, Shirui Pan, and Wenbin Hu. Gradformer: Graph transformer with exponential decay.arXiv preprint arXiv:2404.15729, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    K. Lu, C. Ma, C. Hori, and D. Romeres. KitchenVLA: Iterative vision-language corrections for robotic execution of human tasks. InProceedings of the IEEE International Conference on Robotics and Automa- tion Workshop on Safely Leveraging Vision-Language F oundation Models in Robotics (SafeLVMs@ICRA), 2025

    work page 2025

  34. [34]

    Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors

    Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, and Andrew Markham. Spatialpin: Enhancing spatial reasoning capabilities of vision-language models through prompting and interacting 3d priors. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  35. [35]

    Coopera: Continual open-ended human-robot assistance

    Chenyang Ma, Kai Lu, Ruta Desai, Xavier Puig, Andrew Markham, and Niki Trigoni. Coopera: Continual open-ended human-robot assistance. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2025

    work page 2025

  36. [36]

    Cyclevla: Proactive self-correcting vision-language-action models via subtask backtracking and minimum bayes risk decoding.arXiv preprint arXiv:2601.02295, 2026

    Chenyang Ma, Guangyu Yang, Kai Lu, Shitong Xu, Bill Byrne, Niki Trigoni, and Andrew Markham. Cyclevla: Proactive self-correcting vision-language-action models via subtask backtracking and minimum bayes risk decoding.arXiv preprint arXiv:2601.02295, 2026. 11

    work page arXiv 2026

  37. [37]

    SQA3D: Situated Question Answering in 3D Scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes.arXiv preprint arXiv:2210.07474, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  38. [38]

    DINOv2: Learning Robust Visual Features without Supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  39. [39]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  40. [40]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018

    work page 2018

  41. [41]

    Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

    Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  42. [42]

    Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [43]

    GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

    Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models.arXiv preprint arXiv:2501.01428, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  44. [44]

    Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.arXiv preprint arXiv:2109.08238, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  45. [45]

    SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning

    Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning.arXiv preprint arXiv:2307.06135, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  46. [46]

    Mask3d: Mask transformer for 3d semantic instance segmentation

    Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8216–8223. IEEE, 2023

    work page 2023

  47. [47]

    Qwen3 Technical Report

    Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  48. [48]

    Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models

    Hugues Thomas, Chen Chen, and Jian Zhang. Pts3d-llm: Studying the impact of token structure for 3d scene understanding with large language models.arXiv preprint arXiv:2506.05689, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud

    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang, and Lu Sheng. Vl-sat: Visual-linguistic semantics assisted training for 3d semantic scene graph prediction in point cloud. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21560–21569, 2023

    work page 2023

  50. [50]

    Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation

    Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hier- archical open-vocabulary 3d scene graphs for language-grounded robot navigation. InProceedings of Robotics: Science and Systems, 2024

    work page 2024

  51. [51]

    CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversations

    Pengying Wu, Yao Mu, Kangjie Zhou, Ji Ma, Junting Chen, and Chang Liu. Camon: Cooperative agents for multi-object navigation with llm-based conversations.arXiv preprint arXiv:2407.00632, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    3ur-llm: An end-to-end multimodal large language model for 3d scene understanding.IEEE Transactions on Multimedia, 27: 2899–2911, 2025

    Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, and Huchuan Lu. 3ur-llm: An end-to-end multimodal large language model for 3d scene understanding.IEEE Transactions on Multimedia, 27: 2899–2911, 2025. doi: 10.1109/TMM.2025.3557620

    work page doi:10.1109/tmm.2025.3557620 2025

  53. [53]

    Descrip3d: Enhancing large language model-based 3d scene understanding with object-level text descriptions.arXiv preprint arXiv:2507.14555, 2025

    Jintang Xue, Ganning Zhao, Jie-En Yao, Hong-En Chen, Yue Hu, Meida Chen, Suya You, and C-C Jay Kuo. Descrip3d: Enhancing large language model-based 3d scene understanding with object-level text descriptions.arXiv preprint arXiv:2507.14555, 2025

    work page arXiv 2025

  54. [54]

    Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. InCVPR, 2025

    work page 2025

  55. [55]

    HomeRobot: Open-Vocabulary Mobile Manipulation

    Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open- vocabulary mobile manipulation.arXiv preprint arXiv:2306.11565, 2023. 12

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  56. [56]

    Do transformers really perform badly for graph representation?Advances in neural information processing systems, 34:28877–28888, 2021

    Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation?Advances in neural information processing systems, 34:28877–28888, 2021

    work page 2021

  57. [57]

    Ferret: Refer and Ground Anything Anywhere at Any Granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity.arXiv preprint arXiv:2310.07704, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  58. [58]

    Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning

    Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 14147–14157, 2025

    work page 2025

  59. [59]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283–17297, 2020

    work page 2020

  60. [60]

    3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

    Tatiana Zemskova and Dmitry Yudin. 3dgraphllm: Combining semantic graphs and large language models for 3d scene understanding.arXiv preprint arXiv:2412.18450, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  61. [61]

    Multi3drefer: Grounding text description to multiple 3d objects

    Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15225– 15236, 2023

    work page 2023

  62. [62]

    3dvg-transformer: Relation modeling for visual grounding on point clouds

    Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2928–2937, 2021

    work page 2021

  63. [63]

    BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

    Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal llms.arXiv preprint arXiv:2307.08581, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  64. [64]

    Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences

    Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, and Chuang Gan. Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3761–3771, 2025

    work page 2025

  65. [65]

    Uni3D: Exploring Unified 3D Representation at Scale

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale.arXiv preprint arXiv:2310.06773, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  66. [66]

    3d-vista: Pre-trained transformer for 3d vision and text alignment

    Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre-trained transformer for 3d vision and text alignment. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2911–2921, 2023

    work page 2023

  67. [67]

    Unifying 3d vision-language understanding via promptable queries

    Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-language understanding via promptable queries. InEuropean Conference on Computer Vision, pages 188–206. Springer, 2024. 13 Appendix for CAIRN A Overview This appendix provides supplementary materials supporting the main ...

    work page 2024

  68. [68]

    the tablebetweenthe sofaandthe bookshelf,next tothe lamp

    Multi-reference.For a target object, we re- trieve 2–3 relations that each directly involve the target, and combine them into a single expression via conjunction templates. SceneVerse’s multi- object relations (e.g., aligned, in the middle of) naturally fall into this category. Example: “the tablebetweenthe sofaandthe bookshelf,next tothe lamp.”

    work page

  69. [69]

    the chairnext tothe deskthat supports the monitornearthe window

    Multi-hop chain.We identify a chain of three consecutive relations within the same room, where the target object appears at any position along the chain. The chain provides progressively more spatial context for disambiguation. Exam- ple: “the chairnext tothe deskthat supports the monitornearthe window.”

    work page

  70. [70]

    the lampabovethe nightstand, where the nightstandis next tothe bedthat facesthe window

    Chain with branch.We first identify a rela- tion chain of two or more hops (as in form 2), then connect the target object to any node on the chain via an additional relation. The chain serves as room-level context, while the branch relation localizes the target. Example: “the lampabovethe nightstand, where the nightstandis next tothe bedthat facesthe wind...

    work page

  71. [71]

    Q:” and ending with “?

    Output one line only, starting with “Q:” and ending with “?”

    work page

  72. [72]

    what” or “which object

    Replace the target class with “what” or “which object”

    work page

  73. [73]

    Keep all spatial constraints from the description

    work page

  74. [74]

    In {room}, how many {class} are there?

    If you cannot comply, output: INVALID Example 1: Input: the chair is behind the lamp and in front of the cabinet Class: chair Output: Q: What is behind the lamp and in front of the cabinet? Example 2: Input: the lamp next to the desk that supports the monitor near the window Class: lamp Output: Q: What is next to the desk that supports the monitor near th...

    work page