Pith. sign in

REVIEW 3 major objections 5 minor 61 references

IGGT4D turns 4D scene reconstruction into a causal streaming task, jointly predicting camera motion, geometry, and persistent object identities from video, and claims to beat streaming baselines while keeping memory constant.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:01 UTC pith:V3T4YEYY

load-bearing objection A serious streaming instance-geometry model with a real dataset, but the headline streaming-baseline win is confounded by DA3 initialization and DA3-generated pseudo-labels. the 3 major comments →

arxiv 2607.19228 v1 pith:V3T4YEYY submitted 2026-07-21 cs.CV

IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer

Zhengyu Zou , Hao Li , Kuixuan Jiao , Liu Liu , Tingyang Xiao , Xiaolin Zhou , Fangzhou Hong , Zhizhong Su
show 2 more authors
Dingwen Zhang Ziwei Liu
This is my paper
classification cs.CV
keywords streaming 4D reconstructioninstance segmentationcausal transformergeometry-instance learningdynamic scenesinstance trackingopen-vocabulary segmentationvideo stream understanding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a single feed-forward model can understand long dynamic video streams in an online, frame-by-frame way without sacrificing object identity or geometric consistency. Instead of reprocessing the whole sequence, IGGT4D maintains a causal spatial-temporal state and updates it incrementally, predicting camera pose, depth, ray maps, and per-pixel instance features for each new frame. The authors claim this streaming approach outperforms existing streaming baselines on reconstruction, pose estimation, instance tracking, and open-vocabulary segmentation, while being the only method that scales to long sequences without running out of memory. A reader should care because embodied agents (robots, AR, autonomous vehicles) need exactly this kind of continuous, identity-aware understanding, not offline batch reconstruction.

Core claim

The paper's central claim is that geometry and instance understanding can be unified in a causal streaming architecture and that this unification is what enables persistent object identity in long dynamic videos. IGGT4D encodes each frame, applies causal attention so it only sees current and past frames, decodes depth, rays, and instance features through a coupled Tri-DPT head, and then merges local instance clusters into a global codebook that is updated in constant time. The authors demonstrate that this design maintains temporally consistent object IDs across occlusions and reappearance without explicit motion modeling, and that the memory footprint of the clustering step stays at about 0

What carries the argument

The central mechanism is a causal geometry-instance Transformer with three coupled components: (1) causal intra-view and cross-view attention with camera and cross-view KV caches, so each frame reuses historical context without recomputation; (2) a Tri-DPT head that jointly decodes depth, ray maps, and instance features, injecting geometric features into the instance branch via geometry-aware attention; and (3) a streaming clustering strategy that maintains a lightweight global instance codebook (feature center plus accumulated pixel count) and updates it in constant time per frame via area-weighted fusion. First-frame geometric normalization is used during training to remove the scale ambig

Load-bearing premise

The load-bearing premise is that the depth, pose, and instance labels used for training are trustworthy, especially the DA3-generated geometry pseudo-labels; if those are biased in dynamic, egocentric, or small-object scenes, the model's gains may come from inheriting the teacher's bias rather than from its streaming design.

What would settle it

Train IGGT4D on a purely synthetic dataset with ground-truth depth, pose, and instance masks (e.g., Kubric or RoboTwin), and compare it against a streaming baseline also trained on the same ground-truth labels. If the causal streaming model no longer outperforms the baseline, the claim that causal streaming geometry-instance modeling drives the improvement is falsified. Alternatively, evaluate on sequences where objects fully leave the view for many frames and then reappear; if identity tracking fails at reappearance, the claim of persistent identity under reappearance is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, long-sequence 4D reconstruction becomes practical in online settings: a single GPU can process arbitrarily long video streams without unbounded memory growth, which current streaming baselines cannot do.
  • Instance identities can be maintained across occlusions and viewpoint changes without explicit tracking or optical flow, because the instance features are grounded in reconstructed geometry and clustered incrementally.
  • A unified geometry-instance representation enables downstream tasks like open-vocabulary segmentation and 4D QA grounding to work from streaming input, rather than requiring offline 3D reconstruction first.
  • The proposed dataset with geometry-consistent instance masks provides a way to supervise such models at scale, potentially shifting the field from semantic-lifting pipelines to end-to-end streaming prediction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The performance gap over non-DA3 streaming baselines may partly measure teacher transfer rather than the streaming architecture itself; a cleaner test would compare against a baseline fine-tuned on the same DA3 geometry labels without the causal streaming design.
  • Inference: The evaluation protocol reorders sequences so every frame overlaps a previous one, removing the hardest streaming failure mode (total re-observation after disappearance). A natural extension is to evaluate on truly unconstrained streams where objects leave and re-enter the field of view with no overlap, testing whether the incremental codebook truly recovers identities.
  • Inference: Because the clustering memory is constant in sequence length and time scales linearly, the approach is directly applicable to real-time robotic and embodied perception, but its robustness to rapidly changing lighting, motion blur, and long-term drift remains untested on real egocentric data.
  • Inference: The instance features are learned from pseudo-labels generated by a teacher model; if those pseudo-labels contain systematic bias in dynamic or small-object scenes, the model will inherit that bias. A testable extension is to pretrain with fully ground-truth synthetic supervision and measure whether the streaming advantage persists.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes IGGT4D, a streaming Transformer that jointly predicts camera pose, depth/ray maps, and per-pixel instance features from video streams under causal attention, plus an online clustering stage that maintains persistent instance IDs. It also introduces InsScene4D-147K, a large dataset of real and synthetic, static and dynamic sequences with geometry-guided instance annotations. Experiments cover pose estimation, 3D reconstruction (F1 and Chamfer), instance spatial tracking, open-vocabulary segmentation, and an ablation, and report consistent improvements over streaming baselines while avoiding the OOM failure of the offline IGGT model.

Significance. If the claims hold, the paper makes a useful step: it combines the streaming-reconstruction paradigm with object-level instance persistence in a single feed-forward model, and provides a large training corpus with instance masks. The architecture is sensible, the causal-masking and KV-cache design is a natural extension of existing streaming geometry models, and the streaming clustering with a constant-size codebook is a practical improvement over HDBSCAN. The evaluation spans several external benchmarks and multiple tasks, which is broader than typical. However, the central empirical claim—'outperforms existing streaming baselines'—is weakened by a teacher-chain confound: the model is initialized from DA3-Giant and trained on DA3-generated pseudo-labels, while the compared streaming baselines are neither DA3-initialized nor trained on this pseudo-label distribution. In addition, the instance-tracking evaluation relies on manually curated masks and reordered sequences that remove the hardest streaming failure mode. These issues do not invalidate the architecture but make the headline comparison ambiguous.

major comments (3)
  1. [Sec. 5.1, Table 1; Sec. 4; A.5] The main comparison against streaming baselines is confounded by the DA3 teacher chain. A.5 states that IGGT4D is initialized from DA3-Giant, and Sec. 4 constructs the static and dynamic geometry labels using DA3 depth and pose predictions (TSDF-fused for static scenes). The streaming baselines in Table 1 (CUT3R, StreamVGGT, Wint3R, Stream3R, LingBot-Map) are neither DA3-initialized nor trained on this pseudo-label distribution. The reported margins (e.g., Avg AUC@3 0.446 vs 0.306, Avg F1 0.668 vs 0.523) may therefore reflect inherited DA3 priors and label alignment rather than the proposed causal streaming and instance clustering. A concrete test is to initialize or fine-tune a strong streaming baseline on InsScene4D-147K under the same conditions, or to ablate IGGT4D's initialization from DA3-Giant. Without such a control, the headline 'outperforms existing streaming baselines' is not
  2. [A.6; Table 2] The instance-tracking evaluation is partly based on manual curation and a protocol that removes the central streaming challenge. A.6 reports that HOI4D masks are 'manually annotated by us', that Waymo and ScanNet++ have 'manually remove[d] several small objects', and that PointOdyssey masks are 'manually select[ed]' and merged. Moreover, all geometry evaluation sequences are reordered so that 'every frame shares visual overlap with at least one preceding frame', which eliminates the total re-observation scenario (an object disappears and later reappears) that a streaming instance-consistency model should handle. The paper should evaluate on the original sequence order and on sequences with explicit reappearance, and should release the manually curated masks/protocol so that the reported T-mIoU/T-SR numbers are independently verifiable.
  3. [Sec. 3.2, Sec. 3.3, Table 5] The constant-memory claim is only demonstrated for the clustering stage, not for the full model. Table 5 and the surrounding text state that 'our method maintains a constant memory footprint (~0.7 GB)', but this measurement is for the streaming clustering algorithm only. The model itself maintains 'camera and cross-view KV caches' (Sec. 3.2) whose size is not described or bounded; if all historical tokens are cached, memory grows linearly with sequence length and the long-sequence scalability claim is weaker than implied. Please specify whether these caches are compressed or bounded, and report full-model GPU memory as a function of sequence length alongside the clustering-only numbers.
minor comments (5)
  1. [General] No error bars or variance information is reported for any table. Given the manual curation and reordering, at least a few repeated runs or per-sequence standard deviations would help the reader assess the stability of the reported margins.
  2. [Eq. (4)] The contrastive loss notation is dense: it is unclear whether the sums over v and k are nested or run over all valid pairs, and the definition of prototypes mu_v_k should be stated explicitly in the main text rather than only in words.
  3. [A.6] The reordering of evaluation sequences is described as ensuring visual overlap, but the amount of overlap and the original ordering are not quantified. Please report statistics (e.g., average number of overlapping frames) and, ideally, results on the original ordering.
  4. [References] Several cited works are given as 2026 preprints (e.g., LingBot-Map [60], LangSurf [35], Iris-SLAM [7], Spa3R [29]) without arXiv identifiers or availability notes. Please add stable identifiers or state if they are under review.
  5. [Figures 3 and 11] Figures 3 and 11 appear to show very similar qualitative visualization of instance features/masks. Consider merging or differentiating them to avoid redundancy.

Circularity Check

0 steps flagged

No circular derivation: the reported geometry and instance predictions are tested against external ground truth, not against the model's own outputs or fitted inputs.

full rationale

I find no circular step that reduces a claimed prediction to the paper's inputs by construction. The geometry benchmark in Table 1 is evaluated on hold-out external datasets (HiRoom, ETH3D, 7Scenes, ScanNet++), and A.6 states explicitly: 'Our evaluation datasets (HiRoom, ETH3D, 7Scenes, and ScanNet++) and protocol follow the DA3 benchmark.' Ground-truth depth and poses for these datasets are not DA3 pseudo-labels, so reporting pose AUC and reconstruction F1 against them is an independent test. The fact that IGGT4D is initialized from DA3-Giant (A.5) and that InsScene4D-147K static and dynamic geometry labels are generated by DA3 (Section 4: 'We first estimate multi-view consistent depth with DA3... A static 3D mesh is then reconstructed via TSDF fusion') is a legitimate concern about teacher transfer and label bias, but it is not circularity: the model's outputs are not being compared to DA3's predictions, and no equation in the paper makes the benchmark metric equal to a fitted parameter or to the training-label generator. The instance tracking evaluation on HOI4D uses author-supplied masks (A.6: 'HOI4D contains about 40 frames per scene and is manually annotated by us'), which is an annotation-independence concern, but again the reported T-mIoU/T-SR are computed against those masks, not derived from the model's own masks by construction. The open-vocabulary evaluation aggregates external 2D VLM features with the model's predicted masks; this tests mask quality in a downstream task rather than assuming the conclusion. Self-citations to IGGT [37] appear as prior context and as the source of the offline HDBSCAN comparison, not as a load-bearing uniqueness theorem or as the sole justification for the streaming contribution. No enumerated circularity pattern is exhibited, so a non-finding with score 0 is appropriate; the DA3-initialization and pseudo-label concerns belong under correctness risk or benchmark fairness, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The ledger is dominated by hand-set hyperparameters (loss weights, margins, thresholds) whose values are not reported, and by the domain assumption that DA3 pseudo-labels are adequate ground truth for 4D instance supervision. No new physical or conceptual entities are postulated; the instance codebook and 8-dim instance features are learned representations, not entities with external falsifiable handles.

free parameters (6)
  • Contrastive margins delta_in_pull, delta_in_push, delta_cr_pull, delta_cr_push = not reported
    Eq. (4): margins in the multi-view contrastive instance loss; hand-set, values omitted in the paper; directly shape instance-feature compactness and identity switching.
  • Loss weights lambda_ins, lambda_conf, lambda_grad; scale alpha = not reported
    Eqs. (3)-(4): depth confidence/regularization and instance-loss weights; hyperparameters chosen by hand, values not stated.
  • Clustering thresholds tau_s, tau_l = not reported
    Sec. 3.3: cosine-similarity core-expansion thresholds for intra-frame clustering; hand-chosen; control mask granularity and codebook purity.
  • Disappearance threshold N=5 frames = 5
    Sec. 4: consecutive-frame small-projection criterion to stop stale ID propagation in the dataset pipeline; hand-set.
  • Instance feature dimensionality = 8
    Sec. 3.1: S_t in R^{H x W x 8}; architecture choice that bounds clustering discriminability.
  • GT-pose injection probability = 20%
    A.5: probability of injecting ground-truth camera poses during training; a training hyperparameter that affects how strongly the model learns pose prediction.
axioms (5)
  • domain assumption DA3's offline geometry predictions are accurate enough to serve as ground-truth depth/pose/mesh for 4D supervision
    Sec. 4: static splits are labeled by DA3 predictions + TSDF fusion; the trained model is initialized from DA3 (A.5). Errors in DA3 bound the ceiling of the supervision.
  • domain assumption Causal attention with cached history preserves the geometry-instance consistency of full attention
    Sec. 3.2: causal masks restrict frames to current/past views; the scale at which KV-cache reuse degrades long-range geometric consistency is not measured beyond ~100 frames.
  • domain assumption Geometry-aware attention in the Tri-DPT head grounds instance embeddings meaningfully
    A.7: depth/ray features are injected into the instance branch; the ablation (Tab. 4) shows a modest effect (mIoU 0.3098 vs 0.2970), so the mechanism's benefit is limited.
  • domain assumption Matching SAM2 masks to projected-mesh IDs yields temporally consistent instance labels
    Sec. 4: ID inheritance via IoU matching; failure modes are acknowledged in A.1 (over-segmentation, identity switching) but the cleanup assumes the statistics of the chosen sources.
  • domain assumption 2D VLM features aggregated over instance masks constitute a valid open-vocabulary segmentation metric
    Sec. 3.4: language features from CLIP/OpenSeg averaged per instance; the resulting mIoU is largely a function of mask quality and VLM feature quality.

pith-pipeline@v1.3.0-alltime-deepseek · 17213 in / 17396 out tokens · 180077 ms · 2026-08-01T13:01:41.887755+00:00 · methodology

0 comments
read the original abstract

Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.

Figures

Figures reproduced from arXiv: 2607.19228 by Dingwen Zhang, Fangzhou Hong, Hao Li, Kuixuan Jiao, Liu Liu, Tingyang Xiao, Xiaolin Zhou, Zhengyu Zou, Zhizhong Su, Ziwei Liu.

Figure 1
Figure 1. Figure 1: Online Geometry-Instance Prediction with IGGT4D. IGGT4D processes a dynamic video stream frame by frame to incrementally build a unified 4D representation that jointly captures camera motion, 3D geometry, and temporally consistent instance features. This representation supports diverse downstream applications. We also construct InsScene4D-147K, a large-scale dataset with geometry-guided instance masks, to … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of IGGT4D. Given a dynamic video sequence and optional camera poses, IGGT4D extracts a spatial-temporally consistent representation. A Tri-DPT head incrementally predicts geometry and instance features, and a streaming clustering algorithm derives instance masks for online 4D scene understanding. sequentially streamed in, frame-wise predictions are integrated into a spatial-temporal consistent sce… view at source ↗
Figure 3
Figure 3. Figure 3: Instance Feature and Mask Visualization. We visualize the 3D-consistent instance feature PCA results alongside the corresponding instance masks generated by our streaming clustering. 3.4 4D Scene Understanding The 4D-consistent masks Mt,k and temporally linked instance features ck provide a reusable object￾level representation for downstream scene understanding tasks. First, the persistent instance corre￾s… view at source ↗
Figure 4
Figure 4. Figure 4: InsScene4D-147K data curation pipeline. Real/synthetic and static/dynamic sources are processed through 3D reconstruction, projection-based ID inheritance, and segmentation-based global ID refinement. The right panel summarizes the dataset splits. 4 InsScene4D-147K Dataset We construct InsScene4D-147K, a large-scale dataset for online 4D scene understanding, comprising 147K curated video sequences. The dat… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of 3D reconstruction on the 7Scenes and ETH3D datasets. 3D Reconstruction. Tab. 1(b) reports 3D reconstruction F1-scores with predicted poses (w/o p.) and ground-truth poses (w/ p.). MapAnything, Pi3X, DA3, and our method accept ground-truth poses during inference, while the other methods use them only for evaluation-time fusion. In both settings, our method outperforms all streaming… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative visualization results of Instance Spatial Tracking. LingBot-Map and Stream3R using TSDF-fused point clouds, with invalid ground-truth depth regions masked out. 5.2 Evaluation of Instance Spatial Tracking For instance spatial tracking and open-vocabulary semantic segmentation, all evaluation sequences are sampled from held-out splits that are disjoint from the training data at the scene or seque… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative visualization results of Open-Vocabulary Semantic Segmentation. 5.4 Ablation Study and Streaming Clustering Efficiency We conduct the ablation study on ScanNet++ using a lightweight model, as shown in Tab. 4. Removing Geometry-Aware Attention (Geo-Attn) degrades instance and semantic performance, demonstrating that instance features benefit from geometric priors. Furthermore, without First-Fram… view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative evaluation of the automated geometry-guided annotation pipeline across three [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: 4D QA Scene Grounding. Comparison of spatial-temporal reasoning capabilities using LMMs. Given only RGB frames (top), the LMM fails to track the moving bottle as it blends into the background, erroneously concluding it has left the view. In contrast, by incorporating our 4D-consistent instance features (bottom), the LMM successfully tracks the object across frames, accurately grounds its location, and corr… view at source ↗
Figure 10
Figure 10. Figure 10: First-frame geometric normalization (FF-Norm). We compare streaming predictions without versus with first-frame geometric normalization. A.4 Visualization of Streaming Clustering Masks We further visualize instance-aware representations and the instance masks produced by our efficient streaming clustering [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Streaming clustering masks. PCA visualization of instance features alongside masks obtained by our streaming clustering, demonstrating 4D-consistent instance segmentation in dynamic sequences. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Tri-DPT head. The head jointly decodes depth, ray, and instance features, with geometry￾aware attention injecting depth and ray priors into the instance branch. We design a Tri-DPT head to jointly decode geometry and instance representations from streaming multi-scale tokens {F (l) t } 4 l=1. Rather than treating instance prediction as an independent add-on, Tri-DPT uses a unified DPT-style decoder with t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 17 linked inside Pith

  1. [1]

    Structure-from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016

    2016

  2. [2]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. InEuropean conference on computer vision, pages 501–518. Springer, 2016

    2016

  3. [3]

    Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam

    Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE transactions on robotics, 37(6):1874–1890, 2021

    2021

  4. [4]

    Geoflow-slam: A robust tightly-coupled rgbd-inertial and legged odometry fusion slam for dynamic legged robotics, 2025

    Tingyang Xiao, Xiaolin Zhou, Liu Liu, Wei Sui, Wei Feng, Jiaxiong Qiu, Xinjie Wang, and Zhizhong Su. Geoflow-slam: A robust tightly-coupled rgbd-inertial and legged odometry fusion slam for dynamic legged robotics, 2025

    2025

  5. [5]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.Advances in neural information processing systems, 34:16558–16569, 2021

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.Advances in neural information processing systems, 34:16558–16569, 2021

    2021

  6. [6]

    Nice-slam: Neural implicit scalable encoding for slam

    Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12786–12796, 2022

    2022

  7. [7]

    Iris-slam: Unified geo-instance representations for robust semantic localization and mapping, 2026

    Tingyang Xiao, Liu Liu, Wei Feng, Zhengyu Zou, Xiaolin Zhou, Wei Sui, Hao Li, Dingwen Zhang, and Zhizhong Su. Iris-slam: Unified geo-instance representations for robust semantic localization and mapping, 2026

    2026

  8. [8]

    Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation

    Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open-vocabulary 3d scene graphs for language-grounded robot navigation. InFirst Workshop on Vision-Language Models for Navigation and Manipulation at ICRA 2024, 2024

    2024

  9. [9]

    Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021–5028. IEEE, 2024

    2024

  10. [10]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025

    2025

  11. [11]

    Mapanything: Universal feed-forward metric 3d reconstruction.arXiv preprint arXiv:2509.13414, 2025

    Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fischer, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, et al. Mapanything: Universal feed-forward metric 3d reconstruction.arXiv preprint arXiv:2509.13414, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    π3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025

    Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiangmiao Pang, Chunhua Shen, and Tong He. π3: Permutation-equivariant visual geometry learning.arXiv preprint arXiv:2507.13347, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Haotong Lin, Sili Chen, Junhao Liew, Donny Y Chen, Zhenyu Li, Guang Shi, Jiashi Feng, and Bingyi Kang. Depth anything 3: Recovering the visual space from any views.arXiv preprint arXiv:2511.10647, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    3d reconstruction with spatial memory

    Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. In2025 Interna- tional Conference on 3D Vision (3DV), pages 78–89. IEEE, 2025

    2025

  15. [15]

    Continuous 3d perception model with persistent state

    Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 10510–10522, 2025

    2025

  16. [16]

    Stream3r: Scalable sequential 3d reconstruction with causal transformer.arXiv preprint arXiv:2508.10893, 2025

    Yushi Lan, Yihang Luo, Fangzhou Hong, Shangchen Zhou, Honghua Chen, Zhaoyang Lyu, Shuai Yang, Bo Dai, Chen Change Loy, and Xingang Pan. Stream3r: Scalable sequential 3d reconstruction with causal transformer.arXiv preprint arXiv:2508.10893, 2025. 10

    Pith/arXiv arXiv 2025

  17. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  18. [18]

    Language- driven semantic segmentation.arXiv preprint arXiv:2201.03546, 2022

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation.arXiv preprint arXiv:2201.03546, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Scaling open-vocabulary image segmen- tation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. InEuropean conference on computer vision, pages 540–557. Springer, 2022

    2022

  20. [20]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  21. [21]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. InProceedings of the IEEE/CVF international conference on computer vision, pages 19729–19739, 2023

    2023

  22. [22]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815–824, 2023

    2023

  23. [23]

    Openmask3d: Open-vocabulary 3d instance segmentation.arXiv preprint arXiv:2306.13631, 2023

    Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation.arXiv preprint arXiv:2306.13631, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Uni3r: Unified 3d reconstruction and semantic understanding via generalizable gaussian splatting from unposed multi-view images.arXiv preprint arXiv:2508.03643, 2025

    Xiangyu Sun, Haoyi Jiang, Liu Liu, Seungtae Nam, Gyeongjin Kang, Xinjie Wang, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang, et al. Uni3r: Unified 3d reconstruction and semantic understanding via generalizable gaussian splatting from unposed multi-view images.arXiv preprint arXiv:2508.03643, 2025

    arXiv 2025

  25. [25]

    Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields

    Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21676–21685, 2024

    2024

  26. [26]

    3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494, 2023

    2023

  27. [27]

    Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness.arXiv preprint arXiv:2409.18125, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction, 2025

    Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, and Rakesh Ranjan. Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction, 2025

    2025

  29. [29]

    Spa3r: Predictive spatial field modeling for 3d visual reasoning, 2026

    Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He, Wei Sui, Zhizhong Su, Wenyu Liu, and Xinggang Wang. Spa3r: Predictive spatial field modeling for 3d visual reasoning, 2026

    2026

  30. [30]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20697–20709, 2024

    2024

  31. [31]

    Grounding image matching in 3d with mast3r

    Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. InEuropean conference on computer vision, pages 71–91. Springer, 2024. 11

    2024

  32. [32]

    Omnivggt: Omni-modality driven visual geometry grounded transformer.arXiv preprint arXiv:2511.10560, 2025

    Haosong Peng, Hao Li, Yalun Dai, Yushi Lan, Yihang Luo, Tianyu Qi, Zhengshen Zhang, Yufeng Zhan, Junfei Zhang, Wenchao Xu, and Ziwei Liu. Omnivggt: Omni-modality driven visual geometry grounded transformer.arXiv preprint arXiv:2511.10560, 2025

    arXiv 2025

  33. [33]

    Must3r: Multi-view network for stereo 3d reconstruction

    Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy. Must3r: Multi-view network for stereo 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1050–1060, 2025

    2025

  34. [34]

    Langsplat: 3d language gaussian splatting

    Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024

    2024

  35. [35]

    Langsurf: Language-embedded surface gaussians for 3d scene understanding, 2026

    Hao Li, Minghan Qin, Zhengyu Zou, Diqi He, Xinhao Ji, Bohan Li, Bingquan Dai, Dingewn Zhang, and Junwei Han. Langsurf: Language-embedded surface gaussians for 3d scene understanding, 2026

    2026

  36. [36]

    Large spatial model: End-to-end unposed images to semantic 3d.Advances in neural information processing systems, 37:40212–40229, 2024

    Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, et al. Large spatial model: End-to-end unposed images to semantic 3d.Advances in neural information processing systems, 37:40212–40229, 2024

    2024

  37. [37]

    Iggt: Instance-grounded geometry transformer for semantic 3d reconstruction.arXiv preprint arXiv:2510.22706, 2025

    Hao Li, Zhengyu Zou, Fangfu Liu, Xuanyang Zhang, Fangzhou Hong, Yukang Cao, Yushi Lan, Manyuan Zhang, Gang Yu, Dingwen Zhang, et al. Iggt: Instance-grounded geometry transformer for semantic 3d reconstruction.arXiv preprint arXiv:2510.22706, 2025

    arXiv 2025

  38. [38]

    hdbscan: Hierarchical density based clustering

    Leland McInnes, John Healy, Steve Astels, et al. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205, 2017

    2017

  39. [39]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Stereo magnifi- cation: Learning view synthesis using multiplane images.arXiv preprint arXiv:1805.09817, 2018

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images.arXiv preprint arXiv:1805.09817, 2018

    Pith/arXiv arXiv 2018

  42. [42]

    Scannet++: A high- fidelity dataset of 3d indoor scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023

    2023

  43. [43]

    Aria digital twin: A new benchmark dataset for egocentric 3d machine perception

    Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20133–20143, 2023

    2023

  44. [44]

    John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison. Scenenet rgb-d: Can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In Proceedings of the IEEE international conference on computer vision, pages 2678–2687, 2017

    2017

  45. [45]

    Infinigen indoors: Photorealistic indoor scenes using procedural generation

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, et al. Infinigen indoors: Photorealistic indoor scenes using procedural generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21783–21794, 2024

    2024

  46. [46]

    Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. InProceedings of the IEEE/CVF international conference on computer vision, pages 10912–10922, 2021. 12

    2021

  47. [47]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013–21022, 2022

    2022

  48. [48]

    Waymo open dataset: Panoramic video panoptic segmentation

    Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Liang-Chieh Chen, and Henrik Kretzschmar. Waymo open dataset: Panoramic video panoptic segmentation. In European Conference on Computer Vision, pages 53–72. Springer, 2022

    2022

  49. [49]

    Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.arXiv preprint arXiv:2506.18088, 2025

    Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation.arXiv preprint arXiv:2506.18088, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Kubric: A scalable dataset generator

    Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3749–3761, 2022

    2022

  51. [51]

    Dynamicstereo: Consistent dynamic depth from stereo videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dynamicstereo: Consistent dynamic depth from stereo videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13229–13239, 2023

    2023

  52. [52]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19855–19865, 2023

    2023

  53. [53]

    Virtual kitti 2.arXiv preprint arXiv:2001.10773, 2020

    Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2.arXiv preprint arXiv:2001.10773, 2020

    Pith/arXiv arXiv 2001

  54. [54]

    A volumetric method for building complex models from range images

    Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. InProceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303–312, 1996

    1996

  55. [55]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    A multi-view stereo benchmark with high-resolution images and multi-camera videos

    Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3260–3269, 2017

    2017

  57. [57]

    Scene coordinate regression forests for camera relocalization in rgb-d images

    Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2930–2937, 2013

    2013

  58. [58]

    Streaming 4d visual geometry transformer.arXiv preprint arXiv:2507.11539, 2025

    Dong Zhuo, Wenzhao Zheng, Jiahe Guo, Yuqi Wu, Jie Zhou, and Jiwen Lu. Streaming 4d visual geometry transformer.arXiv preprint arXiv:2507.11539, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Wint3r: Window-based streaming reconstruction with camera token pool.arXiv preprint arXiv:2509.05296, 2025

    Zizun Li, Jianjun Zhou, Yifan Wang, Haoyu Guo, Wenzheng Chang, Yang Zhou, Haoyi Zhu, Junyi Chen, Chunhua Shen, and Tong He. Wint3r: Window-based streaming reconstruction with camera token pool.arXiv preprint arXiv:2509.05296, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    Geometric context transformer for streaming 3d reconstruction.arXiv preprint arXiv:2604.14141, 2026

    Lin-Zhuo Chen, Jian Gao, Yihang Chen, Ka Leong Cheng, Yipengjing Sun, Liangxiao Hu, Nan Xue, Xing Zhu, Yujun Shen, Yao Yao, et al. Geometric context transformer for streaming 3d reconstruction.arXiv preprint arXiv:2604.14141, 2026

    Pith/arXiv arXiv 2026

  61. [61]

    Spatialtrackerv2: 3d point tracking made easy

    Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy. arXiv preprint arXiv:2507.12462, 2025. 13 A Technical Appendices and Supplementary Material A.1 Visualization of the Automated Geometry-Guided Annotation Pipeline To demonstrate the ...

    Pith/arXiv arXiv 2025