Pith. sign in

REVIEW 4 major objections 8 minor 17 cited by

Continuous 3D Perception Model with Persistent State

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CUT3R maintains a persistent latent scene state that updates with every incoming image and reads out online metric-scale pointmaps, camera poses, and even geometry for unseen virtual views.

desk verdict A credible and genuinely novel online 3D reconstruction system, but the two headline capabilities—online refinement and unseen-view inference—are less proven than the abstract implies. read the letter →

arxiv 2501.12387 v1 pith:HFDRWFJ4 submitted 2025-01-21 cs.CV

Qianqian Wang , Yifei Zhang , Aleksander Holynski , Alexei A. Efros , Angjoo Kanazawa This is my paper · ORCID
classification cs.CV
keywords persistentstateonline3Dreconstructionmetricpointmapsrecurrenttransformerdynamicscenesraymapqueryscenepriorscameraposeestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a single stateful recurrent model can act as an online 3D perception system: as a stream of images arrives, the model updates a persistent internal state and, for each frame, outputs a confidence-weighted, metric-scale pointmap (a 3D point for every pixel) plus camera pose in one shared world coordinate system. No test-time optimization or global alignment is needed. The same state can also be queried with a virtual camera, encoded as a raymap, to predict the geometry of scene regions never observed. The authors report competitive or state-of-the-art results on depth, pose, and reconstruction benchmarks, including dynamic scenes, and claim this is the first method to infer unseen structures at metric scale for general scenes. If true, this unifies online dense reconstruction, visual odometry, and novel-view geometry in a single feedforward model.

What carries the argument

The load-bearing mechanism is the persistent state: 768 learnable tokens, each of dimension 768, initialized once per scene before any image is seen. A ViT image encoder converts the current frame into tokens, and two interconnected transformer decoders jointly process image tokens and state tokens, performing state-update and state-readout in the same pass. A learnable pose token attached to the image carries ego-motion information, and the decoder heads output confidence-weighted pointmaps in both the current camera's frame and the world frame, together with the camera-to-world transform. For unobserved views, a virtual camera is represented as a 6-channel raymap (per-pixel ray origin and direction), encoded by a lightweight transformer, and passed through the same decoder without updating the state; a color head renders the queried view. The state thus acts as an implicit, continuously refined scene memory that supports both reconstruction and completion.

What would settle it

Run CUT3R online over a long closed-loop video of a static scene and measure the alignment error between the latest frame's pointmap and the reconstruction accumulated from the first frames; if the error grows steadily with sequence length instead of staying bounded, the persistent-consistency claim fails. Separately, query the state with a virtual raymap far outside the observed viewing frustum and compare the predicted pointmap with a ground-truth scan under metric alignment; failure to hold scale and position there would refute the claim of metric-scale inference for unseen structures.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a compressed latent state can serve as the scene memory for continuous 3D perception. CUT3R represents everything learned about a scene as a fixed set of state tokens; each new image both writes new information into those tokens and reads context back out, and the readout is decoded into dense pointmaps in a metric world frame defined by the first image. Because alignment is implicit in the state rather than performed by a post-processing stage, the model stays online and handles varying numbers of images, unordered photo collections, and dynamic scenes with moving objects. The further discovery is that the state can be probed by a virtual raymap to produce pointmaps and colors for unobserved views, which the paper validates on held-out datasets with metric ground truth. The authors state that this makes their method the first to enable inference of unseen structures in metric scale for general scenes without camera intrinsics or poses for the input.

Load-bearing premise

The paper's load-bearing assumption is that a fixed set of 768 latent tokens, updated causally and without global alignment, can hold enough scene information to keep every new metric-scale pointmap in one consistent world frame indefinitely; if the state silently discards long-range context, reconstruction drifts—a risk the authors themselves acknowledge for very long sequences.

Editorial extensions

If this is right

  • A video or photo collection can be reconstructed online: each new frame immediately updates the scene model, so dense reconstruction, camera pose, and depth appear without bundle adjustment or offline alignment.
  • The metric-scale pointmaps in a common world frame can be accumulated directly into a coherent 3D model, making the output usable for robotics, AR, and navigation without post-processing.
  • Dynamic scenes are handled by the same machinery because the implicit state does not assume a static world, which the paper demonstrates on datasets with moving objects.
  • Few observed images can be enough: raymap queries produce metric-scale pointmaps for unobserved regions, so the model can complete a scene from sparse input.
  • Zero-shot generalization follows from training on 32 datasets with varied and partial annotations, and the paper shows competitive results on held-out depth and pose benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the state's fixed size sets a capacity ceiling: if the state is a genuine compressed scene memory, very long sequences should eventually lose earlier information, and ablating the number of state tokens should measurably change long-range consistency—an experiment the paper does not report.
  • We infer an extension the authors leave implicit: an agent could propose a virtual camera whose readout maximises expected information gain and use the raymap query to decide where to look next, turning the model from a passive reconstructor into a planner.
  • We infer that the 20% raymap masking during training suggests the model may be more reliable at interpolating between observed views than at extreme extrapolation, and that performance will degrade as query cameras move outside the distribution of training viewpoints.
  • We infer that a generative counterpart to the deterministic raymap readout would be needed to keep hallucinated views sharp, addressing the blurriness the paper acknowledges for far extrapolation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper presents CUT3R, a recurrent transformer for online 3D perception from streams of RGB images. The model maintains a fixed-size latent state (768 tokens) that is simultaneously updated by and read from at each time step; from the enriched image tokens it predicts metric-scale pointmaps in the initial frame's coordinate system, camera poses, and, when a virtual raymap query is provided, pointmaps and colors for unobserved views. The method is trained with a multi-stage curriculum on 32 datasets with varied 3D annotation types. Experiments cover monocular and video depth (Tabs. 1–2), camera pose (Tab. 3), sparse-view reconstruction (Tab. 4), an online-versus-revisiting state analysis (Tab. 5), and qualitative demonstrations of unseen-structure inference (Fig. 6). The paper reports competitive or state-of-the-art results on the depth and reconstruction benchmarks and is the only online method in the comparisons that handles dynamic scenes.

Significance. If the central claims hold, CUT3R is a substantial step: a single feedforward model that performs online metric depth, pose, dynamic-scene reconstruction, and unobserved-view completion in a common coordinate frame without test-time optimization or global alignment. The benchmark results (Tabs. 1–4) are credible on their own terms, and the paper has genuine strengths: the state-update/readout design is clean and flexible; the raymap-query formulation for unseen-view completion is elegant; training with partial annotations over 32 datasets is a substantial engineering contribution; and the limitation statements are honest and specific. The method outperforms the concurrent online baseline Spann3R by a large margin on reconstruction, matches optimization-based global-alignment methods at much higher throughput, and is the only online counterpart in the comparisons handling dynamic scenes. The weaknesses are evidentiary: long-horizon common-frame consistency, the causal benefit of state updates, and the quantitative accuracy of unseen-region inference are not established by the current experiments.

major comments (4)
  1. [Sec. 3.1, Sec. 4.3, Limitations] The central claim that predicted pointmaps 'reside within a common coordinate system' over an incoming stream rests entirely on the fixed 768-token state, since no global alignment is performed. The reconstruction evaluation (Tab. 4) uses only 2–5 frames per scene, the pose and depth evaluations (Tabs. 2–3) use short sequences with per-sequence Sim(3) or scale alignment that can mask systematic drift, and the Limitations section concedes that the method 'may eventually drift over very long sequences due to the absence of global alignment.' As written, the common-coordinate-system claim is therefore only established for short horizons, which weakens the stated advantage over pairwise methods with global alignment. I recommend adding an experiment that measures error growth as a function of sequence length (for example, ATE and per-frame reconstruction error on streams of 50–100+ frames, with and without periodic global alignment), or alternatively explicitly qualifying the abstract and Sec. 1 claims to a bounded horizon supported by the evidence.
  2. [Sec. 4.4, Tab. 5] The 'revisiting' ablation freezes the final state, which has seen the entire sequence including frames after the one currently being predicted, and then re-predicts all frames (Fig. 5). The improvement of 'Ours Revisit' over 'Ours' therefore demonstrates that a state with full global context helps; it does not demonstrate that causal incremental updates refine predictions, which is the claim made in the text ('This verifies that the state representation effectively updates with additional observations'). A causal test is needed, for example per-frame error as a function of time step t, or an ablation that resets the state to s0 at fixed intervals, to support the online-refinement interpretation.
  3. [Sec. 4.4, Sec. 3.2, Fig. 6] The claim in Sec. 4.4 that the method 'is the first to enable the inference of unseen structures in metric scale for general scenes' is supported only by four qualitative examples (Fig. 6). No quantitative metric is reported for the virtual-view pointmaps (e.g., depth Abs Rel or chamfer distance at the query viewpoint), even though the MapFree and ARKitScenes validation sets used here provide metric camera poses and depth for the query views, and no baseline is compared for this capability. Because the raymap is constructed from ground-truth camera poses, the evaluation should also report how prediction error scales with the viewpoint baseline between input and query. Without such measurements, the strong 'first to enable' novelty claim is not yet supported.
  4. [Appendix A, Sec. 4.4] The training depth labels for MapFree and DL3DV are MVS reconstructions computed from the provided camera parameters (Appendix A), and the unseen-region demonstration is performed on held-out scenes from MapFree, a dataset in the training pool (Tab. 6). This creates a modest circularity risk: systematic MVS artifacts in the training labels could be learned and then re-emerge as 'new structure' in the qualitative results. A quantitative evaluation on data with independent ground truth (e.g., ScanNet++ or a renderable synthetic scene), as recommended above, would rule out this explanation.
minor comments (8)
  1. [Tab. 1] The caption contains a duplicated word: 'leading on Bonn and and NYU-v2.'
  2. [Tab. 1] In the MonST3R row, two numbers are run together: '0.35854.8' should read '0.358 54.8'.
  3. [Sec. 4.4, Fig. 6] The sentence 'our method generates new structures beyond what what is observed in the input' contains a duplicated 'what'.
  4. [Tab. 4] The entry 'MonST3R-GA [51]' cites the wrong reference; MonST3R is [125] and MASt3R is [51].
  5. [Sec. 4.3, Tab. 4] Please state whether the Acc/Comp/NC metrics are computed in metric scale or after a per-scene rigid or similarity alignment, since this bears directly on the 'metric-scale' claim.
  6. [Sec. 4.2, Tab. 3] ScanNet appears in the training dataset list (Tab. 6); please state whether the pose evaluation on ScanNet overlaps with training data and, if so, report the training-excluded subsets separately.
  7. [Sec. 3.1, Eqs. (3)–(5)] The abstract and introduction claim the model estimates camera intrinsics and extrinsics, but Sec. 3.1 only describes pose prediction; please clarify how camera intrinsics are obtained from the predicted pointmaps.
  8. [Tabs. 2 and 4] The FPS comparison mixes resolutions (512×144 for the authors' method versus 224×224 for Spann3R); please add a footnote stating the resolution policy and whether the raymap query path is included in the reported FPS.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CUT3R's online pointmap and unseen-view predictions are trained with external supervision and tested on held-out data; none of the outputs is defined in terms of its own target.

full rationale

The paper makes empirical claims rather than deriving predictions from a closed-form model. The recurrent state in Sec. 3.1 is updated and read by transformer decoders (Eq. 2), and the pointmap, pose, and color heads (Eqs. 3-5, Sec. 3.2) are trained with confidence-aware regression, pose, and RGB losses (Sec. 3.3) against external ground truth from 32 datasets, including metric MVS/RGB-D annotations. The world frame is defined as the coordinate frame of the first image, but the consistency of pointmaps in that frame is a learned property measured on held-out benchmarks (Tabs. 1-4), not an identity imposed by the equations. The 'revisiting' ablation in Sec. 4.4 is an explicit test of whether additional context improves predictions, and the raymap query evaluation in Sec. 4.4 feeds the model only the query camera parameters, not the ground-truth image, so the predicted unseen-region pointmap is not read off from its target. The raymap representation is cited to prior work (Cat3D/CAT4D/cameras-as-rays), but that citation only supplies a coordinate encoding; it does not smuggle in the geometric prediction. The authors' self-citations (e.g., MegaSaM as concurrent work, raymap-related papers) are not load-bearing evidence for the paper's central claims. The conceded drift limitation in the Limitations section is a genuine correctness risk for long-sequence common-frame claims, but it is a performance caveat, not circular reasoning. The MVS-generated labels for MapFree/DL3DV in Appendix A create potential label noise, but they are external supervision, so no output is forced to equal its input by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 1 invented entities

The paper's central claim is an empirical property of a trained network. The ledger lists the hand-chosen design parameters, including state size, raymap masking probability, loss balance, and curriculum, and the domain assumptions about data and transfer. The only invented artifact is the persistent latent state, which has no external falsifiable handle; its effect is shown only through the paper's own ablations. Note that for MapFree and DL3DV, the training depth comes from MVS run with the same camera parameters used to build raymap queries, which lightly couples the novel-view supervision to the query representation.

free parameters (6)
  • Trained network weights (ViT-L encoder, ViT-B decoders, heads) = Not enumerated (hundreds of millions of parameters)
    All reported results are outputs of these weights, which are fitted to 32 datasets with the stated losses. The paper's claims are therefore empirical properties of this fitted model, not derived quantities.
  • State token count = 768 tokens of dimension 768
    Fixes the capacity of the scene memory; no ablation varies state size to demonstrate sufficiency for the claims.
  • Raymap masking probability = 20%
    Chosen without reported tuning; determines how often the model sees virtual-view queries during training.
  • Confidence loss log-weight alpha (Eq. 7) = Not specified in paper
    Controls the confidence calibration in the regression loss; its value is inherited from MASt3R but not stated, and it affects the balance between fitting good pixels and suppressing bad ones.
  • Pose loss weight = Implicitly 1 (no weighting stated)
    The total loss is presented as the sum of L_conf, L_pose, and L_rgb with no coefficients, so relative weighting is a hidden design choice.
  • Curriculum stages and resolution schedule = 224x224 then 512px longer side, 4 stages
    Hand-designed schedule affects what the model learns at each phase; no ablation shows the contribution of each stage.
assumptions (6)
  • domain assumption Initial state tokens are learnable and shared across scenes (Sec. 3.1).
    The state must start from a fixed prior and specialize through recurrent updates; the model has no per-scene initialization.
  • domain assumption The world frame is defined as the coordinate frame of the initial image (Sec. 3.1).
    Fixes the gauge for all pointmaps and poses; a convention rather than a derived result.
  • domain assumption When ground truth is metric, scale normalization factors are set equal (s_hat := s) to learn metric pointmaps (Sec. 3.3).
    Metric scale is learned from appearance alone; inherited from MASt3R.
  • domain assumption MVS-generated depths for MapFree and DL3DV are accurate enough for supervision (Appendix A).
    Some training labels are produced by running MVS; errors in these labels become noise in the learned prior.
  • domain assumption Raymap queries built from GT poses at training time teach the model to infer unobserved geometry (Sec. 3.2-3.3).
    The 6-channel raymap encodes camera pose; the model must learn the mapping from rays to geometry, which is a learned prior.
  • domain assumption Zero-shot transfer from 32 training datasets to the evaluation benchmarks is valid (Sec. 4).
    Benchmark claims presuppose that the training distribution covers the test domains; standard but unproven.
invented entities (1)
  • Persistent scene state (768 learnable tokens)
    purpose: Stores a compressed representation of the observed scene and is updated and read by each image; enables online reconstruction and virtual-view querying.
    The state is an internal latent mechanism. Its existence and usefulness are evidenced only by in-paper ablations (Tab. 5) and qualitative results; there is no external falsifiable handle independent of the paper's own experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuous 3D Perception Model with Persistent State." pith.science (2026). https://pith.science/paper/HFDRWFJ4

@misc{pith2026250112387,
  author       = {Pith},
  title        = {Pith review of: Continuous 3D Perception Model with Persistent State},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFDRWFJ4}},
  note         = {Machine review of arXiv:2501.12387}
}
read the original abstract

We present a unified framework capable of solving a broad range of 3D tasks. Our approach features a stateful recurrent model that continuously updates its state representation with each new observation. Given a stream of images, this evolving state can be used to generate metric-scale pointmaps (per-pixel 3D points) for each new input in an online fashion. These pointmaps reside within a common coordinate system, and can be accumulated into a coherent, dense scene reconstruction that updates as new images arrive. Our model, called CUT3R (Continuous Updating Transformer for 3D Reconstruction), captures rich priors of real-world scenes: not only can it predict accurate pointmaps from image observations, but it can also infer unseen regions of the scene by probing at virtual, unobserved views. Our method is simple yet highly flexible, naturally accepting varying lengths of images that may be either video streams or unordered photo collections, containing both static and dynamic content. We evaluate our method on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each. Project Page: https://cut3r.github.io/

Figures

Figures reproduced from arXiv: 2501.12387 by the authors.

Figure 1
Figure 1. Continuous 3D Perception. Given a stream of RGB images as input, our approach enables dense 3D reconstruction in an online, continuous manner, estimating both camera parameters and dense 3D geometry with each incoming frame. Our framework supports various 3D tasks, processes inputs from video sequences and sparse photo collections, and can handle both static and dynamic scenes. Abstract We present a unified framewor… view at source ↗
Figure 2
Figure 2. Querying Unseen Regions. In addition to reconstructing a scene from images, our method can also infer structure for unseen parts of the scene, given a virtual camera query (shown in blue). maintains and incrementally updates a persistent internal state that encodes the scene content. With each new observa￾tion, the model simultaneously updates this state and reads from it to predict the current view’s 3D properties,… view at source ↗
Figure 3
Figure 3. Method Overview. Our method performs online dense 3D reconstruction from a stream of images (video frames or a photo collection) by using a persistent state. Each input image is encoded into visual tokens via a shared-weight ViT encoder. These tokens interact with state tokens, where state update integrates the current image into the state, and state readout retrieves the past context stored in the state for predict… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative Results on In-the-wild Internet Videos. We compare our method with concurrent works Spann3R [101] and MonST3R [125]. Our method achieves the best qualitative results. 7 scenes [83] NRGBD [4] Acc↓ Comp↓ NC↑ Acc↓ Comp↓ NC↑ Method Optim. Onl. Mean Med. Mean Me…
Figure 6
Figure 6. Figure 6: Inferring New Structure via. State Readout. From top to bottom: the input image; the ground truth (GT) image, used to query the state via its camera parameters (note: GT image is not given to the model); the depth map from the predicted pointmap; the pointmap predictio…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rig3R: Rig-Aware Conditioning for Learned 3D Reconstruction

    cs.CV 2025-06 conditional novelty 7.0 of 10

    Rig3R conditions learned 3D reconstruction on optional rig metadata and predicts rig-relative raymaps, enabling state-of-the-art pose estimation and rig calibration discovery from images.

  2. Syn4D: A Multiview Synthetic 4D Dataset

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Syn4D supplies multiview synthetic dynamic scenes with dense geometric, tracking and pose ground truth that lets any pixel be unprojected to any time and camera.

  3. LongSplat: Robust Unposed 3D Gaussian Splatting for Casual Long Videos

    cs.CV 2025-08 conditional novelty 6.0 of 10

    An incremental 3D Gaussian Splatting pipeline that jointly optimizes camera poses and scene geometry using MASt3R priors and density-adaptive octree anchors achieves state-of-the-art novel view synthesis on casual lon...

  4. STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A decoder-only Transformer with causal attention and cached past-frame features performs incremental 3D reconstruction from streaming images, beating the RNN-based CUT3R on several benchmark metrics.

  5. LongSplat: Online Generalizable 3D Gaussian Splatting from Long Sequence Images

    cs.CV 2025-07 reject novelty 6.0 of 10

    A feed-forward 3D Gaussian Splatting pipeline that incrementally fuses and compresses historical Gaussians using a 2D image-like representation.

  6. SpatialTrackerV2: 3D Point Tracking Made Easy

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single feed-forward model jointly estimates video depth, camera poses, and 3D point trajectories from monocular video, setting a new state of the art on TAPVid-3D.

  7. Puzzles: Unbounded Video-Depth Augmentation for Scalable End-to-End 3D Reconstruction

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Puzzles synthesizes posed video-depth clips from single images and keyframes, letting 3D reconstruction models match full-data accuracy using only 10% of the data.

  8. Test3R: Learning to Reconstruct 3D at Test Time

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Test3R improves 3D reconstruction by optimizing visual prompts at test time so that pointmaps from different image pairs are geometrically consistent.

  9. EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Mesh

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EX-4D uses a depth watertight mesh and simulated occlusion masks to condition a video diffusion model for extreme-viewpoint 4D video synthesis from monocular input.

  10. RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A ray-conditioned transformer turns single-image 3D shape completion into novel-view depth prediction, achieving state-of-the-art chamfer distance on synthetic and real benchmarks.

  11. UniGeo: Taming Video Diffusion for Unified Consistent Geometry Estimation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Fine-tuning a pretrained video diffusion transformer to predict geometry in one shared global frame produces consistent, camera-free surface normals and coordinates across entire video clips.

  12. X-GRM: Large Gaussian Reconstruction Model for Sparse-view X-rays to Computed Tomography

    eess.IV 2025-05 conditional novelty 6.0 of 10

    A large transformer with fixed-voxel Gaussian splatting reconstructs CT volumes from 6-10 X-ray projections in under a second, substantially beating prior sparse-view methods in simulation.

  13. RoadVGGT: Road-Structure-Aware Feed-Forward Road Surface Reconstruction

    cs.CV 2026-07 conditional novelty 5.5 of 10

    A feed-forward Gaussian head on OmniVGGT plus road-plane grid fusion and structure-aware grouping reconstructs compact road surfaces that beat RoGS and AnySplat on Waymo and zero-shot nuScenes.

  14. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

  15. InstantSfM: Towards GPU-Native SfM for the Deep Learning Era

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A fully GPU-native, PyTorch-based global Structure-from-Motion pipeline using sparse-aware Levenberg-Marquardt with optional metric depth priors reports ~8-40× speedups over COLMAP at comparable accuracy on several be...

  16. UAV4D: Dynamic Neural Rendering of Human-Centric UAV Imagery using Gaussian Splatting

    cs.CV 2025-06 conditional novelty 5.0 of 10

    UAV4D reconstructs 4D scenes from monocular drone video by fitting a single global scale to align human meshes with the background mesh, then renders with separate Gaussian splats.

  17. Reconstructing 4D Spatial Intelligence: A Survey

    cs.CV 2025-07 accept novelty 4.0 of 10

    A review that classifies 4D scene reconstruction methods into five progressive levels: low-level cues, scene components, dynamic scenes, interactions, and physics.

Reference graph

Works this paper leans on

134 extracted references · 49 canonical work pages · cited by 17 Pith papers

  1. [1]

    Bundle adjustment in the large

    Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle adjustment in the large. In Com- puter Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II 11, pages 29–42. Springer, 2010. 2

    2010

  2. [2]

    Building rome in a day

    Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54 (10):105–112, 2011. 2

    2011

  3. [3]

    Map-free visual relocalization: Metric pose relative to a single image

    Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, ´Aron Monszpart, Victor Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocalization: Metric pose relative to a single image. In ECCV, 2022. 5, 8, 15, 16

    2022

  4. [4]

    Neural rgb-d surface reconstruction

    Dejan Azinovi´c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6290– 6301, 2022. 6, 7, 8

    2022

  5. [5]

    ARK- itscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARK- itscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Tra...

    2021

  6. [6]

    Uasol, a large- scale high-resolution outdoor stereo dataset

    Zuria Bauer, Francisco Gomez-Donoso, Edmanuel Cruz, Sergio Orts-Escolano, and Miguel Cazorla. Uasol, a large- scale high-resolution outdoor stereo dataset. Scientific data, 6(1):162, 2019. 16

    2019

  7. [7]

    Speeded-up robust features (surf)

    Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-up robust features (surf). Computer vision and image understanding, 110(3):346–359, 2008. 2

    2008

  8. [8]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 2

    arXiv 2023

Show all 134 references
  1. [9]

    Bedlam: A synthetic dataset of bodies exhibit- ing detailed lifelike animated motion

    Michael J Black, Priyanka Patel, Joachim Tesch, and Jin- long Yang. Bedlam: A synthetic dataset of bodies exhibit- ing detailed lifelike animated motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8726–8737, 2023. 15, 16

    2023

  2. [10]

    Richter, and Vladlen Koltun

    Aleksei Bochkovskii, Ama¨el Delaunoy, Hugo Germain, Mar- cel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv, 2024. 2

    2024

  3. [11]

    Transformerfusion: Monocular rgb scene reconstruction using transformers

    Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural In- formation Processing Systems, 34:1403–1414, 2021. 3

    2021

  4. [12]

    D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In European Conf. on Computer Vision (ECCV), pages 611–

  5. [13]

    Virtual kitti 2, 2020

    Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2, 2020. 16

    2020

  6. [14]

    Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age

    Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Da- vide Scaramuzza, Jos´e Neira, Ian Reid, and John J Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transac- tions on robotics, 32(6):1309–1332, 2016. 2

    2016

  7. [15]

    Matterport3d: Learning from rgb-d data in indoor environments

    Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017. 16

    2017 arXiv

  8. [16]

    Tensorf: Tensorial radiance fields

    Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision , pages 333–350. Springer,

  9. [17]

    Leap-vo: Long-term effective any point tracking for visual odometry

    Weirong Chen, Le Chen, Rui Wang, and Marc Pollefeys. Leap-vo: Long-term effective any point tracking for visual odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19844– 19853, 2024. 2, 6, 16, 17

    2024

  10. [18]

    3d-r2n2: A unified approach for single and multi-view 3d object reconstruction

    Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, ...

    2016

  11. [19]

    Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE,

  12. [20]

    Monoslam: Real-time single camera slam

    Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE transactions on pattern analysis and machine intelli- gence, 29(6):1052–1067, 2007. 2

    2007

  13. [21]

    Superpoint: Self-supervised interest point detection and description

    Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224–236, 2018. 2

    2018

  14. [22]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3, 5

    2010 arXiv

  15. [23]

    Simultaneous local- ization and mapping: part i

    Hugh Durrant-Whyte and Tim Bailey. Simultaneous local- ization and mapping: part i. IEEE robotics & automation magazine, 13(2):99–110, 2006. 2

    2006

  16. [24]

    D2-net: A trainable cnn for joint description and detection of lo- cal features

    Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Polle- feys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint description and detection of lo- cal features. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition,...

  17. [25]

    Finding structure in time

    Jeffrey L Elman. Finding structure in time. Cognitive sci- ence, 14(2):179–211, 1990. 2

    1990

  18. [26]

    Lsd- slam: Large-scale direct monocular slam

    Jakob Engel, Thomas Sch ¨ops, and Daniel Cremers. Lsd- slam: Large-scale direct monocular slam. In European conference on computer vision , pages 834–849. Springer,

  19. [27]

    Svo: Semidirect visual odometry for monocular and multicamera systems

    Christian Forster, Zichao Zhang, Michael Gassner, Manuel Werlberger, and Davide Scaramuzza. Svo: Semidirect visual odometry for monocular and multicamera systems. IEEE Transactions on Robotics, 33(2):249–265, 2016. 2

    2016

  20. [28]

    Plenoxels: Radiance fields without neural networks

    Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5501–5510, 2022. 2

    2022

  21. [29]

    Cat3d: Create any- thing in 3d with multi-view diffusion models

    Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole*. Cat3d: Create any- thing in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314, 2024. 4

    2024 arXiv

  22. [30]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231– 1237, 2013. 5

    2013

  23. [31]

    Digging into self-supervised monocular depth estimation

    Cl´ement Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 3828–3838,

  24. [32]

    G ´omez, Manuel Silva, Antonio Seoane, Agn `es Borr´as, Mario Noriega, Germ ´an Ros, Jose A

    Jose L. G ´omez, Manuel Silva, Antonio Seoane, Agn `es Borr´as, Mario Noriega, Germ ´an Ros, Jose A. Iglesias- Guitian, and Antonio M. L´opez. All for one, and one for all: Urbansyn dataset, the third musketeer of synthetic driving scenes, 2023. 16

    2023

  25. [33]

    Multiple view geometry in computer vision

    Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press,

  26. [34]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 6

    2016

  27. [35]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 16000–16009, 2022. 4

    2022

  28. [36]

    Long short-term memory.Neural Computation MIT-Press, 1997

    S Hochreiter. Long short-term memory.Neural Computation MIT-Press, 1997. 2

    1997

  29. [37]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023. 15

    2023 arXiv

  30. [38]

    Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation

    Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint arXiv:2404.15506, 2024. 2

    2024 arXiv

  31. [39]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 16, 17

    2024 arXiv

  32. [40]

    Deepmvs: Learning multi-view stereopsis

    Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 16

    2018

  33. [41]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 2

    2021

  34. [42]

    Learning a multi-view stereo machine

    Abhishek Kar, Christian H¨ane, and Jitendra Malik. Learning a multi-view stereo machine. Advances in neural informa- tion processing systems, 30, 2017. 2, 3

    2017

  35. [43]

    Dy- namicstereo: Consistent dynamic depth from stereo videos

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dy- namicstereo: Consistent dynamic depth from stereo videos. CVPR, 2023. 5, 16

    2023

  36. [44]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 15, 17

    2024

  37. [45]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1,

  38. [46]

    Parallel tracking and map- ping for small ar workspaces

    Georg Klein and David Murray. Parallel tracking and map- ping for small ar workspaces. In 2007 6th IEEE and ACM international symposium on mixed and augmented reality, pages 225–234. IEEE, 2007. 2

    2007

  39. [47]

    Ro- bust consistent video depth estimation

    Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Ro- bust consistent video depth estimation. In Proceedings of 10 the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611–1621, 2021. 3, 6, 16, 17

    2021

  40. [48]

    Smartportraits: Depth powered handheld smartphone dataset of human portraits for state estimation, reconstruction and synthesis

    Anastasiia Kornilova, Marsel Faizullin, Konstantin Pakulev, Andrey Sadkov, Denis Kukushkin, Azat Akhmetyanov, Timur Akhtyamov, Hekmat Taherinejad, and Gonzalo Ferrer. Smartportraits: Depth powered handheld smartphone dataset of human portraits for state estimation, reconstruct...

    2022

  41. [49]

    Video autoencoder: self-supervised disentanglement of static 3d structure and motion

    Zihang Lai, Sifei Liu, Alexei A Efros, and Xiaolong Wang. Video autoencoder: self-supervised disentanglement of static 3d structure and motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9730–9740, 2021. 3

    2021

  42. [50]

    Eden: Multimodal synthetic dataset of enclosed garden scenes

    Hoang-An Le, Thomas Mensink, Partha Das, Sezer Karaoglu, and Theo Gevers. Eden: Multimodal synthetic dataset of enclosed garden scenes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1579–1589, 2021. 15, 16

    2021

  43. [51]

    Ground- ing image matching in 3d with mast3r

    Vincent Leroy, Yohann Cabon, and J´erˆome Revaud. Ground- ing image matching in 3d with mast3r. arXiv preprint arXiv:2406.09756, 2024. 5, 6, 7, 16, 17

    2024 arXiv

  44. [52]

    Megadepth: Learning single- view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. Megadepth: Learning single- view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018. 5

    2018

  45. [53]

    Megadepth: Learning single- view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. Megadepth: Learning single- view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041–2050, 2018. 2, 16

    2018

  46. [54]

    Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos

    Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holyn- ski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.04463, 2024. 3

    2024 arXiv

  47. [55]

    Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision

    Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pa...

    2024

  48. [56]

    Zero-1-to-3: Zero-shot one image to 3d object, 2023

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023. 3

    2023

  49. [57]

    Hoi4d: A 4d egocentric dataset for category-level human- object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human- object interaction. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition,...

    2022

  50. [58]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  51. [59]

    Distinctive image features from scale- invariant keypoints.International journal of computer vision, 60:91–110, 2004

    David G Lowe. Distinctive image features from scale- invariant keypoints.International journal of computer vision, 60:91–110, 2004. 2

    2004

  52. [60]

    Spring: A high-resolution high- detail dataset and benchmark for scene flow, optical flow and stereo

    Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nali- vayko, and Andr´es Bruhn. Spring: A high-resolution high- detail dataset and benchmark for scene flow, optical flow and stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

  53. [61]

    Realfusion: 360 reconstruction of any object from a single image

    Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 reconstruction of any object from a single image. In CVPR, 2023. 3

    2023

  54. [62]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM, 65(1):99–106, 2021. 2

    2021

  55. [63]

    Instant neural graphics primitives with a mul- tiresolution hash encoding

    Thomas M¨uller, Alex Evans, Christoph Schied, and Alexan- der Keller. Instant neural graphics primitives with a mul- tiresolution hash encoding. ACM transactions on graphics (TOG), 41(4):1–15, 2022. 2

    2022

  56. [64]

    Orb-slam: a versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147–1163,

  57. [65]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012. 5

    2012

  58. [66]

    Dtam: Dense tracking and mapping in real-time

    Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. Dtam: Dense tracking and mapping in real-time. In 2011 international conference on computer vision, pages 2320–2327. IEEE, 2011. 2

    2011

  59. [67]

    3d ken burns effect from a single image

    Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3d ken burns effect from a single image. ACM Transactions on Graphics, 38(6):184:1–184:15, 2019. 16

    2019

  60. [68]

    Palazzolo, J

    E. Palazzolo, J. Behley, P. Lottes, P. Gigu`ere, and C. Stach- niss. ReFusion: 3D Reconstruction in Dynamic Environ- ments for RGB-D Cameras Exploiting Residuals. 2019. 5

    2019

  61. [69]

    Unidepth: Universal monocular metric depth estimation

    Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106–10116, 2024. 2

    2024

  62. [70]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022. 3

    2022 arXiv

  63. [71]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 3

    2021

  64. [72]

    Vi- sion transformers for dense prediction

    Ren´e Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vi- sion transformers for dense prediction. ICCV, 2021. 2, 3, 15

    2021

  65. [73]

    Towards robust monocu- lar depth estimation: Mixing datasets for zero-shot cross- dataset transfer

    Ren´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocu- lar depth estimation: Mixing datasets for zero-shot cross- dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022. 2

    2022

  66. [74]

    Com- mon objects in 3d: Large-scale learning and evaluation of 11 real-life 3d category reconstruction

    Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Com- mon objects in 3d: Large-scale learning and evaluation of 11 real-life 3d category reconstruction. In International Con- ference on Computer Vision, 2021. 5, 16

    2021

  67. [75]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021,

    2021

  68. [76]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  69. [77]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- ti...

    2022

  70. [78]

    SuperGlue: Learning feature matching with graph neural networks

    Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020. 2

    2020

  71. [79]

    Simplere- con: 3d reconstruction without 3d convolutions

    Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Cl´ement Godard. Simplere- con: 3d reconstruction without 3d convolutions. InEuropean Conference on Computer Vision, pages 1–19. Springer, 2022. 3

    2022

  72. [80]

    Structure-from-motion revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 2, 15

    2016

  73. [81]

    A comparison and eval- uation of multi-view stereo reconstruction algorithms

    Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A comparison and eval- uation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), pages 519–528. IEEE,

    2006

  74. [82]

    Learning tem- porally consistent video depth from video diffusion priors

    Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning tem- porally consistent video depth from video diffusion priors. arXiv preprint arXiv:2406.01493, 2024. 16, 17

    2024 arXiv

  75. [83]

    Scene coordinate regression forests for camera relocalization in rgb-d images

    Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2930–2937,

  76. [84]

    Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories

    Samarth Sinha, Roman Shapovalov, Jeremy Reizenstein, Ig- nacio Rocco, Natalia Neverova, Andrea Vedaldi, and David Novotny. Common pets in 3d: Dynamic new-view synthesis of real-life deformable categories. CVPR, 2023. 15, 16

    2023

  77. [85]

    Scene representation networks: Continuous 3d-structure- aware neural scene representations

    Vincent Sitzmann, Michael Zollh¨ofer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019. 3

    2019

  78. [86]

    Light field networks: Neu- ral scene representations with single-evaluation rendering

    Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neu- ral scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems , 34: 19313–19325, 2021. 3

    2021

  79. [87]

    Photo tourism: exploring photo collections in 3d

    Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pages 835–846. 2006. 2

    2006

  80. [88]

    Model- ing the world from internet photo collections

    Noah Snavely, Steven M Seitz, and Richard Szeliski. Model- ing the world from internet photo collections. International journal of computer vision, 80:189–210, 2008. 2

    2008

  81. [89]

    A benchmark for the evalua- tion of rgb-d slam systems

    J¨urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evalua- tion of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573–580. IEEE, 2012. 6, 17

    2012

  82. [90]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  83. [91]

    Loftr: Detector-free local feature matching with transformers

    Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 8922–8931, 2021. 2

    2021

  84. [92]

    Neuralrecon: Real-time coherent 3d re- construction from monocular video

    Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d re- construction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 15598–15607, 2021. 2, 3

    2021

  85. [93]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Et- tinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhan...

    2020

  86. [94]

    Ba-net: Dense bundle ad- justment network

    Chengzhou Tang and Ping Tan. Ba-net: Dense bundle ad- justment network. arXiv preprint arXiv:1806.04807, 2018. 2

    2018 arXiv

  87. [95]

    Cnn-slam: Real-time dense monocular slam with learned depth prediction

    Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. Cnn-slam: Real-time dense monocular slam with learned depth prediction. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 6243–6252, 2017. 2

    2017

  88. [96]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems , 34:16558–16569,

  89. [97]

    Deep patch vi- sual odometry

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch vi- sual odometry. Advances in Neural Information Processing Systems, 2023. 16, 17

    2023

  90. [98]

    Deep patch vi- sual odometry

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch vi- sual odometry. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  91. [99]

    Smd-nets: Stereo mixture density networks

    Fabio Tosi, Yiyi Liao, Carolin Schmitt, and Andreas Geiger. Smd-nets: Stereo mixture density networks. In Proceedings 12 of the IEEE/CVF conference on computer vision and pattern recognition, pages 8942–8952, 2021. 16

    2021

  92. [100]

    Bundle adjustment—a mod- ern synthesis

    Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment—a mod- ern synthesis. In Vision Algorithms: Theory and Prac- tice: International Workshop on Vision Algorithms Corfu, Greece, September 21–22, 1999 Proceedings, pages 298–

    1999

  93. [101]

    3d reconstruction with spatial memory

    Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408.16061, 2024. 3, 5, 6, 7, 16, 17

    2024 arXiv

  94. [102]

    Co- slam: Joint coordinate and sparse parametric encodings for neural real-time slam

    Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co- slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13293–13302, 2023. 6

    2023

  95. [103]

    Vggsfm: Visual geometry grounded deep structure from motion

    Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Vggsfm: Visual geometry grounded deep structure from motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21686–21697, 2024. 2

    2024

  96. [104]

    Neus: Learning neural im- plicit surfaces by volume rendering for multi-view recon- struction

    Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural im- plicit surfaces by volume rendering for multi-view recon- struction. arXiv preprint arXiv:2106.10689, 2021. 2

    2021 arXiv

  97. [105]

    Irs: A large natural- istic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation

    Qiang Wang, Shizhen Zheng, Qingsong Yan, Fei Deng, Kaiyong Zhao, and Xiaowen Chu. Irs: A large natural- istic indoor robotics stereo dataset to train deep models for disparity and surface normal estimation. In 2021 IEEE In- ternational Conference on Multimedia and Expo (ICME) ...

    2021

  98. [106]

    Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision, 2024

    Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision, 2024. 2

    2024

  99. [107]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 20697–20709, 2024. 2, 3, 5, 6, 7, 8, 15, 16, 17

    2024

  100. [108]

    Tartanair: A dataset to push the limits of visual slam

    Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. 2020. 5, 16

    2020

  101. [109]

    Neural video depth stabilizer

    Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV) , pages 9466–9476, 2023. 16, 17

    2023

  102. [110]

    Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion

    Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Ro- main Br´egier, Yohann Cabon, Vaibhav Arora, Leonid Ants- feld, Boris Chidlovskii, Gabriela Csurka, and J ´erˆome Re- vaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neura...

    2022

  103. [111]

    Synscapes: A pho- torealistic synthetic dataset for street scene parsing

    Magnus Wrenninge and Jonas Unger. Synscapes: A pho- torealistic synthetic dataset for street scene parsing. arXiv preprint arXiv:1810.08705, 2018. 5, 15, 16

    2018 arXiv

  104. [112]

    Cat4d: Create anything in 4d with multi-view video diffusion mod- els

    Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion mod- els. arXiv preprint arXiv:2411.18613, 2024. 4

    2024 arXiv

  105. [113]

    Reconfusion: 3d reconstruction with diffusion priors

    Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  106. [114]

    Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation

    Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Liang Pan Jiawei Ren, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In IEEE/CVF Conference on Computer V...

    2023

  107. [115]

    Rgbd objects in the wild: Scaling real-world 3d object learning from rgb-d videos

    Hongchi Xia, Yang Fu, Sifei Liu, and Xiaolong Wang. Rgbd objects in the wild: Scaling real-world 3d object learning from rgb-d videos. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22378–22389, 2024. 16

    2024

  108. [116]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2

    2024

  109. [117]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xi- aogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.09414, 2024. 15, 17

    2024 arXiv

  110. [118]

    D3vo: Deep depth, deep pose and deep uncer- tainty for monocular visual odometry

    Nan Yang, Lukas von Stumberg, Rui Wang, and Daniel Cremers. D3vo: Deep depth, deep pose and deep uncer- tainty for monocular visual odometry. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1281–1292, 2020. 2

    2020

  111. [119]

    Mvsnet: Depth inference for unstructured multi-view stereo

    Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pages 767–783, 2018. 2

    2018

  112. [120]

    Blendedmvs: A large- scale dataset for generalized multi-view stereo networks

    Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large- scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020. 16

    2020

  113. [121]

    Scannet++: A high-fidelity dataset of 3d in- door scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d in- door scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12–22, 2023. 5, 16

    2023

  114. [122]

    pixelnerf: Neural radiance fields from one or few images

    Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4578–4587, 2021. 3

    2021

  115. [123]

    Mvimgnet: A large-scale dataset of multi-view images

    Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognit...

    2023

  116. [124]

    Mip-splatting: Alias-free 3d gaussian splat- 13 ting

    Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splat- 13 ting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19447–19456,

  117. [125]

    Monst3r: A simple approach for estimat- ing geometry in the presence of motion

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jam- pani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming- Hsuan Yang. Monst3r: A simple approach for estimat- ing geometry in the presence of motion. arXiv preprint arxiv:2410.03825, 2024. 3, 5, 6, 7, 16, 17

    arXiv

  118. [126]

    Cameras as rays: Pose estimation via ray diffusion

    Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. In International Confer- ence on Learning Representations (ICLR), 2024. 4

    2024

  119. [127]

    Nerfusion: Fusing radiance fields for large- scale scene reconstruction

    Xiaoshuai Zhang, Sai Bi, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Nerfusion: Fusing radiance fields for large- scale scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5449–5458, 2022. 2, 3

    2022

  120. [128]

    Structure and motion from casual videos

    Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Ru- binstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In European Conference on Computer Vision, pages 20–37. Springer, 2022. 3, 6, 16, 17

    2022

  121. [129]

    Particlesfm: Exploiting dense point trajecto- ries for localizing moving cameras in the wild

    Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajecto- ries for localizing moving cameras in the wild. In European Conference on Computer Vision, pages 523–542. Springer,

  122. [130]

    Pointodyssey: A large-scale synthetic dataset for long-term point tracking

    Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wet- zstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 19855–19865, 2023. 15, 16

    2023

  123. [131]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 37, 2018. 5, 15, 16

    2018

  124. [132]

    Nice-slam: Neural implicit scalable encoding for slam

    Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 12786–12796,

  125. [133]

    camera only

    Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Marc Pollefeys. Nicer-slam: Neural implicit scene encoding for rgb slam. In 2024 International Conference on 3D Vision (3DV), pages 42–52. IEEE, 2024. 2 14 Appendix A. Training Datasets...

    2024

  126. [625]

    5, 6, 17

    Springer-Verlag, 2012. 5, 6, 17

    2012

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.