Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

DROID-Splat: Combining end-to-end SLAM with 3D Gaussian Splatting

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DROID-Splat combines an end-to-end optical-flow tracker with 3D Gaussian Splatting and reports state-of-the-art tracking and rendering on common SLAM benchmarks.

desk verdict Solid hybrid SLAM integration with an honest ablation study; the SotA claim only holds in P-RGBD mode with an external depth prior. read the letter →

arxiv 2411.17660 v2 pith:H47AGKRM submitted 2024-11-26 cs.CV

Christian Homeyer , Leon Begiristain , Christoph Schnörr This is my paper · ORCID
classification cs.CV
keywords simultaneouslocalizationandmapping3DGaussiansplattingend-to-endvisualSLAMphotorealisticreconstructionmonoculardepthpriorloopclosurecameraself-calibrationdifferentiablerendering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Until now, SLAM systems that reconstruct a photorealistic scene from video tended to lag behind on camera tracking, while the best trackers produced at best sparse or semi-dense maps. This paper proposes a system that keeps a dense end-to-end optical-flow tracker as its backbone and runs a 3D Gaussian Splatting renderer in parallel, so that accurate odometry and photo-realistic mapping are optimized at the same time. The authors claim this combination achieves state-of-the-art tracking and rendering on the TUM-RGBD and Replica benchmarks, across monocular, pseudo-RGBD, and RGB-D input, at near real-time speed on a consumer GPU. They also show that with a metric monocular depth prior and camera self-calibration, the same system reconstructs in-the-wild cellphone video even when camera intrinsics are unknown. If the claim holds, a single system can serve applications that need both precise pose and dense, viewable scene geometry from ordinary video.

What carries the argument

The machinery is a covisibility graph over keyframes: the end-to-end tracker maintains per-frame depth maps and poses, and a recurrent network plus differentiable bundle adjustment refines them by minimizing reprojection error against learned optical-flow residuals. After a covisibility and confidence filter, the depth map is triangulated and downsampled to seed 3D Gaussian hyperprimitives, each carrying position, rotation, scale, opacity, and spherical-harmonic color. Those primitives are optimized with a rendering loss that combines RGB, SSIM, and rendered depth against the tracking map or a metric depth prior, using MCMC-guided densification to add or remove Gaussians; loop closure is detected with learned place-recognition features and adds long-range edges to the backend graph. The same bundle-adjustment objective can also optimize camera intrinsics and the scale and shift of a monocular depth prior, which is how the system handles unknown-calibration video.

What would settle it

Run DROID-Splat on a monocular video of a low-texture, reflective scene with strong lighting changes, using a depth prior known to be unreliable there, and compare its trajectory error and held-out-view rendering quality against the tracker alone and a monocular baseline; if the renderer cannot prevent drift and novel-view quality collapses, the claim of simultaneous state-of-the-art tracking and rendering fails for exactly the conditions the paper names as its limits.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that dense tracking and dense rendering reinforce each other rather than being competing objectives. The tracker's dense depth-from-motion map, produced by differentiable bundle adjustment over a covisibility graph, is filtered for covisible, high-confidence points, downsampled, and used to initialize 3D Gaussian hyperprimitives; the renderer then optimizes appearance and depth together with a mixed RGB, SSIM, and depth loss. Because both modules are differentiable and run in parallel, the system reports state-of-the-art tracking and rendering metrics on TUM-RGBD and Replica. The ablations locate the value in specific components: scale-optimized monocular priors and conservative graph building matter most for tracking, MCMC-guided densification is the main rendering improvement, and feeding rendering gradients back into the tracker helps only when perfect groundtruth depth is available, not on real monocular data.

Load-bearing premise

The system depends on the pretrained tracker's depth estimates, optionally corrected by a monocular depth model, being accurate enough to seed and supervise the 3D scene; if those estimates are unreliable, the rendering step cannot fix the geometry.

Editorial extensions

If this is right

  • Photorealistic mapping no longer has to be a post-processing stage: the paper's parallel frontend, backend, renderer, and loop detector run on a single consumer GPU at near real-time rates.
  • Monocular video with unknown camera intrinsics is reconstructable when a temporally consistent metric depth prior is available, because intrinsics and prior scale and shift are absorbed into bundle adjustment.
  • Rendering-to-tracking feedback is not a free lunch: gradients through the rendering loss improved pose estimates only under perfect depth supervision and hurt on noisier monocular data.
  • For rendering, the densification strategy matters more than covisibility pruning; MCMC-guided densification consistently outperforms the original gradient-based density control at matched Gaussian counts.
  • Dense tracking maps generalize to non-training frames, so the system's rendering advantage shows most on held-out views rather than on the keyframes used for optimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the prior ablation implies that improving temporal consistency of monocular depth models would give larger SLAM gains than improving single-image depth accuracy.
  • Beyond the paper, a testable extension is to route tracker uncertainty into Gaussian densification so that unreliable depth regions are not treated as ground truth.
  • Beyond the paper, the pose-feedback result suggests that RGB rendering losses are too noisy for odometry; backpropagating only depth or feature-metric residuals might stabilize renderer-to-tracker feedback.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DROID-Splat, a dense SLAM system that couples the end-to-end DROID-SLAM tracker with a 3D Gaussian Splatting renderer. The system processes monocular, RGB-D, and a "Pseudo-RGBD" mode in which monocular depth priors (Metric3D) are integrated through per-frame scale/shift optimization. The authors run frontend, backend, loop closure, and renderer in parallel on a single consumer GPU, and they report ablations of tracking components (Table 1), rendering techniques (Table 2), and a feedback experiment in which renderer poses/disparity are fed back into the tracker (Table 7). Evaluations on TUM-RGBD and Replica claim state-of-the-art tracking and rendering, with additional qualitative in-the-wild reconstructions.

Significance. The paper is a solid systems contribution: it combines two mature components in a flexible, parallelized framework, provides detailed ablations averaged over five runs, reports failure cases and a 'what did not work' section, and promises open-source code. The honest reporting of the unsuccessful feedback experiment (Table 7) is a strength. If the state-of-the-art claims were accurately qualified, the work would be a useful reference for practitioners building dense SLAM systems. The main weakness is that the headline claim is overstated relative to the paper's own tables: the tracking SotA only holds in P-RGBD mode with an external Metric3D prior, and rendering SotA does not hold on Replica for monocular or RGB-D input.

major comments (3)
  1. [Abstract; Section 4.1, Tables 3 and 4] The abstract's unqualified claim that DroidSplat 'achieves both SotA tracking and rendering results on common SLAM benchmarks' is contradicted by the paper's own data. In Table 3, Ours Mono averages 3.5 cm ATE on TUM-RGBD, behind GlORIE-SLAM (2.1 cm) and GO-SLAM (2.3 cm); only Ours P-RGBD (1.8 cm) achieves the best average. On Replica rendering, Table 4 shows NeRF-SLAM's PSNR of 41.40 dB above Ours Mono's 39.47 dB, and Table 5 shows Gaussian-SLAM's PSNR of 42.08 dB above Ours RGBD's 39.66 dB. The state-of-the-art claim should be restricted to the specific modes and datasets for which it actually holds, and the abstract and contributions should be revised accordingly.
  2. [Section 4, Table 1; Section 4.1, Table 3] The tracking improvement is largely inherited from the monocular depth prior rather than from the combination of the end-to-end tracker with 3D Gaussian Splatting. Table 1 shows that adding scale optimization to the P-RGBD mode reduces TUM-RGBD ATE from 4.88 cm to 1.92 cm, and Table 3 shows that P-RGBD mode (which injects Metric3D priors and optimizes per-frame scale/shift via Eq. 5) is what yields the best tracking numbers. The paper should clearly state that the SotA tracking result depends on an external depth prior, and that the contribution of the tracker-renderer combination to tracking is limited. This is a load-bearing qualification for the central claim.
  3. [Section 4, Failure Cases; Section 3.3, Eq. (7)] The paper's own limitation statement — 'we perform much worse in sparser scenarios or when our priors are unreliable' — directly affects the generality of the reported results. Since Gaussians are initialized from the tracking map (Section 3.3) and trained with depth supervision (Eq. 7), an unreliable prior or sparse scene degrades both tracking and rendering, and the rendering objective cannot repair the geometry. The in-the-wild evaluation is only qualitative, so the benchmark numbers should not be presented as evidence for robust in-the-wild performance. The authors should either provide quantitative in-the-wild evaluation or explicitly scope the performance claims to scenarios with reliable depth priors.
minor comments (6)
  1. [Section 4, Runtime analysis] The term 'near real-time' is never quantified in the text. Please report concrete frame rates (e.g., from Figure 4) or a small table with per-mode timings, including the depth-prior bottleneck.
  2. [Table 2] In the row 'MCMC + Covis. Pruning', the L1 value of '0.82' for the KF column appears to be a typo; it should likely be 0.082 to be consistent with the surrounding values (0.0843 and 0.0841).
  3. [Section 1, Introduction] There is a typo: 'it's parent system' should be 'its parent system'.
  4. [Section 3.3] The sentence 'Each Gaussian possesses a rotation r ∈ SO(3), scaling s ∈ R3, density σ ∈ R1...' uses set membership notation correctly, but 'a set of points P ∈ R3' on line 2 should be 'a set of points P ⊂ R3'.
  5. [Table 3 caption] The caption says 'We achieve SotA tracking with our framework,' but the Mono row is not state-of-the-art. Please clarify that this refers to the P-RGBD mode.
  6. [Section 4, Tracking Ablation] The statement 'we achieve SotA results by simply utilizing a more conservative graph building strategy' is vague; please specify the exact graph-building parameters and how they differ from the default DROID-SLAM settings.

Circularity Check

0 steps flagged · score 0.0 of 10

DROID-Splat builds on external pretrained components and external benchmarks; no derivation is equivalent to its own inputs, so there is no circularity.

full rationale

The paper's contribution is an engineering combination: a pretrained end-to-end tracker (DROID-SLAM, [41]), an optional external monocular depth prior (Metric3D, [51]) with scale/shift alignment (Eq. 5), and a 3D Gaussian Splatting renderer initialized from the tracking map and supervised by the mixed rendering loss (Eq. 7). None of these components is defined in terms of the benchmark outcomes that the paper reports. Tracking accuracy is measured by ATE RMSE against ground-truth trajectories on TUM-RGBD and Replica, rendering quality by PSNR/SSIM/LPIPS and depth L1 against ground-truth images and sensor depth, and the ablations compare alternative published techniques (3DGS, MCMC, 2DGS, covisibility pruning) against one another. The Gaussians are initialized from the tracker's disparity map and depth-supervised by it, but this is an architectural design choice, not a statistical shortcut: the rendering evaluation is against external ground truth, and the tracking evaluation does not use the renderer's output as its criterion. The optional Metric3D prior is a fitted external model, but the paper does not claim to derive it or to predict its output; it explicitly attributes it to prior work and even abates different priors in the supplementary (Table 11). The main paper also candidly states limitations: 'we perform much worse in sparser scenarios or when our priors are unreliable' and, in Table 7, that render-to-tracker feedback does not improve results. The 'SotA tracking' claim is qualified by input mode, and the pure-mono numbers in Table 3 trail GlORIE-SLAM and GO-SLAM; this is a correctness/scope concern about overclaiming, not a circularity concern. No self-citations are used as load-bearing evidence, no uniqueness theorems are imported from the authors' own prior work, and no known result is renamed as a new derivation. The performance claims are supported by external benchmarks and code-release promises, not by construction from the inputs. Therefore the circularity score is 0.

Assumptions & free parameters 13 free parameters · 7 assumptions · 0 invented entities

The system's central claims rest on borrowed pretrained components (DROID-SLAM, Metric3D, EigenPlaces), standard multi-view geometry, and many hand-tuned hyperparameters. No new physical entities, forces, or dimensions are introduced; the contribution is an integration, so the ledger mostly tracks dependencies on external models and benchmark assumptions.

free parameters (13)
  • lambda1 appearance-geometry weight = 0.9 TUM-RGBD, 0.8 Replica
    Balances color and depth loss in Equation 7; authors note that tuning it shifts metrics between appearance and geometry.
  • lambda2 SSIM weight = 0.2
    Weight between L1 and SSIM in the appearance loss, set in the supplementary.
  • Keyframe insertion motion threshold = 3.0
    Controls when new keyframes are added; authors identify graph building as a critical factor for tracking.
  • Frontend max keyframe age = 30, versus DROID-SLAM default 25
    Longer local bundle adjustment window; authors found that keeping keyframes longer matters.
  • Frame distance weight beta = 0.5 TUM-RGBD, 0.7 Replica
    Used for graph building edge selection and tuned per dataset.
  • Backend run frequency = every 8 frontend passes
    Sync schedule of global bundle adjustment, affecting drift and speed.
  • Backend graph window and edge count = 150 frames, up to 1500 edges
    Conservative global graph construction; authors attribute tracking success largely to this graph building strategy.
  • Loop closure thresholds tau_f, tau_t, tau_r = 0.5, 10 frames, 15 degrees
    Feature distance, minimum frame gap, and orientation threshold for loop candidates; chosen qualitatively.
  • Renderer schedule = every 20 frontend calls, 100 iterations, 5 frame delay
    Frequency and optimization budget of the mapper; authors state this can be tuned arbitrarily.
  • Gaussian map downsample factor = 64 Replica, 16 TUM-RGBD
    Point cloud density after covisibility filtering; authors report similar results as long as total Gaussian count is similar.
  • MCMC densification noise level lrnoise = 1e4
    Noise parameter for the MCMC densification strategy from [19].
  • Intrinsics heuristic initialization = fx=fy=(H+W)/2, cx=W/2, cy=H/2
    Initial pinhole intrinsics for unknown-camera calibration; the method starts from this heuristic.
  • Feedback consistency gate = absolute relative disparity error at most 0.2, at least 50 percent of pixels
    Gate for feeding renderer poses and disparity back into the tracker; only used in the feedback experiments.
assumptions (7)
  • domain assumption The pretrained DROID-SLAM tracker generalizes to the evaluation datasets and produces reliable dense optical flow, disparity, and pose updates.
    The entire tracking module is inherited from [41] and is not retrained or independently verified beyond the reported runs. See Sections 3.1 and 4.
  • domain assumption The pretrained Metric3D monocular depth prior supplies metric depth of sufficient temporal consistency for Pseudo-RGBD mode.
    The paper selects Metric3D because of temporal consistency and ablates alternatives, but the central P-RGBD results depend on this external model. See Section 4 and Table 11.
  • domain assumption The scenes are static and approximately Lambertian for the rendering objective.
    The rendering loss in Equation 7 compares rendered images to observed images; the method does not model dynamic objects or strong lighting changes, acknowledged in Section 4 failure cases.
  • domain assumption The pinhole camera model with the heuristic initialization is a sufficient camera model for the demonstrated reconstructions.
    All experiments use a pinhole model and heuristic intrinsics; the paper mentions support for other models in theory but does not demonstrate it. See Section 3.1 and supplementary Section 4.
  • standard math Iterative differentiable bundle adjustment converges to a usable local optimum for the graph structures used.
    No convergence guarantees are given; the system relies on iterative optimization from DROID-SLAM [41]. See Section 3.1.
  • domain assumption EigenPlaces descriptors are sufficient to detect loop closures in the indoor benchmark scenes.
    Loop detection relies on nearest neighbors in a FAISS database; no recall or precision analysis is reported. See Section 3.2.
  • standard math Benchmark ground-truth poses and depths are accurate enough for the reported ATE, PSNR, and L1 metrics.
    TUM-RGBD and Replica ground truth are treated as error-free references in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DROID-Splat: Combining end-to-end SLAM with 3D Gaussian Splatting." pith.science (2026). https://pith.science/paper/H47AGKRM

@misc{pith2026241117660,
  author       = {Pith},
  title        = {Pith review of: DROID-Splat: Combining end-to-end SLAM with 3D Gaussian Splatting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H47AGKRM}},
  note         = {Machine review of arXiv:2411.17660}
}
read the original abstract

Recent progress in scene synthesis makes standalone SLAM systems purely based on optimizing hyperprimitives with a Rendering objective possible. However, the tracking performance still lacks behind traditional and end-to-end SLAM systems. An optimal trade-off between robustness, speed and accuracy has not yet been reached, especially for monocular video. In this paper, we introduce a SLAM system based on an end-to-end Tracker and extend it with a Renderer based on recent 3D Gaussian Splatting techniques. Our framework \textbf{DroidSplat} achieves both SotA tracking and rendering results on common SLAM benchmarks. We implemented multiple building blocks of modern SLAM systems to run in parallel, allowing for fast inference on common consumer GPU's. Recent progress in monocular depth prediction and camera calibration allows our system to achieve strong results even on in-the-wild data without known camera intrinsics. Code will be available at \url{https://github.com/ChenHoy/DROID-Splat}.

Figures

Figures reproduced from arXiv: 2411.17660 by the authors.

Figure 1
Figure 1. DROID-Splat allows to reconstruct a scene with known/unknown intrinsics. By combining an optical flow tracking objective and a fast, dense Renderer, we can achieve photo-realistic Reconstructions while optimizing accurate odometry. In this paper, we introduce DROID-Splat: A SotA SLAM system based on dense, end-to-end optical flow and a dense Rendering objective using 3D Gaussian Splatting [18]. Our system offers the… view at source ↗
Figure 2
Figure 2. DROID-Splat. We make use of an end-to-end SLAM system with an optical flow based objective to perform tracking and reconstruct odometry and a dense initial map. The tracking objective is flexible, which allows us to optimize intrinsics or prior scale and shift as well if wanted. We make use of SotA Gaussian Splatting techniques to learn a photo-realistic reconstruction based on a Rendering objective. Since all compo… view at source ↗
Figure 3
Figure 3. Rendering Results on TUM-RGBD [38]. We show views, that were not in the training set, i.e. our keyframe buffer. Top two rows show monocular methods, bottom shows RGBD (We show the results with prior for ours). We achieve a higher rendering and depth quality than [13, 24, 56] due to initializing with a dense tracking system and using dense hyperprimitives. Using a monocular prior can even improve upon a sparse laser … view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Compute-Performance Trade-off. We take the average across TUM RGBD [38] and Replica [37] in RGBD mode. We added the baseline Tracker at the bottom for perspective, this does not have a meaningful Metric attached to it. practice we perform a check to make sure that the …
Figure 6
Figure 6. Figure 6: More Rendering Results on TUM-RGBD [38]. Top four rows are from monocular input, bottom from RGBD. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Rendering Results on Replica [37]. We show non-training frames in multiple input modalities. Note how visually close the predictions are to the groundtruth. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Geometry vs. appearance. We found, that 2D Gaussian Splatting [12] can quickly create smooth surfaces and does not accumulate many floaters in outdoor scenes. However, the rendering quality lacks behind 3D Gaussian Splatting [18] and as long as good supervision exists …
Figure 9
Figure 9. Figure 9: Monocular depth prediction in-the-wild. Models exhibit very clear differences w.r.t captured details and scale consistency on self-captured video. Problems can arise in particular for reflective surfaces or paintings. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Camera calibration and prior integration matter. Distortion effects and artifacts both on the map and camera odometry can be observed without calibration. Using our strategy, we can get rid of distortions. The scale-optimized prior integration allows accurate structur…
Figure 11
Figure 11. Figure 11: Common failure cases. Since we are heavily dependent on depth priors on in-the-wild video, our method can fail when priors get unreliable. Similarly, if the geometry supervision is not good enough, we accumulate floaters on outdoor scenes. Challeng￾ing lighting condit…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GeoGS-SLAM: Online Monocular Reconstruction Using Gaussian Splatting with Geometric Priors

    cs.RO 2026-07 conditional novelty 5.0 of 10

    An online monocular SLAM system that samples 3D Gaussians from RGB plus VGGT geometric priors and jointly optimizes poses and map with photometric and geometric losses plus loop closure, beating prior monocular 3DGS a...

Reference graph

Works this paper leans on

73 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    Eigenplaces: Training viewpoint robust models for visual place recognition

    Gabriele Berton, Gabriele Trivigno, Barbara Caputo, and Carlo Masone. Eigenplaces: Training viewpoint robust models for visual place recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11080–11090, 2023. 4, 12

    work page 2023

  2. [2]

    Zoedepth: Zero-shot trans- fer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot trans- fer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023. 2, 3, 5, 17, 18

    arXiv 2023

  3. [3]

    Bundlefusion: Real-time glob- ally consistent 3d reconstruction using on-the-fly surface rein- tegration

    Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time glob- ally consistent 3d reconstruction using on-the-fly surface rein- tegration. ACM Transactions on Graphics (ToG), 36(4):1,

    work page

  4. [4]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff John- son, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024. 4

    arXiv 2024

  5. [5]

    Mvgs: Multi-view- regulated gaussian splatting for novel view synthesis, 2024

    Xiaobiao Du, Yida Wang, and Xin Yu. Mvgs: Multi-view- regulated gaussian splatting for novel view synthesis, 2024. 19

    work page 2024

  6. [6]

    Mvgs: Multi-view- regulated gaussian splatting for novel view synthesis, 2024

    Xiaobiao Du, Yida Wang, and Xin Yu. Mvgs: Multi-view- regulated gaussian splatting for novel view synthesis, 2024. 2

    work page 2024

  7. [7]

    Lsd-slam: Large-scale direct monocular slam

    Jakob Engel, Thomas Schöps, and Daniel Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pages 834–849. Springer, 2014. 2, 3

    work page 2014

  8. [8]

    Direct sparse odometry

    Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE transactions on pattern analysis and machine intelligence, 40(3):611–625, 2017. 2

    2017

Show all 73 references
  1. [9]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354–3361. IEEE, 2012. 17

    2012

  2. [10]

    Deep geometry-aware camera self-calibration from video

    Annika Hagemann, Moritz Knorr, and Christoph Stiller. Deep geometry-aware camera self-calibration from video. In Pro- ceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 3438–3448, 2023. 2, 3, 18

    2023

  3. [11]

    Lotus: Diffusion-based visual foundation model for high-quality dense prediction

    Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying- Cong Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409.18124, 2024. 17, 18

    2024 arXiv

  4. [12]

    2d gaussian splatting for geometrically accu- rate radiance fields

    Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accu- rate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024. 2, 4, 5, 8, 12, 16

    2024

  5. [13]

    Photo-slam: Real-time simultaneous localization and photo- realistic mapping for monocular stereo and rgb-d cameras

    Huajian Huang, Longwei Li, Hui Cheng, and Sai-Kit Yeung. Photo-slam: Real-time simultaneous localization and photo- realistic mapping for monocular stereo and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21584–21593...

    2024

  6. [14]

    Olaf Kähler, Victor Adrian Prisacariu, Carl Yuheng Ren, Xin Sun, Philip H. S. Torr, and David William Murray. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Trans. Vis. Comput. Graph., 21(11): 1241–1250, 2015. 2

    2015

  7. [15]

    Repurpos- ing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492–9502,

  8. [16]

    Splatam: Splat track & map 3d gaussians for dense rgb-d slam

    Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition,...

    2024

  9. [17]

    Real-time 3d recon- struction in dynamic scenes using point-based fusion

    Maik Keller, Damien Lefloch, Martin Lambers, Shahram Izadi, Tim Weyrich, and Andreas Kolb. Real-time 3d recon- struction in dynamic scenes using point-based fusion. In 2013 International Conference on 3D Vision-3DV 2013, pages 1–8. IEEE, 2013. 2

    2013

  10. [18]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023. 1, 2, 4, 5, 8, 12, 13, 16, 18, 19

    2023

  11. [19]

    3d gaussian splatting as markov chain monte carlo, 2024

    Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Wei- wei Sun, Jeff Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. 3d gaussian splatting as markov chain monte carlo, 2024. 2, 4, 5, 12, 13, 16

    2024

  12. [20]

    g 2 o: A general framework for graph optimization

    Rainer Kümmerle, Giorgio Grisetti, Hauke Strasdat, Kurt Konolige, and Wolfram Burgard. g 2 o: A general framework for graph optimization. In 2011 IEEE international confer- ence on robotics and automation, pages 3607–3613. IEEE,

    2011

  13. [21]

    Deep patch visual slam

    Lahav Lipson, Zachary Teed, and Jia Deng. Deep patch visual slam. arXiv preprint arXiv:2408.01654, 2024. 1, 2, 4, 5

    2024 arXiv

  14. [22]

    Loopy-slam: Dense neural slam with loop closures

    Lorenzo Liso, Erik Sandström, Vladimir Yugay, Luc Van Gool, and Martin R Oswald. Loopy-slam: Dense neural slam with loop closures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20363–20373, 2024. 2, 4

    2024

  15. [23]

    Robust dynamic radiance fields

    Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13–23, 2023. 3

    2023

  16. [24]

    Gaussian splatting slam

    Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and An- drew J Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18039–18048, 2024. 1, 2, 4, 5, 6, 7, 12, 14

    2024

  17. [25]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: 9 Representing scenes as neural radiance fields for view synthe- sis, 2020. 2

    2020

  18. [26]

    Instant neural graphics primitives with a multires- olution hash encoding

    Thomas Müller, Alex Evans, Christoph Schied, and Alexan- der Keller. Instant neural graphics primitives with a multires- olution hash encoding. ACM transactions on graphics (TOG), 41(4):1–15, 2022. 2

    2022

  19. [27]

    Orb-slam: a versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147–1163,

  20. [28]

    Real-time 3d reconstruction at scale using voxel hashing

    Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG), 32,

  21. [29]

    Q-slam: Quadric representations for monocular slam

    Chensheng Peng, Chenfeng Xu, Yue Wang, Mingyu Ding, Heng Yang, Masayoshi Tomizuka, Kurt Keutzer, Marco Pavone, and Wei Zhan. Q-slam: Quadric representations for monocular slam. arXiv preprint arXiv:2403.08125, 2024. 5, 6, 7

    2024 arXiv

  22. [30]

    Nerf- slam: Real-time dense monocular slam with neural radiance fields

    Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf- slam: Real-time dense monocular slam with neural radiance fields. In 2023 IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pages 3437–3444. IEEE,

    2023

  23. [31]

    Orb: An efficient alternative to sift or surf

    Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 2011 International conference on computer vision , pages 2564–

    2011

  24. [32]

    Point-slam: Dense neural point cloud-based slam

    Erik Sandström, Yue Li, Luc Van Gool, and Martin R Os- wald. Point-slam: Dense neural point cloud-based slam. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18433–18444, 2023. 6

    2023

  25. [33]

    Splat-slam: Globally optimized rgb-only slam with 3d gaussians

    Erik Sandström, Keisuke Tateno, Michael Oechsle, Michael Niemeyer, Luc Van Gool, Martin R Oswald, and Federico Tombari. Splat-slam: Globally optimized rgb-only slam with 3d gaussians. arXiv preprint arXiv:2405.16544, 2024. 2, 3, 4, 5, 6, 7, 13

    2024 arXiv

  26. [34]

    R3d3: Dense 3d reconstruction of dy- namic scenes from multiple cameras

    Aron Schmied, Tobias Fischer, Martin Danelljan, Marc Polle- feys, and Fisher Yu. R3d3: Dense 3d reconstruction of dy- namic scenes from multiple cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3216–3226, 2023. 1

    2023

  27. [35]

    Bad slam: Bundle adjusted direct rgb-d slam

    Thomas Schops, Torsten Sattler, and Marc Pollefeys. Bad slam: Bundle adjusted direct rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 134–144, 2019. 2

    2019

  28. [36]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7- 13, 2012, Proceedings, Part V 12, pages 746–760. Springer,

    2012

  29. [37]

    The replica dataset: A digital replica of indoor spaces

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797,

    1906 arXiv

  30. [38]

    A benchmark for the evalua- tion of rgb-d slam systems

    Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evalua- tion of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573–580. IEEE, 2012. 4, 5, 6, 7, 8, 12, 13, 14, 16, 18

    2012

  31. [39]

    High-fidelity slam using gaussian splatting with rendering-guided densification and regularized optimization

    Shuo Sun, Malcolm Mielle, Achim J Lilienthal, and Martin Magnusson. High-fidelity slam using gaussian splatting with rendering-guided densification and regularized optimization. arXiv preprint arXiv:2403.12535, 2024. 2, 19

    2024 arXiv

  32. [40]

    Raft: Recurrent all-pairs field transforms for optical flow

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020. 2

    2020

  33. [41]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558–16569, 2021. 1, 2, 3, 4, 12, 19

    2021

  34. [42]

    Deep patch visual odometry

    Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. Advances in Neural Information Processing Systems, 36, 2024. 1, 2

    2024

  35. [43]

    How nerfs and 3d gaussian splatting are reshaping slam: a survey

    Fabio Tosi, Youmin Zhang, Ziren Gong, Erik Sandström, Stefano Mattoccia, Martin R Oswald, and Matteo Poggi. How nerfs and 3d gaussian splatting are reshaping slam: a survey. arXiv preprint arXiv:2402.13255, 4, 2024. 2, 6

    2024 arXiv

  36. [44]

    Dn-splatter: Depth and normal priors for gaussian splatting and meshing, 2024

    Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, and Juho Kannala. Dn-splatter: Depth and normal priors for gaussian splatting and meshing, 2024. 2

    2024

  37. [45]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004. 5, 12

    2004

  38. [46]

    Elasticfusion: Dense slam without a pose graph

    Thomas Whelan, Stefan Leutenegger, Renato F Salas- Moreno, Ben Glocker, and Andrew J Davison. Elasticfusion: Dense slam without a pose graph. In Robotics: science and systems, page 3. Rome, Italy, 2015. 2

    2015

  39. [47]

    Sparsegs: Real-time 360° sparse view synthesis using gaussian splatting

    Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time 360° sparse view synthesis using gaussian splatting. Arxiv,

  40. [48]

    Gs-slam: Dense visual slam with 3d gaussian splatting

    Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 19595–19604, 2024. 2

    2024

  41. [49]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371–10381, 2024. 2, 3, 17, 18

    2024

  42. [50]

    V ox-fusion: Dense tracking and mapping with voxel-based neural implicit representation

    Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. V ox-fusion: Dense tracking and mapping with voxel-based neural implicit representation. In 2022 IEEE International Symposium on Mixed and Aug- mented Reality (ISMAR), pages 499–507. IEEE, 2022. 6

    2022

  43. [51]

    Metric3d: 10 Towards zero-shot metric 3d prediction from a single image

    Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaix- uan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: 10 Towards zero-shot metric 3d prediction from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9043–9053, 2023. 2, 3,...

    2023

  44. [52]

    Mip-splatting: Alias-free 3d gaussian splat- ting

    Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splat- ting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19447–19456,

  45. [53]

    Gaussian opacity fields: Efficient adaptive surface reconstruction in unbounded scenes, 2024

    Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient adaptive surface reconstruction in unbounded scenes, 2024. 2

    2024

  46. [54]

    Gaussian-slam: Photo-realistic dense slam with gaus- sian splatting

    Vladimir Yugay, Yue Li, Theo Gevers, and Martin R Os- wald. Gaussian-slam: Photo-realistic dense slam with gaus- sian splatting. arXiv preprint arXiv:2312.10070, 2023. 2, 4, 5, 6, 7

    2023 arXiv

  47. [55]

    Geolrm: Geometry-aware large re- construction model for high-quality 3d gaussian generation

    Chubin Zhang, Hongliang Song, Yi Wei, Yu Chen, Jiwen Lu, and Yansong Tang. Geolrm: Geometry-aware large re- construction model for high-quality 3d gaussian generation. arXiv preprint arXiv:2406.15333, 2024. 8

    2024 arXiv

  48. [56]

    Glorie-slam: Globally optimized rgb-only implicit encoding point cloud slam

    Ganlin Zhang, Erik Sandström, Youmin Zhang, Manthan Patel, Luc Van Gool, and Martin R Oswald. Glorie-slam: Globally optimized rgb-only implicit encoding point cloud slam. arXiv preprint arXiv:2403.19549, 2024. 2, 3, 5, 6, 7, 14

    2024 arXiv

  49. [57]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 5

    2018

  50. [58]

    Hi-slam: Monocular real-time dense mapping with hybrid implicit fields

    Wei Zhang, Tiecheng Sun, Sen Wang, Qing Cheng, and Nor- bert Haala. Hi-slam: Monocular real-time dense mapping with hybrid implicit fields. IEEE Robotics and Automation Letters, 2023. 2, 3, 4

    2023

  51. [59]

    Go-slam: Global optimization for consistent 3d instant reconstruction

    Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimization for consistent 3d instant reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3727–3737, 2023. 4, 5, 6

    2023

  52. [60]

    Pixel-gs: Density control with pixel- aware gradient for 3d gaussian splatting

    Zheng Zhang, Wenbo Hu, Yixing Lao, Tong He, and Hengshuang Zhao. Pixel-gs: Density control with pixel- aware gradient for 3d gaussian splatting. arXiv preprint arXiv:2403.15530, 2024. 2

    2024 arXiv

  53. [61]

    Mod-slam: Monocular dense mapping for unbounded 3d scene reconstruction

    Heng Zhou, Zhetao Guo, Shuhong Liu, Lechen Zhang, Qihao Wang, Yuxiang Ren, and Mingrui Li. Mod-slam: Monocular dense mapping for unbounded 3d scene reconstruction. arXiv preprint arXiv:2402.03762, 2024. 5, 6, 7

    2024 arXiv

  54. [62]

    Loopsplat: Loop closure by registering 3d gaus- sian splats

    Liyuan Zhu, Yue Li, Erik Sandström, Konrad Schindler, and Iro Armeni. Loopsplat: Loop closure by registering 3d gaus- sian splats. arXiv preprint arXiv:2408.10154, 2024. 2

    2024 arXiv

  55. [63]

    Nice-slam: Neural implicit scalable encoding for slam

    Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12786–12796, 2022. 6

    2022

  56. [64]

    Nicer- slam: Neural implicit scene encoding for rgb slam

    Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Marc Pollefeys. Nicer- slam: Neural implicit scene encoding for rgb slam. In 2024 International Conference on 3D Vision (3DV), pages 42–52. IEEE, 2024. 6 11 Supplementary Material In th...

    2024

  57. [66]

    Inference settings and hyperparameters We run our system at resolution 320 × 432 on TUM- RGBD [38] and 360 × 640 on Replica [37]. 1.1. Tracking Tracking is configured by the frontend and backend parame- ters for graph building, optimization and our loop detector. Since the con...

  58. [67]

    The reported rendering metrics for our comparison with related work are computed on the keyframe images based on the estimated poses, as is standard

    Extended Evaluation In this section, we want to provide more insights into how our system performs quantitatively and show more qualitative results. The reported rendering metrics for our comparison with related work are computed on the keyframe images based on the estimated p...

  59. [68]

    We show some qualitative comparisons between selected models in Figure 9

    Monocular Depth Prediction Monocular depth prediction is a longstanding task with very impressive in-the-wild results of recent SotA models [2, 11, 49, 51]. We show some qualitative comparisons between selected models in Figure 9. Due to training on massive datasets, current s...

  60. [69]

    As stated in the main paper, we perform a two-stage reconstruction:

    How important is camera calibration really? In this section we want to show some qualitative examples of in-the-wild footage with unknown intrinsics. As stated in the main paper, we perform a two-stage reconstruction:

  61. [70]

    Run the system without scale-optimization and optimize the camera intrinsics θ

  62. [71]

    (9) The benefit of camera calibration was quantitatively shown in [10]

    Use the now calibrated camera to run in P-RGBD mode and additionally optimize si and oi Since we need an initial estimate of the intrinsics, we assume a heuristic where for a pinhole camera f x= f y= (H + W ) /2 cx = W/2 cy = H/2 . (9) The benefit of camera calibration was qua...

  63. [72]

    We notice that even though monocular depth prediction networks allow highly detailed single-frame predictions, their usage on in-the-wild video is limited

    Failure Cases Due to the challenging unbounded outdoor setting on uncal- ibrated cameras, we quickly observed common limitations of our framework. We notice that even though monocular depth prediction networks allow highly detailed single-frame predictions, their usage on in-t...

  64. [73]

    We can then perform an intersection test to carve out a 3D volume across multiple views

    What did not work? We tried the following things unsuccessfully: • Multi-View Gaussian Splatting [5] backprojects crops of 2D appearance error into 3D by using the camera ray. We can then perform an intersection test to carve out a 3D volume across multiple views. This test id...

  65. [2019]

    4, 6, 7, 8, 12, 13, 15, 18

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.