Pith. sign in

REVIEW 5 major objections 6 minor 58 references

GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GSGTrack claims that jointly optimizing a 3D Gaussian splatting model and a confidence-weighted geometric graph makes 6DoF pose tracking and reconstruction of unknown objects possible from monocular RGB video, reporting ADD-S on HO3D…

desk verdict A plausible RGB-only pose tracking system with an important capability claim, but the evaluation is too thin and the per-frame mask issue must be resolved. read the letter →

arxiv 2412.02267 v1 pith:K6TLS5SQ submitted 2024-12-03 cs.CV cs.RO

Zhiyuan Chen , Fan Lu , Guo Yu , Bin Li , Sanqing Qu , Yuan Huang , Changhong Fu , Guang Chen This is my paper · ORCID
classification cs.CVcs.RO
keywords 6DoFposetrackingGaussiansplattingmonocularRGBvideoobjectreconstructionconfidence-weightedoptimizationgeometricgraphunknownobjectshand-objectocclusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GSGTrack sets out to prove that the 6DoF pose and the 3D shape of an unknown object can be tracked and reconstructed online from ordinary monocular RGB video, without accurate depth input. Its diagnosis is that the standard pipeline—estimate depth, then run RGBD tracking—fails because depth noise accumulates; the proposed alternative jointly optimizes pose and geometry, using 3D Gaussian Splatting as a continuously updated representation and a graph over historical frames with confidence-weighted geometric losses. On the HO3D benchmark the paper reports ADD-S rising from 24.67% for the best prior approach to 64.60%, with reconstruction Chamfer distance improving from 59.92 cm to 20.06 cm. If the claim holds, robots with monocular cameras could handle and model objects they have never seen before.

What carries the argument

The object representation is an online 3D Gaussian Splatting model—a set of anisotropic Gaussian spheres with position, covariance, color, and opacity, rendered through a differentiable rasterizer—that is extended frame by frame via entropy-guided sampling, photometric and depth supervision, and Gaussian pruning. Around it sits the geometric structure graph: nodes are frames carrying poses and depth maps, and edges carry pixel-aligned 3D pointmaps and confidence maps from a generalized stereo matching network. The load-bearing identity is the confidence-weighted geometry loss $L_{pg}=\sum_{e\in H}\sum_{v\in E_e}\sum_{i} C^{v,e}_i \|\chi^v_i - T_{e2w} X^{v,e}_i\|$, which down-weights untrusted points in the Chamfer distance and, together with a three-part image-pair pruning rule, filters out inconsistent edges. A distance-transform silhouette loss $L_s$ stabilizes pose refinement against photometric ambiguity by providing gradients even at low silhouette overlap.

What would settle it

One direct test is to take an HO3D sequence, keep the stereo pointmaps unchanged, and replace the confidence maps with random or inverted values; if tracking accuracy does not collapse, then the paper's central robustness mechanism—confidence-weighted geometric optimization and pruning—is not doing the claimed work. A second test is the sequence the paper itself flags as difficult, AP10, where the first frame lacks geometric cues: perturb the first-frame mask or scale slightly and show that the whole trajectory degrades, which would confirm the stated reliance on good initial matching.

Watch

Extended reading notes

Core claim

The central claim is that a monocular RGB video contains enough information to simultaneously track the 6DoF pose and build a textured 3D model of an unknown object, provided pose and geometry are optimized together rather than sequentially. The paper's discovery is that 3D Gaussian Splatting can serve as the online object representation, while a directed graph over historical frames—edges carrying stereo-matched pointmaps and confidence maps—provides geometry constraints that tolerate imperfect depth predictions. Confidence-weighted Chamfer distances and a three-way image-pair pruning strategy remove unreliable correspondences, and a differentiable silhouette loss stabilizes pose optimization against photometric ambiguity. The reported numbers on HO3D (ADD-S 64.60%, reconstruction Chamfer distance 20.06 cm) and OnePose (trajectory APE 7.36 cm) are presented as evidence that RGB-only tracking of unseen objects is feasible and accurate.

Load-bearing premise

The load-bearing premise is that the scores the stereo network attaches to each predicted 3D point honestly reflect whether that point's geometry is correct. If those scores are wrong on occluded or textureless pixels, the pruning and weighting steps can delete good image pairs and keep bad ones, and the claimed robustness would disappear.

Editorial extensions

If this is right

  • A robot with only an RGB camera could track and build a textured model of a novel object during manipulation, removing the need for depth sensors or pre-scanned CAD models.
  • The confidence-weighted geometric loss and the three-way image-pair pruning rules are separable components that could be transferred to other online reconstruction or SLAM systems that consume predicted depth.
  • The reconstruction metrics on HO3D (PSNR 25.92, SSIM 0.97) indicate that online monocular reconstruction can support appearance-based downstream tasks such as rendering and view planning.
  • The large gap between RGBD-based baselines and the RGB-only result on the same benchmark argues for adding monocular RGB protocols to future pose-tracking evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The first-frame mask and object scale are the only per-object supervision; an obvious extension is to couple the tracker with automatic segmentation and scale estimation, since errors there would likely dominate failures outside the benchmark.
  • Because the confidence maps come from one pretrained stereo network, a direct test of the paper's general idea is to swap in a different stereo model without retuning the pruning thresholds; success would show the confidence-weighting mechanism generalizes.
  • The paper's stated limitation for transparent objects suggests the next stress test is an explicit outlier rejection layer, since a per-point confidence weight cannot fix points that never had valid correspondences.
  • A broader consequence is that object-tracking benchmarks should include depth-free or depth-corrupted settings, since lightweight robot platforms increasingly rely on monocular perception.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. GSGTrack proposes a monocular RGB 6DoF object pose tracking and online reconstruction method for unknown objects. The method jointly optimizes an online 3D Gaussian Splatting representation and a graph-based geometric optimization, using DUSt3R generalized stereo matching to supply coarse pointmaps and confidence maps. An image pair pruning strategy, a silhouette loss, and a confidence-weighted geometric loss are introduced to handle noisy geometry and pose. The method is evaluated on 6 HO3D sequences and 5 OnePose sequences, with reported ADD-S/ADD, PSNR/SSIM, and Chamfer Distance metrics. The headline result is an HO3D ADD-S of 64.60% versus 24.67% for the best baseline.

Significance. If the claims are sustained, this would be a meaningful advance: RGB-only tracking and reconstruction of unknown objects without accurate depth, with large reported gains over the evaluated baselines. The pose evaluation uses independent ground truth (ADD/ADD-S with ground-truth object models) and is not circular; the ablation study covers the main components. However, the evidence is not yet commensurate with the claims: the dataset is small (11 sequences), the ablation is a single sequence, no error bars are provided, the per-frame mask source is undisclosed, ground-truth scale is used in the evaluation protocol, and the key motivating comparison (RGBD methods with estimated depth) is missing from the baseline list. The central idea is credible and timely, but the paper requires substantial clarification and additional evidence before the headline claims can be accepted.

major comments (5)
  1. [§3.1, §4.2, Supp. §6.1; Eq. (6)] The formulation in §3.1 states that the method receives only the first-frame mask M0 and scale s0, but the implementation as described consumes a mask in every frame: §4.2 says 'For each video frame, we use object segmentation for scaling and cropping,' Supp. §6.1 crops using 'the segmented mask as a reference,' and the silhouette loss in Eq. (6) uses ground-truth masks S for every optimized frame. If these are per-frame annotated masks (which HO3D provides), the tracker receives dense 2D object localization every frame, which is materially different from the first-frame-only problem. This is load-bearing for the 'RGB-only/first-frame-only' claim, and the authors must disclose the mask source (ground-truth per-frame masks, propagation from M0, or a video segmentation network) and quantify sensitivity to mask errors.
  2. [§4.1] The experimental protocol gives the method metric scale from ground truth. The HO3D implementation derives 'object scale estimates from ground-truth point clouds in the first frame,' and OnePose scale comes from '3D bounding box annotations.' This is more than the 2D projected size s0 stated in the problem formulation of §3.1. Since ADD/ADD-S are metric and depth is otherwise absent, using ground-truth first-frame 3D scale may mask a core difficulty of monocular tracking. Please state whether s0 is computed from ground-truth 3D data, and if so, report results with scale estimated from RGB-only cues.
  3. [Tables 1–3; §4.5] The quantitative evaluation rests on six HO3D sequences and five OnePose sequences, and the ablation in Table 3 reports a single sequence (the 'Ours' values 62.51/51.83 match the SB column of Table 1). No variance or error bars are given. With reported differences of tens of percentage points, the absence of variance information and the single-sequence ablation make it difficult to distinguish a robust method from one that is tuned to these sequences. Please report results over the full selected sequences (or a clearly defined larger split), with multiple runs or at least per-sequence standard errors.
  4. [§4.1; Fig. 1] The motivating comparison in Fig. 1 is BundleSDF with DepthAnything, but §4.1 does not include BundleSDF or any RGBD-based object tracker in the baseline list. Neither does it include a dedicated RGB-only object pose tracker. The claim that existing RGBD methods 'degenerate quickly' under estimated depth is therefore not quantitatively established by the experiments. Please add these comparisons or explicitly limit the claim to the SLAM and general-stereo baselines actually tested.
  5. [§3.4; Eq. (8); Supp. §6.2] Confidence maps from DUSt3R are used both to weight the geometric loss (Eq. (8)) and to prune image pairs (Sec. 3.4), with thresholds τ_c, τ_r, τ_t and a confidence cutoff of 2. The paper gives no calibration analysis showing that these confidences correlate with actual geometric error, especially under hand occlusion and low texture in HO3D. If the confidences are miscalibrated, the pruning strategy can remove useful edges and retain bad ones, so the claimed robustness to noisy geometry is not yet supported. Please report confidence-versus-error calibration and sensitivity to the pruning thresholds.
minor comments (6)
  1. [Eq. (5)] The depth loss is written without any absolute value or norm; as written, minimizing it is not a meaningful L1 depth loss. Please add the missing absolute value or clarify the intended norm.
  2. [§1] The phrase 'RGB-basd' in the contributions list should be 'RGB-based.'
  3. [§3.1] The heading 'Perliminary for 3DGS' should be 'Preliminary for 3DGS.'
  4. [§4.1; Ref. [32]] The DROID-SLAM author name is Jia Deng, not 'Jis Deng.'
  5. [Table 1] The column abbreviations AP, MPM, SB, and SM should be defined in the table caption; they are only inferable from Table 4.
  6. [Supp. §9] The supplementary limitation section admits failure on uniformly colored objects and sensitivity to first-frame matching, but the main text's conclusion does not mention these caveats. Please address this tension in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: pose tracking is evaluated against independent ground truth and no fitted parameter is renamed as a prediction.

full rationale

GSGTrack's derivation chain is a joint optimization over external inputs rather than a self-referential reduction. The 3DGS representation is initialized from DUSt3R pointmaps and confidence maps, which are fixed pretrained network outputs not trained on HO3D or OnePose, and pose is optimized by photometric, silhouette, and confidence-weighted geometric losses against these inputs and the first-frame mask/scale. None of Eqs. (4)-(8) defines the target pose as a function of the pose used for evaluation, and no parameter is fit to the ADD/ADD-S ground truth used in Table 1. The supplementary limitation passage (Sec. 9) is an honest caveat, not a circular justification. The one evaluation concern, the apparent use of per-frame segmentation masks for cropping and silhouette loss despite the claimed first-frame-only input, is an assumption and experimental-design gap that should be disclosed, but it does not reduce the pose output to the pose input, so it is not a circularity. Similarly, PSNR/SSIM on reconstruction frames is self-referential as an appearance metric, but the central pose claim does not depend on it.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method rests on two external pillars: DUSt3R's stereo priors (with their confidence estimates) and 3DGS rendering, plus a set of hand-tuned thresholds and hyperparameters. The first-frame scale and mask are treated as given, which is a significant input for a claimed RGB-only tracker. No new physical entities are introduced.

free parameters (5)
  • Pixel confidence cutoff for geometric loss = 2
    Pixels with DUSt3R confidence below 2 are excluded from the geometric graph loss (Supplementary Sec. 6.2); chosen by hand, no sensitivity analysis.
  • Edge confidence threshold tau_c
    Prunes image-pair edges when confidence-weighted mask average is low (Sec. 3.4); value not reported in the text.
  • Pose consistency thresholds tau_r, tau_t
    PnP-RANSAC relative pose deviations exceeding these thresholds cause pair pruning (Sec. 3.4); values not reported.
  • Voxel grid resolution K and max sampled points K/2
    Controls entropy-based Gaussian insertion downsampling (Sec. 3.3); value not reported.
  • Gaussian position learning rate = 0.000032
    Set lower than standard 3DGS because geometry optimization provides initial values (Supplementary Sec. 6.2); chosen by hand.
assumptions (4)
  • domain assumption DUSt3R confidence maps are a reliable proxy for geometric error across object views.
    Confidence-weighted pruning (Eq. 7) and geometry loss (Eq. 8) assume the stereo matcher's per-pixel confidence predicts accuracy; no calibration analysis is provided.
  • domain assumption Object scale and a first-frame mask are known from ground truth or external annotations.
    Scale is computed from first-frame GT depth/3D bounding box (Sec. 4.1); without metric scale the monocular pose is scale-ambiguous.
  • domain assumption Objects are rigid, opaque, sufficiently textured, and have no transparency.
    The paper admits failure on uniform-color and transparent objects (Supplementary Limitations); rigidity and texture are assumed throughout.
  • standard math 3D Gaussian Splatting rendering is an adequate differentiable model for joint pose and geometry optimization.
    The rendering equations (Eqs. 1-2) are taken from Kerbl et al. and used as the optimization backbone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos." pith.science (2026). https://pith.science/paper/K6TLS5SQ

@misc{pith2026241202267,
  author       = {Pith},
  title        = {Pith review of: GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6TLS5SQ}},
  note         = {Machine review of arXiv:2412.02267}
}
read the original abstract

Tracking the 6DoF pose of unknown objects in monocular RGB video sequences is crucial for robotic manipulation. However, existing approaches typically rely on accurate depth information, which is non-trivial to obtain in real-world scenarios. Although depth estimation algorithms can be employed, geometric inaccuracy can lead to failures in RGBD-based pose tracking methods. To address this challenge, we introduce GSGTrack, a novel RGB-based pose tracking framework that jointly optimizes geometry and pose. Specifically, we adopt 3D Gaussian Splatting to create an optimizable 3D representation, which is learned simultaneously with a graph-based geometry optimization to capture the object's appearance features and refine its geometry. However, the joint optimization process is susceptible to perturbations from noisy pose and geometry data. Thus, we propose an object silhouette loss to address the issue of pixel-wise loss being overly sensitive to pose noise during tracking. To mitigate the geometric ambiguities caused by inaccurate depth information, we propose a geometry-consistent image pair selection strategy, which filters out low-confidence pairs and ensures robust geometric optimization. Extensive experiments on the OnePose and HO3D datasets demonstrate the effectiveness of GSGTrack in both 6DoF pose tracking and object reconstruction.

Figures

Figures reproduced from arXiv: 2412.02267 by the authors.

Figure 1
Figure 1. We are tackling a challenging problem: tracking 6DoF [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed GSGTrack. To achieve accurate 6DoF object pose tracking without relying on precise depth infor￾mation, we propose a joint optimization framework. Starting with a video sequence, we preprocess consecutive frames by generating object masks and estimating coarse geometry. Next, we introduce an online 3DGS representation that facilitates continuous object reconstruction from incoming video frame… view at source ↗
Figure 3
Figure 3. Qualitative Comparison of GSGTrack and Baseline on HO3D. Left: 6-DOF pose tracking with green and yellow boxes showing ground truth and estimated poses, respectively. Right: front and back views of reconstruction results, highlighting the object’s geometric structure. Due to hand occlusions, black hand-shaped artifacts appear, obscuring parts of the object. Our reconstruction corrects the color divergence between gr… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: We visualize the Relative Rotation Error (RRE) of dif [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of our Gaussian pruning strategy on reconstruc [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization for the objects of in HO3D dataset [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Visualization for the objects of in OnePose dataset [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Qualitative Comparison of GSGTrack and Baseline on HO3D(Seq-AP11). Left: 6-DOF pose tracking with green and yellow boxes showing ground truth and estimated poses, respectively. Right: front and back views of reconstruction results, highlighting the object’s geometric s…
Figure 9
Figure 9. Figure 9: Recall curve of ADD-S (left) and ADD (right) metric on HO3D Dataset.. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Qualitative Comparison of GSGTrack and Baseline on HO3D(Seq-MPM14). Left: 6-DOF pose tracking with green and yellow boxes showing ground truth and estimated poses, respectively. Right: front and back views of reconstruction results, highlighting the object’s geometric…
Figure 11
Figure 11. Figure 11: Qualitative Comparison of GSGTrack and Baseline on OnePose. Our method demonstrates superior object reconstruction quality [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 50 canonical work pages

  1. [1]

    F ´acil, Javier Civera, and Jos ´e Neira

    Berta Bescos, Jos ´e M. F ´acil, Javier Civera, and Jos ´e Neira. DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1–9. IEEE, 2018. 2

    work page 2018

  2. [2]

    F ´acil, Javier Civera, and Jos ´e Neira

    Berta Bescos, Jos ´e M. F ´acil, Javier Civera, and Jos ´e Neira. Dynaslam2: Real-time dense monocular slam with dynamic object removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–9, 2020. 2

    work page 2020

  3. [3]

    Gs-pose: Generalizable segmentation-based 6d object pose estimation with 3d gaussian splatting.arXiv preprint arXiv:2403.10683,

    Dingding Cai, Janne Heikkil ¨a, and Esa Rahtu. Gs-pose: Generalizable segmentation-based 6d object pose estimation with 3d gaussian splatting.arXiv preprint arXiv:2403.10683,

    arXiv

  4. [4]

    Reconstruct locally, localize glob- ally: A model free method for object pose estimation

    Ming Cai and Ian Reid. Reconstruct locally, localize glob- ally: A model free method for object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 3153–3163, 2020. 2

    work page 2020

  5. [5]

    3d-r2n2: A unified approach for single and multi- view 3d object reconstruction

    Christopher B Choy, Yinda Xu, Junhyuk Gwak, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi- view 3d object reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3352–3361, 2016. 2

    work page 2016

  6. [6]

    3D Reconstruction of Simple Objects from A Single View Silhouette Image

    Xinhan Di and Pengqian Yu. 3d reconstruction of simple objects from a single view silhouette image. arXiv preprint arXiv:1701.04752, 2017. 5

    work page Pith review arXiv 2017

  7. [7]

    Supersplat: 3d gaussian splat editor, 2024

    Will Eastcott. Supersplat: 3d gaussian splat editor, 2024. Accessed: 2024-11-19. 1

    work page 2024

  8. [8]

    Instantsplat: Sparse-view sfm-free gaussian splatting in seconds, 2024

    Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, Zhangyang Wang, and Yue Wang. Instantsplat: Sparse-view sfm-free gaussian splatting in seconds, 2024. 2, 4

    work page 2024

Show all 58 references
  1. [9]

    Efros, and Xiaolong Wang

    Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. Colmap-free 3d gaussian splat- ting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 20796– 20805, 2024. 2

    2024

  2. [10]

    Mobile ar depth estimation: Challenges & prospects – extended ver- sion

    Ashkan Ganj, Yiqin Zhao, Hang Su, and Tian Guo. Mobile ar depth estimation: Challenges & prospects – extended ver- sion. arXiv preprint arXiv:2310.14437, 2023. 1

    2023 arXiv

  3. [11]

    Honnotate: A method for 3d annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vin- cent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 3196–3206, 2020. 2, 6

    2020

  4. [12]

    Fs6d: Few-shot 6d pose estimation of novel objects

    Yisheng He, Yao Wang, Haoqiang Fan, Jian Sun, and Qifeng Chen. Fs6d: Few-shot 6d pose estimation of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6814–6824,

  5. [13]

    Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes

    Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Ste- fan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian Conference on Computer Vision (ACCV) , pa...

  6. [14]

    Real-time perception meets reactive motion gener- ation

    Daniel Kappler, Franziska Meier, Jan Issac, Jim Main- price, Cristina Garcia Cifuentes, Manuel W ¨uthrich, Vin- cent Berenz, Stefan Schaal, Nathan Ratliff, and Jeannette Bohg. Real-time perception meets reactive motion gener- ation. IEEE Robotics and Automation Letters , 3(3):...

    2018

  7. [15]

    Sd-6d: Making rgb-based 3d de- tection and 6d pose estimation great again

    Wadim Kehl, Fabian Manhardt, Federico Tombari, Nassir Navab, and Slobodan Ilic. Sd-6d: Making rgb-based 3d de- tection and 6d pose estimation great again. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1521–1529, 2017. 2

    2017

  8. [16]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4):1–14, 2023. 2, 3, 7, 8

    2023

  9. [17]

    PTAM: Real-time tracking and mapping for augmented reality

    Georg Klein and David Murray. PTAM: Real-time tracking and mapping for augmented reality. In IEEE/ACM Interna- tional Symposium on Mixed and Augmented Reality , pages 225–234. IEEE, 2007. 2

    2007

  10. [18]

    CosyPose: Consistent multi-view multi- object 6d pose estimation

    Yann Labb ´e, Mathieu Caron, Mathieu Aubry, Josef Sivic, and Ivan Laptev. CosyPose: Consistent multi-view multi- object 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 574–591,

  11. [19]

    Nerf-pose: A first-reconstruct-then- regress approach for weakly-supervised 6d object pose esti- mation

    Fu Li, Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, and Slobodan Ilic. Nerf-pose: A first-reconstruct-then- regress approach for weakly-supervised 6d object pose esti- mation. arXiv preprint arXiv:2203.04802, 2022. 2

    2022 arXiv

  12. [20]

    Barf: Bundle-adjusting neural radiance fields

    Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Si- mon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6548–6557, 2021. 2

    2021

  13. [21]

    Onepose: One-shot ob- ject pose estimation without cad models

    Jingwen Lin, Ziang Wang, Xinyu Yu, Siyu Zhu, Hujun Bao, Xiaowei Zhou, and Guofeng Wang. Onepose: One-shot ob- ject pose estimation without cad models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6827–6836, 2022. 2, 6

    2022

  14. [22]

    Object gaussian for monocu- lar 6d pose estimation from sparse views

    Luqing Luo, Shichu Sun, Jiangang Yang, Linfang Zheng, Jinwei Du, and Jian Liu. Object gaussian for monocu- lar 6d pose estimation from sparse views. arXiv preprint arXiv:2409.02581, 2024. 2

    2024 arXiv

  15. [23]

    Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and An- drew J. Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1234–1243, 2024. 4, 6, 7, 8, 2

    2024

  16. [24]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), pages 676–691, 2020. 2

    2020

  17. [25]

    Raul Mur-Artal and J. D. Tard ´os. ORB-SLAM2: An open- source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 33(5):1255–1262,

  18. [26]

    Quadricslam: Dual quadrics from object detections as land- marks in object-oriented slam

    Lachlan Nicholson, Michael Milford, and Niko S ¨underhauf. Quadricslam: Dual quadrics from object detections as land- marks in object-oriented slam. IEEE Robotics and Automa- tion Letters, 3(4):3540–3547, 2018. 2

    2018

  19. [27]

    Pix2Pose: Pixel-wise coordinate regression of objects for 6d pose es- timation

    Kiru Park, Timothy Patten, and Markus Vincze. Pix2Pose: Pixel-wise coordinate regression of objects for 6d pose es- timation. In Proceedings of the IEEE International Confer- ence on Computer Vision (ICCV) , pages 7668–7677, 2019. 2

    2019

  20. [28]

    Pvnet: Pixel-wise voting network for 6dof pose estima- tion

    Sida Peng, Yuan Liu, Qixing Huang, Xibin Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estima- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 4561– 4570, 2019. 1

    2019

  21. [29]

    Cornet: Generic 3d corners for 6d pose estimation of new objects without retraining

    Giorgia Pitteri, Slobodan Ilic, and Vincent Lepetit. Cornet: Generic 3d corners for 6d pose estimation of new objects without retraining. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision Workshops (ICCVW), pages 0–0, 2019. 1

    2019

  22. [30]

    A unified mpc framework for whole- body dynamic locomotion and manipulation

    Jean-Pierre Sleiman, Farbod Farshidian, Maria Vittoria Min- niti, and Marco Hutter. A unified mpc framework for whole- body dynamic locomotion and manipulation. IEEE Robotics and Automation Letters, 6(2):4688–4695, 2021. 1

    2021

  23. [31]

    Evaluation for odometry

    J ¨urgen Sturm, Nils Engelhard, Frank Endres, Wolfram Bur- gard, and Daniel Cremers. Evaluation for odometry. Pro- ceedings of the IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS) , pages 4206–4212, 2012. 7

    2012

  24. [32]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras

    Zachary Teed and Jis Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8348–8357, 2021. 6, 7, 8

    2021

  25. [33]

    He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),...

    2019

  26. [34]

    DUSt3R: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3R: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 20697–20709, 2024. 2, 4, 5, 6, 7, 8

    2024

  27. [35]

    Pixel2mesh: Generating 3d mesh models from single rgb images

    Xiu Li Wang, Zhi Zhang, Zhen Zhang, Yi Yang, Yichen Yu, and Lei Zhang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision, pages 54–70, 2018. 2

    2018

  28. [36]

    Image quality assessment: From error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si- moncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Process- ing, 13(4):600–612, 2004. 7

    2004

  29. [37]

    Splat3r: Zero-shot gaussian splatting from uncali- brated image pairs, 2024

    Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Ro- main Br´egier, Yohann Cabon, Vaibhav Arora, Leonid Ants- feld, Boris Chidlovskii, Gabriela Csurka, and J ´erˆome Re- vaud. Splat3r: Zero-shot gaussian splatting from uncali- brated image pairs, 2024. 2

    2024

  30. [38]

    Bowen Wen and Kostas E. Bekris. Bundletrack: 6d pose tracking for novel objects without instance or category-level 3d models. In IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), 2021. 1, 2

    2021

  31. [39]

    Catgrasp: Learning category-level task-relevant grasping in clutter from simulation

    Bowen Wen, Wenzhao Lian, Kostas Bekris, and Stefan Schaal. Catgrasp: Learning category-level task-relevant grasping in clutter from simulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 6401–6408. IEEE, 2022. 1

    2022

  32. [40]

    Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects

    Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ¨uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte...

    2023

  33. [41]

    Visualsfm: A visual structure from motion sys- tem

    Cheng Wu. Visualsfm: A visual structure from motion sys- tem. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8, 2011. 7, 8

    2011

  34. [42]

    Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes

    Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Proceedings of Robotics: Science and Systems (RSS), 2018. 1, 7

    2018

  35. [43]

    Sparsegs: Real- time 360° sparse view synthesis using gaussian splatting

    Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real- time 360° sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206, 2023. 4

    2023 arXiv

  36. [44]

    Uniquadric: A slam backend for un- known rigid object 3d tracking and light-weight modeling

    Linghao Yang, Yanmin Wu, Yu Deng, Rui Tian, Xinggang Hu, and Tiefeng Ma. Uniquadric: A slam backend for un- known rigid object 3d tracking and light-weight modeling. arXiv preprint arXiv:2309.17036, 2023. 1

    2023 arXiv

  37. [45]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1

    2024

  38. [46]

    Cubeslam: Monocular 3-d object slam

    Shichao Yang and Sebastian Scherer. Cubeslam: Monocular 3-d object slam. IEEE Transactions on Robotics, 35(4):925– 938, 2019. 2

    2019

  39. [47]

    Deepmvs: Learning multi-view stereopsis

    Yao Yao, Zhi Li, Yifan Yang, Ming-Hsuan Yang, and Zhen Xu. Deepmvs: Learning multi-view stereopsis. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821–2829, 2018. 2

    2018

  40. [48]

    Noposplat: No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images

    Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng. Noposplat: No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. In arXiv preprint arXiv:2410.24207,

    arXiv

  41. [49]

    Shape-iou: More accurate metric considering bounding box shape and scale

    Hao Zhang and Shuaijie Zhang. Shape-iou: More accurate metric considering bounding box shape and scale. arXiv preprint arXiv:2312.17663, 2023. 5

    2023 arXiv

  42. [50]

    Neuralrecon: Real-time coherent 3d reconstruc- tion from monocular video

    Zhi Zhang, Sifei Liu, Yi Yang, Zhen Zhang, Yichen Yu, and Lei Zhang. Neuralrecon: Real-time coherent 3d reconstruc- tion from monocular video. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15510–15519, 2021. 2

    2021

  43. [51]

    Drivefeedforward: Unleashing generalization of end-to-end autonomous driving with controllable long video generation

    Zhi Zhang, Sifei Liu, Yi Yang, Zhen Zhang, Yichen Yu, and Lei Zhang. Drivefeedforward: Unleashing generalization of end-to-end autonomous driving with controllable long video generation. In arXiv preprint arXiv:2406.01349, 2023. 2

    2023 arXiv

  44. [52]

    Zhao et al

    Y . Zhao et al. Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9378–9388,

  45. [54]

    Data Preprocessing During the data preprocessing stage, for the segmented video image Ft, we first enlarge the image to reduce the relative proportion of background noise

    Implementation details 6.1. Data Preprocessing During the data preprocessing stage, for the segmented video image Ft, we first enlarge the image to reduce the relative proportion of background noise. Subsequently, us- ing the segmented mask as a reference, we crop the image so...

  46. [55]

    Metrics To evaluate the results of the algorithm, we assess both 6DoF pose tracking and object reconstruction. For the 6DoF pose tracking results, we compute the Area Under the Curve (AUC) percentages for the ADD and ADD-S metrics, ADD = 1 |M| X x∈M ∥(Rx + t) − ( ˜Rx + ˜t)∥2, ...

  47. [56]

    6, we selected 6 representative video sequences from the HO3D dataset, which include 4 dy- namic objects

    Datasets As shown in the Fig. 6, we selected 6 representative video sequences from the HO3D dataset, which include 4 dy- namic objects. Each scene contains approximately 1,000 frames of data, featuring dynamic objects and hands inter- acting with them. The scale information fo...

  48. [57]

    8 and Fig

    Limitation Although our method demonstrates greater robustness than the baseline algorithm in handling low-textured objects and occlusions (as shown in Fig. 8 and Fig. 10), it performs poorly when dealing with uniformly colored objects that lack geometric, color, or texture fe...

  49. [58]

    Broader Impact The GSGTrack framework introduces a significant leap for- ward in the field of 6-DoF pose tracking and 3D object re- construction, particularly for applications relying solely on monocular RGB video data. By eliminating the reliance on accurate depth information...

  50. [2023]

    Furthermore, we provide qualitative results for challenging scenarios in the datasets, analyze the limitations of the proposed algorithm, and discuss its broader impact

    1 GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos Supplementary Material In this supplementary material, we provide the imple- mentation details of the experiments, along with the al- gorithm evaluation metrics and comprehensive information about the d...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.