{"id":"bf1df946-c661-452d-8958-3852cd5b4a0f","arxiv_id":"2412.02267","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"GSGTrack jointly optimizes Gaussian Splatting geometry and object pose to track unknown objects in RGB video, reporting large accuracy gains over SLAM baselines.","lead":"A new method, GSGTrack, tracks the 6D pose and builds a 3D model of an unknown object from a monocular RGB video, without needing depth sensors. It jointly optimizes Gaussian Splatting geometry and object pose, and reports large gains over general SLAM baselines on HO3D and OnePose.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed first-frame-only input contradicts the apparent per-frame mask usage in preprocessing and in the silhouette loss; if HO3D masks are per-frame annotations, the evaluation solves an easier problem than the one claimed.","rationale":"Reader's weakest assumption is DUSt3R confidence calibration. I partially agree: Eq. (7)-(8) and Sec. 3.4's pruning do lean on confidences, and no calibration analysis is given. However, the per-frame mask issue is more directly evidenced and more consequential for the central claim. The paper's own text shows segmentation masks are used for every frame, yet the problem formulation says only M0 is provided. If those masks are per-frame annotations, the method is not purely RGB + first-frame mask; it receives dense per-frame object localization through the silhouette loss and cropping. That would change the problem being solved rather than merely degrade a component. The confidence concern is contingent: even if confidences are miscalibrated, the ablations show pruning and confidence weighting help on the reported sequences, and photometric/silhouette losses may compensate for imperfect geometry selection. The mask concern is directly checkable and should be resolved first. The reader already asks for mask-source specification and code; my read reinforces that as a condition, so the verdict remains CONDITIONAL (UNCHANGED). Given the small sequence count and lack of error bars, extra care is warranted before accepting the headline numbers.","tokens_in":16079,"tokens_out":8480,"duration_ms":95600,"concrete_test":"Ask the authors to (a) state whether the HO3D masks used in Sec. 4.2 are per-frame ground-truth/annotations or propagated from M0, and (b) rerun the HO3D evaluation with masks propagated from M0 using an off-the-shelf video segmentation model (e.g., SAM2) while keeping all other components fixed; if ADD-S drops substantially relative to Table 1, the reported numbers reflect per-frame mask supervision rather than the claimed first-frame-only input. A second, cheaper check is to corrupt the per-frame masks with random occlusions or affine noise and measure the drop in ADD-S.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GSGTrack tracks and reconstructs unknown objects from monocular RGB video with only the first-frame mask M0 and scale s0 as extra inputs (Sec. 3.1). The evaluation pipeline, however, appears to consume a mask in every frame. Sec. 4.2 states 'For each video frame, we use object segmentation for scaling and cropping'; the supplement (Sec. 6.1) says 'for the segmented video image F_t ... using the segmented mask as a reference, we crop the image'; and Sec. 4.4 says a video segmentation network is used to isolate the object for OnePose. The paper never states whether the HO3D masks taken 'from BundleSDF bounding boxes' are per-frame ground-truth/annotated masks or are propagated automatically from M0. This is load-bearing because the silhouette loss Eq. (6) and the mask-based cropping give the tracker per-frame object localization: if the masks are annotated, the method receives dense 2D object correspondences every frame, which is materially different from the first-frame-only problem formulated in Sec. 3.1. The authors should disclose the mask source and quantify mask-error sensitivity before the headline RGB-only/first-frame-only claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"GSGTrack proposes a monocular RGB 6DoF object pose tracking and online reconstruction method for unknown objects. The method jointly optimizes an online 3D Gaussian Splatting representation and a graph-based geometric optimization, using DUSt3R generalized stereo matching to supply coarse pointmaps and confidence maps. An image pair pruning strategy, a silhouette loss, and a confidence-weighted geometric loss are introduced to handle noisy geometry and pose. The method is evaluated on 6 HO3D sequences and 5 OnePose sequences, with reported ADD-S/ADD, PSNR/SSIM, and Chamfer Distance metrics. The headline result is an HO3D ADD-S of 64.60% versus 24.67% for the best baseline.","tokens_in":16291,"tokens_out":7290,"duration_ms":75245,"significance":"If the claims are sustained, this would be a meaningful advance: RGB-only tracking and reconstruction of unknown objects without accurate depth, with large reported gains over the evaluated baselines. The pose evaluation uses independent ground truth (ADD/ADD-S with ground-truth object models) and is not circular; the ablation study covers the main components. However, the evidence is not yet commensurate with the claims: the dataset is small (11 sequences), the ablation is a single sequence, no error bars are provided, the per-frame mask source is undisclosed, ground-truth scale is used in the evaluation protocol, and the key motivating comparison (RGBD methods with estimated depth) is missing from the baseline list. The central idea is credible and timely, but the paper requires substantial clarification and additional evidence before the headline claims can be accepted.","major_comments":[{"comment":"The formulation in §3.1 states that the method receives only the first-frame mask M0 and scale s0, but the implementation as described consumes a mask in every frame: §4.2 says 'For each video frame, we use object segmentation for scaling and cropping,' Supp. §6.1 crops using 'the segmented mask as a reference,' and the silhouette loss in Eq. (6) uses ground-truth masks S for every optimized frame. If these are per-frame annotated masks (which HO3D provides), the tracker receives dense 2D object localization every frame, which is materially different from the first-frame-only problem. This is load-bearing for the 'RGB-only/first-frame-only' claim, and the authors must disclose the mask source (ground-truth per-frame masks, propagation from M0, or a video segmentation network) and quantify sensitivity to mask errors.","section":"§3.1, §4.2, Supp. §6.1; Eq. (6)"},{"comment":"The experimental protocol gives the method metric scale from ground truth. The HO3D implementation derives 'object scale estimates from ground-truth point clouds in the first frame,' and OnePose scale comes from '3D bounding box annotations.' This is more than the 2D projected size s0 stated in the problem formulation of §3.1. Since ADD/ADD-S are metric and depth is otherwise absent, using ground-truth first-frame 3D scale may mask a core difficulty of monocular tracking. Please state whether s0 is computed from ground-truth 3D data, and if so, report results with scale estimated from RGB-only cues.","section":"§4.1"},{"comment":"The quantitative evaluation rests on six HO3D sequences and five OnePose sequences, and the ablation in Table 3 reports a single sequence (the 'Ours' values 62.51/51.83 match the SB column of Table 1). No variance or error bars are given. With reported differences of tens of percentage points, the absence of variance information and the single-sequence ablation make it difficult to distinguish a robust method from one that is tuned to these sequences. Please report results over the full selected sequences (or a clearly defined larger split), with multiple runs or at least per-sequence standard errors.","section":"Tables 1–3; §4.5"},{"comment":"The motivating comparison in Fig. 1 is BundleSDF with DepthAnything, but §4.1 does not include BundleSDF or any RGBD-based object tracker in the baseline list. Neither does it include a dedicated RGB-only object pose tracker. The claim that existing RGBD methods 'degenerate quickly' under estimated depth is therefore not quantitatively established by the experiments. Please add these comparisons or explicitly limit the claim to the SLAM and general-stereo baselines actually tested.","section":"§4.1; Fig. 1"},{"comment":"Confidence maps from DUSt3R are used both to weight the geometric loss (Eq. (8)) and to prune image pairs (Sec. 3.4), with thresholds τ_c, τ_r, τ_t and a confidence cutoff of 2. The paper gives no calibration analysis showing that these confidences correlate with actual geometric error, especially under hand occlusion and low texture in HO3D. If the confidences are miscalibrated, the pruning strategy can remove useful edges and retain bad ones, so the claimed robustness to noisy geometry is not yet supported. Please report confidence-versus-error calibration and sensitivity to the pruning thresholds.","section":"§3.4; Eq. (8); Supp. §6.2"}],"minor_comments":[{"comment":"The depth loss is written without any absolute value or norm; as written, minimizing it is not a meaningful L1 depth loss. Please add the missing absolute value or clarify the intended norm.","section":"Eq. (5)"},{"comment":"The phrase 'RGB-basd' in the contributions list should be 'RGB-based.'","section":"§1"},{"comment":"The heading 'Perliminary for 3DGS' should be 'Preliminary for 3DGS.'","section":"§3.1"},{"comment":"The DROID-SLAM author name is Jia Deng, not 'Jis Deng.'","section":"§4.1; Ref. [32]"},{"comment":"The column abbreviations AP, MPM, SB, and SM should be defined in the table caption; they are only inferable from Table 4.","section":"Table 1"},{"comment":"The supplementary limitation section admits failure on uniformly colored objects and sensitivity to first-frame matching, but the main text's conclusion does not mention these caveats. Please address this tension in the main text.","section":"Supp. §9"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The main risk is the gap between the stated input assumptions (first-frame mask and scale only) and the evaluation protocol (per-frame masks and ground-truth scale). This is a factual issue that the authors can resolve by disclosure and additional experiments. I would not reject at this stage, but the next revision must resolve it before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GSGTrack is a sensible engineering integration that deserves serious review, but the current evidence is thinner than the headline numbers suggest. The core idea—jointly optimizing an online 3DGS representation, a BundleSDF-style geometry graph, and DUSt3R stereo priors for RGB-only tracking of unknown objects—is a real combination that I have not seen before. The reported ADD-S jump on HO3D (24.67 to 64.60) is striking, and the pose evaluation uses independent ground truth. The limitations section is honest about failure cases. So this is not a toy paper.\n\nThe soft spots are mostly in the evaluation. Six HO3D sequences and five OnePose sequences is a small testbed. The ablation is on a single sequence, with no error bars. The baseline set omits the most natural comparisons: dedicated RGBD trackers like BundleSDF fed with estimated depth, which is exactly the motivating scenario in the introduction. The OnePose evaluation uses camera trajectory metrics rather than object pose, which is a reasonable proxy but not the same thing. These issues make the headline gains hard to trust, though they do not sink the method.\n\nThe bigger issue is the mask. The problem statement says only the first-frame mask and scale are given (Sec. 3.1), but Sec. 4.2 says 'For each video frame, we use object segmentation for scaling and cropping,' and the supplement describes cropping each frame using a segmented mask. The silhouette loss (Eq. 6) requires a per-frame mask. If the HO3D masks are annotated per-frame, the method receives dense 2D object localization every frame, which is materially easier than the claimed first-frame-only setting. The paper never states the mask source. This is a load-bearing ambiguity, not a minor omission. The authors need to disclose it and ideally ablate with predicted masks only.\n\nA smaller concern: the DUSt3R confidence maps are used heavily, both for pruning and for weighting the geometry loss, but there is no calibration analysis. That could be a minor weakness, but it is not central.\n\nBottom line: the paper is worth engaging with. I would send it to peer review, with the expectation that the authors clarify the mask situation, add task-appropriate baselines, and report variance across sequences and runs. Until then, I would not cite the headline numbers as established.","headline":"A plausible RGB-only pose tracking system with an important capability claim, but the evaluation is too thin and the per-frame mask issue must be resolved.","tokens_in":16900,"tokens_out":3018,"would_cite":false,"duration_ms":31693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GSGTrack claims that jointly optimizing a 3D Gaussian splatting model and a confidence-weighted geometric graph makes 6DoF pose tracking and reconstruction of unknown objects possible from monocular RGB video, reporting ADD-S on HO3D…","keywords":["6DoF pose tracking","Gaussian splatting","monocular RGB video","object reconstruction","confidence-weighted optimization","geometric graph","unknown objects","hand-object occlusion"],"falsifier":"One direct test is to take an HO3D sequence, keep the stereo pointmaps unchanged, and replace the confidence maps with random or inverted values; if tracking accuracy does not collapse, then the paper's central robustness mechanism—confidence-weighted geometric optimization and pruning—is not doing the claimed work. A second test is the sequence the paper itself flags as difficult, AP10, where the first frame lacks geometric cues: perturb the first-frame mask or scale slightly and show that the whole trajectory degrades, which would confirm the stated reliance on good initial matching.","tokens_in":15820,"feed_emoji":"🎯","tokens_out":9643,"duration_ms":93321,"temperature":0.7,"pith_summary":"GSGTrack sets out to prove that the 6DoF pose and the 3D shape of an unknown object can be tracked and reconstructed online from ordinary monocular RGB video, without accurate depth input. Its diagnosis is that the standard pipeline—estimate depth, then run RGBD tracking—fails because depth noise accumulates; the proposed alternative jointly optimizes pose and geometry, using 3D Gaussian Splatting as a continuously updated representation and a graph over historical frames with confidence-weighted geometric losses. On the HO3D benchmark the paper reports ADD-S rising from 24.67% for the best prior approach to 64.60%, with reconstruction Chamfer distance improving from 59.92 cm to 20.06 cm. If the claim holds, robots with monocular cameras could handle and model objects they have never seen before.","feed_headline":"RGB-only tracker lifts ADD-S from 24.7% to 64.6%","feed_subtitle":"A Gaussian-splatting joint optimization tracks and reconstructs unseen objects without depth cameras.","key_machinery":"The object representation is an online 3D Gaussian Splatting model—a set of anisotropic Gaussian spheres with position, covariance, color, and opacity, rendered through a differentiable rasterizer—that is extended frame by frame via entropy-guided sampling, photometric and depth supervision, and Gaussian pruning. Around it sits the geometric structure graph: nodes are frames carrying poses and depth maps, and edges carry pixel-aligned 3D pointmaps and confidence maps from a generalized stereo matching network. The load-bearing identity is the confidence-weighted geometry loss $L_{pg}=\\sum_{e\\in H}\\sum_{v\\in E_e}\\sum_{i} C^{v,e}_i \\|\\chi^v_i - T_{e2w} X^{v,e}_i\\|$, which down-weights untrusted points in the Chamfer distance and, together with a three-part image-pair pruning rule, filters out inconsistent edges. A distance-transform silhouette loss $L_s$ stabilizes pose refinement against photometric ambiguity by providing gradients even at low silhouette overlap.","core_discovery":"The central claim is that a monocular RGB video contains enough information to simultaneously track the 6DoF pose and build a textured 3D model of an unknown object, provided pose and geometry are optimized together rather than sequentially. The paper's discovery is that 3D Gaussian Splatting can serve as the online object representation, while a directed graph over historical frames—edges carrying stereo-matched pointmaps and confidence maps—provides geometry constraints that tolerate imperfect depth predictions. Confidence-weighted Chamfer distances and a three-way image-pair pruning strategy remove unreliable correspondences, and a differentiable silhouette loss stabilizes pose optimization against photometric ambiguity. The reported numbers on HO3D (ADD-S 64.60%, reconstruction Chamfer distance 20.06 cm) and OnePose (trajectory APE 7.36 cm) are presented as evidence that RGB-only tracking of unseen objects is feasible and accurate.","pith_inferences":["The first-frame mask and object scale are the only per-object supervision; an obvious extension is to couple the tracker with automatic segmentation and scale estimation, since errors there would likely dominate failures outside the benchmark.","Because the confidence maps come from one pretrained stereo network, a direct test of the paper's general idea is to swap in a different stereo model without retuning the pruning thresholds; success would show the confidence-weighting mechanism generalizes.","The paper's stated limitation for transparent objects suggests the next stress test is an explicit outlier rejection layer, since a per-point confidence weight cannot fix points that never had valid correspondences.","A broader consequence is that object-tracking benchmarks should include depth-free or depth-corrupted settings, since lightweight robot platforms increasingly rely on monocular perception."],"forward_implications":["A robot with only an RGB camera could track and build a textured model of a novel object during manipulation, removing the need for depth sensors or pre-scanned CAD models.","The confidence-weighted geometric loss and the three-way image-pair pruning rules are separable components that could be transferred to other online reconstruction or SLAM systems that consume predicted depth.","The reconstruction metrics on HO3D (PSNR 25.92, SSIM 0.97) indicate that online monocular reconstruction can support appearance-based downstream tasks such as rendering and view planning.","The large gap between RGBD-based baselines and the RGB-only result on the same benchmark argues for adding monocular RGB protocols to future pose-tracking evaluations."],"supporting_citations":[{"why":"Supplies the differentiable 3D Gaussian splatting representation and rasterizer that the online object model is built on.","marker":"[16]"},{"why":"Provides the generalized stereo matching network that predicts pixel-aligned pointmaps and confidence maps used for initialization, pruning, and geometry optimization.","marker":"[34]"},{"why":"Defines the closest RGBD tracking-and-reconstruction baseline and the evaluation protocol for pose and shape metrics.","marker":"[40]"},{"why":"Supplies the GS-SLAM baseline and the Gaussian pruning and density-control ideas adapted for online object tracking.","marker":"[23]"},{"why":"DROID-SLAM is a monocular SLAM baseline whose optical-flow and depth-prior tracking is compared on object-centered scenes.","marker":"[32]"},{"why":"HO3D dataset provides hand-object RGBD videos from which only RGB is used, the main benchmark for the claimed results.","marker":"[11]"},{"why":"OnePose dataset provides monocular object-centered videos used to evaluate camera-trajectory accuracy and reconstruction quality.","marker":"[21]"}],"fun_headline_variants":["RGB-only pose tracking hits 64.6% ADD-S on HO3D","Gaussian splatting lets RGB video track unseen objects","Joint optimization of pose and geometry beats depth","Silhouette loss stabilizes RGB-based 6DoF tracking","No depth needed: GSGTrack reconstructs while tracking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the scores the stereo network attaches to each predicted 3D point honestly reflect whether that point's geometry is correct. If those scores are wrong on occluded or textureless pixels, the pruning and weighting steps can delete good image pairs and keep bad ones, and the claimed robustness would disappear.","fun_headline_variants_meta":{"raw":{"variants":["RGB-only pose tracking hits 64.6% ADD-S on HO3D","Gaussian splatting lets RGB video track unseen objects","Joint optimization of pose and geometry beats depth","Silhouette loss stabilizes RGB-based 6DoF tracking","No depth needed: GSGTrack reconstructs while tracking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1739,"prompt_tokens":944,"completion_tokens":795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":710}},"tokens_in":560,"tokens_out":795,"duration_ms":8069,"temperature":1.0,"reasoning_tokens":710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:38:50.225000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One direct test is to take an HO3D sequence, keep the stereo pointmaps unchanged, and replace the confidence maps with random or inverted values; if tracking accuracy does not collapse, then the paper's central robustness mechanism—confidence-weighted geometric optimization and pruning—is not doing the claimed work. A second test is the sequence the paper itself flags as difficult, AP10, where the first frame lacks geometric cues: perturb the first-frame mask or scale slightly and show that the whole trajectory degrades, which would confirm the stated reliance on good initial matching.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the differentiable 3D Gaussian splatting representation and rasterizer that the online object model is built on."},{"cited_title":"DUSt3R: Geometric 3d vision made easy","cited_arxiv_id":null,"evidence_quote":"Provides the generalized stereo matching network that predicts pixel-aligned pointmaps and confidence maps used for initialization, pruning, and geometry optimization."},{"cited_title":"Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects","cited_arxiv_id":null,"evidence_quote":"Defines the closest RGBD tracking-and-reconstruction baseline and the evaluation protocol for pose and shape metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GS-SLAM baseline and the Gaussian pruning and density-control ideas adapted for online object tracking."},{"cited_title":"Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras","cited_arxiv_id":null,"evidence_quote":"DROID-SLAM is a monocular SLAM baseline whose optical-flow and depth-prior tracking is compared on object-centered scenes."},{"cited_title":"Honnotate: A method for 3d annotation of hand and object poses","cited_arxiv_id":null,"evidence_quote":"HO3D dataset provides hand-object RGBD videos from which only RGB is used, the main benchmark for the claimed results."},{"cited_title":"Onepose: One-shot ob- ject pose estimation without cad models","cited_arxiv_id":null,"evidence_quote":"OnePose dataset provides monocular object-centered videos used to evaluate camera-trajectory accuracy and reconstruction quality."}],"review_version":1}