{"id":"b04b7b55-f24d-43b4-af30-9ded79668cfe","arxiv_id":"2505.11439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A stereo-based pipeline with Mask R-CNN segmentation and RAFT-Stereo depth lets a modified SAM-6D beat FoundationPose for zero-shot 6D pose estimation of a surgical needle driver in occluded scenes, on a new single-tool dataset.","lead":"This paper adapts two existing zero-shot 6D pose estimation models, FoundationPose and SAM-6D, to track a surgical needle driver using stereo camera images from a surgical robot. It replaces the built-in segmentation with a fine-tuned Mask R-CNN and reports that the modified SAM-6D beats FoundationPose when the tool is occluded.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evidence that the pipeline generalises to unseen instruments: only one tool is tested, and the mask generator is fine-tuned on it.","rationale":"The reader's weakest assumption concerns circular pseudo-label masks, which is a real and important issue for the segmentation evaluation (Table I). However, the central pose comparison in Table II uses the same Mask R-CNN masks for both SAM-6D and FoundationPose, so the circularity does not directly bias the relative pose results. The more load-bearing gap is that the paper's headline claim is about generalisation to unseen instruments, yet no unseen instrument is ever evaluated and the only task-specific component (Mask R-CNN) is trained on the test tool. This is not a tuning issue; it is a mismatch between the claim and the evidence. The reader did mention this point in the rationale (item 4), but it was not the identified weakest assumption, hence partial agreement. The implausible 2D projection mean values also warrant a correction or clarification, but the unseen-tool gap is more fundamental because it undermines the central claim of the paper even if all reported numbers are correct for the LND tool.","tokens_in":10638,"tokens_out":7832,"duration_ms":81119,"concrete_test":"Evaluate the full pipeline on a second, genuinely unseen surgical instrument (e.g., laparoscopic scissors or a grasper), without any fine-tuning or retraining of Mask R-CNN on that instrument, using an occluded test set with marker-based ground truth. Report ADD and 2D projection metrics in the same Table II format; if segmentation fails or pose accuracy drops to the SAM-level baseline, the 'unseen tool' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the enhanced SAM-6D surpasses FoundationPose in zero-shot pose estimation of unseen surgical instruments. That claim is not supported by the experiments because the full pipeline is never evaluated on an unseen instrument. Section IV.A shows that all data (Datasets A, B, C) are collected for a single tool, the Endowrist Large Needle Driver (LND). More importantly, the mask generation stage—identified as the key improvement—is a Mask R-CNN fine-tuned on LND (Section III.C), with training labels derived from the LND CAD model. At test time, the same LND tool is evaluated, so the segmentation model has seen the test object. The 'zero-shot' property applies only to the pose estimation backbones (FoundationPose, SAM-6D, etc.), not to the full SurgPose pipeline. For a genuinely unseen surgical instrument, the fine-tuned Mask R-CNN would not produce a mask, and the pipeline would degrade to the SAM-based version that the paper shows performs very poorly (0.88% at 5 mm ADD in the non-occluded case). Table II contains no unseen-tool column and no cross-instrument experiment. Thus the headline claim of generalisability to unseen surgical instruments is not empirically established; the paper only demonstrates a comparison on a single, segmentation-seen tool.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SurgPose, a pipeline for 6DOF pose estimation of surgical instruments in robot-assisted minimally invasive surgery. The pipeline estimates depth from stereo pairs with RAFT-Stereo, generates instrument masks with a Mask R-CNN fine-tuned on synthetic and real data, and feeds RGB-D plus mask into zero-shot pose estimation backbones such as FoundationPose, SAM-6D, OVE-6D, and MegaPose. The authors contribute a new dataset of a Da Vinci Endowrist Large Needle Driver (LND) with Keydot-marker ground truth, and report experiments comparing original SAM-6D against a version where SAM is replaced by the fine-tuned Mask R-CNN, as well as against FoundationPose, OVE-6D, and MegaPose. The headline claim is that the enhanced SAM-6D surpasses FoundationPose in zero-shot pose estimation of unseen surgical instruments, with the strongest reported result being 49.06% vs. 6.02% accuracy at the 5 mm ADD threshold in the occluded scenario.","tokens_in":10791,"tokens_out":5469,"duration_ms":53831,"significance":"If the results are supported, the paper would be a useful first application of zero-shot RGB-D pose estimation to surgical instruments, and the use of stereo-based depth instead of a depth sensor would be practically relevant for reflective, close-range surgical scenes. The authors deserve credit for benchmarking four pose backbones under the same mask inputs, for collecting a real surgical-imaging dataset with marker-based ground truth, and for reporting runtime (15 fps after initialization). However, the significance is currently limited by three issues: only a single instrument is tested, the segmentation model is fine-tuned on that same instrument, and the segmentation ground truth used for validation is generated by the same pseudo-label procedure used to create training labels. These issues directly affect the paper's central generalisability claim, so the contribution as stated is not yet established.","major_comments":[{"comment":"The claim that SurgPose generalises to 'unseen surgical instruments' is not supported by the experiments. All evaluation is performed on a single tool, the Endowrist Large Needle Driver (LND), and Dataset C is used to fine-tune Mask R-CNN on this same tool, with training masks derived from the LND CAD model. At test time the segmentation model has therefore already seen the object, so the zero-shot property applies only to the pose backbones, not to the full mask-generation-plus-pose pipeline. Table II contains no column for a held-out instrument. To support the headline claim, the authors need to evaluate on at least one instrument not used in fine-tuning, or they must substantially revise the claim to a single-instrument study.","section":"IV.A and III.C"},{"comment":"The segmentation validation in Table I is circular. The pseudo-label masks defined by Eq. (1), which retain points where the depth of the projected CAD model agrees with RAFT-Stereo disparity depth within epsilon = 1 mm, are used both to fine-tune Mask R-CNN (Section III.C.1) and as ground truth for computing AP in Section IV.D. Because the same depth source and the same epsilon are used on both sides, a systematic error in RAFT-Stereo disparity would inflate the reported AP without indicating true segmentation accuracy. Independent mask annotations, or at least a manual validation subset, are needed before the claim that Mask R-CNN outperforms SAM can be accepted.","section":"III.C.1 and IV.D"},{"comment":"The abstract and conclusion claim that the enhanced SAM-6D 'surpasses FoundationPose' in zero-shot pose estimation, but Table II shows that in the non-occluded scenario FoundationPose has higher ADD accuracy at 1 mm, 2.5 mm, and 5 mm thresholds, and higher 2D projection accuracy at 5 px and 20 px thresholds. The proposed method leads only at the 50 px threshold in non-occluded scenes and at all thresholds in the occluded scene. Section IV.E.1 itself acknowledges that 'FoundationPose achieves the best overall performance' in the non-occluded case. The claims should be aligned with the evidence, for example by reporting a single aggregate metric with statistical significance testing, or by explicitly restricting the superiority claim to occluded scenarios.","section":"Table II and Abstract"},{"comment":"The contribution of RAFT-Stereo depth estimation is not directly validated. The paper claims that RAFT-Stereo is robust in reflective and textureless surgical environments, but no quantitative comparison is provided between RAFT-Stereo depth and any alternative depth source, and no ground-truth depth is used to measure depth error. Since the depth map is a core input to pose estimation and directly enters the pseudo-label generation of Eq. (1), an ablation that replaces RAFT-Stereo with another stereo method (or with a depth sensor where feasible) is needed to support the claim that the stereo-depth component is beneficial.","section":"III.B and IV.E"}],"minor_comments":[{"comment":"The row labels are ambiguous: the second data row is labelled only '(Ours)' even though the text refers to it as 'SAM-6D (Mask R-CNN)', and the formatting suggests the label may be merged with the preceding row. The table should be re-typeset so each method name is clearly associated with its columns.","section":"Table II"},{"comment":"Several entries in the 2D Projection mean and standard deviation columns (e.g., 9.77e8 and 4.32e9 for FoundationPose) are implausible if the units are pixels. These values need to be corrected or explained, since as printed they would indicate that the average projection error is hundreds of millions of pixels.","section":"Table II"},{"comment":"The paper does not specify the train/validation/test split for Dataset C, nor whether the images in Datasets A and B are disjoint from the real images used for fine-tuning Mask R-CNN. This information is needed to rule out data leakage between the fine-tuning and evaluation stages.","section":"IV.A"},{"comment":"The AP metrics in Table I are computed only on Dataset B, while the pose metrics in Table II are also computed on Dataset A. It would be helpful to report segmentation performance on Dataset A as well, since mask quality there may behave differently without occlusions.","section":"IV.C and IV.D"},{"comment":"The pipeline is described as 'end-to-end' in the introduction, but it consists of separately trained components (RAFT-Stereo, Mask R-CNN, and a pose backbone) and is not end-to-end differentiable. This wording should be changed to avoid overstating the architecture.","section":"I and III.D"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a recognized research group and the experimental setup with Keydot markers is a reasonable effort, but the gap between the title/abstract claim (generalisation to unseen instruments) and the actual single-tool experiments is substantial. The circularity in the segmentation evaluation is also a concern that the authors should be asked to address with independent annotations. I would be willing to reconsider after a major revision that adds a cross-instrument evaluation and clarifies or strengthens the validation of the pseudo-label masks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent engineering paper that makes one point convincingly—replacing SAM with a domain-fine-tuned Mask R-CNN inside SAM-6D helps a lot in occluded scenes—and then overclaims the headline. The ablative comparison in Table II is the real contribution, along with the stereo-depth adaptation. If you need evidence that segmentation quality is the bottleneck for zero-shot RGB-D pose in RMIS, this is useful.\n\nWhat's actually new: first benchmark of FoundationPose, SAM-6D, OVE-6D, MegaPose on a surgical stereo dataset with RAFT-Stereo depth, plus a modified SAM-6D. The result that SAM-6D with Mask R-CNN beats FoundationPose at 5mm ADD under occlusion (49% vs 6%) is stark and likely real. The paper also correctly reports that original SAM-6D with SAM is near-useless (0.88%). That is a clean ablation.\n\nSoft spots, in order of severity. First, the 'surpasses FoundationPose' claim in the abstract is contradicted by the non-occluded ADD columns: FoundationPose leads at all three ADD thresholds there. The only non-occluded metric the modified model wins is 50px 2D projection. The headline should be 'competitive in non-occluded, better in occlusion.' Second, the reported mean 2D projection errors (1e7 to 1e10 px) are physically impossible. Something is wrong in how means were computed (likely squared errors not rooted). This undermines confidence in the statistical reporting, though it does not invalidate the thresholded accuracies. Third, the segmentation evaluation is circular: the pseudo-label masks used to fine-tune Mask R-CNN and the ground-truth masks used to compute AP both come from the same GT-pose plus RAFT-Stereo depth pipeline, with the same 1mm epsilon. Without independent mask or depth validation, the 86.9 AP measures agreement with a shared label generator, not true accuracy. Fourth—the load-bearing problem for the title—the pipeline is never tested on an unseen instrument. Mask R-CNN is fine-tuned on the Large Needle Driver and evaluated on the same tool. For a new tool it would either need retraining or fall back to SAM, which performs terribly here. So the 'zero-shot' claim applies only to the pose backbones, not to the full SurgPose pipeline. The paper should either add a cross-instrument experiment or drop the generalisation claim.\n\nMinor: only one tool, no code or dataset release, and the comparison to PVNet is mentioned in text but not shown in Table II. The related work section is adequate.\n\nRecommendation: worth sending to serious peer review. The core ablation is publishable after revision, but the authors must fix the metric bug, de-overclaim the abstract, and either add an unseen-tool experiment or temper the generalisation claim. The circularity needs an explicit discussion and ideally an independent segmentation check.","headline":"A useful ablation showing that mask quality is the bottleneck for zero-shot surgical pose estimation, but the headline generalisation claim is unsupported and some reported metrics are unphysical.","tokens_in":11460,"tokens_out":2507,"would_cite":false,"duration_ms":23655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that mask quality is the bottleneck for zero-shot surgical pose estimation: swapping SAM for a fine-tuned Mask R-CNN lifts accuracy at the 5 mm threshold from under 1% to roughly 49% in occluded scenes, surpassing…","keywords":["pose estimation","zero-shot learning","surgical instruments","stereo vision","RGB-D","Mask R-CNN","RAFT-Stereo","robot-assisted surgery"],"falsifier":"Hand-label the instrument masks on the same occluded frames and recompute the AP of Mask R-CNN and SAM; if the 86.9 vs 46.4 gap does not persist against independent labels, the mask-quality explanation for the pose gains is undercut.","tokens_in":10316,"feed_emoji":"🩺","tokens_out":7919,"duration_ms":68774,"temperature":0.7,"pith_summary":"This paper tries to establish that zero-shot RGB-D pose estimation, previously confined to tabletop objects, can be made to work on surgical instruments in robot-assisted minimally invasive surgery. It argues that the two obstacles are unreliable depth from reflective, textureless metal tools and poor segmentation under occlusion. The solution is a pipeline that substitutes RAFT-Stereo disparity for a depth sensor and replaces SAM-6D's segmenter, SAM, with a Mask R-CNN fine-tuned on pseudo-labelled synthetic and real data. If the central claim is right, a surgical robot can estimate the 6D pose of a tool it has never seen before, with accuracy that in occluded scenes exceeds the best existing zero-shot model.","feed_headline":"Mask swap lets zero-shot surgical pose beat FoundationPose","feed_subtitle":"Fine-tuned Mask R-CNN instead of SAM lifts occluded-scene accuracy to 49% at 5 mm, versus 6% for the prior best.","key_machinery":"The load-bearing mechanism has two parts. First, pseudo-label mask generation of Eq. (1): the CAD model is projected through the known ground-truth pose into the image, and each pixel is kept only if the projected depth $Z_{\\mathrm{proj}}(u,v)$ agrees with the RAFT-Stereo disparity depth $Z_{\\mathrm{disp}}(u,v)$ to within $\\epsilon = 1$ mm; this is what erases occluded regions and produces visible-only instrument masks. Second, the enhanced SAM-6D, a zero-shot 6D pose estimator: its original SAM segmenter is replaced by a Mask R-CNN fine-tuned on these pseudo-labels plus synthetic renders, and that refined mask is what drives the pose hypothesis and refinement steps.","core_discovery":"On the author's own framing, the discovery is that mask quality, not the pose refinement machinery, is the binding constraint for zero-shot pose estimation in surgical scenes. In their experiments, swapping SAM for a fine-tuned Mask R-CNN lifts SAM-6D's accuracy at the 5 mm ADD threshold from 0.88% to 46.86% in non-occluded scenes and from 0.86% to 49.06% in occluded scenes, and the occluded-scene score surpasses FoundationPose's 6.02% despite FoundationPose leading in the non-occluded setting. The same swap raises 2D projection accuracy at 50 px to 98.87% under occlusion, versus 28.73% for FoundationPose. They also show that RAFT-Stereo's disparity can replace a physical depth sensor, since their pipeline uses stereo depth throughout.","pith_inferences":["One testable extension is to feed FoundationPose the same fine-tuned Mask R-CNN mask; if its occluded-scene accuracy jumps as much as SAM-6D's did, the decisive factor is mask quality rather than SAM-6D's specific refinement machinery.","Because the pseudo-label generator in Eq. (1) is used both to train the segmenter and to score it, an independent hand-labelled mask set would be a stricter test of whether the reported 86.9 AP reflects true segmentation skill or agreement with a shared, unvalidated label source.","If RAFT-Stereo's disparity has a systematic bias on specular metal surfaces, then using a calibrated depth camera as a substitute should change pose accuracy in a measurable way; a null result there would suggest the depth estimate is not the bottleneck.","The paper only considers one instrument type, the Endowrist Large Needle Driver; whether the same margin holds for curved or articulated tools is left open."],"forward_implications":["Zero-shot pose estimation can be applied to surgical instruments using stereo depth rather than a depth sensor.","Replacing SAM with a fine-tuned Mask R-CNN raises SAM-6D from under 1% to roughly 47-49% accuracy at the 5 mm ADD threshold in both non-occluded and occluded scenes.","In occluded scenes, the enhanced SAM-6D outperforms FoundationPose at every reported threshold, despite FoundationPose leading in the non-occluded setting.","Subsequent frames are processed at about 15 fps after a 2.52 s first-frame initialisation, putting the pipeline near real time."],"supporting_citations":[{"why":"FoundationPose: the leading zero-shot RGB-D baseline whose occluded-scene accuracy the enhanced SAM-6D is claimed to surpass.","marker":"[8]"},{"why":"SAM-6D: the base zero-shot pose estimator whose SAM segmenter is replaced by the fine-tuned Mask R-CNN.","marker":"[9]"},{"why":"RAFT-Stereo: supplies the disparity-based depth used in place of a depth sensor and in the pseudo-label mask filter of Eq. (1).","marker":"[16]"},{"why":"Mask R-CNN: the supervised segmenter fine-tuned on pseudo-labels and synthetic data; its masks are the central intervention.","marker":"[17]"},{"why":"Photorealistic renderer used to generate synthetic training images for the Mask R-CNN fine-tuning.","marker":"[31]"},{"why":"SAM: the original segmenter in SAM-6D, measured as the weaker alternative in Table I.","marker":"[32]"},{"why":"Graph-based pose estimation used to obtain the marker-to-tool transformation for ground-truth pose.","marker":"[7]"}],"fun_headline_variants":["Mask R-CNN swap beats FoundationPose in occluded zero-shot surgical pose","Fine-tuned mask lifts zero-shot surgical pose to 49% at 5 mm","Zero-shot surgical pose soars when SAM swapped for Mask R-CNN","Occluded-scene surgical pose jumps 50x with better instance mask","Better mask, not pose refinement, drives zero-shot surgical pose accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-label masks, built by projecting the CAD model through the known tool pose and keeping only pixels whose projected depth agrees with the stereo depth within 1 mm, are accurate enough to serve both as training labels for Mask R-CNN and as the segmentation ground truth; any systematic error in the stereo depth would be invisible because it contaminates both sides equally.","fun_headline_variants_meta":{"raw":{"variants":["Mask R-CNN swap beats FoundationPose in occluded zero-shot surgical pose","Fine-tuned mask lifts zero-shot surgical pose to 49% at 5 mm","Zero-shot surgical pose soars when SAM swapped for Mask R-CNN","Occluded-scene surgical pose jumps 50x with better instance mask","Better mask, not pose refinement, drives zero-shot surgical pose accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000169,"raw_usage":{"total_tokens":1292,"prompt_tokens":1003,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":619,"tokens_out":289,"duration_ms":3028,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:18.045646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-label the instrument masks on the same occluded frames and recompute the AP of Mask R-CNN and SAM; if the 86.9 vs 46.4 gap does not persist against independent labels, the mask-quality explanation for the pose gains is undercut.","supporting_citations":[{"cited_title":"FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects","cited_arxiv_id":null,"evidence_quote":"FoundationPose: the leading zero-shot RGB-D baseline whose occluded-scene accuracy the enhanced SAM-6D is claimed to surpass."},{"cited_title":"Sam-6d: Segment anything model meets zero-shot 6d object pose estimation","cited_arxiv_id":null,"evidence_quote":"SAM-6D: the base zero-shot pose estimator whose SAM segmenter is replaced by the fine-tuned Mask R-CNN."},{"cited_title":"Raft-stereo: Multilevel recurrent field transforms for stereo matching","cited_arxiv_id":null,"evidence_quote":"RAFT-Stereo: supplies the disparity-based depth used in place of a depth sensor and in the pseudo-label mask filter of Eq. (1)."},{"cited_title":"Mask r-cnn","cited_arxiv_id":null,"evidence_quote":"Mask R-CNN: the supervised segmenter fine-tuned on pseudo-labels and synthetic data; its masks are the central intervention."},{"cited_title":"Nvisii: A scriptable tool for photorealistic image generation","cited_arxiv_id":null,"evidence_quote":"Photorealistic renderer used to generate synthetic training images for the Mask R-CNN fine-tuning."},{"cited_title":"Segment anything","cited_arxiv_id":null,"evidence_quote":"SAM: the original segmenter in SAM-6D, measured as the weaker alternative in Table I."},{"cited_title":"Graph-based Pose Estimation of Texture-less Surgical Tools for Autonomous Robot Control","cited_arxiv_id":null,"evidence_quote":"Graph-based pose estimation used to obtain the marker-to-tool transformation for ground-truth pose."}],"review_version":1}