{"id":"4734aa5b-8b5e-4bad-afb7-b57adb4bcc29","arxiv_id":"2411.11196","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An interaction-guided, class-agnostic pipeline that discovers and reconstructs each manipulated object as a separate 3D model from egocentric RGB-D video.","lead":"PickScan detects objects a person picks up during an RGB-D scan, tracks them in 2D, and reconstructs each as an independent 3D model without needing pretrained object classes. It reduces the need for semantic segmentation for object discovery, so arbitrary graspable objects can be scanned for robotics and augmented reality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 73% chamfer-distance improvement over Co-Fusion is confounded by different reconstruction backends; a fair masked-controlled comparison is needed.","rationale":"The paper's central contribution is an interaction-based object discovery mechanism, and the strongest claim is that it beats the only comparable class-agnostic baseline Co-Fusion on both detection and reconstruction quality. The detection comparison (precision, false positives) is appropriately tied to the discovery output. The reconstruction comparison, however, is not isolated: chamfer distance is computed after a full reconstruction pipeline, and the text only says PickScan uses BundleSDF whereas Co-Fusion's backend is unspecified. Since the SAM* control is explicitly run through the identical XMem+BundleSDF backend, the absence of a similar statement for Co-Fusion is conspicuous. This is the single most load-bearing concern because the headline 'reduction in chamfer distance of 73%' may be an artefact of using a modern neural reconstruction method rather than a benefit of the proposed masks. It is also directly testable with a small code change. The reader's identified weakest assumption (moving-mask heuristic requiring a static background) is a valid scope limitation but does not threaten the reported numbers on the paper's own tabletop dataset; it is a generalizability issue. The paper does have independent strengths: the interaction-detection principle is novel, the integration with XMem and BundleSDF is sensible, and the qualitative results, including the brick-grid failure of SAM, support the motivation. But the quantitative comparison needs the controlled rerun before the headline claim should be taken as established.","tokens_in":9474,"tokens_out":5391,"duration_ms":50605,"concrete_test":"Run Co-Fusion on the same three scans, take its discovered object masks, feed them through XMem and BundleSDF exactly as done for PickScan, and recompute per-scene chamfer distances. If the gap narrows to a level close to the reported 0.90 cm, the 73% improvement is an artefact of the reconstruction module; if a large gap persists, the interaction-based discovery is confirmed as the source of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative results in Table I compare PickScan's 0.90 cm mean chamfer distance to Co-Fusion's 3.33 cm, but the paper never specifies how Co-Fusion's reconstructions are obtained. PickScan's reconstructions use BundleSDF (Sec. III intro), and for the SAM* baseline the authors explicitly reuse the same XMem+BundleSDF backend (Sec. IV-E). No equivalent statement is made for Co-Fusion, so it appears Co-Fusion was run as a complete system with its own superpixel-based fusion, meaning Table I conflates object-discovery quality with the much stronger reconstruction backend. The claimed 73% chamfer-distance reduction could then be driven by BundleSDF's superior tracking and fusion rather than by the proposed interaction-based masks. The paper's own Sec. IV-D attributes the difference to mask quality and Co-Fusion's superpixel discretization, but this is an assertion, not a controlled comparison. The false-positive and precision metrics are not affected by this confound, but the central claim of 'outperforming Co-Fusion in every metric' depends on a fair apples-to-apples reconstruction test.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PickScan, an interaction-guided and class-agnostic pipeline for discovering and reconstructing manipulated objects from egocentric RGB-D/IMU streams. The method builds an initial static point cloud with gradSLAM, computes per-frame 'moving masks' by reprojecting current depth into that cloud and thresholding at a distance rho_moving, removes an arm mask, selects blobs near the hand, detects pick-up/lay-down interactions from crossings of the hand-initial and hand-object distance trajectories, and chooses one stable mask per interaction. The mask is tracked with XMem and the object is reconstructed with BundleSDF. On a custom dataset of three tabletop scans (nine manipulated objects), the paper reports a mean chamfer distance of 0.90 cm (versus Co-Fusion's 3.33 cm), 78.3% precision at 100% recall, and an average of 1.0 false positive per scan. A segmentation-based baseline (SAM*) is also compared.","tokens_in":9855,"tokens_out":8515,"duration_ms":78658,"significance":"The core idea is attractive: by exploiting the user's physical interaction with an object, PickScan avoids class-specific appearance priors and can separate objects that static segmentation merges, as illustrated by the toy-brick example in Fig. 8. The system is modular, reuses established components (XMem, BundleSDF, gradSLAM), and the authors share code. The paper is also candid about its limitations, including false positives, mask-tracking failures, and reconstruction gaps attributed to BundleSDF. If the quantitative claims are confirmed by a controlled evaluation, the work would be a useful step toward practical compositional scanning. However, the current evidence is preliminary: the dataset is small, threshold values are not reported, and the Co-Fusion chamfer comparison is confounded by the reconstruction backend. The precision and false-positive metrics are less affected by the backend confound but are based on only nine positive examples.","major_comments":[{"comment":"The comparison with Co-Fusion is confounded by the reconstruction backend. PickScan uses BundleSDF (Sec. III, pipeline overview) and the SAM* baseline explicitly reuses XMem+BundleSDF (Sec. IV-E), but the paper never specifies how Co-Fusion's reconstructions were obtained. If Co-Fusion was run as a complete system with its own superpixel-based fusion, the reported 73% chamfer-distance reduction cannot be attributed to the proposed interaction-based masks; BundleSDF's tracking and fusion could account for much of the gap. Please add a controlled experiment in which the same reconstruction backend (e.g., XMem+BundleSDF) is fed with masks produced by PickScan and by Co-Fusion, and report the resulting chamfer distances. Without this, the central claim of outperforming Co-Fusion in every metric is not established.","section":"Sec. IV-D, Table I"},{"comment":"The method has five free thresholds that are described only as 'tunable': rho_moving (III-A), the blob area threshold (III-B), the interaction-duration threshold (III-C), tau_iou (III-D), and tau_nms (III-E). Their values are not reported, and the 'topmost 10%' hand-mask fraction is stated without supporting evidence. The Table I numbers (precision, false positives, chamfer distance) are directly controlled by these parameters. Please report the exact values used for all thresholds and include a sensitivity analysis that varies each threshold over a plausible range and shows the effect on precision, recall, and chamfer distance. Without this, the headline results are not reproducible.","section":"Sections III-A through III-E and Table I"},{"comment":"The evaluation uses only three scans and nine manipulated objects drawn from a set of five foreground objects. There are no error bars, no per-object results, and no significance test; the per-scene precision values (100%, 75%, 60%) show considerable variance. A claim of 100% recall based on nine positive examples is fragile. Please provide per-object or per-scene results with error bars and, ideally, evaluate on a larger and more diverse set of scenes and objects to support the general conclusions.","section":"Sec. IV-A, Table I"},{"comment":"The core discovery heuristic assumes that the only moving entities in the egocentric stream are the user's arm and the manipulated object, and that a manipulated object's points are displaced farther than rho_moving from the initial point cloud. The paper does not test what happens when this assumption is violated, for example when a second person or another dynamic object enters the view, or when an object is slid by only a few centimeters. Since this assumption is the foundation of the object-discovery mechanism, the paper should either explicitly scope the claimed generality to this setting or include a stress-test experiment with such violations. This is a correctness-risk concern rather than an objection to the overall idea.","section":"Sec. III-A and III-C"}],"minor_comments":[{"comment":"The phrase 'beginnings and ends of individual user-object detections' appears to be a typo; it should read 'interactions' rather than 'detections'.","section":"Introduction, contributions paragraph"},{"comment":"Co-Fusion is cited as [10] in the Introduction, but reference [10] is the semantic visual SLAM survey by Chen et al.; the correct citation for Co-Fusion is [5] (Rünz and de Agapito).","section":"Introduction and References"},{"comment":"The reprojection equation is difficult to parse because the homogeneous-coordinate notation and the definitions of p_t(u,v), p_0(u,v), and T_{0,t} are not written cleanly. Please rewrite the equation with explicit 4-vector notation so that the displacement heuristic is reproducible.","section":"Sec. III-A"},{"comment":"The dataset description does not specify how many times each of the five foreground objects appears across the three scenes; please add a per-scene object list so that the per-scene metrics in Table I can be interpreted.","section":"Sec. IV-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a promising systems contribution with a clear idea and a code release. The main risk is the uncontrolled Co-Fusion comparison and the small evaluation; I do not see a fundamental flaw in the interaction-detection concept itself. The citation error for Co-Fusion should be fixed. I would be willing to review a revised version that adds the controlled comparison, reports threshold values, and expands the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the interaction-detection mechanism is genuinely new and the false-positive reduction over Co-Fusion looks real, but the headline chamfer-distance claim is not a controlled comparison. The paper deserves a serious referee, but the evaluation needs work.\n\nWhat's new: instead of relying on semantic masks or rigid-motion clustering on superpixels, they detect user-object interactions by tracking when the distance between the hand cloud and the initial scene crosses the distance between the hand cloud and the candidate object mask. That is a clever, simple signal, and the brick example in figure 8 shows a case where interaction beats a strong segmentation model. The pipeline is fully automated and they ship code.\n\nWhat's weak: the central quantitative claim — 0.90 cm vs 3.33 cm chamfer distance — mixes together two differences: object discovery quality and the reconstruction backend. PickScan uses BundleSDF; Co-Fusion uses its own superpixel-based fusion. For the SAM* baseline they reuse the same XMem+BundleSDF backend, so that comparison is fair, but they never do that for Co-Fusion. So you cannot attribute the 73% improvement to the proposed masks. The precision and false-positive numbers are probably not affected by this, and they are the more important evidence for the discovery claim. But the paper should either re-run Co-Fusion with the same tracking/reconstruction backend or discuss the system-level comparison as such, not as evidence for mask quality.\n\nOther soft spots: three scenes, no error bars, no ablations of rho_moving, tau_iou, tau_nms, etc. The moving-mask heuristic assumes only the arm and the object move; they acknowledge some of this in the limitations. Also, the manual alignment for chamfer distance (Section IV-C) is a bit of a fiddle, though they use ICP after an initial manual alignment.\n\nThe paper is honest about its limitations and the writing is clear. The core idea is plausible and worth the community's attention. I'd send it to peer review, but I'd expect the reviewers to ask for a controlled comparison and an ablation before the numbers are treated as solid.","headline":"A plausible class-agnostic interaction-based discovery pipeline whose headline reconstruction numbers are undercut by an apples-to-oranges comparison with Co-Fusion, but the core idea and false-positive reduction look real.","tokens_in":10218,"tokens_out":2461,"would_cite":false,"duration_ms":23459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PickScan shows that picking up an object and moving it in front of an RGB-D camera is enough to discover, track, and reconstruct it as its own 3D model, with no training on object classes.","keywords":["interaction-guided object discovery","class-agnostic 3D reconstruction","compositional scene reconstruction","egocentric RGB-D scanning","object manipulation","moving mask heuristic","object mask tracking","unknown object reconstruction"],"falsifier":"Scan a tabletop scene in which a second rigid object moves independently, for example a ball rolls across the table, while the user picks up the target object, and count the pipeline's false-positive object detections: if they rise well above the reported one per scan, the discovery mechanism depends on the scene being otherwise static, and the central claim is correspondingly limited.","tokens_in":9317,"feed_emoji":"📦","tokens_out":12554,"duration_ms":107965,"temperature":0.7,"pith_summary":"PickScan claims that compositional 3D scene reconstruction, where each object gets its own model, can be reduced to finding one reliable mask per manipulated object. The paper's central idea is to use the act of picking an object up as the discovery signal: a displacement heuristic separates moving scene parts from the static background, and the crossing pattern of hand-to-scene versus hand-to-object distances marks the start and end of each interaction. This yields one high-quality mask per object, which a mask tracker spreads over the whole interaction and a neural reconstruction method turns into a 3D model. On a custom tabletop dataset, the pipeline reports 78.3% precision at 100% recall, a mean chamfer distance of 0.90 cm, and a 73% reduction in reconstruction error and 99% fewer false positives compared with the only comparable class-agnostic interaction-based baseline.","feed_headline":"Picking up objects yields one separate 3D model each","feed_subtitle":"Motion alone discovers any graspable object, and reconstruction error drops 73 percent versus the prior baseline.","key_machinery":"The load-bearing mechanism is the moving-mask heuristic followed by a distance-crossing interaction detector. A pixel is labeled moving when its reprojection into the initial point cloud lies farther than a threshold $\\rho_\\mathrm{moving}$ from every initial-scene point; subtracting the estimated arm mask from the moving mask leaves candidate object blobs, and the blob whose points lie closest to the hand is kept as the candidate object mask. Interaction boundaries come from two distance trajectories: the distance between the hand point cloud and the initial scene, and the distance between the hand point cloud and the candidate object. During a pick-up-and-lay-down maneuver the first rises above the second at the start and falls back below it at the end, so those crossings segment individual interactions. A median filter and a minimum-duration rule remove noise, the longest period of stable cross-frame intersection-over-union supplies a 'best frame' whose mask seeds 2D tracking, and a containment-based non-maximum suppression step removes duplicate detections after tracking.","core_discovery":"The central claim is that the act of picking up an object carries enough information to discover and segment that object without any object-class prior. The paper's contribution is an interaction detector that reads the crossing pattern of two distances in an egocentric RGB-D stream: the hand-to-initial-scene distance crosses above the hand-to-object distance when the object is lifted and falls below it when the object is laid down, marking the start and end of one object interaction. From within each interaction, the pipeline selects the frame whose candidate mask is most stable across neighboring frames, tracks that mask through the entire interaction with a 2D mask tracker, and feeds the tracked masks to an unknown-object reconstruction method that outputs a 3D model and per-frame pose. On the paper's three-scene tabletop dataset, this yields 100% recall with 78.3% precision, a mean chamfer distance of 0.90 cm, a 73% reduction in reconstruction error, and 99% fewer false positives compared with the only comparable class-agnostic interaction-based baseline.","pith_inferences":["The crossing-signal detector is a generic signature of grasp-and-move actions, so the same idea could drive object discovery in robot demonstration learning or augmented-reality object editing, where a hand-object distance staying small while the hand-to-scene distance grows marks the manipulation.","An untested stress case is a scene with a second, independent mover (a person walking by or a rolling object); because the moving-mask heuristic assumes only the arm and held object move, such a scene would likely inflate false positives and reveal the boundary of the static-background assumption.","Because the discovery stage only needs one good mask per object, running the interaction detector online could turn the pipeline into an interactive scanner that tells the user when an object has been seen from enough viewpoints; the paper does not pursue real-time feedback.","The reported performance is measured on a three-scene tabletop dataset with five known foreground objects, so the class-agnostic claim would be strengthened by tests on unseen objects, varied supports such as shelves or floors, and lower-quality depth sensors; these are direct extensions the paper leaves open."],"forward_implications":["Users can scan any rigid, graspable object in a scene with a handheld RGB-D camera and receive one complete 3D model per picked-up object, bypassing the training-class limits of segmentation-based discovery.","Interaction-based discovery resolves object boundaries that static segmentation cannot, such as adjacent bricks with identical texture but different shapes, because motion separates the picked-up object from its support.","Per-pixel displacement detection replaces superpixel-center clustering, yielding substantially cleaner masks and a 73% lower mean chamfer distance than the prior class-agnostic baseline.","The pipeline's modularity means any improvement in 2D mask tracking or unknown-object reconstruction translates directly into better end-to-end compositional reconstruction without changing the discovery stage."],"supporting_citations":[{"why":"The baseline the work must beat: the only prior class-agnostic method that allows objects to move while discovering them.","marker":"[5]"},{"why":"The 2D mask tracker that propagates the seed mask from the best frame to all frames of the interaction.","marker":"[11]"},{"why":"The unknown-object reconstruction and 6-DOF tracking method that consumes the tracked masks and outputs the 3D model.","marker":"[12]"},{"why":"Baseline for comparison: a strong segmentation network whose best mask per object is fed through the same tracking and reconstruction stages.","marker":"[9]"},{"why":"The egocentric hand and arm segmentation network used to remove the arm from the moving mask and to detect the end of the static scan.","marker":"[28]"},{"why":"Differentiable SLAM used to reconstruct the initial point cloud and refine camera poses for the displacement heuristic.","marker":"[29]"},{"why":"Static scanner that provides the high-fidelity ground-truth models used to compute chamfer distances.","marker":"[30]"}],"fun_headline_variants":["Lift an object, get its own 3D model","Picking up objects isolates and reconstructs each one","Interaction-guided scanning: no class priors needed","Handheld motion discovers objects and cuts error 73%","Grasp detection enables class-agnostic 3D reconstruction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole discovery step rests on the assumption that in the egocentric video the only moving things are the user's arm and the object being picked up, and that the object moves far enough from its original position to be flagged as moving; if another thing moves or the object is only slightly nudged, the pipeline cannot isolate it.","fun_headline_variants_meta":{"raw":{"variants":["Lift an object, get its own 3D model","Picking up objects isolates and reconstructs each one","Interaction-guided scanning: no class priors needed","Handheld motion discovers objects and cuts error 73%","Grasp detection enables class-agnostic 3D reconstruction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2374,"prompt_tokens":969,"completion_tokens":1405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":585,"tokens_out":1405,"duration_ms":13912,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:47:46.637699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scan a tabletop scene in which a second rigid object moves independently, for example a ball rolls across the table, while the user picks up the target object, and count the pipeline's false-positive object detections: if they rise well above the reported one per scan, the discovery mechanism depends on the scene being otherwise static, and the central claim is correspondingly limited.","supporting_citations":[{"cited_title":"Co-fusion: Real-time segmentation, tracking and fusion of multiple objects,","cited_arxiv_id":null,"evidence_quote":"The baseline the work must beat: the only prior class-agnostic method that allows objects to move while discovering them."},{"cited_title":"Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,","cited_arxiv_id":null,"evidence_quote":"The 2D mask tracker that propagates the seed mask from the best frame to all frames of the interaction."},{"cited_title":"Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects,","cited_arxiv_id":null,"evidence_quote":"The unknown-object reconstruction and 6-DOF tracking method that consumes the tracked masks and outputs the 3D model."},{"cited_title":"Fine-grained egocentric hand- object segmentation: Dataset, model, and applications,","cited_arxiv_id":null,"evidence_quote":"The egocentric hand and arm segmentation network used to remove the arm from the moving mask and to detect the end of the static scan."},{"cited_title":"∇slam: Dense slam meets automatic differentiation,","cited_arxiv_id":null,"evidence_quote":"Differentiable SLAM used to reconstruct the initial point cloud and refine camera poses for the displacement heuristic."},{"cited_title":"Einscan v2 3d scanner webpage,","cited_arxiv_id":null,"evidence_quote":"Static scanner that provides the high-fidelity ground-truth models used to compute chamfer distances."}],"review_version":1}