{"id":"220a0b0e-383f-4612-a10e-86b04ef6abfe","arxiv_id":"2507.10776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"rt-RISeg uses the physics of rigid body motion to segment unseen objects in real time from robot pushes, without a learned segmentation model.","lead":"A robot arm can segment unfamiliar objects by gently pushing them and watching how their parts move together. The method needs no trained segmentation network and works in real time, reporting higher accuracy on its own test scenes than two leading static models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Alg. 3's CreateFrames is only specified to consume 2D effective optical flow Xt and joint angles; if body frames are built from pixel triplets rather than depth-unprojected 3D points, the SE(3) spatial-twist equality underpinning BFIF grouping does not follow.","rationale":"The reader's conditional verdict is appropriate. I focused on the 2D/3D frame construction because it is the single point at which the mathematical invariant connects to the implemented pipeline. Every other weakness—private dataset, no error bars, no code, ambiguous 27.5% phrasing—affects generalizability or presentation, not the validity of the core mechanism. The BFIF invariance itself is standard and correctly stated; the issue is that the paper does not establish that the frames it actually constructs are SE(3) body frames. This is not an external-consensus disagreement; it is an internal gap between Eq. 2 and the pseudocode. Since depth is used elsewhere, the gap is likely fixable, which supports CONDITIONAL rather than REJECT. Thus I do not change the reader's verdict.","tokens_in":13118,"tokens_out":4276,"duration_ms":50878,"concrete_test":"On the same recorded 20 scenes with ground truth, implement two variants of CreateFrames: (A) literal 2D pixel-coordinate frames from X_t; (B) unproject each sampled triplet to metric 3D via depth and camera intrinsics before building axes (and report the exact unprojection equation). Keep all downstream BFIF computation, Mahalanobis grouping, Markov clustering, and flood-fill identical. If variant B reproduces the reported ~75% correct-object overlap at step 5 while variant A yields near-chance grouping, the absence of an explicit depth unprojection in Algorithm 3 is confirmed as load-bearing. Also state explicitly whether any depth input reaches CreateFrames in the implementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that spatial twists of randomly sampled body frames separate objects—rests on the mathematical fact that two body frames attached to the same rigid body have equal spatial twists when expressed in a common space frame (Eq. 2, Section III.A). This is a 3D SE(3) statement. Algorithm 3, however, never shows that the sampled triplets are lifted to metric 3D coordinates. In Section IV.B.1, CreateFrames takes only X_t and θ_t; X_t is defined earlier in the same algorithm as a 2D effective optical flow (H×W×2). The text says 'pick triplets of pixels' and constructs axes with 'the x-axis perpendicular to the plane formed by the triplet.' If those coordinates are pixel coordinates, every three non-collinear pixels are coplanar in the image plane, so the resulting 'frames' are not rigid body frames in SE(3); their motion under perspective projection is not a spatial twist, and Eq. 2 cannot be applied. The invariance that makes BFIFs equal for same-object samples therefore has no demonstrated basis. Depth is available and is used to compute expected flow in Eq. 5, so the missing step is plausibly an omitted implementation detail rather than a false method, but as written the core mathematical premise is unverified. This is the load-bearing assumption: if CreateFrames is genuinely 2D, grouping by Mahalanobis distance in the BFIF space is not measuring what the paper claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes rt-RISeg, a real-time, model-free framework for unseen object instance segmentation (UOIS) based on robot interactions and a Body Frame-Invariant Feature (BFIF) derived from the spatial twists of rigid body motion. The framework samples body frames from effective optical flow, computes BFIFs, groups them via Mahalanobis distances and Markov clustering, and uses the groups to seed and propagate object masks while the robot is still moving. The authors report that rt-RISeg achieves 75.1% correctly segmented objects after interactions, outperforming static baselines MSMFormer and SAM by roughly 27.5 percentage points, and that prompting SAM with rt-RISeg masks further improves boundary quality. The paper positions the work as a paradigm shift from 'observe, interact, observe' to 'observe while interacting.'","tokens_in":13381,"tokens_out":6162,"duration_ms":68407,"significance":"If the central claims hold, this is a conceptually strong and practically relevant contribution: it shows that a physical invariant (spatial twist) can drive object segmentation without a learned segmentation model, achieving state-of-the-art-level results in real robot experiments while preserving scene task formation. The demonstrated use of autonomously generated masks to prompt SAM is a nice downstream illustration, and the proposed continuous, real-time interaction loop is a genuinely different paradigm from static UOIS. The empirical gains over static baselines are substantial, and the work is well-placed for a robotics journal. However, the significance is tempered by the unresolved implementation detail regarding 3D lifting of sampled points, the lack of timing measurements for the central 'real-time' claim, and the absence of reported parameter values; these issues must be addressed before the claims can be fully evaluated.","major_comments":[{"comment":"The description of CreateFrames is critically underspecified: the paper says 'we sample n random pixels from ||X_t|| >> 0 and pick triplets of pixels to create each frame' and defines the axes using 'the plane formed by the triplet of sampled points.' It is never stated that the sampled pixels are unprojected to 3D metric coordinates using the depth channel before constructing the frame. The BFIF invariance in Eq. (2) is a 3D SE(3) property: it holds only for body frames that are rigidly attached to a moving object in 3D. If the frames are built directly from 2D pixel coordinates, every non-collinear triplet is coplanar in the image plane, the resulting frame is not a rigid body frame in SE(3), and its motion under perspective projection is not a spatial twist. The equality of BFIFs for points on the same object would then have no theoretical basis, and the Markov clustering on Mahalanobis distances would not be grouping by true spatial twists. Since depth is already used in Eq. (5) for expected flow computation, the missing step is plausibly an omitted implementation detail, but as written the load-bearing mathematical premise is unverified. The authors must explicitly state that triplets are lifted to 3D (e.g., using dMap_{t-1}) before frame construction, or provide a different justification if the implementation is genuinely 2D.","section":"Section IV.B.1, Algorithm 3, line 4"},{"comment":"The paper repeatedly claims 'real-time' operation, yet no timing measurements are reported anywhere. There is no average wall-clock time per frame, no latency between robot motion and mask update, and no comparison with the duration of robot interactions. Since real-time performance is a central claimed advantage of rt-RISeg over prior interactive methods, the authors should report at least the average computation time for SEGMENT_OBJS per time step and specify the hardware on which it was measured.","section":"Title, Abstract, Section V.D"},{"comment":"Several core parameters that the method depends on are never given: the number of sampled body frames n, the maximum pairwise distance da, the threshold for '||X_t|| >> 0' used to select moving pixels, and the parameters of the Mahalanobis kernel / Markov clustering (e.g., inflation parameter). Without these values, the experiments are not reproducible and the sensitivity of the results to these choices cannot be assessed. Please report the exact values used in the experiments or, if they are tuned, provide a sensitivity analysis.","section":"Section IV.B and Experiments V.A"}],"minor_comments":[{"comment":"The introduction states that 'rt-RISeg leverages BFIFs to derive object-level understanding without any learning,' but the pipeline uses RAFT, a learned optical flow model, as acknowledged in Section V.A.1. The abstract and later text correctly say 'without any learned segmentation model.' Please revise for consistency.","section":"Introduction"},{"comment":"The '27.5%' improvement mentioned in the abstract refers to the percentage of objects correctly segmented (Overlap F-measure >= 75%), but this metric is not clearly defined at first use. Please define 'object segmentation accuracy rate' explicitly in Section V.B and state whether the improvement is in percentage points or relative percentage.","section":"Section V.B and Fig. 7"},{"comment":"The comparisons are limited to two static baselines, MSMFormer and SAM. While SAM is a strong and relevant baseline, the claim of surpassing 'state-of-the-art UOIS methods' is broad; consider comparing against additional recent UOIS methods (e.g., UCN or other RGB-D segmentation models) or softening the wording to reflect the specific baselines used.","section":"Section V.A.2"},{"comment":"The sentence 'Because X_t represents absolute motion in the real world as a result of a robot interaction' is misleading: X_t is a 2D effective optical flow field after subtracting the expected camera motion, not an absolute 3D motion. Please rephrase to avoid confusion.","section":"Section IV.B.1"},{"comment":"The notation 'υx\\{s\\}' is ambiguous; the superscripts indicating the different body frames are missing or unclear. Please use consistent notation such as 'υ_s^{a1}' and 'υ_s^{a2}'.","section":"Fig. 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a solid extension of the authors' prior RISeg work, but the main theoretical premise is currently ambiguous in a way that is easily fixable. The lack of timing measurements is the most surprising omission given the title. I would encourage the editor to insist on the clarification of the 3D lifting step before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a plausible extension of the authors' RISeg work, and the real-time model-free direction is worth engaging with. But the paper as written leaves a load-bearing detail ambiguous enough that the math doesn't quite close.\n\nWhat is actually new: removing the learned base segmentation model and doing BFIF grouping in real time, then using the resulting masks to prompt SAM. That is a real extension, not just a rehash. The underlying spatial-twist principle is sound and well-grounded in Lynch and Park. The incremental segmentation curve in Fig. 7 is a nice demonstration that interaction helps, and the SAM-prompting idea is practical.\n\nThe soft spot is the construction of body frames in Algorithm 3. The effective optical flow X_t is defined as a 2D HxWx2 quantity, and the text says the algorithm picks \"triplets of pixels\" and builds frame axes from them. It never states that these triplets are unprojected to 3D metric coordinates using the depth channel. If they are not, then every triplet is coplanar in the image plane, the resulting \"frames\" are not SE(3) body frames, and the spatial-twist equality in Eq. 2 does not apply. That equality is the entire basis for the BFIF grouping. Depth is available and is used to compute expected flow, so the fix is likely one line in the pseudocode, but as written the core premise is unverified. This is not a manufactured nit; it is the difference between the method working as advertised and not.\n\nOther issues are real but less severe. The evaluation is thin: 20 scenes, no error bars, no code or data, and no comparison against the authors' own RISeg baseline. The headline \"27.5% greater\" is a percentage-point difference, not a relative improvement. These do not refute the method, but they make it hard to trust the quantitative claims.\n\nWho this is for: people working on interactive perception and unseen object instance segmentation. It deserves a serious referee, because the idea is interesting and the fix is clarifying the 3D construction plus a solid evaluation. I would not cite it in its current form, but I would read the revision closely.","headline":"A plausible real-time model-free interactive segmentation pipeline, but the paper leaves the central 3D construction ambiguous and the evaluation is thin.","tokens_in":13970,"tokens_out":1787,"would_cite":false,"duration_ms":22163,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"rt-RISeg claims that random reference frames on a pushed object expose its identity through identical spatial twists, enabling model-free real-time segmentation.","keywords":["interactive perception","unseen object instance segmentation","spatial twist","body frame-invariant feature","optical flow","model-free segmentation","rigid body motion","Markov clustering"],"falsifier":"A reader could test whether one rigid object, pushed so that it rotates about an axis not parallel to the camera plane, is still returned as a single cluster: compute the BFIFs from 2D pixel coordinates alone, without depth, and check whether points at different depths on the same object give equal spatial twists. If the invariance fails under this 2D construction, the reported grouping rests on an unstated 3D step.","tokens_in":12863,"feed_emoji":"🤖","tokens_out":5620,"duration_ms":60651,"temperature":0.7,"pith_summary":"rt-RISeg claims that a robot can segment previously unseen objects in a cluttered scene without any learned segmentation model, by pushing objects slightly and watching how sampled points on them move. The key idea is that two small reference frames attached to the same rigid body show the same spatial twist when viewed from a fixed world frame, even if their individual motions differ, so grouping frames by twist separates objects. Because the feature is computed from motion rather than appearance, segmentation happens in real time during the push and generalizes to out-of-distribution objects. On roughly 100 manually annotated images across 20 tabletop scenes, the method reaches 75.1% correctly segmented objects after all interactions, about 27.5 percentage points above the static baselines it compares against, and its masks can be used to prompt a foundation model for sharper boundaries.","feed_headline":"Pushing objects once lets a robot segment them with no training","feed_subtitle":"Frame-invariant motion features beat static segmentation models by 27.5% in clutter.","key_machinery":"The central object is the Body Frame-Invariant Feature (BFIF), a spatial twist $V_s = [\\omega_s, \\upsilon_s]^\\top \\in \\mathbb{R}^6$ expressed in a fixed space frame. For a body frame $\\{b\\}$ on a rigid body, the spatial twist is computed from the transformation $T_{sb}$ via $\\dot{T}_{sb} T_{sb}^{-1} = [V_s]$; frames on the same rigid body yield the same $V_s$, while frames on different objects yield different ones. The pipeline's components are effective optical flow $X_t = O_t - E_t$ (observed flow minus expected flow from the wrist camera's own motion, computed from forward kinematics and depth), random non-collinear pixel triplets that define body frames $\\{F^k_{t-1}\\}$, tracked through the flow to $\\{F^k_t\\}$, and unsupervised grouping of the resulting BFIFs by Mahalanobis distance followed by Markov clustering. The BFIFs do the identification; the flood fill and mask propagation do the boundary assignment.","core_discovery":"The paper's central discovery is that object identity can be read off from motion alone during a minimally disruptive robot interaction. It defines the Body Frame-Invariant Feature (BFIF): for any two body frames rigidly attached to the same moving object, the spatial twists expressed in a fixed space frame are identical, regardless of the frames' absolute rotations and translations. rt-RISeg samples random triplets of pixels whose effective optical flow is large, builds a body frame from each triplet, tracks it to the next frame, computes its spatial twist, and clusters the resulting twist vectors using Mahalanobis distances and Markov clustering. Each cluster seeds a flood fill in the optical flow to produce and update an object mask during the interaction. The method needs no base segmentation mask and no training, and it outperforms the learned unseen-object segmentation models it is compared against in overlap and boundary metrics after two to three interactions.","pith_inferences":["Editorial inference: if the BFIF grouping is genuinely 3D, the same machinery should work with any dense motion cue, such as scene flow from RGB-D or lidar, and could segment objects during non-push interactions like grasping or pouring, not just pushes.","Editorial inference: the autonomously generated masks could be collected as pseudo-ground truth to train a conventional static segmentation model, turning the interactive pipeline into a data engine rather than an end product.","Editorial inference: a direct stress test would be a rigid object undergoing pure rotation about the camera axis with no translation; under 2D optical flow, points at different depths trace different pixel velocities, so the BFIF equality would only hold if frames are lifted to metric 3D coordinates."],"forward_implications":["Unseen object segmentation becomes a byproduct of manipulation: any robot that can push an object slightly can segment it, with no pre-trained segmentation network and no dataset of object appearances.","Because masks update during the action, downstream manipulation planning can react to segmentation while the push is still in progress, rather than waiting for an observe–act–observe cycle.","The method transfers to out-of-distribution objects by construction, since the grouping signal is rigid-body motion, not static visual features that can overfit.","rt-RISeg's masks can be fed as prompts to a foundation model like SAM, substantially improving boundary precision without losing the object-level grouping.","The reported 75.1% versus 45–50% correct-object rate after interactions implies that interactive perception can beat static segmentation on exactly the cluttered scenes where learned models are weakest."],"supporting_citations":[{"why":"Introduces the BFIF and the interactive segmentation idea that rt-RISeg builds on while removing the learned model.","marker":"[1]"},{"why":"Supplies the spatial-twist definition and the rigid-body fact that frames on the same object share a twist in a fixed frame.","marker":"[8]"},{"why":"Provides the optical flow estimates (RAFT) used as the observed motion input in Algorithm 3.","marker":"[39]"},{"why":"Serves as the state-of-the-art learned unseen-object segmentation baseline that rt-RISeg compares against.","marker":"[6]"},{"why":"Serves as the SAM baseline and the foundation model that rt-RISeg masks prompt for refined boundaries.","marker":"[10]"},{"why":"Provides the Markov clustering algorithm used to group BFIFs into object identities.","marker":"[36]"},{"why":"Provides RANSAC for tabletop plane fitting that defines which pixels are candidate objects in action selection.","marker":"[30]"}],"fun_headline_variants":["Robot segments objects by motion alone, no training required","Model-free robot segmentation beats trained networks by 27.5%","Real-time object segmentation from robot interactions, zero learning","Motion-based instance segmentation outperforms learned models in clutter","rt-RISeg: interactive robot vision that needs no static features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random pixel triplets used to build body frames are placed in metric 3D coordinates via the depth channel before their axes and twists are computed, so the BFIF comparison is a true 3D rigid-body comparison; the paper never explicitly says this 3D lift happens in Algorithm 3.","fun_headline_variants_meta":{"raw":{"variants":["Robot segments objects by motion alone, no training required","Model-free robot segmentation beats trained networks by 27.5%","Real-time object segmentation from robot interactions, zero learning","Motion-based instance segmentation outperforms learned models in clutter","rt-RISeg: interactive robot vision that needs no static features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1532,"prompt_tokens":966,"completion_tokens":566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":582,"tokens_out":566,"duration_ms":6905,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:25:36.075788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test whether one rigid object, pushed so that it rotates about an axis not parallel to the camera plane, is still returned as a single cluster: compute the BFIFs from 2D pixel coordinates alone, without depth, and check whether points at different depths on the same object give equal spatial twists. If the invariance fails under this 2D construction, the reported grouping rests on an unstated 3D step.","supporting_citations":[{"cited_title":"Riseg: Robot interactive object segmentation via body frame-invariant features,","cited_arxiv_id":null,"evidence_quote":"Introduces the BFIF and the interactive segmentation idea that rt-RISeg builds on while removing the learned model."},{"cited_title":"Raft: Recurrent all-pairs field transforms for optical flow,","cited_arxiv_id":null,"evidence_quote":"Provides the optical flow estimates (RAFT) used as the observed motion input in Algorithm 3."},{"cited_title":"Mean shift mask trans- former for unseen object instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Serves as the state-of-the-art learned unseen-object segmentation baseline that rt-RISeg compares against."},{"cited_title":"Segment anything,","cited_arxiv_id":null,"evidence_quote":"Serves as the SAM baseline and the foundation model that rt-RISeg masks prompt for refined boundaries."},{"cited_title":"Graph clustering by flow simulation,","cited_arxiv_id":null,"evidence_quote":"Provides the Markov clustering algorithm used to group BFIFs into object identities."}],"review_version":1}