{"id":"bef02747-a663-4ea2-9dc4-bae2e3c11f25","arxiv_id":"2505.02232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A student-teacher framework with a memory-augmented student policy over SAM 2 detections learns prompt-responsive grasping from clutter in simulation and transfers to a real robot on tabletop tasks.","lead":"The authors train a robot to pick a specific object out of clutter when told which one to get, using the SAM 2 segmentation model to turn the prompt into a target signal and a memory-based student network to cope with imperfect detections. It matters because it shows a practical path to making reinforcement-learned dexterous manipulation controllable by high-level prompts, with small real-robot tabletop tests.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt-responsiveness is only validated for ground-truth bounding-box prompts; real user prompts (points or text) are never tested, so the central claim rests on an untested proxy.","rationale":"In good faith, the paper delivers a coherent student-teacher pipeline that keeps SAM 2 out of the RL loop, and the internal evidence is consistent: the teacher performs well, the student tracks the teacher, and the context-length experiment supports the claim that history helps the student compensate for imperfect detections. The released code and videos are also independently checkable. The most load-bearing gap is the prompt proxy. The reader identified the same concern, but I would sharpen it: the discrepancy is not merely that real prompts may be noisier; for language prompts there is no described mechanism at all, and for point prompts the training distribution differs from the deployment distribution in a way that the memory mechanism is specifically supposed to address but is never tested against. That makes the headline claim of prompt-responsive retrieval conditional on unshown prompt-robustness. I do not think this overturns the paper, because the core student-teacher contribution and the bbox-prompted grasping results stand on their own, but the advertised user-facing capability should be verified before the claim is accepted at face value. The reader's CONDITIONAL verdict already captures this; my concern reinforces it rather than changing it.","tokens_in":9380,"tokens_out":5418,"duration_ms":72716,"concrete_test":"Using the released code and trained student policies, run the tabletop and bin simulation benchmarks with (a) the current ground-truth bounding-box prompt, (b) point prompts sampled uniformly from visible target-object pixels, and (c) text-derived prompts if a language module exists in the released code; otherwise state explicitly that text prompting is not implemented. Report lift and goal success for the LSTM and Transformer students over the same training/held-out split used in Table IV, with at least three seeds. Additionally, report the prompt modality used for each Table V real-robot trial and rerun the real-robot evaluation with point prompts if the simulation point-prompt results are comparable to the bounding-box results. If success under (b) or (c) drops substantially below Table IV, the central claim should be scoped to bounding-box prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the robot can be conditioned by 'human-understandable prompts' to retrieve specified objects and that the student observation space transfers zero-shot. The only prompt modality actually used in training is the automated bounding box described in Section III.D: ground-truth mesh points are projected into the camera frame and a tight image-space box is fed to SAM 2. The student policy therefore learns to interpret SAM 2 detections generated under an oracle-quality box perfectly aligned with the target. Section III.A and Fig. 2 advertise text or point prompts at deployment, and Table V reports real-robot success, but the paper never states which prompt modality produced Table V and never evaluates point or language prompting in simulation or on the robot. This matters because SAM 2 is not a text-conditioned model; no mechanism is described for converting an open-vocabulary text description into a SAM 2 prompt, so the 'language' branch of the claim has no implemented path in the paper. Point prompts, in contrast, are implementable but create a different detection-error distribution (partial masks, background leakage, wrong-object selection) than the tight ground-truth boxes used during training. Since the whole method compensates for imperfect detections via history, a mismatch between training-time prompt error and deployment-time prompt error is exactly the failure mode the memory mechanism would need to handle, and it is unmeasured. Thus the load-bearing assertion that the approach 'successfully learns prompt-responsive policies' is currently supported only for a privileged proxy of prompting, not for the user-facing interface the abstract and introduction promise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a two-stage student-teacher framework for prompt-conditioned grasping from clutter. A teacher policy is trained with PPO on privileged simulator state (target OBB, heightmap, fingertip states), and a memory-augmented student policy is trained with DAgger to imitate the teacher from proprioception plus detections produced by SAM 2 from automated bounding-box prompts. The student architectures considered are a 1D-CNN, an LSTM, and a Transformer. Experiments are conducted in Isaac Lab on 48 training and 12 held-out YCB objects in both tabletop and bin scenarios, reporting 80-86% student goal success in simulation and 50-60% success in 10-trial real-robot tabletop runs. The authors claim prompt-responsive policies and zero-shot real-robot transfer, with the core idea that a history of imperfect SAM 2 detections enables implicit state estimation.","tokens_in":9609,"tokens_out":8543,"duration_ms":104136,"significance":"If the claims hold, the paper makes a useful engineering contribution: it shows a tractable way to use SAM 2 outputs as observations for an RL policy without backpropagating through the foundation model, and the context-length ablation is a clean, internally consistent test of the memory hypothesis. The release of videos and code is a strength, and the explicit evaluation of held-out objects is good practice. The broader significance is currently limited, however, by the gap between the claimed prompt modalities (points and language) and the only implemented modality (ground-truth bounding boxes), as well as by the thin real-robot evaluation and the absence of a direct comparison with prior SAM-based grasping work.","major_comments":[{"comment":"The only prompt modality actually implemented and evaluated is the automated tight bounding box computed by projecting the ground-truth target mesh into the camera frame (Section III.D). The paper's central claim, however, is framed around user prompts of point, text, or bounding-box type (Section III.A and Fig. 2). SAM 2 is not a text-conditioned model, and no mechanism is described for converting an open-vocabulary text description into a SAM 2 prompt; Table V does not state which prompt modality was used for the real-robot trials. Because the student's belief state is built from SAM 2 detections generated under oracle-quality boxes, a deployment with point or language prompts would face a different detection-error distribution than anything seen in training. As written, the experiments demonstrate responsiveness to ground-truth bounding boxes, not to 'human-understandable prompts' as claimed in the abstract.","section":"III.D, Fig. 2, Table V"},{"comment":"The real-robot evidence is limited to 10 trials per condition, with no variance estimates, no statistical test, and no bin-picking deployment; the conclusion acknowledges that bin-picking deployment is future work. The statement that 'the strong performance of the policies in simulation transfers to our real robot system' is not supported at the claimed precision: 5/10 versus 6/10 differences between train and test objects are within sampling noise, and the 50-60% real-robot success is substantially below the 84-86% simulated student success for the same tabletop task. The zero-shot transfer claim needs either more trials, error bars, or a more modest interpretation.","section":"V.B, Table V"},{"comment":"The architecture ablation does not isolate the memory mechanism. All three student architectures (1D-CNN, LSTM, Transformer) consume a history of observations, and no non-memory student is included, so the claim that history-awareness is a core mechanism is only partially tested. The context-length curve in Fig. 3(C) is a useful qualitative check, but it is reported for a single teacher-student setting, without numerical loss values or error bars. I would also like to see a student variant without SAM 2 detections to evaluate the contribution of the VFM backbone to the reported performance.","section":"V.A, Table IV, Fig. 3(C)"},{"comment":"No direct comparison is made to the authors' prior Grasp Anything method [21], which also uses SAM for prompt-based tabletop grasping. Because [21] is cited as related work and addresses the same task family, the absence of a baseline makes it difficult to assess what the memory-augmented student-teacher formulation adds over the prior instance-segmentation-plus-RL pipeline. This is particularly relevant for evaluating the paper's novelty and the practical significance of the reported gains.","section":"II.A, V"},{"comment":"The necessity of the student-teacher decomposition is asserted but not empirically tested. The authors motivate it by compute constraints, but no comparison is shown to an alternative policy trained directly on SAM 2 detections with reward, or to an end-to-end variant. Without such an ablation, the first of the two 'core mechanisms' listed in the introduction remains a design choice rather than a demonstrated requirement.","section":"III.D, I"}],"minor_comments":[{"comment":"Typo: 'follwoing' should be 'following'.","section":"III.D"},{"comment":"The spacing in the results table is inconsistent ('6/10' vs '6 /10'), and the table would benefit from a column indicating the prompt modality used for each trial.","section":"Table V"},{"comment":"The text and figure caption refer to an 'imitation loss' while the y-axis in the bottom-right panel reads 'DAgger Loss'; the values are too small to read, and the relationship between 'Visible Timesteps' and the history length H in Section III.D should be stated explicitly.","section":"Fig. 3"},{"comment":"The entry 'SAM2 detected point-cloud 4D* Npoints' does not define the fourth feature dimension or the value of Npoints; this is needed to reproduce the student observation space.","section":"Table I"},{"comment":"DAgger hyperparameters (number of iterations, dataset size, history length H, and the choice of teacher used for the context-length experiment) are not reported; please add them.","section":"III.D"}],"recommendation":"major_revision","confidential_remarks":"The overlap with the authors' own prior work [21] is substantial; a direct comparison would help the editor and reviewers judge the incremental contribution. The absence of any report of the real-robot prompt modality should be resolved before acceptance. I am not questioning the authors' integrity, but the narrowness of the prompt evaluation should be treated as a central issue rather than a presentation nit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Send it to review. The paper describes a clean student-teacher pipeline for prompt-conditioned grasping from clutter: train a teacher with privileged state, distill to a student that sees only proprioception plus SAM 2 segmentations, and give the student a history (LSTM or Transformer) so it can tolerate imperfect detections. The new bit is the combination—SAM 2 as a promptable perception backbone inside a memory-augmented distillation loop—and it is a sensible way to keep the foundation model out of the RL loop. The teacher numbers (82–94%) and student numbers (80–86%) are internally consistent, and the context-length loss curve is a decent direct test of the memory hypothesis. Real-robot transfer at 5–6/10 on held-out objects is encouraging, and the project page with code and videos helps reproducibility.\n\nThe main soft spot is one the stress-test flags, and it is real. All training uses an automated prompt: ground-truth mesh points projected into the camera frame, then a tight bounding box fed to SAM 2. The paper advertises text and point prompts at deployment, but no text mechanism is described (SAM 2 is not a language-conditioned model) and no point-prompt experiments appear. The student therefore learns to interpret detections produced by a perfectly aligned oracle box. Real user prompts—especially points—will produce a different error distribution (partial masks, wrong objects, background leakage), and that is exactly what the memory mechanism is supposed to fix. The gap is on the central claim. It is not fatal; bounding-box prompts are a legitimate prompt modality, and the method could work with points, but the paper needs to test at least one non-oracle prompt modality and state what it used on the real robot.\n\nOther issues are smaller. There is no baseline against their prior Grasp Anything [21] or a no-history student; the 1D-CNN is a weak baseline, not a proper ablation. The number of seeds for simulation is not stated (standard errors imply multiple seeds, but say so). Real-robot evidence is 10 trials per condition and only tabletop; bin-picking hardware is deferred, which the limitations section acknowledges. None of these are load-bearing flaws.\n\nThe citation pattern is fine; they position clearly against their own earlier work. This paper deserves a serious referee: the gap between the prompt claim and the prompt test is addressable and the engineering is solid. I'd ask for revision, not reject.","headline":"A clean, well-engineered student-teacher pipeline for prompt-conditioned grasping that deserves review, but the prompt-responsiveness claim rests solely on ground-truth bounding-box prompts.","tokens_in":10221,"tokens_out":2778,"would_cite":true,"duration_ms":32497,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a robot can be given a prompt — a text description, a point in an image, or a box around an object — and then retrieve that specific object from a cluttered scene without retraining for the object.","keywords":["prompt-responsive manipulation","Segment Anything 2","student-teacher learning","memory-augmented policy","reinforcement learning","object retrieval from clutter","implicit state estimation","zero-shot real-robot transfer"],"falsifier":"Run the deployed student policy on the real robot with deliberately noisy prompts — a text description that fits two objects in the scene, a click point on the wrong object, or a bounding box misaligned by several pixels — and measure success rates against the ground-truth-derived box prompts used in training. A large drop would show that the simulated automatic prompting is not a faithful proxy for real user prompts and that the policy's prompt-responsiveness is narrower than claimed.","tokens_in":9127,"feed_emoji":"🤖","tokens_out":7402,"duration_ms":75632,"temperature":0.7,"pith_summary":"This paper claims that a robot can be given a prompt — a text description, a point in an image, or a box around an object — and then retrieve that specific object from a cluttered scene without retraining for the object. The method couples the promptable segmentation model SAM 2 with reinforcement learning through a student-teacher scheme: a teacher policy learns from privileged simulator state, and a memory-augmented student policy learns to imitate it from SAM 2's imperfect detections. Because occlusions and misdetections make the detections non-Markovian, the student consumes a history of detections through an LSTM or Transformer to implicitly reconstruct the target's state. The authors report high success rates in simulation for tabletop and bin picking, and similar success rates on a real robot for seen and unseen objects. If correct, this offers a path to warehouse-style 'pick this item' commands without per-object retraining or explicit object models.","feed_headline":"Robots follow SAM 2 prompts to retrieve objects from clutter","feed_subtitle":"Student-teacher learning with memory turns noisy detections into reliable grasping, in simulation and on a real robot.","key_machinery":"The load-bearing mechanism is memory-augmented student-teacher learning with SAM 2 as the prompt-responsive perception backbone. SAM 2 maps a user prompt to a segmentation mask; the mask is projected onto a depth-derived point cloud and encoded by a PointNet-like encoder, and this encoding is concatenated with proprioception and fed to a history-aware policy (an LSTM or Transformer) that consumes a window of recent detections. Keeping SAM 2 out of the RL loop — training the teacher on privileged state and only distilling to the visual student — makes training tractable, and the history processing lets the student compensate for missing or unstable detections by implicitly tracking the target across time.","core_discovery":"The central discovery is that imperfect, prompt-derived detections from a vision foundation model can serve as the observation space for a dexterous manipulation policy, provided the policy can integrate detection history. The authors formulate the problem as a partially observable Markov decision process: SAM 2's outputs are non-Markovian, so the student policy receives a sequence of detected point clouds, with a feature marking which points SAM 2 assigns to the target, and must implicitly infer the underlying object state. Training proceeds in two stages. First, a PPO-trained teacher policy solves the task using privileged simulator information, including oriented bounding boxes, a heightmap of the clutter, and fingertip states. Then, a student policy is trained with DAgger to imitate the teacher from proprioception and SAM 2-based detections alone. The student's memory-augmented architectures recover most of the teacher's performance, and the learned policies transfer zero-shot to a real robot. The authors interpret the decreasing imitation loss with longer visible context as evidence that the student learns to infer the true object state from the detection history rather than memorizing a reactive mapping.","pith_inferences":["Because the student policy's observation space is independent of prompt modality — the prompt only shapes the SAM 2 mask — the same trained policy could be exposed to text, point, or box prompts at deployment without retraining, but this is untested: the real-robot section does not report which modality was used.","The implicit state estimation carried by the student's memory could be probed directly: train a linear classifier on the student's recurrent state to predict the target's true pose, testing whether the model genuinely tracks state rather than merely reacting to recent masks.","The approach suggests a general recipe for grounding any promptable foundation model into RL policies: factor the problem into a privileged teacher, a foundation-model observer, and a memory-augmented student, which could apply to other perception backbones and to tasks where sensory signals are lossy or intermittent.","One stress test not reported: bin picking on the real robot. The simulation results for bins are strong, but the real-robot experiments cover only tabletop scenes, so the 'cluttered bin' claim currently rests on simulation evidence alone."],"forward_implications":["A single policy can handle many object types: in simulation, the teacher and student generalize across dozens of training objects, and the real-robot results show similar success for held-out objects.","Occlusion robustness follows from memory: because the student consumes detection history, targets can disappear from view or be misdetected without breaking the policy, unlike earlier prompt-based grasping that required unobstructed tracking.","The same pipeline should extend to other manipulation skills beyond retrieval, since the formulation is a generic POMDP solved by history-aware imitation from imperfect foundation-model detections.","Prompt-conditioning removes the need for explicit object models at deployment: the robot never sees a CAD model or ground-truth state at test time, only the user's prompt and the RGB-D stream."],"supporting_citations":[{"why":"Supplies SAM 2, the promptable perception backbone that converts user prompts into the detection masks the student policy observes.","marker":"[14]"},{"why":"Supplies DAgger, the imitation-learning algorithm used to distill the teacher into the memory-augmented student from imperfect observations.","marker":"[29]"},{"why":"Provides the memory-augmented student-teacher formulation for implicit state inference from observation history that this method builds on.","marker":"[23]"},{"why":"Introduces learning-by-cheating privileged-information distillation, the two-stage training scheme this paper adapts.","marker":"[22]"},{"why":"Provides PPO, the reinforcement-learning algorithm that trains the privileged teacher policy.","marker":"[27]"},{"why":"Supplies the simulation framework used to build the parallel multi-task manipulation environments.","marker":"[30]"},{"why":"Provides the object set used for training and held-out generalization evaluation.","marker":"[31]"}],"fun_headline_variants":["Memory turns noisy AI vision into reliable robot grasping","Robots learn to grab objects from imperfect SAM 2 prompts","Student-teacher learning lets robots follow visual prompts","Dexterous robots retrieve objects using memory-augmented prompts","Prompt-driven robot picking with memory-augmented vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automated prompting in simulation, which projects the ground-truth target mesh into the camera frame to build SAM 2's bounding-box prompt, is assumed to be a faithful proxy for real user prompts, whether text, points, or boxes; the real-robot experiments do not report which prompt modality was used.","fun_headline_variants_meta":{"raw":{"variants":["Memory turns noisy AI vision into reliable robot grasping","Robots learn to grab objects from imperfect SAM 2 prompts","Student-teacher learning lets robots follow visual prompts","Dexterous robots retrieve objects using memory-augmented prompts","Prompt-driven robot picking with memory-augmented vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1429,"prompt_tokens":920,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":536,"tokens_out":509,"duration_ms":6041,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:53.150157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deployed student policy on the real robot with deliberately noisy prompts — a text description that fits two objects in the scene, a click point on the wrong object, or a bounding box misaligned by several pixels — and measure success rates against the ground-truth-derived box prompts used in training. A large drop would show that the simulated automatic prompting is not a faithful proxy for real user prompts and that the policy's prompt-responsiveness is narrower than claimed.","supporting_citations":[{"cited_title":"A reduction of imitation learning and structured prediction to no-regret online learning,","cited_arxiv_id":null,"evidence_quote":"Supplies DAgger, the imitation-learning algorithm used to distill the teacher into the memory-augmented student from imperfect observations."},{"cited_title":"Learning by cheating,","cited_arxiv_id":null,"evidence_quote":"Introduces learning-by-cheating privileged-information distillation, the two-stage training scheme this paper adapts."},{"cited_title":"Orbit: A unified simulation framework for interactive robot learning environments,","cited_arxiv_id":null,"evidence_quote":"Supplies the simulation framework used to build the parallel multi-task manipulation environments."},{"cited_title":"The YCB object and model set: Towards common benchmarks for manipulation research,","cited_arxiv_id":null,"evidence_quote":"Provides the object set used for training and held-out generalization evaluation."}],"review_version":1}