{"id":"50c1f7d8-a3d0-42e3-94c6-ee1368c6688f","arxiv_id":"2508.08748","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A pick-and-place system overlays bounding boxes on camera images, trains an ACT transformer on human demonstrations, and reports 80% to 100% success rates across three retail scenarios.","lead":"This paper pairs bounding-box visual prompts with the ACT imitation learning algorithm for robotic pick-and-place tasks in convenience stores. It reports success rates across three task complexity levels and claims the annotations improve grasp accuracy and adaptability.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing issue: the method requires bounding-box annotations at inference (Algorithm 1, Fig. 3) but §III.C says images are fed to ACT 'without explicit bounding box annotations'; no detector or no-annotation ablation resolves this.","rationale":"The reader's weakest assumption identifies the same internal contradiction: the method claims to use bounding-box annotations, but §III.C says images are fed without them, and no detector or ablation supplies the missing link. I regard this as the single most load-bearing issue because the entire novelty claim—'annotation-guided visual prompting'—collapses if the prompts are not actually used at inference. The paper reports plausible success rates and includes attention heatmaps, but these do not resolve the contradiction or isolate the effect of prompts. Absent code, data, or a no-prompt baseline, the central claim cannot be verified. Since my concern matches the reader's, and the reader already recommends REJECT, my read does not change the verdict.","tokens_in":8755,"tokens_out":2650,"duration_ms":27432,"concrete_test":"Instrument the deployed inference pipeline (or, if code were released, inspect the data loader and model forward pass) to log whether bounding-box overlays are present in the images passed to ACT at test time. Then run the same 10-trial protocol from §V.D under three conditions: (a) ground-truth boxes drawn, (b) no boxes, and (c) boxes randomly shifted by 20 px. If (b) matches (a), or if (c) is omitted, the paper's central claim that annotation-guided prompting improves success is unsupported. Repeat across the three scenarios of Table I.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that annotation-guided visual prompting improves ACT-based pick-and-place. But the method description is internally inconsistent. Algorithm 1 (line 8) constructs Ivp_t by adding visual prompts to the RGB image, and Fig. 3 shows the prompted image as the ACT encoder input. Yet §III.C states that captured images are 'directly fed into ACT's CNN network ... without explicit bounding box annotations.' If true at inference, then the reported success rates in Table I do not demonstrate any benefit from visual prompting; they may simply be ACT operating on raw images. If false, the paper must explain how bounding boxes are obtained at test time—no automatic detector is proposed or evaluated, and §V.D does not test robustness to noisy or missing boxes. The absence of a with-prompt vs. without-prompt ablation makes it impossible to attribute the results to the annotation mechanism. This is not a matter of disputed consensus; it is an unresolved contradiction inside the paper's own methodology, and it directly undermines the central contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an imitation-learning pipeline for robotic pick-and-place in retail settings, combining Action Chunking with Transformers (ACT) with annotation-guided visual prompting: bounding boxes identify the target object and the destination, and the prompted RGB image is fed into the ACT encoder. The authors evaluate the approach in three scenarios of increasing complexity using five product types, reporting success rates between 70% and 100% and presenting attention heatmaps and failure-mode visualizations. The central claim is that this annotation-guided visual prompting improves grasp accuracy and adaptability compared to conventional ACT-based pick-and-place.","tokens_in":8992,"tokens_out":4945,"duration_ms":51071,"significance":"If the central claim were supported, the contribution would be practically useful: bounding-box prompting is a lightweight way to inject task-level spatial information into an imitation-learning policy without retraining on new objects, and the progressive-scenario evaluation is a reasonable way to stress-test the idea. The paper also builds on the public RoboManipBaselines framework and the existing ACT algorithm, and it provides qualitative attention and failure analyses. However, the evidence as presented does not substantiate the claimed improvement: there is no comparison with ACT without visual prompts, the method description is internally inconsistent about whether annotations are used at inference, and the reported success rates are statistically fragile and in places inconsistent with the accompanying table.","major_comments":[{"comment":"The method description is internally contradictory. Algorithm 1 (line 8) and Fig. 3 construct Ivp_t by adding visual prompts to the RGB image and use it as the ACT encoder input; Fig. 4 shows green/red bounding boxes as the task specification. Yet Section III.C states that captured images are 'directly fed into ACT's CNN network ... without explicit bounding box annotations.' If annotations are not available at inference, then Table I does not demonstrate any effect of visual prompting; if they are available, the paper must say how they are obtained — no automatic detector is proposed or evaluated, and there is no ablation for noisy or missing boxes. This is a load-bearing gap for the paper's main claim and must be resolved with a with-prompt vs. without-prompt comparison and a clear inference-time annotation protocol.","section":"Section V.D, Table I"},{"comment":"The reported numbers do not consistently match the table. In Scenario 2, the text says the success rate 'initially dropped to 70%' and later 'achieving a 100% success rate,' but Table I lists category-level rates of 70%, 0%, 100%, and 0% (average 42.5%) for the initial setting and 90%, 100%, 100%, and 100% (average 97.5%) for the more-data setting. Similarly, in Scenario 3 the text says the success rate 'initially decreased to 70%,' while the table's average is 85%. These discrepancies need to be reconciled, and the number of trials per category and per product should be stated explicitly.","section":"Section V.D"},{"comment":"There is no baseline condition: the experiments never run ACT without visual prompting under the same task definitions, data collection, and evaluation protocol. The abstract and conclusion claim 'improved grasp accuracy and adaptability,' but 'improved' is not measured against any control. At minimum, the paper needs an ACT-only ablation with raw RGB images, and ideally a comparison to a standard ACT policy or a non-prompted policy with the same demonstration budget. Without this, the reported success rates cannot be attributed to the annotation mechanism.","section":"Section V.D"},{"comment":"It is unclear whether the results in Table I come from the real UR5e/Robotiq setup or from simulation. Section III says 'Both real-world and simulation experiments were conducted using the RoboManipBaselines framework,' but Section V.C describes simulation (MuJoCo) only for hyperparameter tuning, and Section V.D does not specify the evaluation environment. This distinction is essential for interpreting the success rates and for reproducibility; the authors should state per experiment whether it was real or simulated and describe the evaluation protocol accordingly.","section":"Section III, Section V.C"}],"minor_comments":[{"comment":"Typo: 'handle and diverse product types' should likely be 'handle diverse product types.'","section":"Section III.D"},{"comment":"The category 'Rigid 1 (Choco Box, Biscuit Box)' groups two distinct products, but the text says 'Each product was tested with 10 trials.' Please clarify whether the success rate is per product or per category, and report the denominator for each row.","section":"Table I"},{"comment":"The visual-prompt rendering is underspecified: line width, color, coordinate frame, and whether prompts are overlayed on the 480x640 image or resized are not given. This affects reproducibility.","section":"Algorithm 1"},{"comment":"The attention heatmaps are described only qualitatively; the method for extracting and aggregating them across encoder layers is not stated. Please specify how the heatmaps are generated.","section":"Fig. 5"},{"comment":"Reference [19] (Dosovitskiy et al., 2014) appears not to be cited in the text; either cite it where relevant or remove it.","section":"References"},{"comment":"Phrases like 'before adding more data' are vague; please give the actual demonstration counts for the initial and augmented datasets in both Scenario 2 and Scenario 3.","section":"Section V.D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant problem and the idea of conditioning ACT on bounding-box prompts is worth investigating, but the current evidence does not support the paper's central claim. The two most urgent fixes are (1) adding an ACT-without-prompt baseline and an annotation-robustness study, and (2) resolving the contradiction between Section III.C and the prompting pipeline. The table/text discrepancies are also damaging and must be corrected. If the authors can supply this missing evidence, the paper may become acceptable; in its present form, the contribution is not demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper's central claim is not supported as written. Algorithm 1 and Figure 3 clearly show bounding boxes drawn on the input image fed into ACT, but Section III.C says the captured images are 'directly fed into ACT's CNN network... without explicit bounding box annotations.' Those two statements cannot both be true for the reported experiments. If the boxes are present, the paper never explains how they are obtained at test time—no detector is proposed, and the experiments don't test noise or missing boxes. If the boxes are absent, then the 70–100% success rates in Table I are just ACT on raw images, and the visual prompting is a no-op. There is no ablation comparing ACT with and without prompts, so the central contribution is unmeasured either way.\n\nWhat is genuinely here: a straightforward integration of established components—ACT imitation learning plus bounding-box-style visual prompting in the spirit of MOKA or CLIPort—applied to a convenience-store product set. The experimental structure (three difficulty levels, attention heatmaps, a failure taxonomy for misalignment and slippery surfaces) is reasonable and could be the skeleton of a solid empirical study.\n\nThe soft spots extend beyond the contradiction. Evaluation is 10 trials per product/condition with no error bars or statistical tests, so the reported distinctions (90% vs 80%, etc.) are within the noise of a binomial with n=10. The real-versus-simulation question is muddy: Section III says both setups were used, but Section V only describes simulation for hyperparameter tuning; the main results appear to come from the real robot, and that should be stated plainly. No code, data, or hyperparameter settings are released, which is disappointing for a system paper that cites an open framework. The paper also leans on self-citation (RoboManipBaselines) but that is not itself a problem; the cited tool is public.\n\nWho gets value? Someone looking for a quick example of ACT applied to a specific retail manipulation task might skim it. But as a contribution to the manipulation literature, the internal contradiction and missing control make the reported numbers uninterpretable as evidence for the method.\n\nRecommendation: I would desk reject this version. The contradiction is fixable and a proper ablation with more trials could make it a modest workshop-level paper, but as submitted it is not ready for referee time.","headline":"Central claim is undermined by an internal contradiction: the method section says images are fed without bounding boxes, while Algorithm 1 and Figure 3 show visual prompts as the input; no ablation measures the prompting effect.","tokens_in":9455,"tokens_out":3486,"would_cite":false,"duration_ms":31282,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that drawing bounding boxes on the camera image—green for the object to pick, red for the destination—is enough visual prompting for a transformer-based imitation policy to perform cluttered pick-and-place with 70–100% suc","keywords":["robotic manipulation","pick-and-place","visual prompting","bounding-box annotations","Action Chunking with Transformers","imitation learning","retail robotics","attention heatmaps"],"falsifier":"Run the trained policy on the same scenes with no bounding boxes drawn, or with boxes shifted by 10–20 pixels, and measure success over a sufficiently large number of trials. If success barely changes, the visual prompt is not doing the claimed causal work; if it collapses, the method requires an external annotation mechanism the paper does not provide.","tokens_in":8667,"feed_emoji":"🤖","tokens_out":5549,"duration_ms":60803,"temperature":0.7,"pith_summary":"The paper tries to establish that bounding-box annotations drawn directly on the camera image—green around the item to pick, red around the place location—are sufficient spatial prompts for a transformer-based imitation-learning policy to perform tabletop pick-and-place in cluttered convenience-store-like scenes. The authors pair these prompts with Action Chunking with Transformers (ACT), which predicts smooth chunks of future actions instead of one incremental step, and report success rates of 90% in the simple scenario, 70% initially in harder scenarios, and 90–100% after adding targeted demonstration data. If true, the practical payoff is that a robot can be retargeted to new products and layouts without expensive object models or full scene segmentation; the bottleneck shifts to obtaining accurate annotations and diverse human demonstrations.","feed_headline":"Box prompts steer robot pick-and-place to 90%+ success","feed_subtitle":"A retail robot using ACT imitation learning follows green and red bounding-box cues to grasp and place store products in cluttered scenes.","key_machinery":"The central object is the combination of annotation-guided visual prompting with Action Chunking with Transformers (ACT): bounding boxes are drawn on the RGB frames, and a transformer encoder–decoder predicts chunks of future joint and gripper actions conditioned on those prompted images, joint states, and a latent style variable. The visual prompt does the perceptual work by marking the target and destination, while ACT does the temporal work by outputting coherent multi-step action sequences instead of step-by-step commands.","core_discovery":"The paper's central claim is that a pick-and-place policy can be driven by minimal visual annotations: a green bounding box marks the object to grasp and a red bounding box marks the destination, and these prompts are overlaid on the RGB image that feeds the ACT transformer. ACT then predicts chunked action sequences—smooth, multi-step motions—rather than per-timestep commands. The authors show this pipeline works across three levels of task complexity using a UR5e arm and a two-finger gripper: in a uniform 3x3 arrangement of boxes the success rate is 90%; with diverse products it initially drops to 70%, but adding 20% more demonstrations for the failure cases raises it to 100%; in a more co","pith_inferences":["A testable extension the paper leaves implicit is coupling the prompted ACT policy with an automatic object detector that produces the bounding boxes at runtime; the paper assumes these annotations are given, so end-to-end performance under imperfect detections is unknown.","A reader could probe the causal role of the visual prompt by shifting or jittering the boxes at test time; if success does not drop, the policy may be relying on scene layout rather than the annotations, and if it does drop, the method's real-world utility hinges on annotation accuracy.","The box dimensions themselves could carry unused information: since the paper treats boxes as location cues only, training with boxes that also encode object size or approach direction might improve grasp selection without any architectural change.","The data-demands acknowledged in the conclusion suggest the main bottleneck is human demonstration cost, so a synthetic data-augmentation pipeline that mimics human trajectories could be the highest-value next step."],"forward_implications":["If the claim holds, full object segmentation or explicit pose estimation is unnecessary for this class of retail pick-and-place tasks; a simple box overlay is enough spatial guidance.","Targeted data collection works as a corrective: adding only 20% more demonstrations for failed cases lifted the complex scenario's success rate to 100%, suggesting failure-informed data is more valuable than uniformly more data.","The attention heatmaps imply the learned policy internalizes a pick-then-place itinerary, using the green box as the entry cue and the red box as the goal cue, which explains why chunked action prediction is a natural fit for the task.","Object material properties remain the hard boundary of the method: reflective and slippery products need extra demonstrations, and soft flexible objects are excluded entirely, so visual prompting alone does not solve grasp mechanics."],"supporting_citations":[{"why":"Supplies the ACT imitation-learning algorithm—the action-chunking transformer whose predictions form the learned policy.","marker":"[11]"},{"why":"Supplies the robot-learning software platform used to set up and run both simulated and real-robot experiments.","marker":"[15]"},{"why":"Supplies the physics simulator in which ACT hyperparameters (chunk size, epoch count, batch size) were tuned before deployment.","marker":"[20]"},{"why":"Supplies the deep-learning library used to implement and train the ACT model.","marker":"[21]"}],"fun_headline_variants":["Colored box prompts steer robot to 90% pick-and-place success","Retail robot uses bounding boxes to guide pick-and-place","Visual annotations boost robot grasp accuracy to 100%","ACT transformer follows box cues for smooth robot motion","Annotation-guided picking lifts success from 70% to 100%"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The reported success rates assume that accurate bounding boxes for the object to pick and the place location are available at test time, but the paper provides no automatic detector to supply them; if the boxes must be hand-drawn or are noisy, the claimed performance is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Colored box prompts steer robot to 90% pick-and-place success","Retail robot uses bounding boxes to guide pick-and-place","Visual annotations boost robot grasp accuracy to 100%","ACT transformer follows box cues for smooth robot motion","Annotation-guided picking lifts success from 70% to 100%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1303,"prompt_tokens":673,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":417,"tokens_out":630,"duration_ms":6643,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:19:46.830875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained policy on the same scenes with no bounding boxes drawn, or with boxes shifted by 10–20 pixels, and measure success over a sufficiently large number of trials. If success barely changes, the visual prompt is not doing the claimed causal work; if it collapses, the method requires an external annotation mechanism the paper does not provide.","supporting_citations":[{"cited_title":"Z., Kumar, V ., et al","cited_arxiv_id":null,"evidence_quote":"Supplies the ACT imitation-learning algorithm—the action-chunking transformer whose predictions form the learned policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the robot-learning software platform used to set up and run both simulated and real-robot experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the physics simulator in which ACT hyperparameters (chunk size, epoch count, batch size) were tuned before deployment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the deep-learning library used to implement and train the ACT model."}],"review_version":1}