{"id":"08dd2bb0-ccc1-4272-80e4-3b89ed9b7826","arxiv_id":"2506.17328","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reflective, memory-augmented VLM planner coordinating dual Franka arms achieves 87.2% task completion on simulated desktop cleaning, a 28.8 percentage point gain over a static VLM baseline.","lead":"A memory-augmented vision-language planner tells two simulated robot arms how to clean a desktop, reaching 87.2% task completion compared to 58.4% when the planner cannot reflect on its mistakes. The result matters because it shows that simple structured feedback loops can make large language models more useful for physical robot tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central result is load-bearing on Grounded-SAM2/scene-graph perception accuracy, which the paper never reports; if Isaac Sim supplies oracle masks and poses, the 87.2% success rate validates only the planner/controller, not the claimed open-vocabulary perception bridge.","rationale":"I read the paper as a simulation-only system demonstration whose headline contribution is the integrated pipeline, not any single module. The reader's conditional verdict is appropriate, and the weakest assumption is correctly identified: the perception component is load-bearing but completely unmeasured. Without perception metrics or an oracle-vs-real-perception ablation, the 87.2% success rate cannot be attributed to the open-vocabulary perception bridge, and the reported comparison to a static VLM baseline does not isolate perception quality. I do not see evidence of circular reasoning or internal contradiction; the concern is an empirical gap. The proposed test directly settles whether that gap matters. Since the reader already recommended CONDITIONAL acceptance with perception validation as a requirement, my analysis does not change the verdict, hence UNCHANGED.","tokens_in":2903,"tokens_out":2769,"duration_ms":34662,"concrete_test":"Instrument the Isaac Sim evaluation to log Grounded-SAM2 masks, class labels, and estimated 3D poses against simulator ground-truth instance/pose data for all 17 scenarios across the 3 seeds, and report mean mask IoU, label F1, and pose translation/rotation error. Then rerun the full pipeline twice with the planner and controller held fixed: once with actual Grounded-SAM2 outputs and once with oracle perception (ground-truth masks/poses). If the two success rates are statistically indistinguishable and perception metrics are high, the perception concern is resolved; if success drops materially with real perception, the central claim of bridging open-vocabulary perception is unsupported and the title-level contribution must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the full hierarchy—Grounded-SAM2 perception, scene graph construction, reflective VLM planning, and dual-arm execution—achieves 87.2% task completion and that this demonstrates a bridge from open-vocabulary perception to precise manipulation. That claim breaks if the perception layer is not actually exercised. Section II-A says RGB-D observations are processed by Grounded-SAM2 to yield masks, labels, and spatial relations, but Section III reports no perception metrics: no mask IoU, label accuracy, 3D pose error, or spatial-relation precision on any of the 17 scenarios. In Isaac Sim, ground-truth instance IDs, poses, and semantics are directly accessible, and nothing in the method or experiments rules out their use in constructing the scene graph or in supplying the 'current bounding boxes and masks' used during replanning. If perception is oracle, the 87.2% figure tests only the VLM planner and dual-arm controller under perfect perception, so the open-vocabulary perception component advertised in the title and abstract remains unvalidated. Moreover, the claimed 28.8 percentage point improvement over static VLM could be contingent on this idealized perception: a real RGB-D perception layer with noisy masks or labels might degrade the reflective planner's ability to recover, shrinking or eliminating the advantage. This is not an internal inconsistency, but it is a correctness risk for the generalization claim: the system may not transfer to real desktop scenes where perception errors are inevitable and where the 'bridging' contribution is supposed to matter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hierarchical architecture for dual-arm desktop cleaning on a simulated desktop. RGB-D observations are processed by Grounded-SAM2 to produce masks and labels, a scene graph is constructed, a memory-augmented VLM (Gemini-2.0-flash) generates JSON plans using five manipulation primitives, and two Franka arms execute these plans under impedance control with collision avoidance. The system is evaluated in Isaac Sim across 17 scenarios with 3 seeds each, reporting 87.2% task completion, a 28.8 percentage point gain over a static VLM baseline, a 36.2 percentage point gain over a single-arm baseline, and a 'Recovery' metric of 0.82. The paper claims that structured memory integration and reflective planning are crucial for robust manipulation.","tokens_in":3176,"tokens_out":3906,"duration_ms":44909,"significance":"The design has clear practical appeal: separating high-level VLM reasoning from low-level geometric execution, using a scene graph as a structured interface, and adding a rolling memory buffer for failure-driven replanning are sensible ideas that could generalize to other long-horizon manipulation tasks. The ablations against static-VLM and single-arm baselines directly isolate the contributions of reflection and bimanual coordination, which is a strength. However, the central 'bridging open-vocabulary perception and precise manipulation' claim is currently unsupported because the perception layer is never quantitatively validated, and the reported superiority lacks statistical grounding. The paper would be a useful systems contribution if these gaps were filled.","major_comments":[{"comment":"The paper never reports any accuracy metric for Grounded-SAM2 or the scene graph constructor: no mask IoU, no label accuracy, no 3D pose error, and no spatial-relation precision on any of the 17 scenarios. Since Isaac Sim provides ground-truth instance IDs, poses, and semantics, it is unclear whether the 'current bounding boxes and masks' used during replanning come from the perception pipeline or from oracle data. If the perception is oracle, then the 87.2% success rate validates only the VLM planner and dual-arm controller, and the paper's title and abstract claim of bridging open-vocabulary perception remains unvalidated. Please state explicitly what the perception pipeline receives and reports per trial, and provide quantitative perception accuracy (or, if unavailable, a clear statement that perception is assumed perfect).","section":"Section II-A and Section III-A"},{"comment":"The reported success rates (87.2% vs 58.4% vs 51.0%) are presented without any variance information. With only 3 seeds per scenario, a 28.8 percentage point gap could in principle be driven by a few hard scenarios or by high variance in the simulator. Please report per-scenario success counts, the standard error or confidence interval across seeds, and a paired significance test (e.g., Wilcoxon signed-rank test across the 17 scenarios) to support the claim that the reflective dual-arm system is statistically better than the baselines.","section":"Section III-A and Table I"},{"comment":"The text states that single-arm operation 'increases episode time by 28%', but Table I reports 164 s for the single-arm condition versus 118 s for the reflective dual-arm condition, which is a 39% increase (and only an 18% increase relative to the 139 s static-VLM baseline). This numerical inconsistency should be corrected, and the efficiency claim should be restated in a way that aligns with the tabulated numbers.","section":"Section III-A(a) and Table I"}],"minor_comments":[{"comment":"The 'Recovery' column in Table I is not defined. If it is the fraction of failed trials that are subsequently recovered by the reflective loop, please define it explicitly and report how it is computed; otherwise the column is uninterpretable.","section":"Section III-A and Table I"},{"comment":"The statement that the reflective loop runs in 'approximately 250 ms' is presented without measurement details. Please specify whether this is a measured mean, a latency bound, or an estimate, and mention the hardware/software stack used for timing.","section":"Section II-B"},{"comment":"The title contains a typographical artifact 'Open-V ocabulary Perception' and the abstract uses 'our system achieving' instead of 'our system achieves'; both should be corrected.","section":"Title and Abstract"},{"comment":"The paper refers to 'Grounded-SAM2' but cites reference [1], which describes Grounded SAM, not specifically SAM2. The authors should cite the correct model or clarify the relationship between the two.","section":"References"},{"comment":"The claim of 'real-time control performance' is not supported by any quantitative measure of control frequency or latency during execution. Consider reporting the servo control rate or stating what 'real-time' means in the context of this simulator.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"This reads like a short conference-style paper rather than a full journal submission. The most serious gap is the absence of any perception validation, which is directly relevant to the title's claim. In addition, the statistical support is thin. If the authors are resubmitting elsewhere, they should be encouraged to add perception metrics and significance tests. The numerical mismatch in the episode-time comparison should be fixed before any further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a plausible integration paper with a real gap between what the title promises and what the experiments can support. If you read it as a demonstration of a reflective VLM planner on a dual-arm cleaning task, it's fine. If you read it as evidence of a working open-vocabulary perception bridge, it's not supported.\n\nWhat's new: the specific combination of Grounded-SAM2, a JSON-schema VLM planner with rolling memory, and five parameterized primitives on dual Franka arms is not in the cited literature. The hierarchical separation between semantic planning and geometric execution is a reasonable pattern, and the task design—17 scenarios with fragile items, crumbs, and paper shreds—is more realistic than many sim-only benchmarks. The 28.8 point gap over the static baseline is directionally consistent with the prior reflective planning result, so the mechanism is believable.\n\nThe soft spots are mostly about evidence, not method. The biggest one is perception. The paper never reports a single metric on Grounded-SAM2's masks, labels, or scene graph accuracy. In Isaac Sim, ground truth is right there; nothing rules out the scene graph coming from the simulator rather than from the perception stage. If that's the case, the 87.2% number says nothing about open-vocabulary perception, and the title's bridging claim collapses. The authors need to state explicitly what perception outputs were used and report accuracy on a held-out split.\n\nSecond, the numbers are sloppy. The text says single-arm operation increases episode time by 28%, but Table I implies (164-118)/118 = 39%. Also, no error bars or significance tests are shown; 17 scenarios x 3 seeds is 51 trials, and a few failures move the percentage a lot. Third, the \"Recovery\" column is undefined in the text—I can guess it's the fraction of failed trials that were recovered, but the paper should say so.\n\nNone of this is fatal to the core idea, and I don't see circularity. The improvements are empirical, not fitted. The paper just overclaims the strength of its evidence.\n\nVerdict: this deserves a serious referee if the venue is open to simulation-only system papers. A referee should ask for three things: perception validation or an explicit statement about oracle use, error bars or per-scenario results, and a definition of recovery. With those, the paper would be a useful data point. As is, it's a promising demo with an unproven perception bridge.","headline":"A promising integration of reflective VLM planning with dual-arm cleaning, but the perception bridge is unvalidated and the numbers need tightening.","tokens_in":3724,"tokens_out":2954,"would_cite":false,"duration_ms":33200,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simulated dual-arm robot clears desktops with 87% success","keywords":["robotic desktop cleaning","vision-language model planning","reflective planning with memory","dual-arm manipulation","open-vocabulary perception","scene graph representation","Isaac Sim simulation"],"falsifier":"Run the same pipeline on a real cluttered desktop or in simulation with injected perception errors, such as random label flips on 10% of objects or eroding instance masks by 20%. If task completion stays near 87% under noisy perception, the reflective planner is carrying the load; if it falls sharply, the reported performance is bounded by perception accuracy rather than by the reflective planning mechanism.","tokens_in":1358,"feed_emoji":"🧹","tokens_out":1498,"duration_ms":47332,"temperature":0.7,"pith_summary":"This paper proposes a three-layer robotic system for cleaning a cluttered desktop: a perception stage using Grounded-SAM2 turns RGB-D images into a structured scene graph, a vision-language model (VLM) generates a JSON manipulation plan from that graph, and two Franka arms execute the plan through five parameterized primitives. The central claim is that adding a rolling memory of past plans and execution outcomes lets the VLM reflect on failures and revise its strategy, lifting task completion to 87.2% across 17 simulated scenarios, versus 58.4% for a static VLM and 51.0% for a single arm. The authors argue that structured intermediate representations, rather than end-to-end language generation, are what make VLM-based manipulation both geometrically precise and adaptable. If this holds, domestic robots could handle arbitrary debris and valuables without retraining, simply by re-prompting a VLM and letting it learn from its own execution mistakes.","feed_headline":"Simulated dual-arm robot clears desktops with 87% success","feed_subtitle":"Memory-based replanning lifts success 29 points over a static VLM plan.","key_machinery":"The reflective planning loop with a rolling memory buffer of the five most recent plans and their execution outcomes. When a primitive fails, detected through force feedback, pose errors, or timeouts, the VLM receives updated visual information, the memory context, and a structured prompt that tells it to reflect on the failure and revise its strategy. This loop closes in about 250 ms while real-time servo control continues uninterrupted. The other load-bearing element is the scene graph, serialized into text, which serves as the structured interface connecting open-vocabulary perception to language-based planning.","core_discovery":"The paper reports that a memory-augmented reflective VLM planner, fed by a structured scene graph from Grounded-SAM2, can drive coordinated dual Franka arms to clear a desktop of mixed debris and fragile valuables in simulation, achieving 87.2% task completion. Removing the memory buffer drops success by 28.8 percentage points and eliminates failure recovery entirely, while replacing the dual arms with a single arm drops success by 36.2 points and increases episode time by 28%. The authors interpret these results as evidence that structured VLM integration with reflection is a viable path toward generalizable domestic manipulation.","pith_inferences":["The 87.2% result is measured in Isaac Sim, and the paper reports no perception accuracy; if the simulator supplies near-perfect masks and labels, the reported gains may mostly reflect the reflective planner rather than open-vocabulary perception. A natural test is to inject mask noise or label flips and see whether task completion holds.","The reflective memory mechanism resembles in-context learning from execution history; this suggests that VLM planners could improve from their own rollouts without gradient updates, a property that might transfer to other manipulation domains.","The authors' framing implies that the scene graph, not the VLM's raw visual reasoning, is what carries geometric precision; if so, the approach's success on real robots will hinge on the quality of the scene graph constructor, not on the VLM's language ability.","A direct comparison against a system that feeds raw images instead of scene graphs to the VLM would isolate the contribution of the structured representation, which the current experiments do not provide."],"forward_implications":["Robotic cleaning can handle heterogeneous, unseen debris categories without retraining, provided the underlying detector recognizes the categories from text prompts.","Reflection from execution feedback can make VLM planners robust to failures without external supervisory signals, enabling self-improvement from mistakes during a single episode.","Dual-arm coordination with a shared occupancy map allows sweeping and pick-and-place to run in parallel, cutting task time by 28% relative to serialized single-arm operation.","Structured scene graphs offer a general interface for VLM-controlled manipulation beyond cleaning, potentially applying to other long-horizon household tasks.","The roughly 250 ms replanning latency suggests that language-model reasoning is not the bottleneck for real-time control, leaving perception and servo as the limiting factors."],"supporting_citations":[{"why":"Supplies Grounded-SAM2, the open-vocabulary detector that produces instance masks and labels for the scene graph.","marker":"[1]"},{"why":"Provides the reflective planning paradigm that the memory-augmented critique-and-revise loop is built on.","marker":"[2]"},{"why":"Grounds language-model actions in robot affordances, informing the design of the primitive-based execution interface.","marker":"[3]"}],"fun_headline_variants":["Reflective VLM planning lifts dual-arm cleaning to 87.2% success","Memory-augmented VLM improves dual-arm cleaning by 28.8%","Dual-arm robotic cleaner uses reflective VLM planning for 87% success","Reflective planning with memory improves dual-arm desktop cleaning","VLM reflection and memory achieve 87% dual-arm cleaning"],"cache_read_input_tokens":5888,"weakest_assumption_plain":"The system assumes that Grounded-SAM2 and the scene graph constructor produce accurate object masks, labels, and spatial relations from RGB-D in the simulator, yet the paper reports no perception accuracy; if the simulator provides idealized perception, the 87.2% success rate does not actually test open-vocabulary perception, and the title's bridging claim remains unvalidated.","fun_headline_variants_meta":{"raw":{"variants":["Reflective VLM planning lifts dual-arm cleaning to 87.2% success","Memory-augmented VLM improves dual-arm cleaning by 28.8%","Dual-arm robotic cleaner uses reflective VLM planning for 87% success","Reflective planning with memory improves dual-arm desktop cleaning","VLM reflection and memory achieve 87% dual-arm cleaning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001306,"raw_usage":{"total_tokens":5248,"prompt_tokens":790,"completion_tokens":4458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":4363}},"tokens_in":406,"tokens_out":4458,"duration_ms":33356,"temperature":1.0,"reasoning_tokens":4363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:37.494112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a real cluttered desktop or in simulation with injected perception errors, such as random label flips on 10% of objects or eroding instance masks by 20%. If task completion stays near 87% under noisy perception, the reflective planner is carrying the load; if it falls sharply, the reported performance is bounded by perception accuracy rather than by the reflective planning mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Grounded-SAM2, the open-vocabulary detector that produces instance masks and labels for the scene graph."}],"review_version":1}