{"id":"ffc31106-f1eb-41cc-9712-84185be3293a","arxiv_id":"2506.18448","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent system with planner, coder, and observer agents achieves zero-shot language-driven grasp detection that outperforms existing baselines on benchmarks and robots.","lead":"GraspMAS combines three AI agents, a planner, a coder, and an observer, to let robots grasp objects from natural language commands without any fine-tuning. It reports higher success rates than existing systems on two standard datasets and on simulated and real robots, which could make human-robot interaction more flexible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Observer feedback loop is the unablated component that distinguishes GraspMAS from ViperGPT; without an Observer-off variant, the closed-loop mechanism claimed to drive the reported gains is not supported.","rationale":"The reader's weakest-assumption analysis correctly identifies the missing Observer ablation as the key under-supported link between the reported numbers and the claimed multi-agent closed-loop mechanism. I do not see an internal inconsistency in the method or a metric error that would overturn the empirical comparisons; the benchmark numbers, real-robot trial protocol, and qualitative in-the-wild evidence are genuine supporting material. The concern is attribution: the full system is never compared against its own no-feedback variant, and ViperGPT is almost exactly that variant with a possibly weaker Planner prompt. If the Observer contributes nothing, the central novelty of the paper reduces to prompt engineering on top of an existing tool-augmented pipeline, even though the headline success rates might remain correct. The requested ablation is cheap and decisive, so the existing CONDITIONAL verdict is the right one; no verdict change is needed. Secondary issues such as missing code, lack of error bars on real-robot trials, and ambiguity around the Harmonic Mean metric on GraspAnything++ would also be addressed by a release with seeds and per-configuration logs, but they are not the single most load-bearing concern.","tokens_in":13257,"tokens_out":6334,"duration_ms":72802,"concrete_test":"Run the full GraspMAS pipeline on a fixed 1,000-sample subset of OCID-VLG (or the full test split) in three configurations: (i) full GraspMAS with Observer feedback; (ii) Planner+Coder with Observer disabled, returning the grasp after the first Coder execution; (iii) full system with Observer feedback replaced by a constant/empty message. Use identical prompts, toolset, and inference settings, and report success rates with paired differences. If configurations (ii) or (iii) match (i) within roughly 0.02-0.03, the Observer is not load-bearing and the paper should be reframed; if (i) clearly exceeds both, the closed-loop claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the closed-loop interaction among Planner, Coder, and Observer (Sec. III-A, III-D), and the results sections attribute the large margins to this reasoning loop. However, no experiment disables the Observer or halts after the first Coder execution. This matters because the closest baseline, ViperGPT, is also a Planner+Coder system equipped with the same toolset (Sec. IV-A), so the structural difference between ViperGPT and a no-Observer GraspMAS is mainly the Planner prompt and in-context examples. The reported margins (0.62 vs 0.44 on OCID-VLG; 0.68 vs 0.57 on GraspAnything++; 0.80 vs 0.64 and 0.76 vs 0.52 on the robot) could therefore come from stronger Planner prompting or tool orchestration rather than from Observer feedback. The qualitative examples show multiple feedback-driven revisions and the text says the loop runs for 'one or several reasoning loops,' but no loop-count distribution, per-step success, or Observer-disabled comparison is reported. The causal claim that the Observer improves plans, which is the stated key novelty over prior tool-augmented zero-shot systems, is thus untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GraspMAS, a training-free multi-agent system for zero-shot language-driven grasp detection. The system has three agents: a Planner (GPT-4) that generates step-by-step grasp strategies, a Coder that turns plans into executable Python using a fixed toolset (object detection, part detection, grasp detection, depth estimation, etc.), and an Observer (GPT-4o) that evaluates intermediate visual results and error logs and feeds textual feedback back to the Planner for refinement (Sec. III). Experiments on OCID-VLG and GraspAnything++ report success rates of 0.62 and 0.68, respectively, outperforming baselines such as ViperGPT (0.44 and 0.57); comparisons with supervised methods (Table III) and simulation/real-robot trials (Tables IV and V) also report large margins. The paper includes qualitative in-the-wild examples, failure cases, and a limitations discussion.","tokens_in":13506,"tokens_out":6673,"duration_ms":73250,"significance":"If the reported results hold, GraspMAS is a meaningful advance: it demonstrates that an orchestrated set of frozen foundation models can handle compositional language commands for grasping without task-specific training, and the margins over ViperGPT with the same toolset are large. The work is an empirical systems comparison against external baselines and ground-truth datasets, so circularity is not a concern. The paper is explicit about its toolset, provides quantitative robot experiments, and acknowledges limitations such as slow inference and failure in dense clutter. However, the central mechanistic claim—that the Observer feedback loop drives the improvements—is not tested by any ablation, and the 'significantly outperforms' claim is not supported by confidence intervals or significance tests, especially given small robot-trial counts. These two omissions currently prevent full confidence in the causal attribution and in the magnitude of the reported gains.","major_comments":[{"comment":"The Observer feedback loop is never ablated. Since ViperGPT, the strongest baseline, is also a Planner+Coder system equipped with the same toolset (Sec. IV-A), the only structural differences are the Planner prompt and the Observer loop. To support the claim that the closed-loop reasoning drives the reported gains (0.62 vs 0.44 on OCID-VLG, 0.68 vs 0.57 on GraspAnything++, and the robot margins in Tables IV and V), please add a GraspMAS variant without the Observer (Planner+Coder only, no refinement) and, if feasible, a variant with a fixed maximum number of refinement loops. Also report the distribution of the number of loops and, on a sample, the per-step success rate so the reader can see how often the Observer's feedback actually changes the final grasp.","section":"III-D and IV"},{"comment":"No confidence intervals or significance tests are reported for any success rate, although the text repeatedly says 'significantly outperforms' (Secs. IV-A, IV-D, IV-E). For the real robot experiments, each condition uses only 25 trials (Sec. IV-E); for example, 0.80 vs 0.64 in Table V corresponds to 20 vs 16 successes, whose 95% binomial confidence intervals overlap substantially. Please report binomial confidence intervals and appropriate statistical tests (e.g., Fisher's exact test or bootstrap) for at least the simulation and robot results, and state the number of independent runs or seeds used for the dataset evaluations.","section":"Tables II, IV, and V"},{"comment":"The metric definitions need clarification. Section IV-A says the GraspAnything++ evaluation uses the 'Harmonic Mean metric,' but Section IV-B says the supervised comparison uses 'success rate as defined in [25]' and Table III reports GraspMAS at 0.68. Please clarify whether the Harmonic Mean and the success rate are the same quantity for this dataset, and if they differ, justify using different metrics for the zero-shot comparison and the supervised comparison. Please also clarify the scope of the 'zero-shot' claim given that the toolset includes RAGT-3/3 (Table I), a trained grasp detector, and whether 'training-free' refers only to language-driven grasp models.","section":"IV-A and IV-B"}],"minor_comments":[{"comment":"Notation is inconsistent: the text uses 'ViberGPT' (Sec. IV-A) while tables use 'ViperGPT'; 'GroundDINO' (Table IV) appears alongside 'GrDINO' (Table II) and 'GroundingDINO' (Sec. IV-A); and 'QWEN2' (Sec. IV-A) appears as 'QWEN2-VL' in Table II. Please unify the names and citations.","section":"IV-A and Tables"},{"comment":"There are typos in the code examples and tool names: 'grasp_detction' and 'image_pacth' appear in Fig. 2, and Table I lists 'graspdetection' while the code uses 'grasp_detection'. Please fix these inconsistencies so the toolset matches the executable code.","section":"Fig. 2 and Table I"},{"comment":"The full system prompts, LLM versions (GPT-4 for Planner and GPT-4o for Observer), sampling temperature, and the loop termination criterion are not specified. Please include the complete prompts and generation settings in an appendix for reproducibility.","section":"III-B and III-D"},{"comment":"The simulation section reports 50 samples per scenario and the real-robot section reports 25 repetitions, but the paper does not state whether the same object arrangements and text prompts were used across all methods for the simulation experiments; Sec. IV-E only says 'the same text prompts are used for all methods.' Please describe the evaluation protocol more precisely.","section":"IV-D and IV-E"},{"comment":"The abstract says 'limiting their generation in real-world applications,' which appears to be a typo for 'generalization.'","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a robotics venue and the reported system is plausible, but the two main requests—an Observer-off ablation and statistical reporting for the small-N robot trials—are essential before the causal claims can be accepted. I would also encourage the editor to ask for the code and prompts to be made available with the revised version, since the paper currently only says the code 'will be released.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GraspMAS is a genuine new application of the visual-programming multi-agent pattern to language-driven grasp detection, and the reported results are strong enough that the paper deserves a real referee. The system composes existing tools (GroundingDINO, VLPart, RAGT, BLIP2, MiDaS, SAM, GPT-4) with a Planner-Coder-Observer loop and gets 0.62 vs 0.44 over ViperGPT on OCID-VLG, 0.68 on GraspAnything++, and 0.80/0.76 on real-robot single/clutter trials. Credit where due: the writing is clear, the toolset is minimal and sensible, the robot experiments are a real step beyond dataset-only papers, and the in-the-wild examples (Kleenex ambiguous brand, knife handle vs blade) are illustrative. The authors also state their own limitations (slow inference, dense clutter) without overclaiming.\n\nThe main soft spot is exactly what the stress-test note says: the Observer feedback loop is the differentiating mechanism from ViperGPT, and it is never ablated. The paper attributes the gains to 'closed-loop reasoning,' but a no-Observer variant (Planner+Coder with the same toolset) is missing. Since ViperGPT is structurally Planner+Coder, the reported margins could in principle come from the Planner prompt, the in-context examples, or the richer toolset. That is not fatal—the system as a whole outperforms baselines—but it means the stated novelty is unverified. A simple Observer-off ablation would settle it, and its absence is surprising.\n\nOther issues are in the same direction. No confidence intervals or significance tests anywhere, despite 25 real-robot repetitions per condition. Code is not released despite 'will be released' in the conclusion. And the GraspAnything++ metric is described as 'success rate using Harmonic Mean metric,' which is vague—the paper should state exactly which metric is being reported and how it is computed. None of these are load-bearing flaws; they are the difference between a strong systems paper and a publishable one.\n\nWho is this for? People working on language-conditioned grasping, tool-augmented LLM agents, and zero-shot robot manipulation. It is a useful reference for how far a composed foundation-model system can go without training.\n\nRecommendation: send it to peer review. The idea is credible, the experiments are substantial, and the missing ablation and statistical detail are exactly what referees should ask for. A serious reviewer would make this paper better.","headline":"GraspMAS is a credible zero-shot grasping system with strong numbers, but the Observer loop that supposedly drives the gains is never ablated, and the paper lacks statistical rigor.","tokens_in":14019,"tokens_out":2184,"would_cite":false,"duration_ms":24292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraspMAS claims that a zero-shot, training-free multi-agent loop—Planner, Coder, Observer—outperforms baselines for language-driven grasping, with 0.62/0.68 benchmark and 0.80/0.76 real-robot success.","keywords":["language-driven grasping","zero-shot grasp detection","multi-agent system","robotic manipulation","foundation models","large language models","grasp pose detection"],"falsifier":"Run GraspMAS on the OCID-VLG and GraspAnything++ test splits with the Observer disabled, so only the Planner and Coder operate in a single pass, and compare success rates with the reported 0.62 and 0.68; if the scores are essentially unchanged, the feedback loop is not carrying the claimed improvement, and tracking how often Observer feedback alters the final grasp pose would measure the mechanism directly.","tokens_in":13083,"feed_emoji":"🤖","tokens_out":9485,"duration_ms":93177,"temperature":0.7,"pith_summary":"The paper tries to establish that language-driven grasp detection can be done zero-shot, with no training or fine-tuning, by composing off-the-shelf vision and language models inside a closed multi-agent loop. The loop has three specialist roles: a Planner that turns an image and a text query into a step-by-step strategy, a Coder that writes and runs Python code calling a small toolset, and an Observer that inspects intermediate crops, grasp rectangles, and error logs and sends textual feedback that refines the plan. On the OCID-VLG and GraspAnything++ benchmarks the authors report success rates of 0.62 and 0.68, beating end-to-end detectors and prior compositional systems such as ViperGPT given the same tools. Simulated and real-robot trials report 0.82/0.72 and 0.80/0.76 success in single-object and cluttered settings. If the results hold, language-conditioned grasping becomes a composition problem rather than a dataset-collection problem.","feed_headline":"Three-agent loop nails language-driven grasping without training","feed_subtitle":"Planner-Coder-Observer loop outperforms trained baselines on benchmarks and real robots with zero fine-tuning.","key_machinery":"The central object is the three-agent closed loop. The Planner (GPT-4) receives the image, the query, and any prior feedback, and emits a step-by-step grasp strategy. The Coder converts that strategy into executable Python that calls nine predefined tool functions, each wrapping a foundation model: open-vocabulary detection (find), part detection (find_part), grasp-pose proposal (grasp_detection), depth estimation (compute_depth), segmentation (masks), and VLM-based verification and grounding (exists, verify_property, best_image_match, llm_query). The Observer (GPT-4o) examines the visualized intermediate results and code error logs and writes textual feedback that starts the next Planner-Coder round. This loop is what distinguishes GraspMAS from a one-shot program generator such as ViperGPT, and it is the component the paper credits for refinement of ambiguous or task-oriented instructions.","core_discovery":"The paper's central claim is that GraspMAS, a closed-loop multi-agent system, achieves state-of-the-art zero-shot language-driven grasp detection without any training or fine-tuning. It reports success rates of 0.62 on OCID-VLG and 0.68 on GraspAnything++, surpassing end-to-end baselines (OWLv2, GroundingDINO, and QWEN2-VL, each paired with a grasp detector) and compositional baselines (OWG and ViperGPT, with ViperGPT given the identical toolset). In robot experiments the method reports 0.82 single-object and 0.72 clutter success in ManiSkill simulation, and 0.80 single-object and 0.76 clutter success with a Kinova Gen3 real robot. The authors further report that GraspMAS outperforms supervised grasp-detection models on GraspAnything++, including GraspSAM, without having been trained on grasp data. The mechanism behind the gains is iterative feedback from the Observer, which lets the Planner re-express ambiguous queries (resolving 'Kleenex' into 'blue rectangle box') and choose task-appropriate grasp points (the handle for cutting, the blade for handing a knife over safely).","pith_inferences":["The paper never reports a Planner+Coder-only version, so whether the Observer feedback loop is the source of the gains, rather than the specific prompts, tool selection, or model choices, is not established by the experiments.","A direct test would record, for each successful query, whether the Observer's feedback changed the final plan; if most successes come from the first Planner-Coder pass, the loop contributes less than the paper's framing suggests.","The same Planner-Coder-Observer pattern could transfer to other language-conditioned manipulation outputs, such as placing, tool use, or non-prehensile pushing, since the loop is task-agnostic apart from the toolset.","The zero-shot claim currently rests on proprietary GPT-4 and GPT-4o APIs; whether the success rates survive with open-weight language models is untested and affects reproducibility."],"forward_implications":["If the reported numbers hold, supervised training on grasp datasets is no longer necessary for language-conditioned grasping in new settings, because the method composes existing foundation models.","Because the toolset is modular, replacing any component (detector, part segmenter, grasp proposer, depth estimator) with a better one should improve overall success without retraining the pipeline.","The Observer's ability to reinterpret queries using world knowledge suggests the approach can handle brand names and functional descriptions that closed-vocabulary detectors cannot.","The reported inference time of about 2.1 seconds per query, versus fractions of a second for end-to-end baselines, means the approach currently trades speed for reasoning and fits non-time-critical human-robot interaction better than industrial pick-and-place.","The failure cases shown for heavily overlapping objects indicate that dense clutter still needs an obstacle-removal or re-arrangement capability before the method can fully solve it."],"supporting_citations":[{"why":"Provides the GraspAnything++ dataset and the supervised LGD baseline that GraspMAS is compared against.","marker":"[8]"},{"why":"Provides the OCID-VLG benchmark for cluttered tabletop language-driven grasp evaluation.","marker":"[9]"},{"why":"ViperGPT is the closest compositional baseline; GraspMAS shares its code-generation approach and is evaluated against it with the same toolset.","marker":"[31]"},{"why":"OWG is the open-world grasping baseline that uses LLM reasoning and is outperformed in all reported settings.","marker":"[27]"},{"why":"GroundingDINO implements the find tool, the main open-vocabulary object localization primitive in the system.","marker":"[61]"},{"why":"VLPart implements the find_part tool, enabling task-oriented grasps on object parts such as handles and blades.","marker":"[62]"},{"why":"RAGT-3/3 implements the grasp_detection tool that proposes grasp rectangles for detected objects.","marker":"[63]"},{"why":"GPT-4 and GPT-4o are the models used as the Planner and Observer agents, respectively.","marker":"[11]"}],"fun_headline_variants":["Zero-shot grasp via Planner-Coder-Observer loop","Robot grasps by language: multi-agent, no training","GraspMAS: three agents, zero training, beats baselines","Language-driven grasping: agents reason, no fine-tuning","Multi-agent loop for zero-shot language-driven grasp"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Observer feedback loop is what produces the reported accuracy; the paper presents the closed-loop refinement as central but never tests a version without the Observer, so if a Planner+Coder-only variant scored the same, the system's distinctive mechanism would be unverified.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot grasp via Planner-Coder-Observer loop","Robot grasps by language: multi-agent, no training","GraspMAS: three agents, zero training, beats baselines","Language-driven grasping: agents reason, no fine-tuning","Multi-agent loop for zero-shot language-driven grasp"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2665,"prompt_tokens":989,"completion_tokens":1676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1595}},"tokens_in":605,"tokens_out":1676,"duration_ms":13530,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:15:40.941914+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GraspMAS on the OCID-VLG and GraspAnything++ test splits with the Observer disabled, so only the Planner and Coder operate in a single pass, and compare success rates with the reported 0.62 and 0.68; if the scores are essentially unchanged, the feedback loop is not carrying the claimed improvement, and tracking how often Observer feedback alters the final grasp pose would measure the mechanism directly.","supporting_citations":[{"cited_title":"Language-driven grasp detection,","cited_arxiv_id":null,"evidence_quote":"Provides the GraspAnything++ dataset and the supervised LGD baseline that GraspMAS is compared against."},{"cited_title":"Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter","cited_arxiv_id":"2311.05779","evidence_quote":"Provides the OCID-VLG benchmark for cluttered tabletop language-driven grasp evaluation."},{"cited_title":"Vipergpt: Visual inference via python execution for reasoning,","cited_arxiv_id":null,"evidence_quote":"ViperGPT is the closest compositional baseline; GraspMAS shares its code-generation approach and is evaluated against it with the same toolset."},{"cited_title":"Towards open-world grasping with large vision-language models,","cited_arxiv_id":null,"evidence_quote":"OWG is the open-world grasping baseline that uses LLM reasoning and is outperformed in all reported settings."},{"cited_title":"Grounding dino: Marrying dino with grounded pre-training for open-set object detection,","cited_arxiv_id":null,"evidence_quote":"GroundingDINO implements the find tool, the main open-vocabulary object localization primitive in the system."},{"cited_title":"Going denser with open-vocabulary part segmentation,","cited_arxiv_id":null,"evidence_quote":"VLPart implements the find_part tool, enabling task-oriented grasps on object parts such as handles and blades."},{"cited_title":"NBMOD: Find It and Grasp It in Noisy Background","cited_arxiv_id":"2306.10265","evidence_quote":"RAGT-3/3 implements the grasp_detection tool that proposes grasp rectangles for detected objects."}],"review_version":1}