{"id":"81f6fbba-e2da-475c-b468-26e7d4552f08","arxiv_id":"2411.14917","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A zero-shot pipeline using GPT-4o to label object subparts and a quality diversity grasp archive to select task-conditioned grasps reports 73.6% weighted IoU against human-annotated grasp regions and 88% human preference in a small user study.","lead":"This paper combines large language models with quality diversity grasp archives to choose where a robot should grip an object based on the upcoming task. The system labels object parts such as 'handle' and 'rim', asks GPT-4o which part to grasp and which part performs the task, then picks a grasp from a precomputed diverse set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 73.6% weighted IoU is arithmetically incompatible with the reported 91.1% precision and 73.6% recall: for any pair of masks, IoU cannot exceed recall, so at least one headline metric is miscalculated or misreported.","rationale":"The reader's weakest assumption concerns external validity: whether human preference predicts task success. That is a real concern, but the paper explicitly frames human intuition as its evaluation reference (Section V-B) and repeatedly claims alignment with human intuition, so it does not by itself invalidate the stated claim. The metric inconsistency, by contrast, is an internal-validity problem with the headline number. The reported IoU, precision, and recall are mutually incompatible under any standard mask-based definition, so the 73.6% figure cannot be trusted unless the authors disclose a nonstandard computation. This is more load-bearing than the absence of baselines or the strawman control group, because it attacks the principal quantitative result directly; the other issues reinforce the need for a conditional verdict rather than changing the category. The verdict is therefore unchanged: continue to require revisions, now specifically including recomputation and transparent reporting of the weighted IoU, precision, and recall.","tokens_in":7813,"tokens_out":10815,"duration_ms":102195,"concrete_test":"Recompute the three metrics from the raw predicted masks and the consolidated survey ground truth for all 65 task-object pairs, using the participant-overlap weights described in Section V-B and a single global formula: weighted IoU = Σw|P∩G|/Σw|P∪G|, precision = Σw|P∩G|/Σw|P|, recall = Σw|P∩G|/Σw|G|. If the recomputed values do not reproduce the printed 73.6/91.1/73.6 triple, or if no self-consistent choice of reported weights can produce it, then identify which quantity was misreported and replace the headline with the corrected value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-B reports weighted IoU 73.6%, precision 91.1%, and recall 73.6% over the 65 task-object pairs. For any predicted mask P, ground-truth mask G, and nonnegative per-point weights w, the standard weighted definitions give IoU = Σw·1{P∩G}/Σw·1{P∪G} and recall = Σw·1{P∩G}/Σw·1{G}. Because P∪G always contains G, the IoU denominator is at least the recall denominator, so IoU ≤ recall, with equality only when P ⊂ G, which would force precision = 100%. Since precision is 91.1%, P extends beyond G and IoU must be strictly below recall. The printed triple therefore cannot arise from any single consistent set-based computation. It may reflect different normalizations, different subsets, or averaging over runs/tasks, but the paper does not say so. This is load-bearing because the 73.6% IoU is the paper's primary quantitative evidence for task-aware region prediction; if it is not reproducible under one consistent definition, the central claim is unsupported. The authors should specify the exact weighted formulas and per-task or per-run results, or recompute all three metrics from the same masks and weights.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot pipeline for task-conditioned robotic grasping that combines semantic subpart segmentation with large language models and quality-diversity (QD) grasp generation. An object point cloud is rendered, segmented with SAM, and labeled with GPT-4o as a vision-language model; an LLM then selects the subpart to grasp (L_grasp) and the subpart relevant to the task (L_task). QD-Grasp provides a diverse grasp archive, and a score function C(g,T) selects the grasp that maximizes Kforce*F + Kdist*dtask among grasps on L_grasp. The authors evaluate grasp-region prediction on 65 task-object pairs against a six-participant human survey, reporting weighted IoU 73.6%, precision 91.1%, and recall 73.6%, and an end-to-end study on 7 task-object pairs with 55 participants, where 88% preferred the task-aware grasp over control grasps.","tokens_in":8017,"tokens_out":6551,"duration_ms":65275,"significance":"If the reported results hold, the paper would make a useful contribution by coupling the diversity of QD-generated grasp archives with LLM semantic knowledge in a training-free manner, and by demonstrating the full pipeline on a real Franka robot. The authors are explicit about several limitations, including the lack of quantitative segmentation benchmarking and the absence of task-execution metrics. The main strengths are the clean system integration, the real-robot validation, and the honest treatment of the difficulty of evaluating task-aware grasping. However, the central quantitative evidence is currently undermined by an arithmetic inconsistency in the headline region-prediction metrics and by the fact that the same human-preference signal is used for both the ground truth and the final validation.","major_comments":[{"comment":"The reported triple of weighted IoU 73.6%, precision 91.1%, and recall 73.6% cannot arise from any single consistent set-based computation. For nonnegative per-point weights, IoU = O/U and recall = O/G with U >= G, so IoU <= recall, with equality only when the predicted region is a subset of the ground truth, which would force precision to be 100%. Since precision is 91.1%, the predicted region must extend beyond the ground truth, and IoU must be strictly less than recall; from the reported precision and recall, the implied IoU is about 68.7%, not 73.6%. The authors must specify the exact weighted definitions used, report per-task or per-run values, and recompute all three metrics from the same masks and weights, or clearly explain why the metrics come from different normalizations or subsets.","section":"Section V-B"},{"comment":"The validation is circular in an important respect: the ground truth for grasp regions is a six-participant human survey, and the end-to-end evaluation asks 55 participants to choose their preferred grasp. The system is therefore measured by its agreement with human intuition in both the training-free design and the final test, and the reported numbers do not establish that the selected grasps lead to successful task execution. The limitation section acknowledges that task performance is not measured, but this is the central claim of the paper, not a peripheral issue. The authors should either add task-execution trials (e.g., success rates for pouring, cutting, or handing over) or explicitly restrict the claims to 'alignment with human preference' rather than 'task-aware grasping' in the abstract and conclusion.","section":"Sections V-B and V-C"},{"comment":"The binomial test is reported as p < 0.001 with 88% preference among 55 participants, but the statistical unit is unclear. If each participant evaluated all seven task-object combinations, the responses are clustered by participant, and a simple binomial test on pooled choices treats dependent observations as independent, which likely overstates significance. The authors should report per-participant preference counts, a mixed-effects model, or a per-participant binomial test. Additionally, the control grasps are defined by C(g_i,T)=0, meaning they are deliberately off the LLM-selected grasp region; this comparison mostly shows that participants prefer the selected grasp region, not that the score function's force/distance trade-off contributes beyond the region choice.","section":"Section V-C"},{"comment":"The score function C(g,T) depends on hand-set gains Kforce and Kdist, and Section V-C fixes Kforce=10 and Kdist=1 without any sensitivity analysis or ablation. Since the selected grasp g* can change with these gains, the reported end-to-end preferences may be contingent on a particular tuning rather than on a robust property of the framework. Please provide an ablation over (Kforce, Kdist) or a principled procedure for setting them, and report how often the selected grasp changes under reasonable variations.","section":"Section III-C"}],"minor_comments":[{"comment":"The sentence beginning 'we determined the optimal task-aware grasp g* according to the proposed approach and for each task-object combination g* = ...' is a run-on fragment; please rephrase for clarity.","section":"Section V-C"},{"comment":"Reference [2] is cited as 'GPT-4 technical report' but the model used throughout the paper is GPT-4o; please clarify the exact model version and access date.","section":"References"},{"comment":"The paper states that the segmentation pipeline should be evaluated against an established segmentation dataset, but no such quantitative evaluation is included; this should be addressed or explicitly listed as future work.","section":"Section VI-A"},{"comment":"The box plot would be easier to interpret with per-participant or per-task distributions, and the text should state whether the 88.05% mean is over participants, responses, or task-object combinations.","section":"Figure 6"},{"comment":"The assignment of a grasp point to the nearest subpart label may be unstable for grasps near subpart boundaries; a brief discussion or a threshold on the distance comparison would clarify the robustness of the L_g determination.","section":"Section III-C"},{"comment":"No code or data release is mentioned; making the survey responses, grasp archives, and evaluation scripts available would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the arithmetic inconsistency in Section V-B, because the reported IoU is the primary evidence for the region-prediction claim. I do not think this requires rejection: the inconsistency may stem from an unreported averaging or weighting convention, and the pipeline itself is plausible and well-motivated. However, the authors must fix the metric definitions and either add task-execution evidence or soften the claims from 'task-aware' to 'human-aligned.' The lack of a comparison to a non-task-aware baseline or chance-level IoU should also be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: This paper has a real idea—coupling QD grasp archives with open-vocabulary part labeling and a part-distance score for zero-shot task-conditioned grasping. The pipeline is coherent and the experiments are real, but the headline 73.6% weighted IoU is arithmetically incompatible with the reported 91.1% precision and 73.6% recall. For any pair of masks under any weighted set-based definition, IoU ≤ recall, so the reported triple cannot arise from one consistent computation. The stress-test is correct. Since that IoU is the paper's primary quantitative evidence, the claim is currently unsupported.\n\nWhat's genuinely new: the specific combination of QD grasp archives (QD-Grasp) with LLM/VLM part labeling and a task-distance reward is not in the cited prior work (GraspGPT, FoundationGrasp, LangGrasp). The segmentation pipeline—PCA render, SAM, GPT-4o labels, projection back to 3D—is a reasonable engineering contribution. The score function is simple, interpretable, and training-free. The human-survey ground truth is a defensible choice given no objective task-aware grasp metric, and the end-to-end study with 55 participants is a real effort.\n\nSoft spots: (1) the metric inconsistency; (2) no baselines or ablations—no comparison to prior LLM-based methods or to a force-only/random baseline, so the added value of task conditioning is not quantified; (3) the control group is strawman: it consists of grasps with C(g,T)=0, i.e., grasps explicitly violating the task-specific part constraint, so participants' preference for the task-aware grasp is unsurprising; (4) Kforce and Kdist are hand-set (10 and 1) with no sensitivity analysis; (5) the same human-preference signal is used for both region prediction and end-to-end validation, so circularity is a risk; task execution success is never measured; (6) prompts and data are not released, limiting reproducibility. The paper's own limitations section mentions segmentation evaluation but not the metric issue.\n\nWho this is for: researchers in task-oriented grasping, LLM grounding, and QD-based manipulation. The paper deserves a serious referee because the core idea is sound and the list of flaws is fixable, but it needs major revision: recompute or clarify the metrics, add baselines, fix the control group, and ideally include a small physical task-execution check. Recommendation: send to peer review, not desk reject, with instructions to push hard on the evaluation.","headline":"A plausible and novel pipeline for zero-shot task-conditioned grasping, but the headline IoU is arithmetically impossible as reported and the evaluation lacks baselines.","tokens_in":8651,"tokens_out":3495,"would_cite":true,"duration_ms":30985,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-shot pipeline using a large language model to choose which object part to grasp, then scoring a diverse grasp archive, matches human task-grasp preferences in 88% of end-to-end tests.","keywords":["task-aware grasping","quality diversity","large language models","vision-language models","zero-shot learning","semantic segmentation","grasp synthesis","human preference"],"falsifier":"Execute each selected grasp on a real robot for the 65 task-object pairs and measure task completion directly—for example, volume successfully poured, cutting force achieved, or insertion success. If grasps that match human preference perform no better than control grasps on these objective measures, the central claim that the method produces task-appropriate grasps is falsified.","tokens_in":7567,"feed_emoji":"🤖","tokens_out":9374,"duration_ms":94052,"temperature":0.7,"pith_summary":"Task-aware robotic grasping usually requires annotated data or task-specific training. This paper argues that a zero-shot pipeline can do it by combining three ingredients: a vision-language model that segments an object into named subparts, a large language model that selects the subpart to grasp for a given task, and a quality-diversity algorithm that supplies many candidate grasps. The paper's score function rewards grasps on the selected subpart with high contact force and sufficient distance from the task-relevant part. Evaluated on 65 task-object pairs, the predicted grasp regions score 73.6% weighted IoU against a human survey, and in an end-to-end robot study 88% of participants preferred the task-aware grasp over control grasps. The central claim is that semantic reasoning from foundation models can be grounded in physical grasp geometry without any training.","feed_headline":"LLM-guided grasps match human preference 88% of the time","feed_subtitle":"Language-model part reasoning plus a diverse grasp archive selects task-appropriate grasps with zero training.","key_machinery":"The key objects are the labeled subpart vocabulary $V(O)$ and the task-compatibility score $C(g,T)$. The vocabulary is produced by a zero-shot, open-vocabulary segmentation pipeline that uses a principal-component-analysis-based rendering, a vision-language model, and 2D-to-3D projection to name each graspable part of the object. The score function $C(g,T)=K_{\\mathrm{force}}F+K_{\\mathrm{dist}}d_{\\mathrm{task}}$ when the grasp touches the LLM-chosen grasp part, and 0 otherwise, converts the LLM's semantic choice into a ranking over the physically diverse set of grasps generated offline by quality-diversity optimization. The distance term $d_{\\mathrm{task}}$, the minimum Euclidean distance from the gripper contact point to the task part's point cloud, encodes the intuition that the gripper should stay clear of the part that does the work.","core_discovery":"The central discovery is that task-conditioned grasping can be decomposed into a semantic selection problem and a geometric scoring problem. The paper segments a 3D object into labeled subparts by rendering it along principal variance axes, segmenting the render with a vision-language model, and projecting masks back to 3D, producing a vocabulary of named regions. Given a task, an LLM returns two labels—the part to grasp and the part that performs the task—and the algorithm scores every grasp in a precomputed quality-diversity archive by $C(g,T)=K_{\\mathrm{force}}F+K_{\\mathrm{dist}}d_{\\mathrm{task}}$ if the grasp contacts the selected grasp part and 0 otherwise, picking the maximum. Against a consolidated human-preference ground truth the predicted regions reach 73.6% weighted IoU with 91.1% precision, and a separate end-to-end preference study finds 88% of participants favor the selected grasp (binomial $p<0.001$).","pith_inferences":["Because both headline metrics are scored against the same human-intuition ground truth, a direct measurement of task execution success (e.g., volume poured, force applied, insertion success) would give independent evidence for the method's claims.","The same segmentation-and-LLM scoring recipe could apply beyond grasping to other part-dependent manipulation choices, such as where to place a tool or which surface to press, whenever a task-relevant subpart can be named.","The PCA-based rendering step suggests the method will be weakest on concave or articulated objects; replacing it with a multi-view or volumetric open-vocabulary segmenter is a natural testable extension that could improve the weighted IoU.","If human preferences themselves vary with the participant's experience or with the framing of the task, the consolidated ground truth may hide systematic biases; re-running the survey with task videos or functional instructions would test the stability of the 73.6% IoU."],"forward_implications":["Predicted task-grasp regions align with human intuition on 65 task-object combinations, reaching 73.6% weighted IoU and 91.1% precision.","In end-to-end tests on a physical robot, 88% of 55 participants preferred the task-aware grasp over control grasps, with a binomial test giving $p<0.001$.","The pipeline needs no task-specific training or annotated grasp labels; the same precomputed quality-diversity archive serves all tasks.","The score function is modular, so additional task constraints such as required grip force or grasp orientation could be added as further terms in $C(g,T)$."],"supporting_citations":[{"why":"Supplies the 2D segmentation masks that the pipeline labels with semantic labels and projects back into 3D subparts.","marker":"[1]"},{"why":"Serves as both the vision-language model that names object subparts and the LLM that selects the grasp part and task part.","marker":"[2]"},{"why":"Introduces the quality-diversity grasp synthesis that generates the diverse archive of stable grasps from which a task-conditioned grasp is chosen.","marker":"[20]"},{"why":"Provides the specific 6-DoF grasp sampling algorithm that builds the quality-diversity archive efficiently.","marker":"[22]"},{"why":"Defines the YCB object subset and the 65 task-object combinations used to evaluate grasp region prediction.","marker":"[25]"},{"why":"Supplies the motion planning used to execute the selected grasps on the physical robot in the end-to-end validation.","marker":"[24]"}],"fun_headline_variants":["LLM picks grasp parts, human raters agree 88%","Zero-shot task grasp via LLM subpart scoring","73.6% IoU for LLM-chosen grasp regions","Human raters prefer LLM-guided grasp in 88% of cases","LLM + diversity archive selects human-liked grasps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that human-preferred grasp locations, as consolidated from a six-participant survey, are a valid proxy for actual task performance; the paper never measures task execution success, so if human intuition does not track task effectiveness, the headline numbers would not establish task-aware grasping quality.","fun_headline_variants_meta":{"raw":{"variants":["LLM picks grasp parts, human raters agree 88%","Zero-shot task grasp via LLM subpart scoring","73.6% IoU for LLM-chosen grasp regions","Human raters prefer LLM-guided grasp in 88% of cases","LLM + diversity archive selects human-liked grasps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1681,"prompt_tokens":971,"completion_tokens":710,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":587,"tokens_out":710,"duration_ms":7613,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:43:03.536714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Execute each selected grasp on a real robot for the 65 task-object pairs and measure task completion directly—for example, volume successfully poured, cutting force achieved, or insertion success. If grasps that match human preference perform no better than control grasps on these objective measures, the central claim that the method produces task-appropriate grasps is falsified.","supporting_citations":[{"cited_title":"Quality diversity under sparse reward and sparse interaction: Application to grasping in robotics,","cited_arxiv_id":null,"evidence_quote":"Introduces the quality-diversity grasp synthesis that generates the diverse archive of stable grasps from which a task-conditioned grasp is chosen."},{"cited_title":"Speeding up 6-dof grasp sampling with quality-diversity,","cited_arxiv_id":null,"evidence_quote":"Provides the specific 6-DoF grasp sampling algorithm that builds the quality-diversity archive efficiently."}],"review_version":1}