{"id":"2f27aefa-5b77-4161-9678-883ff9fb5bf9","arxiv_id":"2411.19626","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GREAT combines multimodal language model reasoning with 3D geometry to ground open-vocabulary object affordances, and introduces the large PIADv2 dataset.","lead":"This paper describes a computer vision system that locates the part of a 3D object you would act on by reading a photo of a person performing an action on that kind of object, even when the action is new. The authors also release a large dataset of 3D objects with action-region labels, aimed at robots that need to handle unfamiliar objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The open-vocabulary claim rests on unverified MLLM reasoning quality for unseen affordances; the ablation deltas are too small to confirm that CoT knowledge, rather than the image branch, drives generalization.","rationale":"The reader's weakest assumption is that MLLM chain-of-thought reasoning remains accurate and transferable for unseen affordances, and the reader notes it is only indirectly tested. My stress-test converges on the same point: the paper's open-vocabulary generalization claim depends on the quality of the MHACoT outputs for categories absent from PIADv2 training, yet no direct evaluation of these outputs is provided. The ablation results are consistent with this being an important component, but the deltas are small enough that the improvement could stem from other factors, such as the image branch or pretrained world knowledge. I also considered data leakage from InternVL pretraining, but the primary unresolved issue is the lack of a direct quality check on the reasoning outputs themselves. The proposed concrete test—human annotation of object-part identification, geometric-attribute correctness, and intention plausibility on unseen-affordance images, followed by a stratified AUC comparison—would settle whether the concern lands. Since the reader already assigned CONDITIONAL on similar grounds, my analysis does not change the verdict; it sharpens the condition that should be satisfied before the SOTA claim is fully accepted.","tokens_in":14797,"tokens_out":5254,"duration_ms":47542,"concrete_test":"Sample 100 test images from the Unseen Affordance partition. Run the MHACoT prompts with the fine-tuned InternVL and have at least two independent annotators judge, per image: (1) whether the 'interaction part' named by the object-head is the correct affordance region, (2) whether the geometric attributes described are actually true of the object in the image, and (3) whether the two listed alternative interactions are plausible for that object. Compute per-image pass/fail and inter-annotator agreement. Then split the test set by whether all three judgments pass and compare GREAT's per-group AUC. If the 'fail' group is large or the AUC gap between 'pass' and 'fail' groups is small, the CoT knowledge is not doing the load-bearing work claimed; additionally, run a near-duplicate search between the test images and InternVL's pretraining set to rule out image-level leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GREAT generalizes to affordance categories unseen in training by leveraging MHACoT-derived geometry attributes and interaction intentions. This claim is load-bearing on the accuracy of the fine-tuned InternVL's outputs for those unseen affordances. However, the paper never measures the quality of these outputs. The introduction states that the multi-step strategy 'eliminates the hallucination and ambiguities raised by MLLMs when reasoning interactions' (Sec. 1), yet no hallucination rate, correctness metric, or failure-case analysis is reported. The only evidence is the ablation in Table 3: removing the affordance head drops unseen-affordance AUC from 69.81 to 67.18, and removing the object head drops it to 64.69. These deltas are modest, and they could reflect changes in the training distribution of text features or a reduced amount of auxiliary supervision rather than the specific correctness of the CoT knowledge. If InternVL hallucinates on out-of-distribution images—naming a non-interactive part, describing geometric properties that are not actually present, or listing irrelevant alternative interactions—then the knowledge dictionaries To and Ta inject noise through Eq. 1-4. In that case, the observed generalization might be driven by the image branch Fi or the point-cloud geometry alone, not by the proposed geometry-intention collaborative mechanism. Independently, the 'Unseen Affordance' partition only guarantees that affordances are absent from PIADv2 training; it does not guarantee that they are absent from InternVL's pretraining corpus. The paper does not test for image-level or concept-level overlap, leaving open the possibility that the model's 'open-vocabulary' behavior is partly memorized world knowledge rather than the result of the proposed inference pipeline. Together, these omissions leave the strongest claim insufficiently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GREAT, a framework for open-vocabulary 3D object affordance grounding. Given a point cloud and an interaction image, GREAT first fine-tunes an MLLM (InternVL) with a Multi-Head Affordance Chain-of-Thought (MHACoT) procedure to produce object geometric attributes and underlying interaction intentions. These textual outputs are encoded with RoBERTa and aggregated into two knowledge dictionaries, which are then fused into point-cloud features and image features through a Cross-Modal Adaptive Fusion Module. The final fused features are decoded into a point-wise affordance heatmap. The paper also introduces PIADv2, a large dataset of paired interaction images and 3D point clouds, and reports experiments on three partitions (Seen, Unseen Object, Unseen Affordance) showing substantial gains over reimplemented baselines. Ablations demonstrate the contribution of each component, and qualitative visualizations support the method's behavior.","tokens_in":15104,"tokens_out":4672,"duration_ms":40546,"significance":"If the reported results are sound, the paper makes a useful contribution to 3D affordance grounding by showing that MLLM-based chain-of-thought reasoning can provide transferable knowledge for unseen affordances. The proposed dataset, PIADv2, is substantially larger than prior benchmarks and could benefit future research. The paper also provides code and dataset release, and the ablations are fairly comprehensive. However, the significance is currently bounded by three limitations: all quantitative evaluation is on the authors' own dataset, the MLLM reasoning outputs are never directly validated for correctness or hallucination, and the fine-tuning data for the MLLM is not described in sufficient detail. These gaps make the central open-vocabulary claim less certain than the paper's wording suggests.","major_comments":[{"comment":"The fine-tuning of InternVL is described only in terms of adapters, epochs, learning rate, and LoRA rank; the paper does not state what data the MHACoT prompt-answer pairs come from, whether the answers are human-annotated or MLLM-generated, or whether any of those training pairs involve affordance categories or object categories that appear in the Unseen Affordance or Unseen Object test partitions. This matters because the method's open-vocabulary capability is attributed to this fine-tuned MLLM, and any overlap between the fine-tuning set and the evaluation partitions would be a form of leakage. Please specify the construction, size, and partition membership of the fine-tuning data, and verify that it is disjoint from the evaluation splits.","section":"Sec. 3.2"},{"comment":"All quantitative comparisons are carried out exclusively on PIADv2, a dataset introduced in the same paper, and the baselines are reimplemented with 'the same feature extractor and settings' rather than taken from official reported results. The claim of state-of-the-art performance therefore lacks external anchoring: a reader cannot tell whether the gains reflect the method's superiority or favorable properties of the new dataset and splits. Please add at least one external benchmark (for example, evaluation on the original PIAD or on 3D AffordanceNet) or, at minimum, report official baseline numbers and cross-dataset transfer experiments. Without such evidence, the SOTA claim is not fully supported.","section":"Sec. 5.2 and Table 2"},{"comment":"The paper states that the multi-step chain-of-thought strategy 'eliminates the hallucination and ambiguities raised by MLLMs when reasoning interactions,' but no direct measurement of MLLM output quality is reported. There is no hallucination rate, no accuracy metric for the predicted geometric attributes or interaction intentions, and no failure-case analysis, especially for affordance categories unseen in training. The Table 3 ablations show that removing the affordance head lowers Unseen Affordance AUC from 69.81 to 67.18 and removing the object head lowers it to 64.69; these deltas are modest and could stem from changes in the text-feature distribution or from reduced auxiliary supervision rather than from the factual correctness of the extracted knowledge. Please report direct evaluations of the MLLM-generated knowledge (e.g., human-rated accuracy or automatic comparison against annotations) and show representative successes and failures for the unseen-affordance setting.","section":"Sec. 3.2, Sec. 5.3, Table 3"},{"comment":"The definition of the Unseen Affordance partition is under-specified. The paper says only that 'affordances in the test set are not present in the training set, and so does certain objects,' but it does not clarify whether the interaction images and the fine-tuned MLLM training data also exclude those affordances, nor how object categories and image sources are controlled across partitions. If, for example, images depicting 'pour' are present in the MLLM fine-tuning set even though the affordance is absent from the main training set, the open-vocabulary evaluation would be confounded. Please provide the exact partition construction rules and a per-category overlap analysis between training, fine-tuning, and test sets.","section":"Sec. 4, Data Partitions"}],"minor_comments":[{"comment":"The annotation description says 'each affordance category possesses such annotation of an instance,' which is ambiguous. Please clarify whether each point-cloud instance is annotated with one affordance heatmap per affordance category or only with the affordance category depicted in the paired image.","section":"Sec. 4"},{"comment":"The ablation labeled '✗ CMAFM' is not described in detail; please specify what replaces the Cross-Modal Adaptive Fusion Module when it is removed (e.g., simple concatenation or a fixed cross-attention layer).","section":"Sec. 5.3 and Table 3"},{"comment":"The phrase 'significantly outperforms' is used without statistical support. Reporting standard deviations across multiple seeds or a significance test would make the comparison more convincing.","section":"Sec. 5.2 and Table 2"},{"comment":"The full MHACoT prompts are said to be in the appendix, but the appendix is not included in the reviewed manuscript. Please ensure the full prompts are part of the submission or provide them in the main text.","section":"Sec. 3.2"},{"comment":"The cross-attention operation fm and the self-attention operation fδ are not defined in terms of their inputs and output shapes. In particular, when To and Ta have different sequence lengths, the implementation of Eq. (1) should be clarified so that the output dimensions of ¯To and ¯Ta are unambiguous.","section":"Sec. 3.2, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with comprehensive ablations and a released dataset, but the central open-vocabulary claim currently rests on self-benchmarking and on an unvalidated MLLM reasoning module. The major comments above ask for evidence that can be added within the paper's scope: external validation, MLLM output quality analysis, and explicit partition/fine-tuning data disclosure. I do not see a load-bearing error that would force rejection, but the current evidence is not yet sufficient for acceptance as a state-of-the-art claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a genuine method+dataset contribution. The MHACoT reasoning chain (object-head for geometry, affordance-head for intentions), the cross-attention knowledge encoding, and the CMAFM fusion are coherent, and the ablations mostly support each piece. PIADv2 is a meaningful resource: 15K images, 38K point clouds, 24 affordances, 43 object categories, over 5x the original PIAD. That alone is worth something.\n\nThe main weakness is not in the architecture but in the evidence for the headline claim. All evaluation is on PIADv2, which the authors built. The baselines are reimplemented, and there is no external benchmark to anchor the 92 AUC / 38 aIOU numbers. The Unseen Affordance partition is the right idea, but the paper never checks whether the unseen affordances overlap with InternVL's pretraining data, so part of the 'open-vocabulary' gain could be memory rather than the proposed reasoning pipeline. Relatedly, the MLLM's chain-of-thought outputs are never evaluated for correctness. The paper says the multi-step strategy eliminates hallucination, but no hallucination rate or failure analysis is reported. The ablation deltas (AUC 69.81 to 67.18 without AffCoT) are real but modest, and they could reflect changes in the text feature distribution as much as the reasoning quality itself. That is a load-bearing gap: if the CoT outputs are noisy for unseen affordances, the knowledge dictionaries inject noise, and the observed generalization might come from the image branch.\n\nThat said, these are standard concerns for this kind of paper. The method is clearly described, the dataset statistics are thorough, and the qualitative results show the model does something beyond the baselines. Minor points: no error bars, no multiple runs, and the multiple-object / multiple-affordance / multiple-instance analyses are lightweight but fine.\n\nWho is this for? People working in 3D affordance grounding and embodied perception. The dataset alone is a useful resource, and the CoT-plus-geometry fusion idea is worth trying in other variants.\n\nRecommendation: send it to serious peer review. I would ask for validation on an external benchmark (or at least on a subset of 3D AffordanceNet with the same protocols), a direct evaluation of MLLM CoT outputs on unseen affordances (human-rated correctness or agreement), and analysis of concept overlap with pretraining data. Then the SOTA claim would be credible.","headline":"A solid method-and-dataset contribution whose open-vocabulary claim is plausible but needs external benchmarks and direct checks of the MLLM's reasoning outputs before the SOTA claim carries weight.","tokens_in":15708,"tokens_out":2279,"would_cite":true,"duration_ms":18841,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mining invariant geometry and interaction intentions lets a model ground 3D object affordances for categories never seen in training.","keywords":["3D object affordance grounding","open-vocabulary affordance","chain-of-thought reasoning","multimodal large language models","cross-modal fusion","point cloud grounding","PIADv2 dataset","affordance generalization"],"falsifier":"Run GREAT on unseen-affordance test images and compare the MLLM's four chain-of-thought answers against human annotations of the interaction part, its geometry, the shown action, and the alternative interactions. If the generated text is frequently wrong or generic for unseen affordances while grounding accuracy stays high, the chain-of-thought is not the mechanism carrying the gain; alternatively, if substituting fixed random text for the chain-of-thought answers preserves unseen-affordance AUC, then the chain-of-thought content is not the active ingredient.","tokens_in":14623,"feed_emoji":"🤖","tokens_out":10284,"duration_ms":74135,"temperature":0.7,"pith_summary":"GREAT addresses open-vocabulary 3D object affordance grounding: given a 3D point cloud and a 2D interaction image, locate the region that supports an arbitrary action. The paper's central claim is that the bottleneck is not cross-modal alignment but the semantic space of interaction priors, and that this space can be expanded by having a multimodal language model reason in steps about which part interacts, why the geometry supports the interaction, what action is shown, and what other actions the object affords. GREAT implements this reasoning with a Multi-Head Affordance Chain-of-Thought fine-tuned on interaction images, then fuses the resulting text knowledge with point cloud and image features to predict point-wise affordance heatmaps. The paper also introduces PIADv2, a dataset with 15K interaction images and more than 38K point clouds covering 24 affordances and 43 object categories, and reports that GREAT outperforms prior methods on seen, unseen-object, and unseen-affordance partitions. This matters because it is a route toward robots that can reason about how to use objects they were not explicitly trained on.","feed_headline":"Multi-step reasoning grounds 3D affordances never seen in training","feed_subtitle":"Reasoning from images beats prior methods on object and affordance categories never seen in training.","key_machinery":"The load-bearing component is the Multi-Head Affordance Chain-of-Thought (MHACoT): a fine-tuned multimodal large language model prompted through four steps: object interaction perception, geometric structure reasoning, interaction detailed description, and interactive analogical reasoning. The object head's first two prompts extract which part interacts and why its geometry permits the interaction; the affordance head's last two prompts extract the shown interaction and brainstorm two additional interactions. These outputs are encoded, cross-attended to align geometry with intention, and compressed into two knowledge dictionaries. The Cross-Modal Adaptive Fusion Module (CMAFM) then injects the geometric dictionary into the deepest point cloud encoder layer and fuses the intention dictionary with image features, so the decoder sees point geometry and interaction context together. The mechanism's job is to make unseen affordances reachable through known geometric primitives and analogical interaction intentions instead of through memorized label mappings.","core_discovery":"The discovery GREAT argues for is that explicitly separating geometric-structure reasoning from interaction-intention reasoning, and letting the two heads cross-reference each other, transfers to affordances whose categories were absent from training. The object head identifies the interacting part and explains its geometry; the affordance head describes the shown interaction and brainstorms two additional plausible interactions. After encoding these descriptions and aligning them via cross- and self-attention into two knowledge features, GREAT injects the geometric knowledge into point cloud features and the intention knowledge into image features, then decodes the fused features into a point-wise affordance map. On the unseen-affordance partition this design reaches an AUC of 69.81 and an aIOU of 12.05, outperforming the strongest baseline by 5.74 AUC and 3.68 aIOU. The authors present this as evidence that invariant geometry plus analogical intention reasoning, rather than a wider vocabulary of labels, is what enables open-vocabulary affordance grounding.","pith_inferences":["An untested implication is that the four-step reasoning text itself is the active ingredient: one could replace the MLLM's answers with the same template filled with random or generic text while keeping fusion fixed, and check whether unseen-affordance accuracy drops.","The geometry-intention decomposition could transfer to robotic grasping-point selection or tool-use planning, where the geometric step names the contact region and the intention step filters candidate actions; that application is outside the paper's experiments.","Because the authors identify multi-step inference cost as the bottleneck, distilling the MLLM's chain-of-thought into a small text encoder or a lightweight reasoning head is the most direct path toward real-time use, and PIADv2's scale would support such distillation.","The multi-to-multi relation between affordances and object categories in PIADv2 suggests the dataset could support compositional or few-shot splits beyond the three partitions explored here."],"forward_implications":["A robot controller using this pipeline could hypothesize an interaction part and a manipulation strategy for an object category it has never seen, as long as the object's geometry resembles shapes whose affordances were described during training.","Reasoning output is not tied to a fixed label vocabulary: the same framework can accept open-ended instructions and produce heatmaps for interactions beyond the 24 affordance categories in PIADv2.","The dataset's three partitions turn open-vocabulary affordance grounding into a measurable benchmark, so future methods can be compared on generalization rather than only on seen-category accuracy.","Removing either chain-of-thought head degrades performance, with the larger drop coming from removing the affordance head on unseen affordances, so both geometry and intention reasoning contribute to the open-vocabulary gain."],"supporting_citations":[{"why":"Defines the prior PIAD dataset and the interaction-guided baseline that GREAT extends; supplies the original task formulation and two of the three evaluation partitions.","marker":"[56]"},{"why":"Provides the strongest prior baseline using text-conditioned affordance queries, which GREAT is compared against on all three partitions.","marker":"[25]"},{"why":"Supplies the multimodal large language model that GREAT fine-tunes with adapters to perform the four chain-of-thought reasoning steps.","marker":"[4]"},{"why":"Supplies the point cloud encoder into whose deepest layer the geometric knowledge is injected.","marker":"[43]"},{"why":"Supplies the text encoder used to turn the chain-of-thought descriptions into object-geometry and affordance-intention knowledge features.","marker":"[28]"},{"why":"Supplies 3D-AffordanceNet, one of the main point-cloud sources for PIADv2 and a prior 3D affordance benchmark.","marker":"[6]"},{"why":"Supplies Objaverse, a major source of the 38K+ 3D object instances in PIADv2.","marker":"[5]"},{"why":"Supplies the image backbone used to extract visual features that are fused with affordance-intention knowledge.","marker":"[9]"}],"fun_headline_variants":["GREAT splits geometry and intent, besting baselines on unseen affordances","Analogical reasoning unlocks open-vocabulary 3D affordance grounding","Invariant geometry plus intention analogies drive 3D affordance grounding","Multi-step reasoning grounds unseen 3D affordance categories","GREAT: geometry-intention split beats unseen affordance baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MLLM's chain-of-thought, fine-tuned on PIADv2 training images, produces accurate geometric attributes and interaction intentions for affordance categories never seen in training; if that reasoning is unreliable or hallucinated, the open-vocabulary gain the paper reports would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["GREAT splits geometry and intent, besting baselines on unseen affordances","Analogical reasoning unlocks open-vocabulary 3D affordance grounding","Invariant geometry plus intention analogies drive 3D affordance grounding","Multi-step reasoning grounds unseen 3D affordance categories","GREAT: geometry-intention split beats unseen affordance baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2655,"prompt_tokens":979,"completion_tokens":1676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1583}},"tokens_in":595,"tokens_out":1676,"duration_ms":11434,"temperature":1.0,"reasoning_tokens":1583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:58:46.597260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GREAT on unseen-affordance test images and compare the MLLM's four chain-of-thought answers against human annotations of the interaction part, its geometry, the shown action, and the alternative interactions. If the generated text is frequently wrong or generic for unseen affordances while grounding accuracy stays high, the chain-of-thought is not the mechanism carrying the gain; alternatively, if substituting fixed random text for the chain-of-thought answers preserves unseen-affordance AUC, then the chain-of-thought content is not the active ingredient.","supporting_citations":[{"cited_title":"Grounding 3d object affordance from 2d interactions in images","cited_arxiv_id":null,"evidence_quote":"Defines the prior PIAD dataset and the interaction-guided baseline that GREAT extends; supplies the original task formulation and two of the three evaluation partitions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the strongest prior baseline using text-conditioned affordance queries, which GREAT is compared against on all three partitions."},{"cited_title":"Roberta: A robustly optimized bert pretraining approach, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the text encoder used to turn the chain-of-thought descriptions into object-geometry and affordance-intention knowledge features."},{"cited_title":"3d affordancenet: A benchmark for visual object affordance understanding","cited_arxiv_id":null,"evidence_quote":"Supplies 3D-AffordanceNet, one of the main point-cloud sources for PIADv2 and a prior 3D affordance benchmark."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the image backbone used to extract visual features that are fused with affordance-intention knowledge."}],"review_version":1}