{"id":"0f40d1ad-872a-42ca-8f1e-3a221fe61753","arxiv_id":"2504.15650","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adapting EVF-SAM with learnable affordance queries and a coarse-to-fine dataset yields strong affordance grounding on AGD20K, with caveats about test-set tuning.","lead":"AffordanceSAM adapts the Segment Anything Model to predict where on an object an action like 'sit' or 'hold' can be performed, and it reports top results on the AGD20K benchmark. The paper also introduces C2F-Aff, a three-stage dataset of masks, pseudo-labels, and human annotations that improves several existing methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard-split category disjointness is asserted only for stages 2–3; Part 1 is trained without category splitting, so novel-object generalization may be inflated by seen object categories.","rationale":"The reader's weakest assumption concerns stage-2 pseudo-labeled images overlapping the test splits. My concern is adjacent but distinct: the hard-split guarantee is explicitly limited to stages 2 and 3, while stage 1 is trained on external datasets without any category-level split. The paper's own appendix admits this, and the public source datasets make the overlap check feasible. If overlap exists, the hard-split numbers no longer support the central 'novel objects' generalization claim, because the model has already seen those object categories in stage 1. This is a load-bearing threat to the strongest claim, but it is not necessarily fatal: the authors could exclude overlapping categories from stage 1 or demonstrate that the intersection is empty. The reader also noted test-set hyperparameter tuning for gamma and filtration passes, which is a secondary methodological concern. Because the appropriate remedy is a conditional revision (split-hygiene verification and retraining if needed), the reader's CONDITIONAL verdict stands unchanged.","tokens_in":15566,"tokens_out":7211,"duration_ms":65597,"concrete_test":"Build the category-name lists of C2F-Aff Part 1 after the paper's merging/reclassification (PADv2, Handal, RGB-D Part Affordance) and the AGD20K hard-split test categories used by AffordanceLLM; compute their intersection. All source datasets and the split are public. If the intersection is non-empty, retrain with those Part 1 categories excluded and re-evaluate on the hard split; if SIM/NSS/KLD drop materially, the reported hard-split advantage is attributable to seen object categories rather than generalizable affordance grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claimed SOTA and generalized capacity rest on the hard split of AGD20K, which is supposed to test unseen object categories. The Evaluation section states: \"The hard split ensures that there is no overlap between the object categories in the train and test set (both in our stage 2 and stage 3).\" The parenthetical is the problem: it guarantees disjointness only for AGD20K-derived data (stages 2 and 3). For stage 1, Appendix 2 says: \"we do not split any object when training, because the output form of this stage is completely different from the output used in the final evaluation.\" Stage 1 trains on PADv2 (103 object classes), Handal (212 classes), and RGB-D Part Affordance (37 classes) against AGD20K's 50 classes; category overlap with the hard-split test list is plausible (common containers, tools, furniture). If any hard-split test category appeared in stage 1, the model saw that object category during training, with object-verb pairings, so the hard-split evaluation no longer measures category-level novelty. The different label format (binary mask vs heatmap) does not remove the object-level prior. Thus the headline generalization result is weakened by an acknowledged gap in split hygiene.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes AffordanceSAM, a fully supervised affordance grounding model built on EVF-SAM (a text-prompted variant of SAM) with the addition of an affordance-adaption module: a set of learnable queries that cross-attend to text features and to multi-layer SAM image features. The authors also curate C2F-Aff, a three-part coarse-to-fine dataset: part 1 combines binary-mask-labeled data from PADv2, Handal, and RGB-D Part Affordance; part 2 uses LOCATE to pseudo-label the unlabeled images of AGD20K, followed by a post-processing filtration algorithm; part 3 uses the human-annotated AGD20K affordance maps. Training proceeds in three stages matching these dataset parts, with progressive unfreezing of the model. Experiments on AGD20K report state-of-the-art SIM and NSS on the easy and hard splits, and qualitative results on internet images suggest generalization to novel objects and actions.","tokens_in":15753,"tokens_out":4748,"duration_ms":41136,"significance":"If the results hold, the contribution is meaningful: it demonstrates that a large segmentation foundation model (SAM) can be adapted to affordance grounding with a lightweight prompt-like module, and it shows that scaling fully supervised data via pseudo-labeling improves all tested fully supervised baselines. The staged ablation is internally consistent and clearly isolates the contribution of each dataset part and module component. The architecture and training recipe are described in enough detail to be reproducible. However, the central SOTA and generalization claims are weakened by the test-set hyperparameter selection and by the incomplete category-disjointness guarantee for stage 1, detailed in the major comments. Once these issues are addressed, the paper could be a solid contribution to the affordance grounding literature.","major_comments":[{"comment":"The post-processing hyperparameters γ and the number of filtration passes in Algorithm 1 are selected using the hard split test set: Table 6 sweeps γ and reports the best NSS at γ=0.45, and Table 5 sweeps the number of filtration steps; the default settings (γ=0.45, Num=3) are then used in the main comparison of Table 2. This is test-set hyperparameter tuning, so the reported hard-split numbers are optimistically biased and are not a clean out-of-sample estimate of the model's generalization. The authors should select these hyperparameters on a validation set held out from the hard split, or alternatively state explicitly that the reported numbers include test-set selection and discuss the resulting optimism in the SOTA claims.","section":"Experiments, Ablation Study (Tables 5 and 6)"},{"comment":"The hard-split guarantee of no object-category overlap is stated only for stages 2 and 3: the Evaluation section says 'The hard split ensures that there is no overlap between the object categories in the train and test set (both in our stage 2 and stage 3).' Appendix 2 reveals that stage 1 does not perform any object-category split: 'we do not split any object when training'. Stage 1 uses PADv2, Handal, and RGB-D Part Affordance, which plausibly contain object categories that also appear in the hard-split test list. Since stage 1 trains object-verb associations, the model has been exposed to those categories before the hard-split evaluation, even though the label format differs (binary masks vs. heatmaps). Consequently, the hard-split results do not strictly measure category-level novelty, weakening the claim of 'strong generalized capacity to novel objects'. The authors should either document that the stage-1 categories are disjoint from the hard-split test categories, or retrain without the overlapping categories and re-evaluate.","section":"Evaluation section and Appendix 2"}],"minor_comments":[{"comment":"The term 'state-of-the-art' is used without qualification, but Table 2 shows that OOAL trained on C2F-Aff achieves a better KLD than AffordanceSAM on both the easy split (0.974 vs 1.083) and the hard split (1.119 vs 1.128). Please qualify the claim, e.g., 'best SIM and NSS'.","section":"Conclusion and Abstract"},{"comment":"There is a typo: 'finaly' should be 'finally'.","section":"Approach, Section 3"},{"comment":"In the caption, 'row maps' and 'row affordance maps' should be 'raw maps' and 'raw affordance maps'.","section":"Figure 6 caption"},{"comment":"The appendix section titled 'Failure Cases and Feature Work' should be 'Failure Cases and Future Work'.","section":"Appendix title"},{"comment":"The paper does not state whether the C2F-Aff dataset, the pseudo-label post-processing code, and the exact train/test category lists will be released; given the split-hygiene concerns raised above, releasing these artifacts is important for verification.","section":"Reproducibility"},{"comment":"The generalization results on internet images are qualitative only; adding a small quantitative evaluation (e.g., human ratings or agreement with hand-annotated maps) would strengthen the claim of generalization to novel objects and actions.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The two major concerns are real and consequential. The test-set tuning of γ and the filtration count is visible from Tables 5 and 6 and directly affects the reported headline numbers. The stage-1 category overlap is acknowledged in Appendix 2 but dismissed with an argument about label format, which is not sufficient for claims about novel-object generalization. If the authors can address these by reporting validation-based selection or demonstrating category disjointness across all stages, the paper would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The most valuable part of this paper is the C2F-Aff recipe. Showing that three-stage coarse-to-fine training improves not just their own model but also OOAL and AffordanceLLM is a solid result. That's independent of the specific architecture, and it's the strongest evidence that the data pipeline has real value. The ablations are consistent, and the staged gains are believable. The affordance-adaption module is a straightforward BLIP-2/DETR-style query module, but it works; the multi-layer SAM feature fusion is a reasonable touch. I'd say the paper makes a legitimate engineering contribution.\n\nThe soft spots are real, though. The biggest one: the hard-split evaluation is supposed to test novel object categories, but stage 1 is trained without any category splitting on PADv2, Handal, and RGB-D Part Affordance. Those datasets overlap with common AGD20K categories. The paper acknowledges this in the appendix: \"we do not split any object when training.\" The parenthetical in the Evaluation section says the hard split guarantees no overlap \"in our stage 2 and stage 3\" — which is exactly the problem. Stage 1 may well have seen the test categories, so the hard-split numbers no longer measure category-level novelty. This is a genuine gap, and it directly weakens the generalization claim.\n\nSecond, the post-processing hyperparameters (gamma and number of filtration passes) are chosen on the hard test split. That's test-set tuning, and it inflates the reported scores. The authors disclose it, but it's still a protocol flaw.\n\nThird, the abstract's unqualified SOTA claim conflicts with Table 2: OOAL has better KLD on both splits when trained on C2F-Aff. The text also says AffordanceSAM \"outperforms OOAL in terms of KLD and NSS\" when trained on AGD20K-Fully, but on KLD it actually loses. That's a fixable misstatement, but it needs fixing.\n\nNo code, no data, no error bars makes the numbers hard to verify. The split-hygiene question can't be checked without the released data.\n\nWho's this for? People working on affordance grounding, especially fully supervised methods and dataset curation. It's a good target for a serious referee — the dataset recipe alone justifies that — but the evaluation protocol needs to be tightened, the overclaims corrected, and the stage-1 overlap issue either fixed or honestly disclosed as a limitation.\n\nI'd send it to peer review with major revision requests.","headline":"A useful dataset recipe and a sensible SAM adaptation, but the SOTA claims are undercut by test-set tuning and a stage-1 category-overlap gap that weakens the novel-object story.","tokens_in":16382,"tokens_out":4339,"would_cite":true,"duration_ms":36756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that SAM, a general image segmenter, can be turned into a state-of-the-art affordance grounding model by adding learnable queries and training on a three-part coarse-to-fine dataset.","keywords":["affordance grounding","Segment Anything Model","text-prompted segmentation","foundation model adaptation","coarse-to-fine dataset","pseudo-labeling","visual generalization"],"falsifier":"Rerun the evaluation after deleting from training any image whose file hash appears in the test set or whose object category appears in the hard-split test categories; if any deletion changes the reported KLD, SIM, or NSS values for AffordanceSAM trained on C2F-Aff, the split-disjointness assumption is violated.","tokens_in":15334,"feed_emoji":"🎯","tokens_out":6846,"duration_ms":56081,"temperature":0.7,"pith_summary":"The paper tries to establish that a text-prompted SAM variant can become a generalized affordance grounding model, pointing to the region of an object where a given action such as 'wear hat' or 'drink with cup' can be performed, rather than merely segmenting the whole object. The authors argue that previous fully supervised methods train key components from scratch on only hundreds of labeled images, while weakly supervised methods need complex two-branch pipelines and cannot handle new actions. Their answer is to keep SAM's image encoder and decoder, insert a small affordance-adaption module, and supervise it with a coarse-to-fine dataset called C2F-Aff built from three sources with increasing label quality. If the central claims hold, a competitive affordance grounding model can be obtained by adapting a general segmentation foundation model instead of designing task-specific architectures from scratch.","feed_headline":"SAM retrained to find action regions on objects","feed_subtitle":"A three-stage coarse-to-fine training recipe turns a text-prompted SAM into a top affordance grounding model.","key_machinery":"The load-bearing component is the affordance-adaption module: a set of learnable 'affordance queries' inserted between the EVF-SAM multimodal encoder and the SAM mask decoder. The queries first cross-attend to text-conditioned image features, then to a learnable weighted fusion of SAM image-encoder layers, and the refined queries are injected into the decoder alongside the original mask features to produce a heatmap-like affordance map. The second pillar is the C2F-Aff dataset and its three-stage recipe: stage 1 supervises the EVF-SAM backbone with binary masks from PADv2, Handal, and RGB-D Part Affordance; stage 2 adds post-processed LOCATE pseudo-labels on unannotated AGD20K images; stage 3 fine-tunes on human-labeled AGD20K maps. The staged schedule itself is part of the mechanism, since the ablation shows that mixing all data at once hurts performance.","core_discovery":"AffordanceSAM reports state-of-the-art results on the AGD20K benchmark, with the best Similarity and Normalized Scanpath Saliency scores among compared fully supervised methods on both the easy and hard splits, and its best KLD on the hard split. The paper attributes this to two complementary transfers: SAM's existing generalization in segmentation carries over to affordance grounding through a lightweight affordance-adaption module, and the C2F-Aff dataset supplies enough supervised signal, in three label-quality stages, to teach the model verb-object relations that do not appear in standard segmentation data. Ablations show each stage matters, since combining all data in one stage degrades performance, and qualitative results on internet images indicate the model maps novel objects and actions to reasonable affordance regions. The paper also claims the C2F-Aff data alone improves prior fully supervised methods, so the dataset is reusable beyond the proposed model.","pith_inferences":["Editorial inference: if the generalization claim survives on broader test sets, AffordanceSAM could be dropped into robot grasping or human-robot interaction loops where one verb-object prompt must map onto unseen object instances; the paper only gestures at robotics and reports no closed-loop test.","Editorial inference: the pseudo-label post-processing pipeline suggests that a weak supervisor plus aggressive thresholding can create usable affordance supervision at scale; replacing LOCATE with a stronger current model in stage 2 is a direct comparative experiment the paper does not run.","Editorial inference: the reported failure cases, multiple objects or multiple actions in one image, point to the single-object and single-action composition of the training data as the main remaining bottleneck, so constructing composite-scene C2F-Aff extensions is the most direct test of further gains."],"forward_implications":["On the paper's evidence, fully supervised affordance grounding can be built directly on a SAM foundation model with a small adapter, reducing the need for task-specific decoders trained from scratch.","C2F-Aff, when used as training data, lifts the hard-split performance of other fully supervised methods such as AffordanceLLM and OOAL, so the dataset is claimed to be a reusable resource beyond AffordanceSAM itself.","Because the text prompt is formed as '<action> <object>', the model can be queried with action-object pairs at inference time; qualitative results on internet images support generalization to novel objects and actions.","The three-stage training order matters: ablations show that the coarse-to-fine schedule, rather than simply adding more data, is necessary for the reported gains.","If the central claims hold, the weakly supervised approach's two-branch training complexity is no longer needed for strong affordance grounding, since a fully supervised SAM adaptation achieves higher scores with simpler supervision."],"supporting_citations":[{"why":"It supplies the SAM segmenter whose generalization the paper claims to reuse for affordance grounding.","marker":"Kirillov et al. 2023"},{"why":"It provides EVF-SAM, the text-prompted SAM baseline that AffordanceSAM starts from and extends.","marker":"Zhang et al. 2024"},{"why":"It provides AGD20K, the benchmark dataset used for evaluation and the source of stage-2 and stage-3 images.","marker":"Luo et al. 2022"},{"why":"It generates the LOCATE pseudo-labels that are post-processed into stage-2 supervision.","marker":"Li et al. 2023a"},{"why":"It defines the easy and hard split protocol that the paper follows and serves as a fully supervised baseline.","marker":"Qian et al. 2024"},{"why":"It is the prior fully supervised baseline, OOAL, whose hard-split performance improves when trained on C2F-Aff.","marker":"Li et al. 2024"},{"why":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff.","marker":"Zhai et al. 2022"},{"why":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff.","marker":"Guo et al. 2023"},{"why":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff.","marker":"Myers et al. 2015"},{"why":"It provides BEIT-3, the multimodal encoder inside EVF-SAM that produces text-conditioned image features.","marker":"Wang et al. 2023b"}],"fun_headline_variants":["SAM gains affordance skills with coarse-to-fine dataset","AffordanceSAM: SAM reimagined for action regions","Three-stage training turns SAM into affordance expert","New dataset and training lift SAM to top affordance model","C2F-Aff data helps SAM ace affordance grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the stage-2 pseudo-labeled images and the object categories they depict do not appear in the easy or hard test splits of AGD20K; if test images or their categories leak into the training pool, the reported generalization numbers stop measuring generalization.","fun_headline_variants_meta":{"raw":{"variants":["SAM gains affordance skills with coarse-to-fine dataset","AffordanceSAM: SAM reimagined for action regions","Three-stage training turns SAM into affordance expert","New dataset and training lift SAM to top affordance model","C2F-Aff data helps SAM ace affordance grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1369,"prompt_tokens":884,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":500,"tokens_out":485,"duration_ms":4874,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:21:07.533013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the evaluation after deleting from training any image whose file hash appears in the test set or whose object category appears in the hard-split test categories; if any deletion changes the reported KLD, SIM, or NSS values for AffordanceSAM trained on C2F-Aff, the split-disjointness assumption is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides AGD20K, the benchmark dataset used for evaluation and the source of stage-2 and stage-3 images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff."},{"cited_title":"L.; Ferm \\\"u ller, C.; and Aloimonos, Y","cited_arxiv_id":null,"evidence_quote":"It is one of the three mask-annotated data sources used in stage 1 of C2F-Aff."}],"review_version":1}