{"id":"f482184f-5b0b-4d3b-ae2b-f74b0ab84682","arxiv_id":"2508.01651","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DAG uses frozen text-to-image diffusion features plus a learned decoder to predict manipulable regions on 3D object point clouds from a reference human-object interaction image and an affordance verb.","lead":"This paper builds a 3D object affordance grounding system that pulls visual features from a frozen text-to-image diffusion model instead of training a vision encoder from scratch. The proposed model, DAG, reports higher accuracy than prior methods on the PIAD benchmark in both seen and unseen object settings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'one-shot setting' claim has no corresponding experiment in the body; the central open-world generalization claim is currently unsupported at its strongest stated setting.","rationale":"The reader's designated weakest assumption was the t=0 diffusion timestep. That concern is real, but it is partly mitigated by the extractor ablation in Table 4, which holds the rest of the pipeline fixed and shows the diffusion extractor outperforming ResNet, CLIP, and DINOv2. A timestep sweep would be informative but would not, by itself, overturn the central claim. By contrast, the abstract's one-shot claim is a standalone empirical assertion with zero supporting evidence anywhere in the body. Because the paper's title and framing emphasize open-world generalization, an unsubstantiated one-shot claim is the most direct threat to the advertised contribution. The reader flagged this gap in the strongest_claim and rationale, though not in the weakest_assumption field, hence partial agreement. The appropriate remedy is either to add the missing experiment or to soften the abstract; neither requires rejecting the core benchmark comparisons. The reader's CONDITIONAL verdict therefore remains appropriate, and the verdict is unchanged.","tokens_in":13150,"tokens_out":9058,"duration_ms":106192,"concrete_test":"Run the missing one-shot evaluation on PIAD: define a protocol (e.g., one annotated point cloud per affordance-object pair from the seen split), train DAG under that protocol, and report mIoU/AUC/SIM/MAE on the unseen split averaged over seeds. Independently, grep the submitted PDF for 'one-shot'/'one shot' outside the abstract; if there is no match, the claim is unsupported. If no one-shot protocol is specified, the minimal fix is to delete the one-shot sentence from the abstract and revise the headline generalization claim accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states that DAG 'exhibits strong open-world generalization, even in the challenging one-shot setting.' A full-text search of the manuscript finds 'one-shot' only in the abstract; Sections 4.1-4.4 report only the standard PIAD seen/unseen splits, with no few-example training protocol, no shot-count definition, and no one-shot results in Table 1 or the ablation tables. This matters because the unseen split is not a one-shot setting: it still trains on the full seen-set annotations and only holds out object/affordance pairings. The one-shot claim is therefore qualitatively stronger than anything the experiments demonstrate. Since the title and framing rest on open-world generalization, this missing evaluation is the most direct gap between the paper's advertised contribution and its evidence. The rest of the empirical core (Table 1, Tables 2-4) is internally consistent, so the finding is an overclaim rather than an internal inconsistency, but it is material: if the one-shot claim is not true, the abstract and the paper's headline framing need revision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DAG, a 3D affordance grounding method that uses the frozen internal features of a text-to-image Stable Diffusion UNet as an affordance prior, fuses them with text and point-cloud features through an affordance block and a multi-source decoder, and predicts dense affordance masks on the PIAD benchmark. The main experiments report consistent improvements over prior methods on both seen and unseen splits, and ablations examine the affordance block, [CLS] token, captioning strategy, and image feature extractor. The paper also claims open-world generalization, including in a one-shot setting, and releases code under a public repository link.","tokens_in":13333,"tokens_out":4362,"duration_ms":52243,"significance":"If the reported gains are robust, the paper makes a useful empirical contribution by showing that frozen text-to-image diffusion features can improve 3D affordance grounding, complementing recent work that repurposes diffusion internals for 2D tasks. The evaluation uses the external PIAD benchmark, the Stable Diffusion and Uni3D encoders are frozen, the training losses are standard Dice and BCE, and the ablations in Tables 2–4 provide useful component-level evidence. The code link and the explicit reporting of standard deviations for DAG are also positive practical elements. However, the strongest advertised claim, one-shot open-world generalization, is not tested, and the design choice t=0 in Eq. (1) is not ablated, leaving uncertainty about whether the gains stem specifically from the diffusion prior.","major_comments":[{"comment":"The abstract claims that DAG 'exhibits strong open-world generalization, even in the challenging one-shot setting,' but the body contains no one-shot experiment. Section 4.1 defines only the standard PIAD seen/unseen splits, and Section 4.3 reports results after full training on the seen-set annotations; the word 'one-shot' does not appear anywhere in Sections 4.1–4.4. The unseen split is not a one-shot protocol because it still trains on all seen-set annotations and only holds out object/affordance pairings. Please add a properly defined one-shot or few-shot experiment (for example, limiting training exemplars per object/affordance class) or remove the one-shot claim from the abstract and from the open-world framing.","section":"Abstract and Section 4"},{"comment":"The method defaults to diffusion timestep t=0, but by the definition in Eq. (1), alpha_bar_0 = 1 gives a clean input x_t = x, which is far from the noisy inputs the diffusion UNet was trained on. The paper provides no ablation over t and no comparison with features extracted at other timesteps, so it is not established that the observed gains come from diffusion-specific affordance knowledge rather than from generic visual features combined with the trained decoder. Please report results for several timesteps (e.g., t=0, t=50, t=200, t=500) and, ideally, a control that replaces the frozen pretrained UNet with a randomly initialized version of the same architecture to isolate the contribution of the pretrained diffusion prior.","section":"Equation (1) and Section 4.2"},{"comment":"Several implementation details are deferred to 'the Appendix,' but no appendix is included in the submitted manuscript. These details include the construction of the self-prompt in Section 3.3, the hierarchical up-sampling and feature propagation in Section 3.5, the specific UNet layers used to build the feature pyramid, and the depths and widths of the affordance block and decoder. Without these details the reported numbers are not reproducible even with the code link. Please include the appendix or move the essential implementation choices into the main text.","section":"Sections 3.3, 3.5, and 4.2"}],"minor_comments":[{"comment":"The sentence 'by passing the need for additional affordance labels' should read 'bypassing the need'.","section":"Section 4.2"},{"comment":"Tables 3 and 4 do not state whether the reported ablation numbers are on the seen or unseen split; please label the split explicitly, since the generalization claims depend on this distinction.","section":"Tables 3 and 4"},{"comment":"Standard deviations are reported only for DAG; for fair comparison, please provide variability estimates for the strongest baselines or clarify that the deviations were not available.","section":"Table 1"},{"comment":"The baselines paragraph contains an unresolved citation marker '?' in the list of open-vocabulary affordance learning works; please replace it with the intended reference.","section":"Section 4.1 references"},{"comment":"The title contains a stray space in 'Open-V ocabulary'; this should be corrected.","section":"Title"}],"recommendation":"major_revision","confidential_remarks":"The paper's core table is internally consistent and the gains over prior methods are plausible, but the abstract's one-shot claim is unsupported by any experiment and the central interpretation of the diffusion timestep needs an ablation. The missing appendix is also a substantive reproducibility issue for a submission that points to it repeatedly. I would like to see the one-shot claim either substantiated or removed, and the timestep/feature-source question addressed with new experiments before the paper is accepted. The editor may also want to verify that the GitHub repository is publicly accessible and that the exact Stable Diffusion checkpoint and code version match the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe short version: this paper has a real, modestly novel idea and an honest empirical core, but the abstract makes a one-shot generalization claim that no experiment in the body supports. With that claim corrected, the rest is a reasonable submission; as is, a referee needs to send it back for revision.\n\nWhat's actually new: DAG is the first work in its cited set to take frozen text-to-image diffusion UNet features and use them for 3D affordance grounding on point clouds. The design is simple: one forward pass through the UNet, with features pooled and fed along with text and point cloud features into a decoder. The modules themselves are not exotic, but the application is new. The ablations are the paper's strongest feature. They test the affordance block, the CLS token, three captioner variants, and, crucially, swap the diffusion extractor for ResNet, CLIP, and DINOv2. Diffusion wins by a clear margin, which gives the central claim some teeth. The PIAD results are consistent across metrics, with DAG beating GREAT by roughly 2 mIoU on the seen split and 0.9 on the unseen split.\n\nThe soft spots. First, the one-shot claim is simply not in the paper. 'One-shot' appears only in the abstract. The unseen split still trains on the full seen set; it omits affordance-object pairings, which is a real generalization test, but it is not few-shot. That is a load-bearing overclaim because the title and framing lean on open-world generalization. Second, the implementation sets the diffusion timestep to t=0 without any ablation. The input to the UNet is then clean, not noisy, which sits awkwardly with Eq. (1). Maybe it works fine, but the paper should show it. Third, the appendix is referenced multiple times but is missing from the submission, which makes reproducibility hard to assess despite the code link. Fourth, the phrase 'affordance knowledge' is an interpretation of why the features help, not a mechanism that has been tested. The DINOv2 comparison partially addresses this, but it is not conclusive.\n\nWho this is for: people working on 3D affordance grounding or robot manipulation will likely want to read it. The idea is a reasonable new baseline.\n\nVerdict on review: yes, send it to peer review. The gaps are fixable—run the one-shot experiment, ablate t, release the appendix. The empirical core holds up. Just make the authors' claims match the evidence.\n\nBest,\n[Your name]","headline":"A real but modest novelty with an unsupported one-shot claim in the abstract; the empirical core is sound, so send to review but require the missing experiments.","tokens_in":13933,"tokens_out":3175,"would_cite":true,"duration_ms":34716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen text-to-image diffusion model encodes general 3D affordance knowledge, and DAG transfers it to point clouds, outperforming prior grounding methods.","keywords":["3D affordance grounding","diffusion models","open-vocabulary","point cloud","human-object interaction","generative priors","cross-modal learning","affordance transfer"],"falsifier":"Run the DAG pipeline on the PIAD unseen split with diffusion timesteps t=0, t=50, and t=200. If t=0 is not the best setting, or if a frozen DINOv2 backbone of comparable capacity closes the mIoU gap when the decoder is identical, the claim that diffusion priors specifically carry affordance knowledge would be weakened.","tokens_in":12954,"feed_emoji":"🎯","tokens_out":11677,"duration_ms":117950,"temperature":0.7,"pith_summary":"3D affordance grounding predicts which regions of a 3D object support a given interaction, a skill needed by robots and embodied agents. Prior methods learn this from human-object interaction images but generalize poorly to unseen object-action pairs. The paper argues that text-to-image diffusion models, trained on internet-scale data to synthesize images, necessarily encode how humans use objects, and that this knowledge can be lifted to 3D. It introduces DAG, which extracts multi-scale features from a frozen diffusion UNet in one forward pass, fuses them with language cues, and decodes per-point affordance masks on point clouds. On the PIAD benchmark, DAG outperforms prior methods on both seen and unseen splits, supporting the idea that generative priors are reusable affordance knowledge.","feed_headline":"Frozen diffusion model beats prior work on 3D affordance grounding","feed_subtitle":"DAG extracts affordance knowledge from a single forward pass of a frozen generative UNet and wins on seen and unseen splits of PIAD.","key_machinery":"The central object is the frozen text-to-image diffusion UNet, the denoising network inside a text-to-image diffusion model, which DAG repurposes as a fixed feature extractor instead of running the full sampling loop. The argument is carried by a feature-pyramid extraction: a single forward pass at timestep t=0 produces multi-scale internal feature maps, an interpretable aggregation network learns mixing weights over those layers, and an Affordance Block fuses the aggregated visual features with text embeddings of the affordance verb via self- and cross-attention, followed by pooling into affordance tokens. A multi-source decoder then cross-attends point tokens from a frozen 3D encoder and a global [CLS] token with these affordance tokens, and an MLP outputs a per-point affordance mask. The load-bearing property is that these frozen features, without any affordance-specific fine-tuning, encode enough human-object interaction knowledge to transfer across object categories and affordance types.","core_discovery":"Diffusion models are not just image generators: their internal representations capture how objects are functionally used. The paper shows that a pretrained text-to-image diffusion model, when fed a human-object interaction image, produces feature maps that highlight affordance-relevant regions even without any affordance-specific training, and that this knowledge transfers to 3D. DAG runs a single forward pass of the frozen diffusion UNet at timestep t=0, aggregates multi-scale features with learned weights, and combines them with text embeddings of the affordance verb and an implicit caption of the image. A multi-source decoder then produces a dense affordance mask on the input point cloud. The diffusion model, point encoder, and text encoder remain frozen; only lightweight aggregation, attention, and decoding modules are learned with binary cross-entropy and Dice losses. This recipe outperforms prior methods on the PIAD benchmark, with the largest margins on the unseen split.","pith_inferences":["If a frozen text-to-image UNet carries affordance knowledge at t=0, then other frozen generative backbones, such as text-to-video or image-editing models, may supply analogous interaction knowledge for tasks like grasp planning or tool-use reasoning, since they are trained on similar internet-scale data.","Because the diffusion timestep is fixed at t=0 without an ablation, a natural next experiment is to map how affordance features vary across timesteps; this would reveal whether the knowledge lives in the clean-image branch or in the denoising dynamics.","The implicit captioner's strong showing suggests that any image-to-text model that can describe an object in interaction terms might replace the learned module, opening a path to fully zero-shot affordance grounding with no task-specific captioner training."],"forward_implications":["A single forward pass of a frozen diffusion model suffices to extract affordance knowledge, so the full multi-step denoising process is never needed at inference.","The same diffusion backbone yields higher mIoU than ResNet, CLIP, or DINOv2 features when plugged into an identical framework, indicating the generative prior is the key ingredient.","DAG generalizes to unseen affordance-object pairings and to partial, noisy point clouds, matching the demands of real-world perception.","Only the lightweight aggregation network, affordance block, and decoder are trained, while the diffusion model, point encoder, and text encoder stay frozen."],"supporting_citations":[{"why":"Introduces the PIAD dataset and the IAGNet baseline, defining the seen/unseen evaluation protocol and metrics that DAG uses.","marker":"Yang et al. 2023b"},{"why":"Provides the text-to-image diffusion model whose frozen UNet features DAG extracts as affordance knowledge.","marker":"Rombach et al. 2022"},{"why":"Supplies the feature-pyramid extraction recipe from a frozen diffusion UNet that DAG adapts for affordance grounding.","marker":"Xu et al. 2023"},{"why":"Presents GREAT, the strongest prior open-vocabulary affordance grounding baseline that DAG must beat on PIAD.","marker":"Shao et al. 2024"},{"why":"Presents MIFAG, a recent baseline learning 2D invariant affordance knowledge, which DAG compares against.","marker":"Gao et al. 2025"},{"why":"Defines the 3D affordance grounding task and the 3D AffordanceNet benchmark used for visualization on partial point clouds.","marker":"Deng et al. 2021"},{"why":"Supplies Uni3D, the frozen point encoder that produces the point features and the [CLS] token used in DAG's decoder.","marker":"Zhang et al. 2023"},{"why":"Provides the self-prompt mechanism that the implicit captioner uses to generate flexible text embeddings from the input image.","marker":"Song, Wang, and Zhong 2024"}],"fun_headline_variants":["Diffusion priors teach 3D models how to interact with objects","DAG: frozen diffusion UNet unlocks open-world 3D affordance","One forward pass: diffusion model grounds 3D affordances","From images to interaction: diffusion knows how objects work","Diffusion encoders hold affordance knowledge for 3D grounding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single forward pass of a frozen image-generation network at the clean-image timestep (t=0) exposes useful knowledge about how objects are used, even though the network was trained for denoising at noisier timesteps.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion priors teach 3D models how to interact with objects","DAG: frozen diffusion UNet unlocks open-world 3D affordance","One forward pass: diffusion model grounds 3D affordances","From images to interaction: diffusion knows how objects work","Diffusion encoders hold affordance knowledge for 3D grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1394,"prompt_tokens":956,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":572,"tokens_out":438,"duration_ms":4655,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:28:12.906966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the DAG pipeline on the PIAD unseen split with diffusion timesteps t=0, t=50, and t=200. If t=0 is not the best setting, or if a frozen DINOv2 backbone of comparable capacity closes the mIoU gap when the decoder is identical, the claim that diffusion priors specifically carry affordance knowledge would be weakened.","supporting_citations":[],"review_version":1}