CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding

Joe Lin; Sathyanarayanan N. Aakur; Zhou Chen

REVIEW 4 major objections 5 minor 16 references

CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CRAFT is a neuro-symbolic framework that grounds functional affordances by fusing commonsense priors with visual evidence through an energy-based iterative loop.

desk verdict Plausible neuro-symbolic assembly for affordance grounding, but the headline top-1 gain is near sampling noise and the iterative reweighting is never ablated. read the letter →

arxiv 2507.14426 v1 pith:IY76CIRJ submitted 2025-07-19 cs.CV

Zhou Chen , Joe Lin , Sathyanarayanan N. Aakur This is my paper · ORCID

classification cs.CV

keywords functionalaffordanceneuro-symbolicreasoninggroundingvision-languagemodelsenergy-basedConceptNetCLIPinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CRAFT is a neuro-symbolic framework that identifies which objects in a scene afford a given action such as “cut”, without relying on object labels. It combines structured commonsense priors from ConceptNet or large language models with visual evidence from CLIP, then refines the candidate list through an energy-based iterative re-ranking loop. The paper argues that this integration makes affordance grounding more accurate and more interpretable than prior-only or purely learned baselines. Its headline result is that CRAFT with GPT-4o priors reaches 46.43% top-1 accuracy and 60.80% MRR / 67.00% nDCG, the best non-oracle performance on the benchmark.

What carries the argument

The central object is an affordance graph $G_v$ built from ConceptNet by traversing relations such as UsedFor and CapableOf from a verb to object candidates, with each object assigned a prior score $\phi(o,v)$. The grounding energy is $E(v,x_i) = -\max_{o \in G_v}[\phi(o,v) \cdot s(o,x_i)]$, where $s(o,x_i)=\cos(g(o), f(x_i))$ is the cosine similarity between a CLIP text embedding of the object label and a CLIP image embedding of the candidate image. The iterative step updates priors as $\phi^{(t+1)}(o,v) \propto \phi^{(t)}(o,v) \cdot \exp(\lambda \cdot s(o,x_t))$, with $x_t$ the current top-ranked image, so visual evidence gradually sharpens the symbolic prior.

What would settle it

Run CRAFT on the same benchmark with $\lambda$ set to values from 0 to 10 and see whether the reported gains persist; if accuracy peaks at one value and collapses nearby, or if setting $\lambda=0$ (no iterative update) already matches the final numbers, the iterative reweighting is not the source of the improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that functional affordance grounding—matching a verb like “cut” to the image that supports it—is best treated as neuro-symbolic inference rather than pure visual similarity. On a benchmark of verb-object affordance labels, combining symbolic candidates with CLIP image-text similarity and an iterative reweighting step consistently beats each prior alone: CRAFT with GPT-4o is the best non-oracle method in single-label accuracy and multi-label ranking, and CRAFT also improves over its ConceptNet and Gemini priors. The authors further claim the framework is interpretable because each prediction carries a traceable reasoning path through the affordance graph, allowing post-hoc inspection of both correct and erroneous decisions.

Load-bearing premise

The load-bearing premise is that exponentially boosting the current top-ranked image's matched concepts, with a fixed adjustment strength $\lambda$, drives the ranking toward the true affordance; the paper does not justify the choice of $\lambda$ or report how sensitive the gains are to it.

Editorial extensions

If this is right

If CRAFT's central claim is right, affordance grounding can be improved without retraining or labels by dressing up symbolic priors with one CLIP pass and a short iterative loop.
The same pipeline should transfer to any verb for which ConceptNet or an LLM can propose object candidates, including verbs unseen during training.
Because the final ranking depends on the prior, CRAFT's gains are largest when the prior is strong (GPT-4o) but still positive when the prior is noisy (ConceptNet, Gemini).
The per-prediction reasoning traces give an operator a concrete path to inspect when the model fails, which is what the authors mean by interpretability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The iterative reweighting rule resembles a temperature-scaled ranking feedback; if the adjustment strength $\lambda$ is not tuned per verb, hard verbs with diffuse ConceptNet graphs may over-amplify the first visual match. Varying $\lambda$ and measuring accuracy would test this directly.
The energy formulation could be extended to region proposals or bounding boxes, allowing the same reasoning loop to pick out a specific object within an image rather than a whole image.
A natural extension is to use the affordance graph's ego-graph structure as a prior over sequences of tool use, which would connect CRAFT to action planning and robot manipulation.
The benchmark uses validation images from a 216-category ImageNet-derived dataset; moving to cluttered real-world scenes with distractors that share visual similarity with the affordant object would stress the method more than the current episode setup.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know about this one. First, the core idea—use ConceptNet or an LLM to generate candidate object concepts for a verb, score images with CLIP, then iteratively reweight the candidates by exponentiating the similarity to the current top image—is a reasonable neuro-symbolic integration and, as far as I can tell, not in the cited prior work. Second, the empirical case is weaker than the abstract suggests. The headline 'enhances accuracy' rests on a 1.13-point top-1 improvement over GPT-4o (46.43 vs 45.30). With roughly 5,000 episodes, that difference sits around one standard error. No confidence intervals, no significance test, no per-verb paired stats.

What deserves credit: the evaluation uses an external benchmark (Nguyen et al., 2020), so there's no circularity. The gains are consistent across three different priors—ConceptNet, Gemini, GPT-4o—which makes the method look robust even if each individual delta is small. The multi-label nDCG improvement (67.0 vs 60.4) is more substantive. And the appendix's reasoning traces and ego-graphs are a genuine interpretability contribution, not just a figure.

The soft spots, in order of importance. (1) The iterative reweighting is the novelty, but the paper gives no ablation without it, and no sweep over the temperature λ. So the observed gains can't be attributed to the energy-based loop; a static fusion of prior and CLIP similarity might do just as well. (2) The contribution list says 'we introduce a rigorous benchmark,' but the experiments run on Nguyen et al.'s dataset. That's a factual inconsistency that has to be fixed. (3) Missing error bars, which matters given the small headline delta. (4) Minor: no code release mentioned, and the text and Table 1 disagree on ALGO's accuracy (42.46 vs 42.52). These are all fixable in revision.

Overall, the method is plausible, clearly written, and the interpretability angle is a plus. The weaknesses are in the evidence, not the logic. I'd send it to review but require the ablations and error bars before accepting. It's a solid workshop-level paper that could become a useful reference if the authors tighten the empirical claims.

Referee Report

4 major / 5 minor

Summary. The paper proposes CRAFT, a neuro-symbolic framework for functional affordance grounding. Given a verb query and a set of unlabeled candidate images, CRAFT builds an affordance graph from ConceptNet or LLM-generated candidates, scores image-label pairs by CLIP cosine similarity, and iteratively updates prior scores with an exponential reweighting rule. The framework is evaluated on the Nguyen et al. benchmark in single- and multi-label settings, reporting top-1 accuracy, MRR, and nDCG. The authors claim that CRAFT+GPT-4o achieves the best non-oracle accuracy and that the method improves interpretability through traceable reasoning paths.

Significance. The problem is well motivated, and the proposed integration of structured commonsense priors with CLIP similarity is simple and plausible. If the accuracy gains are statistically reliable, the framework would be a useful contribution to label-free affordance grounding, with the notable strength of producing inspectable reasoning traces on an external benchmark. However, the current evidence does not yet establish the central claims: the headline single-label gain is small and unaccompanied by significance testing, the iterative reweighting loop is not ablated, and the interpretability claim is supported only by qualitative examples.

major comments (4)

[Section 3, Table 1] The headline single-label improvement of CRAFT+GPT-4o (46.43%) over GPT-4o (45.30%) is 1.13 percentage points across 5,000 episodes (50 verbs × 100 episodes). Under an independence approximation, the standard error of this difference is about 1.0 percentage point, so the gain is within sampling noise. The paper reports no seeds, confidence intervals, per-verb paired tests, or error bars; since the abstract's claim of enhanced accuracy rests on this comparison, the authors should provide significance testing or at least per-verb variance statistics.
[Section 2, Iterative Reasoning and Re-ranking] The update rule φ^{(t+1)}(o,v) ∝ φ^{(t)}(o,v) · exp(λ·s(o,x_t)) introduces a free parameter λ that controls the influence of visual feedback, yet the paper reports no λ value, no sensitivity sweep, and no λ=0 ablation. Without such an ablation, the observed gains cannot be attributed to the iterative energy-based reasoning loop rather than to a single static fusion of the prior and CLIP similarity.
[Section 3, Table 1 and text] The statement that 'CRAFT+Gemini and CRAFT+ConceptNet also show consistent gains over their prior-only counterparts' is not supported by Table 1: for CRAFT+Gemini, multi-label MRR and nDCG are 56.50% and 59.30%, which are below Gemini alone (57.50% and 59.70%). In addition, no ConceptNet prior-only baseline is reported, so the claimed improvement of CRAFT+ConceptNet over its prior cannot be verified. The authors should correct this inconsistency and add the missing baseline.
[Appendix A] The claim of improved interpretability is supported only by qualitative traces of a few correct and incorrect reasoning paths. There is no comparison with the interpretability of baseline models, no quantitative measure such as a human evaluation or faithfulness metric, and no user study. The second half of the central claim—'improving interpretability'—is therefore currently unsubstantiated beyond anecdotal illustration.

minor comments (5)

[Table 1] The table header contains a stray space in 'F unctional Affordance Performance'; please fix the typo.
[Section 3] The ALGO single-label accuracy is given as 42.52% in Table 1 but as 42.46% in the text; these should be reconciled.
[Section 2, Implementation Details] The exact prompts used for GPT-4o and Gemini are not provided, and the convergence criterion and maximum number of iterations for the reweighting loop are not specified; these details are needed for reproducibility.
[Figure 2] The distractor experiment is described in the text, but the figure caption should state the axis labels and the precise definition of 'number of distractors' (5 to 20 candidates per episode).
[Section 3, Baselines] The baseline 'Afford-CLIP' is attributed to Radford et al. (2021), but that reference describes CLIP, not an affordance-specific model; please clarify whether Afford-CLIP is a new model introduced here or an existing one with a proper citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the energy-based grounding is defined from external priors and CLIP similarity, and the evaluation is on an external benchmark; the only same-group citation is a non-load-bearing baseline.

full rationale

The paper's central derivation is self-contained. The grounding energy E(v, xi) = -max_o [phi(o,v) * s(o, xi)] combines a ConceptNet/LLM-derived prior phi with CLIP cosine similarity s; neither is defined in terms of the Nguyen et al. benchmark labels, and no parameter is fitted to the test episodes. The iterative update phi^(t+1)(o,v) proportional to phi^(t)(o,v) * exp(lambda * s(o, x_t)) is an explicitly stated heuristic that uses the model's own previous top prediction as feedback; while this is algorithmically self-referential, it is not a case of a prediction reducing to a fitted input by construction, nor is the evaluation target baked into the update. The only self-citation in the load-bearing sense would be ALGO, but ALGO appears solely as a prior-only baseline and does not justify CRAFT's design or conclusions. Statistical fragility of the 1.13-point top-1 gain over GPT-4o and the absence of a lambda sensitivity analysis are soundness/robustness concerns, not circularity. The paper is evaluated against external benchmarks and external models, so the central claim retains independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. The main free parameters are the temperature in the iterative update, the top-k truncations, and the path depth; these are not systematically justified. The axioms are the assumptions that CLIP scores and lexical priors are aligned with functional affordances, and that the update rule is a valid inference procedure.

free parameters (3)

lambda (temperature) in iterative reweighting = not reported
Controls how strongly the current top image's CLIP scores influence the next round of candidate priors; no sensitivity analysis or principled choice given.
top-k candidate counts = 25 for ConceptNet, 10 for LLM
Candidates per prior source are truncated at these limits; the choices are not justified or ablated.
path depth bound for ConceptNet subgraph = not specified
The local subgraph G_v is extracted by traversing paths of bounded depth, but the bound is not stated.

assumptions (4)

domain assumption CLIP cosine similarity between text labels and image embeddings is a valid measure of functional affordance visual alignment
The energy function E(v, x_i) is defined as a product of prior score and CLIP cosine similarity; this assumes that visual similarity to object labels captures affordances, which is known to be imperfect.
domain assumption ConceptNet path-based scores and LLM outputs provide reliable affordance priors
The candidate set G_v is built from these sources; noise in the priors is acknowledged but the framework's success depends on their partial reliability.
ad hoc to paper The exponential update rule is an appropriate inference procedure
No derivation or theoretical justification is given for phi(t+1) proportional to phi(t) * exp(lambda * s); it is simply asserted as an attention-like update.
domain assumption The Nguyen et al. dataset episodes with 5 candidates and 100 random episodes per verb fairly represent functional affordance grounding
Evaluation relies on this benchmark; no analysis of the randomness or stability across episodes is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding." pith.science (2026). https://pith.science/paper/IY76CIRJ

@misc{pith2026250714426,
  author       = {Pith},
  title        = {Pith review of: CRAFT: A Neuro-Symbolic Framework for Visual Functional Affordance Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IY76CIRJ}},
  note         = {Machine review of arXiv:2507.14426}
}

read the original abstract

We introduce CRAFT, a neuro-symbolic framework for interpretable affordance grounding, which identifies the objects in a scene that enable a given action (e.g., "cut"). CRAFT integrates structured commonsense priors from ConceptNet and language models with visual evidence from CLIP, using an energy-based reasoning loop to refine predictions iteratively. This process yields transparent, goal-driven decisions to ground symbolic and perceptual structures. Experiments in multi-object, label-free settings demonstrate that CRAFT enhances accuracy while improving interpretability, providing a step toward robust and trustworthy scene understanding.

Figures

Figures reproduced from arXiv: 2507.14426 by the authors.

**Figure 1.** CRAFT Overview. Given a verb query v, symbolic priors generate affordance candidates ov, which are grounded in visual input. Energy-based reweighting refines predictions for grounding functional affordances. settings. Given a verb query and a set of unlabeled images, CRAFT retrieves candidate objects from external knowledge sources, such as ConceptNet (Liu and Singh, 2004) or large language models (Anil et al., 202… view at source ↗

**Figure 2.** Impact of distractors. quantified as (a) Accuracy under the single-label setting. (b) nDCG under the multi-label setting, as a function of distractors. oracle baselines perform best, with the Affordance-aware oracle achieving the highest nDCG (82.70%) and MRR (69.90%). Full supervision on in-domain semantics (ResNet-RNN) helps achieve close to oracle-level performance, while web-scale supervision (Afford-CLIP) prov… view at source ↗

**Figure 3.** Illustration of some reasoning traces that are correct (a-c) and irrelevant (d-f). Score refers to the normalized path weight in ConceptNet used to compute the reasoning-based prior score ϕ(o, v) . 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Illustration of ego graphs used to generate affordance reasoning traces for verb query (a) “cut”, (b) “eat”, (c) “play”, and (d) “write.” 10 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[1]

Gemini: a family of highly capable multimodal models

Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

arXiv 2023
[2]

Taskclip: Extend large vision-language model for task oriented object detection

Hanning Chen, Wenjun Huang, Yang Ni, Sanggeon Yun, Yezi Liu, et al. Taskclip: Extend large vision-language model for task oriented object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

work page 2024
[3]

What does CLIP know about peeling a banana?

Claudia Cuttano, Gabriele Rosi, Gabriele Trivigno, and Giuseppe Averta. What does clip know about peeling a banana? arXiv preprint arXiv:2404.12015, 2024

work page Pith review arXiv 2024
[4]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255. Ieee, 2009

work page 2009
[5]

Where is the information for affordances? Ecological Psychology, 12 0 (1): 0 53--56, 2000

Eleanor J Gibson. Where is the information for affordances? Ecological Psychology, 12 0 (1): 0 53--56, 2000

work page 2000
[6]

The theory of affordances:(1979)

James J Gibson. The theory of affordances:(1979). In The people, place, and space reader, pages 56--60. Routledge, 2014

work page 1979
[7]

Gpt-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

arXiv 2024
[8]

Discovering novel actions from open world egocentric videos with object-grounded visual commonsense reasoning

Sanjoy Kundu, Shubham Trehan, and Sathyanarayanan N Aakur. Discovering novel actions from open world egocentric videos with object-grounded visual commonsense reasoning. In European Conference on Computer Vision, pages 39--56. Springer, 2024

work page 2024

Show all 16 references

[9]

Conceptnet—a practical commonsense reasoning tool-kit

Hugo Liu and Push Singh. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22 0 (4): 0 211--226, 2004

2004
[10]

Discovering object attributes by prompting large language models with perception-action apis

Angelos Mavrogiannis, Dehao Yuan, and Yiannis Aloimonos. Discovering object attributes by prompting large language models with perception-action apis. arXiv preprint arXiv:2306.13253, 2023

2023 arXiv
[11]

Robot object retrieval with contextual natural language queries

Thao Nguyen, Nakul Gopalan, Roma Patel, Matt Corsaro, Ellie Pavlick, and Stefanie Tellex. Robot object retrieval with contextual natural language queries. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020

2020
[12]

Knowledge enhanced bottom-up affordance grounding for robotic interaction

Wen Qu, Xiao Li, and Xiao Jin. Knowledge enhanced bottom-up affordance grounding for robotic interaction. PeerJ Computer Science, 10: 0 e2097, 2024

2024
[13]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pa...

2021
[14]

What object should i use? task driven object detection

Johann Sawatzky, Yaser Souri, Christian Grund, and Juergen Gall. What object should i use? task driven object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020

2020
[15]

Conceptnet 5.5: An open multilingual graph of general knowledge

Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017

2017
[16]

Cotdet: Affordance knowledge prompting for task driven object detection

Jiajin Tang, Ge Zheng, Jingyi Yu, and Sibei Yang. Cotdet: Affordance knowledge prompting for task driven object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Gemini: a family of highly capable multimodal models

Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

arXiv 2023

[2] [2]

Taskclip: Extend large vision-language model for task oriented object detection

Hanning Chen, Wenjun Huang, Yang Ni, Sanggeon Yun, Yezi Liu, et al. Taskclip: Extend large vision-language model for task oriented object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

work page 2024

[3] [3]

What does CLIP know about peeling a banana?

Claudia Cuttano, Gabriele Rosi, Gabriele Trivigno, and Giuseppe Averta. What does clip know about peeling a banana? arXiv preprint arXiv:2404.12015, 2024

work page Pith review arXiv 2024

[4] [4]

Imagenet: A large-scale hierarchical image database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255. Ieee, 2009

work page 2009

[5] [5]

Where is the information for affordances? Ecological Psychology, 12 0 (1): 0 53--56, 2000

Eleanor J Gibson. Where is the information for affordances? Ecological Psychology, 12 0 (1): 0 53--56, 2000

work page 2000

[6] [6]

The theory of affordances:(1979)

James J Gibson. The theory of affordances:(1979). In The people, place, and space reader, pages 56--60. Routledge, 2014

work page 1979

[7] [7]

Gpt-4o system card

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

arXiv 2024

[8] [8]

Discovering novel actions from open world egocentric videos with object-grounded visual commonsense reasoning

Sanjoy Kundu, Shubham Trehan, and Sathyanarayanan N Aakur. Discovering novel actions from open world egocentric videos with object-grounded visual commonsense reasoning. In European Conference on Computer Vision, pages 39--56. Springer, 2024

work page 2024