pith. sign in

arxiv: 1906.09836 · v1 · pith:J7DPVVL5new · submitted 2019-06-24 · 💻 cs.RO

Learning Grasp Affordance Reasoning through Semantic Relations

Paola Ard\'on , \`Eric Pairet , Ronald P. A. Petrick , Subramanian Ramamoorthy , Katrin S. Lohan This is my paper

Pith reviewed 2026-05-25 17:26 UTC · model grok-4.3

classification 💻 cs.RO
keywords grasp affordancesemantic relationsMarkov Logic Networksgeneralizationrobotic graspingknowledge baseprototypical patchesmanipulation tasks
0
0 comments X

The pith

Combining multiple semantic attributes in a Markov Logic Network produces probability distributions over grasp affordances that generalize to novel objects.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that grasp affordance reasoning can be improved by encoding relations among semantic attributes rather than relying on single hypotheses. This is done by building a knowledge base with Markov Logic Networks from a new dataset of attribute relations. A sympathetic reader would care because it enables robots to handle manipulation tasks on previously unseen objects. The approach learns prototypical grasping patches to map the predictions onto object surfaces. Robotic evaluations in simulation and reality confirm higher success rates for grasping when guided by these predictions.

Core claim

By defining semantics as combinations of multiple attributes and using Markov Logic Networks to construct a knowledge base graph, the method obtains a probability distribution of grasp affordances for an object. Reliable mappings to the object are achieved by learning prototypical grasping patches from examples, enabling generalization to novel instances and successful execution on a robotic platform.

What carries the argument

Markov Logic Network knowledge base that encodes semantic attribute relations to generate grasp affordance probability distributions.

If this is right

  • Multiple grasp affordances can be detected and extracted from visual input of a single object.
  • The predictions generalize to novel object instances beyond the training set.
  • Comparisons show advantages over similar methods from the literature.
  • Grasping tasks succeed more often on simulated and real robotic platforms when conditioned on the affordance predictions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Robots could use this to infer grasps for objects sharing semantic properties like shape or function without explicit training.
  • The dataset of semantic relations might be extended to other manipulation tasks beyond grasping.
  • Combining this with visual perception systems could lead to fully autonomous object interaction in unstructured environments.

Load-bearing premise

The semantic attributes in the collected dataset and their encoding in the Markov Logic Network are sufficient to generate accurate probability distributions that work on real-world novel objects.

What would settle it

A test set of novel objects where the method's grasp affordance predictions lead to lower grasping success rates than existing single-hypothesis approaches would falsify the generalization claim.

Figures

Figures reproduced from arXiv: 1906.09836 by \`Eric Pairet, Katrin S. Lohan, Paola Ard\'on, Ronald P. A. Petrick, Subramanian Ramamoorthy.

Figure 1
Figure 1. Figure 1: PR2 reasoning about grasp affordances of objects on a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Proposed framework for reasoning about object grasp affordances, composed of the learning, querying and mapping [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mapping grasp affordance patches on 3-D object data. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Hausdorff distance example between two grasp regions [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: The first row shows an example of an image containing [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Examples of learned grasp affordance patch features [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: (a) Comparison of our method with state-of-the-art alternatives [ [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
read the original abstract

Reasoning about object affordances allows an autonomous agent to perform generalised manipulation tasks among object instances. While current approaches to grasp affordance estimation are effective, they are limited to a single hypothesis. We present an approach for detection and extraction of multiple grasp affordances on an object via visual input. We define semantics as a combination of multiple attributes, which yields benefits in terms of generalisation for grasp affordance prediction. We use Markov Logic Networks to build a knowledge base graph representation to obtain a probability distribution of grasp affordances for an object. To harvest the knowledge base, we collect and make available a novel dataset that relates different semantic attributes. We achieve reliable mappings of the predicted grasp affordances on the object by learning prototypical grasping patches from several examples. We show our method's generalisation capabilities on grasp affordance prediction for novel instances and compare with similar methods in the literature. Moreover, using a robotic platform, on simulated and real scenarios, we evaluate the success of the grasping task when conditioned on the grasp affordance prediction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript presents a method for detecting multiple grasp affordances on objects from visual input by encoding semantic attributes (shape, material, function relations) as weighted first-order logic formulas in a Markov Logic Network knowledge base. It collects and releases a novel dataset to populate the KB, learns prototypical grasping patches, claims generalization of affordance predictions to novel object instances, compares against literature methods, and evaluates grasping success on a robotic platform in both simulated and real scenarios.

Significance. If the generalization results hold with rigorous quantitative support, the integration of semantic relations via MLNs could improve robustness of affordance-based manipulation over single-hypothesis visual methods, with the public dataset aiding reproducibility in robotics research.

major comments (1)
  1. [Abstract] Abstract: the central generalization claim (reliable mappings and prediction for novel instances) rests on the sufficiency of the collected semantic attributes and MLN rules to yield transferable probability distributions, yet the abstract supplies no object count, attribute coverage statistics, data-split protocol, or quantitative metrics (e.g., accuracy or success rates) to substantiate that the KB produces accurate marginals outside the training set.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central generalization claim (reliable mappings and prediction for novel instances) rests on the sufficiency of the collected semantic attributes and MLN rules to yield transferable probability distributions, yet the abstract supplies no object count, attribute coverage statistics, data-split protocol, or quantitative metrics (e.g., accuracy or success rates) to substantiate that the KB produces accurate marginals outside the training set.

    Authors: We agree that the abstract would be strengthened by including these quantitative details. In the revised version we will add the number of objects and attributes in the collected dataset, the train/test split protocol used to assess novel instances, and the key performance figures (affordance prediction accuracy and grasping success rates) already reported in the experimental sections. This will make the generalization claims more self-contained in the abstract without altering the manuscript's technical content. revision: yes

Circularity Check

0 steps flagged

No circularity: derivation relies on external dataset, standard MLN inference, and independent generalization tests

full rationale

The paper collects a novel dataset of semantic attributes, encodes them into an MLN knowledge base using first-order logic formulas, learns prototypical grasp patches from examples, and evaluates probability distributions on held-out novel instances. No step reduces a claimed prediction to a fitted parameter by construction, no self-citation is invoked as a uniqueness theorem or load-bearing premise, and the central generalization claim is tested against external benchmarks rather than being definitionally equivalent to the training inputs. The approach is therefore self-contained against the provided evidence.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that semantic attributes form a sufficient knowledge base for probabilistic inference over grasps; no free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption Markov Logic Networks can encode semantic relations as weighted first-order logic formulas to produce a probability distribution over grasp affordances.
    Invoked when building the knowledge base graph from the collected dataset.

pith-pipeline@v0.9.0 · 5725 in / 1175 out tokens · 18498 ms · 2026-05-25T17:26:44.613792+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 1 internal anchor

  1. [1]

    The theory of affordances,

    J. Gibson, “The theory of affordances,” in Perceiving, Acting, and Knowing: Toward and Ecological Psychology (R. Shaw and J. Bransford, eds.), pp. 62–82, Hillsdale, NJ: Erlbaum, 1977

    work page 1977

  2. [2]

    Learning grasping affordances from local visual descriptors,

    L. Montesano and M. Lopes, “Learning grasping affordances from local visual descriptors,” in Development and Learning, 2009. ICDL 2009. IEEE 8th International Conference on , pp. 1–6, IEEE, 2009

    work page 2009

  3. [3]

    Learning to grasp and extract affordances: the Integrated Learning of Grasps and Affordances (ILGA) model,

    J. Bonaiuto and M. A. Arbib, “Learning to grasp and extract affordances: the Integrated Learning of Grasps and Affordances (ILGA) model,” Biological cybernetics , vol. 109, no. 6, pp. 639–669, 2015

    work page 2015

  4. [4]

    The affordance template ros package for robot task programming,

    S. Hart, P. Dinh, and K. A. Hambuchen, “The affordance template ros package for robot task programming,” 2015 IEEE International Conference on Robotics and Automation (ICRA) , pp. 6227–6234, 2015

    work page 2015

  5. [5]

    Affordancenet: An end-to-end deep learning approach for object affordance detection,

    T.-T. Do, A. Nguyen, and I. Reid, “Affordancenet: An end-to-end deep learning approach for object affordance detection,” in International Conference on Robotics and Automation (ICRA) , 2018

    work page 2018

  6. [6]

    Deep learning for detecting robotic grasps,

    I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” International Journal of Robotics Research , vol. 34, no. 4-5, pp. 705–724, 2015

    work page 2015

  7. [7]

    Learning grasping points with shape context,

    J. Bohg and D. Kragic, “Learning grasping points with shape context,” Robotics and Autonomous Systems , vol. 58, no. 4, pp. 362–377, 2010

    work page 2010

  8. [8]

    Object–action com- plexes: Grounded abstractions of sensory–motor processes,

    N. Krüger, C. Geib, J. Piater, R. Petrick, M. Steedman, F. Wörgötter, A. Ude, T. Asfour, D. Kraft, D. Omr ˇcen, et al. , “Object–action com- plexes: Grounded abstractions of sensory–motor processes,” Robotics and Autonomous Systems , vol. 59, no. 10, pp. 740–757, 2011

    work page 2011

  9. [9]

    Object- based affordances detection with convolutional neural networks and dense conditional random fields,

    A. Nguyen, D. Kanoulas, D. G. Caldwell, and N. G. Tsagarakis, “Object- based affordances detection with convolutional neural networks and dense conditional random fields,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2017

    work page 2017

  10. [10]

    Learning task con- straints for robot grasping using graphical models,

    D. Song, K. Huebner, V . Kyrki, and D. Kragic, “Learning task con- straints for robot grasping using graphical models,” in Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on , pp. 1579–1585, IEEE, 2010

    work page 2010

  11. [11]

    Learning relational affordance models for robots in multi- object manipulation tasks,

    B. Moldovan, P. Moreno, M. van Otterlo, J. Santos-Victor, and L. De Raedt, “Learning relational affordance models for robots in multi- object manipulation tasks,” in Robotics and Automation (ICRA), 2012 IEEE International Conference on , pp. 4373–4378, IEEE, 2012

    work page 2012

  12. [12]

    Learning object-specific grasp affordance densities,

    R. Detry, E. Baseski, M. Popovic, Y . Touati, N. Kruger, O. Kroemer, J. Peters, and J. Piater, “Learning object-specific grasp affordance densities,” in Development and Learning, 2009. ICDL 2009. IEEE 8th International Conference on , pp. 1–7, IEEE, 2009

    work page 2009

  13. [13]

    Efficient grasping from rgbd im- ages: Learning using a new rectangle representation,

    Y . Jiang, S. Moseson, and A. Saxena, “Efficient grasping from rgbd im- ages: Learning using a new rectangle representation,” in 2011 IEEE In- ternational Conference on Robotics and Automation (ICRA) , pp. 3304– 3311, IEEE, 2011

    work page 2011

  14. [14]

    Markov logic networks,

    M. Richardson and P. Domingos, “Markov logic networks,” Machine learning, vol. 62, no. 1-2, pp. 107–136, 2006

    work page 2006

  15. [15]

    Reasoning about object affordances in a knowledge base representation,

    Y . Zhu, A. Fathi, and L. Fei-Fei, “Reasoning about object affordances in a knowledge base representation,” in European conference on computer vision, pp. 408–424, Springer, 2014

    work page 2014

  16. [16]

    Describing objects by their attributes,

    A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, “Describing objects by their attributes,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on , pp. 1778–1785, IEEE, 2009

    work page 2009

  17. [17]

    Visuomotor neu- rons: Ambiguity of the discharge or ‘motor’perception?,

    L. Fadiga, L. Fogassi, V . Gallese, and G. Rizzolatti, “Visuomotor neu- rons: Ambiguity of the discharge or ‘motor’perception?,” International journal of psychophysiology , vol. 35, no. 2-3, pp. 165–177, 2000

    work page 2000

  18. [18]

    Towards robust grasps: Using the environment semantics for robotic object affordances,

    P. Ardón, È. Pairet, S. Ramamoorthy, and K. S. Lohan, “Towards robust grasps: Using the environment semantics for robotic object affordances,” in Proceedings on AAAI FS on Reasoning and Learning in Real-World Systems for Long-Term Autonomy , pp. 5–12, AAAI Press, 2018

    work page 2018

  19. [19]

    Modified broyden’s method for accelerating conver- gence in self-consistent calculations,

    D. D. Johnson, “Modified broyden’s method for accelerating conver- gence in self-consistent calculations,” Physical Review B, vol. 38, no. 18, p. 12807, 1988

    work page 1988

  20. [20]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016

    work page 2016

  21. [21]

    State-space models with regime switching: classical and gibbs-sampling approaches with applications,

    C.-J. Kim, C. R. Nelson, et al. , “State-space models with regime switching: classical and gibbs-sampling approaches with applications,” MIT Press Books , vol. 1, 1999

    work page 1999

  22. [22]

    Katz and A

    S. Katz and A. Tal, Hierarchical mesh decomposition using fuzzy clustering and cuts , vol. 22. ACM, 2003

    work page 2003

  23. [23]

    3dnet: Large-scale object class recognition from cad models,

    W. Wohlkinger, A. Aldoma, R. B. Rusu, and M. Vincze, “3dnet: Large-scale object class recognition from cad models,” in 2012 IEEE International Conference on Robotics and Automation , pp. 5384–5391, IEEE, 2012

    work page 2012

  24. [24]

    Recognizing indoor scenes,

    A. Quattoni and A. Torralba, “Recognizing indoor scenes,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on , pp. 413–420, IEEE, 2009

    work page 2009

  25. [25]

    AI2-THOR: An Interactive 3D Environment for Visual AI

    E. Kolve, R. Mottaghi, D. Gordon, Y . Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017