pith. sign in

arxiv: 1906.11114 · v1 · pith:QFO6MZ2Unew · submitted 2019-06-26 · 💻 cs.AI · cs.RO

From Multi-modal Property Dataset to Robot-centric Conceptual Knowledge About Household Objects

Madhura Thosar , Christian A. Mueller , Georg Jaeger , Johannes Schleiss , Narender Pulugu , Ravi Mallikarjun Chennaboina , Sai Vivek Jeevangekar , Andreas Birk
show 2 more authors
Max Pfingsthorn Sebastian Zug
This is my paper

Pith reviewed 2026-05-25 15:29 UTC · model grok-4.3

classification 💻 cs.AI cs.RO
keywords robot-centric conceptual knowledgehousehold objectsmulti-modal property extractionunsupervised clusteringtool substitutionphysical propertiesfunctional propertiesRoCS dataset
0
0 comments X

The pith

Ten physical and functional properties extracted from household objects are clustered into robot-centric symbols that generate conceptual knowledge via frequency distributions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that conceptual knowledge for robots must originate from the robot's own sensing and acting capabilities rather than human-defined categories. It selects ten properties and develops multi-modal extraction methods to collect numerical data on 110 objects. Unsupervised clustering converts these values into symbols, after which bivariate joint frequency distributions and sample proportions derive relational concepts among objects. This matters for tool-use tasks because mismatched human and robot perspectives limit effective decision making during object interactions.

Core claim

Multi-modal extraction of ten physical and functional properties from 110 household objects supplies numerical data that unsupervised clustering converts into robot-centric symbols; bivariate joint frequency distributions and sample proportions then operate on those symbols to produce conceptual knowledge that supports real-world applications such as tool substitution.

What carries the argument

Unsupervised clustering of numerical property data into symbols, combined with bivariate joint frequency distributions and sample proportion calculations to form conceptual relations.

If this is right

  • Robots obtain symbols and concepts grounded directly in their own property measurements.
  • The RoCS dataset supplies a concrete resource for testing property extraction and knowledge generation.
  • Conceptual knowledge produced this way can inform decision making in household tool-use scenarios.
  • The same pipeline evaluates both the semantics of the properties and their practical usefulness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be tested on tasks beyond tool substitution, such as object sorting or grasping planning.
  • Periodic re-clustering on new sensor readings would allow the symbols to adapt when object properties change over time.
  • Direct comparison of success rates between robot-centric and human-centric knowledge on identical tasks would quantify the claimed advantage.

Load-bearing premise

The ten chosen physical and functional properties, once turned into symbols by clustering, capture enough distinctions to support useful conceptual knowledge for robot decisions.

What would settle it

Run the generated conceptual knowledge in a tool-substitution experiment on the 110-object set and measure success rate against a baseline that uses no such knowledge or uses human-defined categories; zero or negative improvement falsifies the utility claim.

Figures

Figures reproduced from arXiv: 1906.11114 by Andreas Birk, Christian A. Mueller, Georg Jaeger, Johannes Schleiss, Madhura Thosar, Max Pfingsthorn, Narender Pulugu, Ravi Mallikarjun Chennaboina, Sai Vivek Jeevangekar, Sebastian Zug.

Figure 1
Figure 1. Figure 1: Symbol grounding approach comparison: the typical approach vs. proposed approach to [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The contribution of ROCS can be separated on two layers while considering the whole [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Proposed property hierarchy and their dependencies (arrow colors chosen to visually [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Light-weight experimental setup consisting of a two cameras and fiducial markers [ [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Light-weight experimental setup consisting of a camera-manipulator combination, for [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The robot-centric conceptual knowledge generation process is illustrated where acquired [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The interface for operating sensors and actuators is provided to our framework by ROS. This [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: RoCS dataset samples: Point cloud and RGB images of a [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Mean variance for physical properties [fl, ri, ro, si, he, ho] illustrated in form of a Box plot (in log-scale to provide insights of respective intra property variances compared to linear-scale shown in [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Category-wise coverage for each physical property [ [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Gradual partitioning of instances to particular concepts given a particular set of properties [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Substitution results w.r.t. human expert selection [PITH_FULL_IMAGE:figures/full_fig_p028_12.png] view at source ↗
read the original abstract

Tool-use applications in robotics require conceptual knowledge about objects for informed decision making and object interactions. State-of-the-art methods employ hand-crafted symbolic knowledge which is defined from a human perspective and grounded into sensory data afterwards. However, due to different sensing and acting capabilities of robots, their conceptual understanding of objects must be generated from a robot's perspective entirely, which asks for robot-centric conceptual knowledge about objects. With this goal in mind, this article motivates that such knowledge should be based on physical and functional properties of objects. Consequently, a selection of ten properties is defined and corresponding extraction methods are proposed. This multi-modal property extraction forms the basis on which our second contribution, a robot-centric knowledge generation is build on. It employs unsupervised clustering methods to transform numerical property data into symbols, and Bivariate Joint Frequency Distributions and Sample Proportion to generate conceptual knowledge about objects using the robot-centric symbols. A preliminary implementation of the proposed framework is employed to acquire a dataset comprising physical and functional property data of 110 houshold objects. This Robot-Centric dataSet (RoCS) is used to evaluate the framework regarding the property extraction methods, the semantics of the considered properties within the dataset and its usefulness in real-world applications such as tool substitution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes a framework for generating robot-centric conceptual knowledge about household objects to support tool-use in robotics. It argues that such knowledge must derive entirely from a robot's perspective (in contrast to hand-crafted human symbols), motivates basing it on physical and functional properties, defines a selection of ten properties with corresponding multi-modal extraction methods, creates the RoCS dataset of 110 objects, applies unsupervised clustering to convert numerical property values into symbols, and uses bivariate joint frequency distributions plus sample proportions to derive conceptual relations among objects. A preliminary implementation is evaluated on the accuracy of property extraction, the semantics of the properties within the dataset, and utility for real-world tool substitution tasks.

Significance. If the central claim holds, the work would supply a reproducible dataset (RoCS) and a data-driven pipeline that avoids human-defined symbols, potentially improving robot decision-making in manipulation scenarios by producing symbols grounded in actual sensor/actuator capabilities. The unsupervised clustering plus frequency-based knowledge generation steps are a concrete methodological contribution that could be tested against baselines in tool-use benchmarks.

major comments (2)
  1. [Abstract] Abstract: the central claim that 'conceptual understanding of objects must be generated from a robot's perspective entirely' is load-bearing yet directly contradicted by the human-authored 'selection of ten properties' whose 'corresponding extraction methods are proposed.' No derivation from robot sensor statistics, actuator limits, or task-performance metrics is supplied to justify why these ten properties (rather than others) become the basis for symbols; unsupervised clustering and bivariate frequency analysis therefore operate only on a pre-filtered human inventory.
  2. [Evaluation] Evaluation section (tool substitution experiments): without quantitative metrics (e.g., success rate deltas versus human-symbol baselines, error bars on clustering stability, or ablation removing individual properties), it is impossible to assess whether the generated symbols actually support improved robot decision-making or merely reproduce the human-chosen distinctions.
minor comments (2)
  1. [Abstract] The abstract states that the framework is 'evaluated regarding the property extraction methods, the semantics of the considered properties within the dataset and its usefulness,' yet supplies no numerical results, tables, or statistical tests; these details should be added to the main text or supplementary material for reproducibility.
  2. [Knowledge Generation] Notation for the bivariate joint frequency distributions and sample proportion calculations is introduced without an explicit equation or pseudocode; adding a short formal definition would clarify how symbols are combined into conceptual knowledge.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond point-by-point to the major comments, indicating revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'conceptual understanding of objects must be generated from a robot's perspective entirely' is load-bearing yet directly contradicted by the human-authored 'selection of ten properties' whose 'corresponding extraction methods are proposed.' No derivation from robot sensor statistics, actuator limits, or task-performance metrics is supplied to justify why these ten properties (rather than others) become the basis for symbols; unsupervised clustering and bivariate frequency analysis therefore operate only on a pre-filtered human inventory.

    Authors: We agree that the selection of the ten properties is human-authored and motivated by considerations of robot sensing and actuation capabilities for household tasks. The symbols themselves, however, are generated via unsupervised clustering on measured numerical values rather than hand-crafted definitions, and the conceptual relations are produced from bivariate frequency distributions of those symbols. We will revise the abstract and introduction to explicitly acknowledge the human role in property selection while emphasizing that the resulting symbols and knowledge are derived from robot-centric data. revision: yes

  2. Referee: [Evaluation] Evaluation section (tool substitution experiments): without quantitative metrics (e.g., success rate deltas versus human-symbol baselines, error bars on clustering stability, or ablation removing individual properties), it is impossible to assess whether the generated symbols actually support improved robot decision-making or merely reproduce the human-chosen distinctions.

    Authors: The manuscript reports accuracy of the multi-modal property extraction, semantic coherence of the properties in the RoCS dataset, and a preliminary demonstration of utility for tool substitution. We acknowledge that these evaluations lack the quantitative comparisons (e.g., success-rate deltas against human-symbol baselines or clustering stability metrics) requested. In revision we will add such quantitative metrics and ablations where the existing data permit. revision: yes

Circularity Check

0 steps flagged

No circularity in derivation chain

full rationale

The paper explicitly states an author-driven selection of ten properties as the starting point, followed by proposed extraction methods, unsupervised clustering to symbols, and bivariate frequency analysis on a new RoCS dataset of 110 objects. No claimed result (e.g., the generated conceptual knowledge) reduces by construction to fitted parameters, prior self-citations, or redefinitions of its own inputs. The framework applies standard unsupervised methods without invoking uniqueness theorems or ansatzes from overlapping prior work. This is a self-contained proposal of a pipeline on fresh data and does not match any enumerated circularity pattern.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the ten properties and clustering step are presented as chosen design decisions without further decomposition.

pith-pipeline@v0.9.0 · 5789 in / 1076 out tokens · 25686 ms · 2026-05-25T15:29:09.018477+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages

  1. [1]

    Introduction

    Christopher Baber. Introduction. In Cognition and Tool Use, chapter 1, pages 1–15. Taylor and Francis, 2003

    work page 2003

  2. [2]

    Cognition and Tool Use

    Christopher Baber. Cognition and Tool Use. Taylor and Francis, 2003

    work page 2003

  3. [3]

    The Design of Tools

    Christopher Baber. The Design of Tools. In Cognition and Tool Use, chapter 6, pages 69–80. Taylor and Francis, 2003. 29

    work page 2003

  4. [4]

    Working With Tools

    Christopher Baber. Working With Tools. In Cognition and Tool Use, chapter 5, pages 51–68. Taylor and Francis, 2003

    work page 2003

  5. [5]

    Corpus Guided Sense Cluster Analysis: a methodology for ontology development (with examples from the spatial domain)

    Brandon Bennett and Claudia Cialone. Corpus Guided Sense Cluster Analysis: a methodology for ontology development (with examples from the spatial domain). In Pawel Garbacz and Oliver Kutz, editors, 8th International Conference on Formal Ontology in Information Systems (FOIS) , volume 267 of Frontiers in Artificial Intelligence and Applications , pages 213–...

    work page 2014

  6. [6]

    Tool use as adaptation

    Dora Biro, Michael Haslam, and Christian Rutz. Tool use as adaptation. Philosophical transactions of the Royal Society of London. Series B, Biological sciences , 368(1630), 2013

    work page 2013

  7. [7]

    Bischoff, U

    R. Bischoff, U. Huggenberger, and E. Prassler. Kuka youbot - a mobile manipulator for research and education. In 2011 IEEE International Conference on Robotics and Automation , pages 1–4, May 2011

    work page 2011

  8. [8]

    Ecology and cognition of tool use in chimpanzees

    Christophe Boesch. Ecology and cognition of tool use in chimpanzees. In Josep Boesch Christophe Sanz, Crickette M. Call, editor, Tool Use in Animals: Cognition and Ecology , chapter 2, pages 21–47. Cambridge University Press, 2013

    work page 2013

  9. [9]

    An introduction to the anchoring problem

    Silvia Coradeschi and Alessandro Saffiotti. An introduction to the anchoring problem. Robotics and Autonomous Systems, 43(2-3):85–96, 2003

    work page 2003

  10. [10]

    Grounding commonsense knowledge in intelligent systems

    Marios Daoutis, Silvia Coradeshi, and Amy Loutfi. Grounding commonsense knowledge in intelligent systems. Journal of Ambient Intelligence and Smart Environments , 1(4):311–321, 2009

    work page 2009

  11. [11]

    What Is a Knowledge Representation ? AI Magazine, 14:17–33, 1993

    Randall Davis, Howard Shrobe, and Peter Szolovits. What Is a Knowledge Representation ? AI Magazine, 14:17–33, 1993

    work page 1993

  12. [12]

    Nathan J. Emery. Insight, imagination and invention: Tool understanding in a non-tool-using corvid. In Josep Boesch Christophe Sanz, Crickette M. Call, editor, Tool Use in Animals: Cognition and Ecology, chapter 4, pages 67–88. Cambridge University Press, 2013

    work page 2013

  13. [13]

    WordNet: An Electronic Lexical Database

    Christiane Fellbaum, editor. WordNet: An Electronic Lexical Database . The MIT Press, Cambridge, MA ; London, 1998

    work page 1998

  14. [14]

    Garrido-Jurado, R

    S. Garrido-Jurado, R. Munoz-Salinas, F.J. Madrid-Cuevas, and M.J. Marin-Jimenez. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6):2280–2292, 2014

    work page 2014

  15. [15]

    The Theory of Affordances

    James J Gibson. The Theory of Affordances. In The Ecological Approach to Visual Perception , chapter 8, pages 127–143. Psychology Press, Taylor & Francis Group, 1986

    work page 1986

  16. [16]

    Common Sense Data Acquisition for Indoor Mobile Robots

    Rakesh Gupta and Mykel J Kochenderfer. Common Sense Data Acquisition for Indoor Mobile Robots. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence , pages 605–610, San Jose, California, USA, 2004

    work page 2004

  17. [17]

    The Symbol Grounding Problem

    Stevan Harnad. The Symbol Grounding Problem. Physica D, 42:335–346, 1990. 30

    work page 1990

  18. [18]

    Cognitive, physical, sensory, and functional affordances in interaction design

    Rex Hartson. Cognitive, physical, sensory, and functional affordances in interaction design. Behaviour & Information Technology, 22(5):315–338, 2003

    work page 2003

  19. [19]

    Functional understanding facilitates learning about tools in human children

    Mikolaj Hernik and Gergely Csibra. Functional understanding facilitates learning about tools in human children. Current Opinion in Neurobiology , 19(1):34–38, 2009

    work page 2009

  20. [20]

    Springer, 2017

    Anis Koubˆ aa.Robot operating system (ros): The complete reference , volume 2. Springer, 2017

    work page 2017

  21. [21]

    An Image-Schematic Account of Spatial Categories

    Werner Kuhn. An Image-Schematic Account of Spatial Categories. Spatial Information Theory, pages 152–168, 2007

    work page 2007

  22. [22]

    ORO, a knowledge management platform for cognitive architectures in robotics

    Sverin Lemaignan, Raquel Ros, Lorenz M¨ osenlechner, Rachid Alami, and Michael Beetz. ORO, a knowledge management platform for cognitive architectures in robotics. IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems, IROS 2010 - Conference Proceedings , (April):3548–3553, 2010

    work page 2010

  23. [23]

    Douglas B. Lenat. Cyc: A large-scale investment in knowledge infrastructure. Commun. ACM , 38(11):33–38, 11 1995

    work page 1995

  24. [24]

    Ontology-based unified robot knowledge for service robots in indoor environments

    Gi Hyun Lim, Il Hong Suh, and Hyowon Suh. Ontology-based unified robot knowledge for service robots in indoor environments. IEEE Transactions on Systems, Man, and Cybernetics Part A:Systems and Humans, 41(3):492–509, 2011

    work page 2011

  25. [25]

    Liu and P

    H. Liu and P. Singh. ConceptNet A Practical Commonsense Reasoning Tool-Kit. BT Technology Journal, 22(4):211–226, 2004

    work page 2004

  26. [26]

    Mandler and Cristobal Pagen Canovas

    Jean M. Mandler and Cristobal Pagen Canovas. On defining image schemas. Language and Cognition, 6(04):510–532, 2014

    work page 2014

  27. [27]

    Pineda, Arturo Rodr´ ıguez, Gibran Fuentes, Caleb Rasc´ on, and Ivan Meza

    Luis A. Pineda, Arturo Rodr´ ıguez, Gibran Fuentes, Caleb Rasc´ on, and Ivan Meza. A light non- monotonic knowledge-base for service robots. Intelligent Service Robotics, 10(3):159–171, 2017

    work page 2017

  28. [28]

    Ruiz and Laurie R

    April M. Ruiz and Laurie R. Santos. Understanding differences in the way human and non-human primates represent tools: The role of teleological-intentional information. In Crickette M. Sanz, Josep Call, and Christophe Boesch, editors, Tool Use in Animals: Cognition and Ecology , chapter 6, pages 119–133. Cambridge University Press, 2013

    work page 2013

  29. [29]

    Sanz, Josep Call, and Christophe Boesch, editors

    Crickette M. Sanz, Josep Call, and Christophe Boesch, editors. Tool Use in Animals: Cognition and Ecology. Cambridge University Press, 2013

    work page 2013

  30. [30]

    Misra, and Hema S

    Ashutosh Saxena, Ashesh Jain, Ozan Sener, Aditya Jami, Dipendra K. Misra, and Hema S. Koppula. RoboBrain: Large-Scale Knowledge Engine for Robots. arXiv, pages 1 – 11, 2014

    work page 2014

  31. [31]

    Why We Need Many Knowledge Representation Formalisms

    Aaron Sloman. Why We Need Many Knowledge Representation Formalisms. Proceedings BCS Expert Systems Conference, pages 163–183, 1984. 31

    work page 1984

  32. [32]

    Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence

    Il Hong Suh, Gi Hyun Lim, Wonil Hwang, Hyowon Suh, Jung Hwa Choi, and Young Tack Park. Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence. IEEE International Conference on Intelligent Robots and Systems , (October):429–436, 2007

    work page 2007

  33. [33]

    Susi and T

    T. Susi and T. Ziemke. On the subject of objects: Four views on object perception and tool use. tripleC-Cognition, Communication, Co-operation, 3(2):619, 2005

    work page 2005

  34. [34]

    A comprehensive characterization of the asus xtion pro depth sensor

    Daniel Maximilian Swoboda. A comprehensive characterization of the asus xtion pro depth sensor. 2014

    work page 2014

  35. [35]

    KNOWROB- Knowledge Processing for Autonomous Personal Robots

    Moritz Tenorth and Michael Beetz. KNOWROB- Knowledge Processing for Autonomous Personal Robots. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 4261–4266, 2009

    work page 2009

  36. [36]

    What Stands-in for a Missing Tool?: A Pro- totypical Grounded Knowledge-based Approach to Tool Substitution

    Madhura Thosar, Christian Mueller, and Sebastian Zug. What Stands-in for a Missing Tool?: A Pro- totypical Grounded Knowledge-based Approach to Tool Substitution. In 11th International Workshop on Cognitive Robotics in 16th International Conference on Principles of Knowledge Representation and Reasoning, Tempe, Arizona, 2018

    work page 2018

  37. [37]

    A Review of Knowledge Bases for Service Robots in Household Environments

    Madhura Thosar, Sebastian Zug, Alpha Mary Skaria, and Akshay Jain. A Review of Knowledge Bases for Service Robots in Household Environments. In 6th International Workshop on Artificial Intelligence and Cognition , 2018

    work page 2018

  38. [38]

    The cognitive bases of human tool use

    Krist Vaesen. The cognitive bases of human tool use. Behavioral and Brain Sciences , 35(04):203–218, 2012

    work page 2012

  39. [39]

    Vauclair and J

    J. Vauclair and J. A. Anderson. Object Manipulation, Tool Use, and The Social Context in Human and Non-Human Primates. Techniques and Culture, 23-24:121136, 1994

    work page 1994

  40. [40]

    Reasoning About Object Affordance in a Knowledge Based Representation

    Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning About Object Affordance in a Knowledge Based Representation. European Conference on Computer Vision , (3):408–424, 2014. 32

    work page 2014