Pith. sign in

REVIEW 3 major objections 5 minor 68 references

Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A robot's probing actions — bend, fold, push, recover — let a large language model classify physical properties of previously unseen deformable objects at 78.57% accuracy, and those labels drive bin-packing plans that succeed 96.84% of…

desk verdict A genuinely new interaction-based LLM pipeline for deformability, but the headline accuracy hides a systematic 0% on rigid objects and the planning metric may not be end-to-end. read the letter →

arxiv 2506.03760 v1 pith:PR65N2QZ submitted 2025-06-04 cs.RO cs.AI

Changmin Park , Beomjoon Lee , Haechan Jung , Haejin Jung , Changjoo Nam This is my paper
classification cs.ROcs.AI
keywords largelanguagemodelstaskplanningdeformableobjectmanipulationphysicalpropertyreasoningrobotprobingactionsbinpackingdetectionfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a robot can discover the physical properties of objects it has never seen by poking them and letting a large language model watch. The proposed loop detects each object, probes it with a bend, fold, or push action followed by a recovery attempt, and reads three photographs of the interaction to classify the object as rigid, bendable, foldable, compressible, or plastic; those labels then become planning constraints. The reported numbers are 78.57% property-probing accuracy on 14 objects and 96.84% bin-packing success after five replanning rounds, with image-only baselines at 34.29% and 22.86%. A sympathetic reader would care because this replaces the closed-world assumption of classical task planning: properties that once required training data or analytic models are obtained on the fly from interaction.

What carries the argument

The load-bearing mechanism is the probing protocol: a dimension-matched deforming action (bend for 1D objects, fold for 2D objects, push for 3D objects) always followed by a recovery attempt, with three labeled still images — before, during, and after recovery — that the LLM reads to judge whether deformation happened and whether the object returned to its original shape. The five target properties are defined in plain language, the protocol can be organized as a decision tree, and the resulting labels become predicates in a Python-coded, PDDL-style domain whose action schemas encode the packing constraints. A plan validator executes the generated code, detects syntax errors and constraint violations, and feeds the output back to regenerate the action schemas and replan until a valid plan is found.

What would settle it

Run the probing loop on the two rigid objects that currently score 0% (Objects 2 and 9) while adding a no-change cue such as a pixel-difference mask between the before and after images, and count how often the LLM returns 'rigid'; if the score stays near zero, the LLM's visual judgment of 'no deformation' is the bottleneck, while a jump above 80% would show the three-still format itself was the problem.

Watch

Extended reading notes

Core claim

The paper's central claim is that physical properties of previously unseen deformable objects need not be learned from data or assumed from a known category: a robot can probe each object and a large language model can classify it from three still images of the interaction into one of five properties — rigid, bendable, foldable, compressible, or plastic — and these labels compile directly into the constraints of a long-horizon plan. On 14 objects across 38 bin-packing instances, the interaction-based method reaches 78.57% property-probing accuracy, against 34.29% and 22.86% for image-only baselines, and the resulting plans reach 96.84% success after five replanning rounds. The intended takeaway is that robotic interactability plus LLM commonsense reasoning can stand in for the closed-world assumption of classical task planning, with no fine-tuning.

Load-bearing premise

The load-bearing premise is that a language model can correctly tell, from three still photographs, whether an object deformed under a robot's action and whether it returned to its original shape afterwards; the paper's own data show this fails for rigid objects, since both rigid test objects score 0% with the robot-interaction methods.

Editorial extensions

If this is right

  • A robot can classify a never-seen object into one of five deformation properties purely by executing scripted probes, with no per-object training and no fine-tuning — the pipeline reports 78.57% overall probing accuracy on 14 objects.
  • The discovered property labels become planning predicates automatically, so pointing the pipeline at a new task domain amounts to swapping the constraint set rather than reprogramming the planner.
  • Plan validation with replanning lifts bin-packing success from 76.05% on the first attempt to 96.84% after five rounds, indicating that most LLM planning errors are detectable and correctable rather than fatal.
  • Physical interaction, not visual appearance, is what carries the classification: image-only baselines reach 34.29% and 22.86%, while interaction-based methods reach 78.57% and 73.57%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 0% rigidity scores on the two rigid test objects hint that the LLM is biased to report deformation whenever the robot acts; a cheap test would be to feed the model a no-change cue such as a pixel-difference mask between the before and after images and see whether 'rigid' is ever returned.
  • The same probe-and-read loop should transfer to other physical properties — thermal response, magnetic attraction, water resistance — as long as the state change is as visually legible as bending or folding; the paper's five-property set is task-driven, not a ceiling of the method.
  • Most of the replanning gain comes after the first round, which suggests the dominant errors are systematic code bugs (such as a place action that forgets to update the in-bin flag) that a single simulator check could catch; five replanning rounds may be an artifact of not compiling the generated Python.
  • Free-form LLM judgment (Robot, 78.57%) edged out the decision-tree scaffold (Robot+Tree, 73.57%), so imposing hard symbolic structure on the perception step can cost accuracy; using the tree only to choose actions while leaving the final label free-form might combine both strengths.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an LLM-based pipeline that probes the physical properties (rigid, bendable, foldable, compressible, plastic) of previously unseen objects through robot interactions, then uses the discovered properties to generate task plans for a bin-packing domain. The pipeline consists of a property reasoner (VLM-based detection/naming plus LLM reasoning over before/during/after probing images), a domain generator, an instance descriptor, an LLM task planner, and an LLM plan validator with iterative replanning. Experiments on 14 objects and 38 randomly generated bin-packing instances report 78.57% property-probing accuracy for the interaction-based Robot method (vs 34.29% for image-only), and 96.84% planning success after five replanning attempts.

Significance. The paper's strength is its clear experimental setup: a real dual-arm robot, a nontrivial object set, a detailed prompt protocol, and a direct comparison between interaction-based and image-only property reasoning. The finding that robot interaction improves property classification for most non-rigid objects is useful evidence for the value of physical probing in LLM-based manipulation. However, the central claim is substantially weakened by a systematic failure on rigid objects: both interaction methods achieve 0% accuracy on the two rigid objects in Table 1c, while image-only methods do better on one of them. Because rigidity is the baseline property in the five-property taxonomy, this is not a minor edge case. In addition, the planning-success metric is computed in a simulation whose action preconditions are themselves generated by the LLM, so it does not independently validate the discovered properties. If these issues are addressed with per-property analysis and a scoped claim, the work would be a solid contribution to LLM-based task and motion planning.

major comments (3)
  1. [§5.2, Table 1c] The interaction-based methods (Robot and Robot+Tree) achieve 0% accuracy on both rigid objects (Objects 2 and 9), whereas the non-interaction Image method achieves 60% and Image+Name achieves 100% for Object 9. Since rigidity is one of the five properties in the taxonomy and is the baseline distinction for deformability, this is a systematic failure in the core mechanism, not a random error. The aggregate 78.57% is driven by the 11 non-rigid objects and hides the fact that the robot interaction actively suppresses the correct answer for rigid 1D/3D objects. The discussion in §5.4 only addresses thin 2D-object failures (Object 6 and Object C) and does not analyze the rigid-object failures. Please report per-property accuracy, provide a failure analysis for Objects 2 and 9, and either improve the probing procedure or temper the claim that robot interactions reveal physical properties.
  2. [§5.3, Fig. 8a] The planning success metric is computed by executing the LLM-generated Python simulation, in which the action preconditions and effects are also generated by the LLM. If a rigid object is mislabeled as compressible, the planner can still 'push' it in the simulation and the plan counts as successful, so the reported 96.84% planning success rate does not provide independent evidence that the probed properties are correct. It only demonstrates self-consistency of the generated code with the (possibly wrong) property labels. Please report planning success conditioned on whether property probing was correct versus incorrect, and clarify that the planning metric does not validate physical property discovery.
  3. [§4.1.2 and Table 1c] The claim that 'robot interactions significantly help the LLM reason about the object properties' is not supported for the rigid category: for Objects 2 and 9 the interaction methods are worse than the image-only methods. The paper should report a confusion matrix across all five properties and test whether the interaction benefit is confined to non-rigid objects. Without this, the headline comparison (78.57% vs 34.29%) overstates the generality of the conclusion and leaves the reader unable to judge when the proposed probing protocol is reliable.
minor comments (5)
  1. [§5.2] There is a typo: 'PbD mathod' should be 'PbD method'.
  2. [Table 1c] The abbreviation 'C: compressive' is inconsistent with the term 'compressible' used throughout the text; please unify the terminology.
  3. [Fig. 3a] The decision tree does not show which deforming action is applied for each dimension (bend/fold/push) at the root; making this explicit would help the reader follow the probing protocol.
  4. [Appendix A.2] The Robot+Tree prompt example only covers a 3D object and asks about compressible versus rigid; please provide the corresponding prompts for 1D and 2D objects so that the tree's behavior is fully specified.
  5. [§5.3] The text says the validation process repeats 'until T without any error is obtained,' but the experiments cap replanning at five attempts; please reconcile this description with the actual experimental protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; property labels are checked against external ground truth and planning success against executable Python state transitions.

full rationale

The paper's derivation chain is not circular. Property probing evaluates LLM judgments against externally defined ground-truth labels (Table 1c), with no parameter fitted to the reported accuracy. The 'Robot' method feeds three images (before/during/after probing) to GPT-4o; the output label is compared to the ground truth, not to the prompt content. Similarly, 'Robot+Tree' applies a fixed decision tree derived from dictionary definitions, not from the experimental outcomes. The planning stage generates Python code whose preconditions and effects encode the constraint set C; the plan validator executes this code and checks for 'Cannot...' tags and goal-state satisfaction, which are independent checks rather than the LLM ratifying its own plan. The strongest apparent self-reference—the LLM serving as both planner and validator—does not reduce the validation to the planner's own output because execution failures (e.g., 'Cannot push') are produced by the Python state machine. Replanning uses the execution trace Exe as feedback, an external signal. The 0% rigidity accuracy for Objects 2 and 9 (Table 1c) is a performance/correctness concern, not a circularity concern; no quantity is defined in terms of the result it predicts. No self-citation is load-bearing, and no prediction is equivalent to its inputs by construction. Therefore no circular steps are identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The method rests on several domain assumptions: sufficiency of the five chosen properties, sufficiency of the probing actions, visual judgment of deformation, and the correctness of the manually defined packing constraints. The most fragile is the visual judgment assumption, which the paper's own experiments show fails for rigid objects. There are no invented physical entities or fitted numeric parameters.

assumptions (5)
  • ad hoc to paper The five properties (rigid, bendable, foldable, compressible, plastic) are sufficient to describe deformability for bin-packing.
    Stated in Sec 4.1.2; the property set is chosen by the authors for this task, and no evidence shows it covers all relevant deformability modes.
  • domain assumption Probing actions bend, fold, push, and recover are sufficient to reveal the five properties.
    Introduced in Sec 4.1.2; if an object's deforming mode differs (e.g., twisting, stretching), the probing would miss it.
  • domain assumption The LLM can correctly infer deformation and recovery from three images.
    This is the crux of the algorithm, validated only empirically; Table 1c shows it fails for rigid objects (0% for objects 2 and 9).
  • domain assumption Object dimension (1D/2D/3D) can be determined visually and determines the appropriate probing action.
    Used in the decision tree in Sec 4.1.2; object naming success is high but not perfect (e.g., object 5 shape 71.88%).
  • ad hoc to paper The manually written bin-packing constraints C are correct and complete for safe packing.
    The five rules in Sec 4.2.1 are provided by the authors; the planner is only as good as these constraints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions." pith.science (2026). https://pith.science/paper/PR65N2QZ

@misc{pith2026250603760,
  author       = {Pith},
  title        = {Pith review of: Understanding Physical Properties of Unseen Deformable Objects by Leveraging Large Language Models and Robot Actions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PR65N2QZ}},
  note         = {Machine review of arXiv:2506.03760}
}
read the original abstract

In this paper, we consider the problem of understanding the physical properties of unseen objects through interactions between the objects and a robot. Handling unseen objects with special properties such as deformability is challenging for traditional task and motion planning approaches as they are often with the closed world assumption. Recent results in Large Language Models (LLMs) based task planning have shown the ability to reason about unseen objects. However, most studies assume rigid objects, overlooking their physical properties. We propose an LLM-based method for probing the physical properties of unseen deformable objects for the purpose of task planning. For a given set of object properties (e.g., foldability, bendability), our method uses robot actions to determine the properties by interacting with the objects. Based on the properties examined by the LLM and robot actions, the LLM generates a task plan for a specific domain such as object packing. In the experiment, we show that the proposed method can identify properties of deformable objects, which are further used for a bin-packing task where the properties take crucial roles to succeed.

Figures

Figures reproduced from arXiv: 2506.03760 by the authors.

Figure 1
Figure 1. An overview of the proposed method for understanding physical properties of unseen objects by using the commonsense knowledge of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall procedure of our method for the bin-packing domain involves autonomously investigating object properties to expand domain [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The decision tree approach for determining object properties. (a) By applying a series of actions, one of the properties at the leaf nodes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The set of 14 objects used in the experiments. The dimensions and physical properties of them are listed in the second and third columns [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: An example of a set of images processed by a VLM and a connection graph. Since the objects are unknown yet, their labels are determined [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The success rates of the property reasoner and its components. (a) The high success rate of object naming shows that the LLM can [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Property probing for similarly shaped objects (e.g., [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The task planning result. (a) The success rates of task planning where replanning has shown the e [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 55 canonical work pages

  1. [1]

    R. E. Fikes, N. J. Nilsson, STRIPS: A new approach to the application of theorem proving to problem solving, Artificial intelligence 2 (3-4) (1971) 189–208

    work page 1971

  2. [2]

    Aeronautiques, A

    C. Aeronautiques, A. Howe, C. Knoblock, I. D. McDermott, A. Ram, M. Veloso, D. Weld, D. W. Sri, A. Barrett, D. Christianson, et al., PDDL— the planning domain definition language, Technical Report, Tech. Rep

    work page

  3. [3]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, A. Zeng, Code as policies: Language model programs for embodied control, in: Proc. of International Conference on Robotics and Automation (ICRA), 2023, pp. 9493–9500

    work page 2023

  4. [4]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al., Do as I can, not as I say: Grounding language in robotic affordances, arXiv preprint arXiv:2204.01691

    arXiv

  5. [5]

    Huang, F

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotar, et al., Inner monologue: Embodied reasoning through planning with language models, arXiv preprint arXiv:2207.05608

    arXiv

  6. [6]

    X. Ma, Y . Bhalgat, B. Smart, S. Chen, X. Li, J. Ding, J. Gu, D. Z. Chen, S. Peng, J.-W. Bian, et al., When LLMs step into the 3D world: A survey and meta-analysis of 3d tasks via multi-modal large language models, arXiv preprint arXiv:2405.10255

    arXiv

  7. [7]

    R. Fu, J. Liu, X. Chen, Y . Nie, W. Xiong, Scene-LLM: Extending language model for 3d visual understanding and reasoning, arXiv preprint arXiv:2403.11401

    arXiv

  8. [8]

    S. Wang, M. Han, Z. Jiao, Z. Zhang, Y . N. Wu, S.-C. Zhu, H. Liu, LLM3: Large language model-based task and motion planning with motion failure reasoning, arXiv preprint arXiv:2403.11552

    arXiv

Show all 68 references
  1. [9]

    Shirai, C

    K. Shirai, C. C. Beltran-Hernandez, M. Hamaya, A. Hashimoto, S. Tanaka, K. Kawaharazuka, K. Tanaka, Y . Ushiku, S. Mori, Vision-language interpreter for robot task planning, in: Proc. of International Conference on Robotics and Automation (ICRA), 2024, pp. 2051–2058

    2024

  2. [10]

    Z. Zhao, W. S. Lee, D. Hsu, Large language models as commonsense knowledge for large-scale task planning, in: Proc. ofAdvances in Neural Information Processing Systems (NeurIPS), V ol. 36, 2024

    2024

  3. [11]

    Z. Wu, Z. Wang, X. Xu, J. Lu, H. Yan, Embodied task planning with large language models, arXiv preprint arXiv:2307.01848

    arXiv

  4. [12]

    R. Wu, C. Ning, H. Dong, Learning foresightful dense visual a ffordance for deformable object manipulation, in: Proc. of International Conference on Computer Vision (ICCV), 2023, pp. 10947–10956

    2023

  5. [13]

    Z. Wang, A. H. Qureshi, DeRi-Bot: Learning to collaboratively manipulate rigid objects via deformable objects, IEEE Robotics and Automa- tion Letters

  6. [14]

    Y . Deng, K. Mo, C. Xia, X. Wang, Learning language-conditioned deformable object manipulation with graph dynamics, in: Proc. of International Conference on Robotics and Automation (ICRA), 2024, pp. 7508–7514

    2024

  7. [15]

    Guler, K

    P. Guler, K. Pauwels, A. Pieropan, H. Kjellstr ¨om, D. Kragic, Estimating the deformability of elastic materials using optical flow and position- based dynamics, in: Proc. of International Conference on Humanoid Robots (Humanoids), 2015, pp. 965–971. 11

    2015

  8. [16]

    Petit, F

    A. Petit, F. Ficuciello, G. A. Fontanelli, L. Villani, B. Siciliano, Using physical modeling and RGB-D registration for contact force sensing on deformable objects, in: Proc. of International Conference on Informatics in Control, Automation and Robotics (ICINCO), V ol. 2, 2017...

    2017

  9. [17]

    Haouchine, W

    N. Haouchine, W. Kuang, S. Cotin, M. Yip, Vision-based force feedback estimation for robot-assisted surgery using instrument-constrained biomechanical three-dimensional maps, IEEE Robotics and Automation Letters 3 (3) (2018) 2160–2165

    2018

  10. [18]

    P. Zhou, P. Zheng, J. Qi, C. Li, H.-Y . Lee, A. Duan, L. Lu, Z. Li, L. Hu, D. Navarro-Alarcon, Reactive human–robot collaborative manipulation of deformable linear objects using a new topological latent control model, Robotics and Computer-Integrated Manufacturing 88 (2024) 102727

    2024

  11. [19]

    Huang, X

    J. Huang, X. Chu, X. Ma, K. W. S. Au, Deformable object manipulation with constraints using path set planning and tracking, IEEE Trans- actions on Robotics

  12. [20]

    Matas, S

    J. Matas, S. James, A. J. Davison, Sim-to-real reinforcement learning for deformable object manipulation, in: Proc. of Conference on Robot Learning (CoRL), 2018, pp. 734–743

    2018

  13. [21]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al., Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.05499

    arXiv

  14. [22]

    A. Wang, T. Kurutach, K. Liu, P. Abbeel, A. Tamar, Learning robotic manipulator visual planning and acting, arXiv preprint arXiv:1905.04411

    1905 arXiv

  15. [23]

    M. Yan, Y . Zhu, N. Jin, J. Bohg, Self-supervised learning of state estimation for manipulating deformable linear objects, IEEE Robotics and Automation Letters 5 (2) (2020) 2372–2379

    2020

  16. [24]

    Lippi, P

    M. Lippi, P. Poklukar, M. C. Welle, A. Varava, H. Yin, A. Marino, D. Kragic, Latent space roadmap for visual action planning of deformable and rigid object manipulation, in: Proc. of International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 5619–5626

    2020

  17. [25]

    T. Fu, C. Li, J. Liu, F. Li, C. Wang, R. Song, Flingflow: Llm-driven dynamic strategies for e fficient cloth flattening, IEEE Robotics and Automation Letters

  18. [26]

    Shridhar, L

    M. Shridhar, L. Manuelli, D. Fox, Cliport: What and where pathways for robotic manipulation, in: Proc. of Conference on Robot Learning (CoRL), 2022, pp. 894–906

    2022

  19. [27]

    Bartsch, A

    A. Bartsch, A. B. Farimani, LLM-craft: Robotic crafting of elasto-plastic objects with large language models, arXiv preprint arXiv:2406.08648

    arXiv

  20. [28]

    Hanheide, M

    M. Hanheide, M. G ¨obelbecker, G. S. Horn, A. Pronobis, K. Sj ¨o¨o, A. Aydemir, P. Jensfelt, C. Gretton, R. Dearden, M. Janicek, et al., Robot task planning and explanation in open and uncertain worlds, Artificial Intelligence 247 (2017) 119–150

    2017

  21. [29]

    C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, Y . Su, LLM-planner: Few-shot grounded planning for embodied agents with large language models, in: Proc. of International Conference on Computer Vision (ICCV), 2023, pp. 2998–3009

    2023

  22. [30]

    M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, B. Ichter, A. Irpan, N. Joshi, R. Julian, et al., Autort: Embodied foundation models for large scale orchestration of robotic agents, arXiv preprint arXiv:2401.12963

  23. [31]

    Shridhar, J

    M. Shridhar, J. Thomason, D. Gordon, Y . Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, D. Fox, Alfred: A benchmark for interpreting grounded instructions for everyday tasks, in: Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10740–10749

    2020

  24. [32]

    Y . Liu, L. Palmieri, S. Koch, I. Georgievski, M. Aiello, Delta: Decomposed e fficient long-term robot task planning using large language models, arXiv preprint arXiv:2404.03275

    arXiv

  25. [33]

    J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, D. Sadigh, Physically grounded vision-language models for robotic manipulation, in: Proc. of International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 12462–12469

    2024

  26. [34]

    W. Xie, J. Lavering, N. Correll, DeliGrasp: Inferring object properties with LLMs for adaptive grasp policies, Arxiv preprint arXiv: 2403.07832

    arXiv

  27. [35]

    X. Zhao, M. Li, C. Weber, M. B. Hafez, S. Wermter, Chat with the environment: Interactive multimodal perception using large language models, in: Proc. of International Conference on Intelligent Robots and Systems (IROS), IEEE, 2023, pp. 3590–3596

    2023

  28. [36]

    W. Lai, T. Zhang, T. L. Lam, Y . Gao, Vision-language model-based physical reasoning for robot liquid perception, in: Proc. of International Conference on Intelligent Robots and Systems (IROS), IEEE, 2024, pp. 9652–9659

    2024

  29. [37]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., GPT-4 technical report, arXiv preprint arXiv:2303.08774

    arXiv

  30. [38]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, in: Proc. of Advances in Neural Information Processing Systems (NeurIPS), V ol. 35, 2022, pp. 24824–24837

    2022

  31. [39]

    1D”, “line

    Cambridge Dictionary, Definitions for search terms: “1D”, “line”, “compressible”, and others, Accessed: Oct. 24, 2024 (Oct. 2024). URL https://dictionary.cambridge.org/

    2024

  32. [40]

    Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929

    A. Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929

    2010 arXiv

  33. [41]

    [Definitions of dimensions and shapes]

    T. Z. Zhao, V . Kumar, S. Levine, C. Finn, Learning fine-grained bimanual manipulation with low-cost hardware, arXiv preprint arXiv:2304.13705. 12 Appendix A. Prompts Appendix A.1. Prompts for Object Detection and Naming (Sec. 4.1.1) Prompts for the object detection and naming...

    arXiv

  34. [42]

    Result - Property: ‘’ —template end— Answer: —template start—

  35. [43]

    When the robot pushes the object, it deforms and does not return to its original shape after the force is removed

    Reasoning - The object is a 3D cuboid. When the robot pushes the object, it deforms and does not return to its original shape after the force is removed. This indicates that the object can be shaped and retains the new form, which aligns with the property of being plastic

  36. [44]

    Result - Property: ‘is plastic’ —template end— Prompt for the property probing: Robot+Tree Message 1: Describe the scenes 1. 2. 3. Answer 1 : 1. The object is a small, red, rectangular box placed on a flat surface. It appears to be made of cardboard or a similar material

  37. [45]

    The box is slightly deformed, indicating pressure from the gripper

    The object is the same red box, but it is now being grasped by a black robotic gripper. The box is slightly deformed, indicating pressure from the gripper. Message 2: We will now probe the object’s properties. This table defines the physical properties of the object we are inv...

  38. [46]

    Description of 1st image: -

  39. [47]

    Description of 2st image: -

  40. [48]

    Result - Property: ‘’ —template end— Answer 3-1: —template start—

  41. [49]

    Description of 1st image: - The red 3D cuboid is intact and maintains its original shape, placed on a flat surface

  42. [50]

    Description of 2nd image: - The red 3D cuboid is being pushed by a robotic gripper, and its shape is visibly deformed

  43. [51]

    Reasoning - The deformation of the cuboid when pushed indicates that it can be compressed, showing a change in shape under pressure

  44. [52]

    Result - Property: ‘is compressible’ —template end— Answer 3-2: —template start—

  45. [53]

    Description of 1st image: - The red 3D cuboid is intact and centered within a marked area on a flat surface

  46. [54]

    Description of 2nd image: - A robotic arm is pressing the red cuboid, causing it to deform visibly

  47. [55]

    This indicates that the object does not recover from deformation

    Reasoning - After the robot releases the object, the cuboid remains deformed and does not return to its original shape. This indicates that the object does not recover from deformation. 5

  48. [56]

    ‘python # only write a code here without example instantiation

    Result - Property: ‘is plastic’ —template end— Appendix A.3. Prompts for Predicate Generator (Sec. 4.2.1) Prompt for the predicate generator Message: Our goal is to define the types of objects and their predicates within the dataclass Object. Here, we have the types, names, an...

  49. [57]

    **Weight**: Determines if the object is heavy or light, impacting how it can be packed and whether it requires special handling

  50. [58]

    **Material**: Identifies if the object is plastic, metal, or another material, affecting its fragility and packing requirements

  51. [59]

    **Rigidity**: Indicates if the object is rigid or flexible, influencing how it can be positioned and stacked

  52. [60]

    **Size**: Defines the dimensions of the object, crucial for determining fit within a box

  53. [61]

    pick obj.name

    **Shape**: Affects how objects can be arranged and packed efficiently. These properties help in planning and executing effective bin-packing strategies. Message 2: This is Object class you have to use. from dataclasses import dataclass @dataclass class Object: # Basic dataclas...

  54. [62]

    Object Class (Start with [@dataclass])

  55. [63]

    Action Class (Start with [class Action:])

  56. [64]

    Object initial State (Start with [# Object Initial State])

  57. [65]

    Planning State (Start with [if name == ” main ”:]) And this is a planning result of the code. bend black 1D line pick black 1D line place black 1D line pick yellow 3D cuboid place yellow 3D cuboid Cannot push yellow 3D cuboid pick white 3D cylinder place white 3D cylinder Cann...

  58. [66]

    The ‘place’ action for the blue 2D rectangle (plastic object) is executed without any issue, which aligns with the constraints

  59. [67]

    pick { obj . name }

    The sequence of actions seems to follow the given rules, but the ‘push’ action fails for compressible objects. Second, check the error part follows the notes !!Note1. Do not assume the physical properties of the object. !!Note2. If a predicate required by the constraints is no...

  60. [68]

    packing all object in the box

    start 1. fold yellow 2. place yellow 3. place red 4. push red 5. place blue0. start 2. place white 3. place green 4. place brown 5. push brown1. pick white Figure D.11: The real-world execution of task planning which leverages probed properties. Given the instruction “packing ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.