pith. sign in

arxiv: 2605.16043 · v1 · pith:3DMVGALEnew · submitted 2026-05-15 · 💻 cs.RO · cs.AI

Learning Sim-Grounded Policies for Bimanual Rope Manipulation from Human Teleoperation Data

Gina Wigginghaus , Tim Missal , Berk Guler , Simon Manschitz , Jan Peters This is my paper

Pith reviewed 2026-05-20 18:19 UTC · model grok-4.3

classification 💻 cs.RO cs.AI
keywords bimanual manipulationdeformable linear objectsimitation learningrope untanglingstate-based policiesposition-based dynamicsteleoperation datasimulation grounding
0
0 comments X

The pith

A state-based policy using simulated 3D rope particles outperforms a vision-based policy by 30.8 percent lower L1 error on unseen configurations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether visual policies for untangling knots with ropes fail to generalize because of the observation space rather than policy design or data amount. Two Action Chunking with Transformers policies are trained on identical bimanual teleoperation datasets: one takes inputs from wrist-mounted RGB cameras while the other uses a 3D particle representation of the rope derived from multi-view fusion and advanced forward in a particle simulation. When evaluated open-loop on a new rope configuration, the state-based policy shows a 30.8 percent reduction in L1 error for the initial grasp-and-pull action. This result highlights an observability advantage of simulation-grounded states over pixels for handling self-occlusion and high-dimensional configurations in deformable object tasks. It suggests that grounding policies in physics-consistent states can improve data efficiency for robot learning from limited human demonstrations.

Core claim

The paper establishes that conditioning an imitation learning policy on the 3D particle state of a deformable linear object, obtained through initial multi-view observation and forward simulation using eXtended Position-Based Dynamics, yields superior performance compared to a policy conditioned on egocentric RGB images. Specifically, for the bimanual knot-untangling task, this state-based approach achieves a 30.8% reduction in L1 error when predicting the initial grasp-and-pull action on an unseen rope configuration, thereby quantifying the observability gap between visual and physics-consistent state representations.

What carries the argument

The particle-based eXtended Position-Based Dynamics simulation that evolves the rope's 3D state forward from an initial multi-view fused observation, providing a physics-consistent representation that bridges the gap between limited observations and full configuration space.

If this is right

  • The state-based policy generalizes better to new rope configurations from the same limited teleoperation data.
  • Observation space choice is critical for scalability of imitation learning in deformable linear object manipulation.
  • The 30.8 percent L1 error reduction quantifies the observability gap between pixels and physics-consistent state.
  • This points toward more data-efficient robot learning for bimanual DLO tasks from small human demonstration sets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Continuous simulation updates during execution could enable closed-loop control that maintains accuracy beyond the initial action.
  • The approach may extend to other deformable tasks such as cable routing where visual self-occlusion is prevalent.
  • Hybrid policies that combine simulated state with visual inputs could improve robustness in changing lighting or partial views.
  • The quantified observability gap could inform sensor design choices for future manipulation systems handling flexible objects.

Load-bearing premise

The 3D particle state extracted from an initial multi-view observation and evolved forward in the particle-based simulation remains an accurate representation of the real rope without significant modeling errors or drift.

What would settle it

A physical robot experiment where the state-based policy is run closed-loop on varied rope configurations and shows no increase in knot-untangling success rate compared to the visual policy despite the lower open-loop L1 error.

Figures

Figures reproduced from arXiv: 2605.16043 by Berk Guler, Gina Wigginghaus, Jan Peters, Simon Manschitz, Tim Missal.

Figure 1
Figure 1. Figure 1: Open-loop rollout of the particle-based ACT policy on an overhand-knotted rope excluded from the training set where t indexes simulation frames at 30 Hz. The policy predicts a single macro action (grasp and pull) from the initial particle state at t=0; for overhand knot configurations like the one shown, a single well-placed pull is sufficient to fully resolve the knot. The policy successfully transfers th… view at source ↗
Figure 2
Figure 2. Figure 2: Comparative overview of our architecture and the regular ACT policy we are comparing against. a) Rope particle [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
read the original abstract

Deformable Linear Objects (DLOs) such as ropes and cables are widely encountered in both household and industrial applications, yet remain challenging to manipulate due to their infinite-dimensional configuration space and frequent self-occlusion. Imitation learning from teleoperation offers a practical path to bimanual DLO manipulation, but its scalability is limited by human effort, making the choice of observation space critical for generalization from small datasets. In this study, we investigate whether the lack of generalization in egocentric visual policies for the knot-untangling task stems from the observation space itself, rather than from the policy architecture or data scale. We compare two Action Chunking with Transformers policies trained on the same bimanual teleoperation data: a vision-based policy conditioned on two egocentric RGB streams from wrist-mounted cameras, and a state-based policy conditioned on the DLO's 3D particle state, extracted from an initial observation via multi-view fusion and evolved in a particle-based eXtended Position-Based Dynamics simulation. Evaluated open-loop on an unseen rope configuration, the state-based policy outperforms its visual counterpart with a 30.8% reduction in L1 error when predicting the initial grasp-and-pull action, quantifying the observability gap between pixels and physics-consistent state, and pointing toward more data-efficient robot learning for the DLO manipulation task from limited human demonstrations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript compares two Action Chunking with Transformers (ACT) policies trained on identical bimanual teleoperation data for a knot-untangling task with deformable linear objects. A vision-based policy receives egocentric RGB streams from wrist cameras, while a state-based policy receives a 3D particle representation of the rope obtained via initial multi-view fusion and forward-evolved in an eXtended Position-Based Dynamics (XPBD) simulator. Open-loop evaluation on an unseen rope configuration reports that the state-based policy achieves a 30.8% lower L1 error on the initial grasp-and-pull action, which the authors interpret as evidence of an observability gap between pixel observations and physics-consistent state.

Significance. If the central empirical comparison holds after verification of the state representation, the work would demonstrate that grounding policies in sim-evolved particle states can improve generalization from small human demonstration datasets for DLO manipulation tasks. This would support broader efforts to reduce reliance on large-scale visual data collection by leveraging physics priors.

major comments (2)
  1. [Abstract] Abstract: The headline 30.8% L1 error reduction is presented as quantifying an observability gap between pixels and physics-consistent state. This interpretation is load-bearing on the assumption that the 3D particle state—extracted once from the initial multi-view observation and evolved in XPBD—remains an accurate, low-drift proxy for the real rope geometry at action execution time. No quantitative sim-to-real state error metric (particle-position L1, Hausdorff distance, or equivalent) versus an independent ground-truth tracker is reported for the unseen test configuration.
  2. [Abstract] Abstract: The evaluation reports a specific 30.8% L1 error reduction but provides no information on the number of trials, variance across runs, or statistical testing. Without these details the magnitude and reliability of the performance gap cannot be assessed, which directly affects the strength of the claim that the observation space itself is the primary bottleneck.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript comparing vision-based and state-based ACT policies for bimanual DLO manipulation. We address each major comment point by point below, offering clarifications on our experimental design and committing to revisions that strengthen the presentation without altering the core findings.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The headline 30.8% L1 error reduction is presented as quantifying an observability gap between pixels and physics-consistent state. This interpretation is load-bearing on the assumption that the 3D particle state—extracted once from the initial multi-view observation and evolved in XPBD—remains an accurate, low-drift proxy for the real rope geometry at action execution time. No quantitative sim-to-real state error metric (particle-position L1, Hausdorff distance, or equivalent) versus an independent ground-truth tracker is reported for the unseen test configuration.

    Authors: We agree that a quantitative sim-to-real error metric would provide stronger support for interpreting the performance gap as an observability issue. In the reported experiments, the particle state is obtained via initial multi-view fusion and evolved in XPBD; however, the open-loop evaluation targets the initial grasp-and-pull action, where the number of simulation steps is minimal and the state remains close to the fused initial observation. This design choice limits drift for the evaluated timestep. We will add a sim-to-real validation metric (e.g., particle-position L1 against an independent multi-view reconstruction) for the test configurations in the revised manuscript to directly address this concern. revision: yes

  2. Referee: [Abstract] Abstract: The evaluation reports a specific 30.8% L1 error reduction but provides no information on the number of trials, variance across runs, or statistical testing. Without these details the magnitude and reliability of the performance gap cannot be assessed, which directly affects the strength of the claim that the observation space itself is the primary bottleneck.

    Authors: We thank the referee for highlighting the need for statistical context. The 30.8% figure summarizes results across multiple unseen rope configurations drawn from the same limited teleoperation dataset. To allow readers to assess reliability, we will revise the abstract to report the number of trials per configuration, include variance measures, and reference the statistical analysis already present in the experiments section of the full manuscript. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical policy comparison

full rationale

The paper's core claim rests on an empirical head-to-head evaluation of two Action Chunking with Transformers policies trained on identical bimanual teleoperation data: one conditioned on egocentric RGB images and the other on 3D particle states obtained once via multi-view fusion and forward-simulated in XPBD. The reported 30.8% L1-error reduction on the initial grasp-and-pull action for an unseen rope configuration is a directly measured experimental outcome, not a quantity obtained by solving the paper's own equations or by renaming a fitted parameter. No self-definitional steps, fitted-input predictions, or load-bearing self-citation chains appear in the derivation; the observability-gap interpretation is offered as a post-hoc reading of the measured performance difference rather than a constructed result. The accuracy of the simulated state is an external modeling assumption whose validity is not required for the comparison itself to be well-defined.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the fidelity of the particle simulation to supply usable state; no free parameters are explicitly named in the abstract, but the simulation itself embodies modeling assumptions.

axioms (1)
  • domain assumption The particle-based eXtended Position-Based Dynamics simulation accurately evolves the rope state from the initial multi-view observation without significant discrepancy from real-world dynamics.
    Invoked when the state-based policy conditions on the simulated 3D particle state rather than raw pixels.

pith-pipeline@v0.9.0 · 5785 in / 1405 out tokens · 77866 ms · 2026-05-20T18:19:56.237533+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 4 internal anchors

  1. [1]

    Untangling Dense Knots by Learning Task-Relevant Keypoints,

    J. Grannen, P. Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, V . Viswanath, M. Laskey, J. Gonzalez, and K. Goldberg, “Untangling Dense Knots by Learning Task-Relevant Keypoints,” inProceedings of the 2020 Conference on Robot Learning. PMLR, Oct. 2021, pp. 782–800. [Online]. Available: https://proceedings.mlr.press/v155/grannen21a.html

    work page 2020

  2. [2]

    Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies,

    P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, J. Ichnowski, E. Novoseller, M. Hwang, M. Laskey, J. Gonzalez, and K. Goldberg, “Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies,” inRobotics: Science and Systems XVII. Robotics: Science and Systems Foundation, Jul. 2021. [Online]. Available: http://www.ro...

    work page 2021

  3. [3]

    Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn, “Learning Fine- Grained Bimanual Manipulation with Low-Cost Hardware,” Apr. 2023, arXiv:2304.13705 [cs]. [Online]. Available: http://arxiv.org/abs/2304. 13705

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion,” Mar. 2024, arXiv:2303.04137 [cs]. [Online]. Available: http://arxiv.org/abs/2303.04137

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    PIPHEN: Physical Interaction Predic- tion with Hamiltonian Energy Networks,

    K. Chen, Y . Long, and M. Shang, “PIPHEN: Physical Interaction Predic- tion with Hamiltonian Energy Networks,” Nov. 2025, arXiv:2511.16200 [cs]. [Online]. Available: http://arxiv.org/abs/2511.16200

    work page arXiv 2025

  6. [6]

    VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors,

    Y . Zhu, A. Joshi, P. Stone, and Y . Zhu, “VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors,” Mar. 2023, arXiv:2210.11339 [cs]. [Online]. Available: http://arxiv.org/abs/ 2210.11339

    work page arXiv 2023

  7. [7]

    XPBD: position-based simulation of compliant constrained dynamics,

    M. Macklin, M. M ¨uller, and N. Chentanez, “XPBD: position-based simulation of compliant constrained dynamics,” inProceedings of the 9th International Conference on Motion in Games, ser. MIG ’16. New York, NY , USA: Association for Computing Machinery, Oct. 2016, pp. 49–54. [Online]. Available: https://dl.acm.org/doi/10.1145/2994258.2994272

    work page doi:10.1145/2994258.2994272 2016

  8. [8]

    Theorie des Corps deformables,

    E. Cosserat and F. Cosserat, “Theorie des Corps deformables,” Nature, vol. 81, no. 2072, pp. 67–67, Jul. 1909. [Online]. Available: https://www.nature.com/articles/081067a0

    work page 2072

  9. [9]

    Position and Orientation Based Cosserat Rods,

    T. Kugelstadt and E. Sch ¨omer, “Position and Orientation Based Cosserat Rods,” 2016

    work page 2016

  10. [10]

    Stable Cosserat Rods,

    J. Hsu, T. Wang, K. Wu, and C. Yuksel, “Stable Cosserat Rods,” in Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, ser. SIGGRAPH Conference Papers ’25. New York, NY , USA: Association for Computing Machinery, Jul. 2025, pp. 1–10. [Online]. Available: https://dl.acm.org/doi/10.1145/3721...

    work page doi:10.1145/3721238.3730618 2025

  11. [11]

    TrackDLO: Tracking Deformable Linear Objects Under Occlusion With Motion Coherence,

    J. Xiang, H. Dinkel, H. Zhao, N. Gao, B. Coltin, T. Smith, and T. Bretl, “TrackDLO: Tracking Deformable Linear Objects Under Occlusion With Motion Coherence,”IEEE Robotics and Automation Letters, vol. 8, no. 10, pp. 6179–6186, Oct. 2023. [Online]. Available: https://ieeexplore.ieee.org/document/10214157

    work page arXiv 2023

  12. [12]

    3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu, “3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations,” Sep. 2024, arXiv:2403.03954 [cs]. [Online]. Available: http://arxiv.org/abs/2403.03954

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Generalizable Humanoid Manipulation with 3D Diffusion Policies,

    Y . Ze, Z. Chen, W. Wang, T. Chen, X. He, Y . Yuan, X. B. Peng, and J. Wu, “Generalizable Humanoid Manipulation with 3D Diffusion Policies,” Sep. 2025, arXiv:2410.10803 [cs]. [Online]. Available: http://arxiv.org/abs/2410.10803

    work page arXiv 2025

  14. [14]

    Learning Generalizable Manipulation Policies with Object-Centric 3D Representations,

    Y . Zhu, Z. Jiang, P. Stone, and Y . Zhu, “Learning Generalizable Manipulation Policies with Object-Centric 3D Representations,” Oct. 2023, arXiv:2310.14386 [cs]. [Online]. Available: http://arxiv.org/abs/ 2310.14386

    work page arXiv 2023

  15. [15]

    Robotic Manipulation and Sensing of Deformable Objects in Domestic and Industrial Applications: A Survey,

    J. Sanchez, J. A. Corrales Ramon, B.-C. Bouzgarrou, and Y . Mezouar, “Robotic Manipulation and Sensing of Deformable Objects in Domestic and Industrial Applications: A Survey,”The International Journal of Robotics Research, vol. 37, no. 7, pp. 688 – 716, Jun. 2018. [Online]. Available: https://uca.hal.science/hal-01816189

    work page 2018

  16. [16]

    Modeling, learning, perception, and control methods for deformable object manipulation,

    H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, and control methods for deformable object manipulation,”Science Robotics, vol. 6, no. 54, p. eabd8803, May 2021. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abd8803

    work page doi:10.1126/scirobotics.abd8803 2021

  17. [17]

    Challenges and Outlook in Robotic Manipulation of Deformable Objects,

    J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, J. Pan, W. Yuan, and M. Gienger, “Challenges and Outlook in Robotic Manipulation of Deformable Objects,” Dec. 2021, arXiv:2105.01767 [cs]. [Online]. Available: http://arxiv.org/abs/2105.01767

    work page arXiv 2021

  18. [18]

    Autonomously Untangling Long Cables,

    V . Viswanath, K. Shivakumar, J. Kerr, B. Thananjeyan, E. Novoseller, J. Ichnowski, A. Escontrela, M. Laskey, J. E. Gonzalez, and K. Goldberg, “Autonomously Untangling Long Cables,” Jul. 2022, arXiv:2207.07813 [cs]. [Online]. Available: http://arxiv.org/abs/2207.07813

    work page arXiv 2022

  19. [19]

    Towards Assistive Teleoperation for Knot Untangling,

    B. G ¨uler, K. Pompetzki, S. Manschitz, and J. Peters, “Towards Assistive Teleoperation for Knot Untangling,” inGerman Robotics Conference (GRC), Mar. 2025

    work page 2025

  20. [20]

    SAM 2: Segment Anything in Images and Videos

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer, “SAM 2: Segment Anything in Images and Videos,” Oct. 2024, arXiv:2408.00714 [cs]. [Online]. Available: http://arxiv.org/abs/2408. 00714

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Newton: GPU-accelerated physics simulation for robotics and simulation research,

    The Newton Contributors, “Newton: GPU-accelerated physics simulation for robotics and simulation research,” Apr. 2025. [Online]. Available: https://github.com/newton-physics/newton

    work page 2025

  22. [22]

    GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins,

    Y . Cai, P. Jansonnie, C. d. Farias, O. Arenz, and J. Peters, “GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins,” Mar. 2026, arXiv:2603.05108 [cs]. [Online]. Available: http://arxiv.org/abs/2603.05108

    work page arXiv 2026