REVIEW 3 major objections 6 minor 30 references
TrackDeform3D claims that high-quality 3D keypoint trajectories of deformable objects can be produced autonomously from a single RGB-D camera, without markers or manual annotation, by solving one unified optimization for both initialization
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A markerless RGB-D tracking pipeline for deformable objects plus a released 110-minute, six-object trajectory dataset.
T0 review reviewed 2026-08-02 challenge →
load-bearing objection Useful dataset and a clean pipeline, but the headline accuracy claims outrun the evaluation. the 3 major comments →
TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that a single constrained optimization can serve both keypoint initialization and frame-by-frame tracking of deformable objects. The method segments the object by differencing two point clouds taken before and after a robot motion, classifies it as 1D or 2D from local eigenvalue ratios, detects anchor points (leaf nodes and junctions for linear objects, contour points for cloth), and then solves for keypoint positions that keep adjacent edge lengths at target values while lying on the segmented surface and staying attached to anchors. Tracking repeats the same solve each frame, warm-started from the previous solution, and applies a temporal moving average. The paper repo
What carries the argument
The central mechanism is the constrained optimization of Eq. (1): minimize the sum over topology edges of the squared deviation between the current keypoint distance and a target edge length, subject to every keypoint lying in the segmented point cloud and anchor keypoints fixed at detected 3D positions. It is solved by a Gauss-Seidel-style iterative projection — projecting each edge to its target length, snapping non-anchor keypoints to their nearest point in the point cloud, and resetting anchors — which is reused for initialization and for every tracking frame, giving the pipeline a single, unified consistency mechanism.
Load-bearing premise
The pipeline assumes the deformable object is never occluded and that, after subtracting two point clouds and masking the robot arms, the object is the largest coherent cluster in the scene; if that segmentation fails, every keypoint and trajectory in the affected frames is corrupted.
What would settle it
Run the released pipeline on sequences where a similarly sized stationary object sits near the manipulation area and measure the fraction of frames in which the selected cluster overlaps that background object. If that fraction is non-trivial (say, above 1-2%), the tracking accuracy numbers in the paper will not reproduce for cluttered scenes.
If this is right
- If correct, large 3D keypoint trajectory datasets for deformable objects can be generated with commodity RGB-D hardware and two robot arms, without markers or human annotation.
- Keypoint indices remain tied to the same physical locations across sessions and frames, making the data directly usable for training dynamics models, sim-to-real transfer, and manipulation policies.
- The unified-optimization design suggests that geometry-aware constraints (anchors, edge lengths, surface projection) can outperform purely learned point trackers for this data-collection setting.
- Branched objects such as wire harnesses, which were absent from earlier public deformable-object benchmarks, become feasible to track and to model.
- The recorded robot control inputs alongside keypoints may enable supervised learning of visual-manipulation mappings, not just forward dynamics.
Where Pith is reading between the lines
- A natural extension is to replace the largest-coherent-cluster segmentation with a learned promptable segmenter, which could make the pipeline robust to clutter and occlusion at the cost of some autonomy; that experiment would reveal how much of the reported accuracy comes from the optimization rather than the segmentation.
- The edge-length constraint is essentially an as-rigid-as-possible prior along the object's topology; extending the same framework to non-rectangular meshes with unknown connectivity is a plausible next step the paper leaves implicit.
- Because the dataset also records the robot arm's grasp points and joint inputs, it could be used to train closed-loop manipulation policies directly from keypoint observations — a downstream use the authors mention but do not demonstrate.
- If multiple synchronized cameras were added, the same optimization could in principle fuse their point clouds, relaxing the unoccluded assumption while keeping the consistency constraints unchanged.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. TrackDeform3D proposes an autonomous RGB-D pipeline for collecting 3D keypoint trajectories of deformable objects. Depth images are lifted to point clouds, the object is segmented by background differencing, arm masking, and DBSCAN, then 3D keypoints are initialized and tracked by solving a constrained optimization (Eq. 1) that enforces anchor constraints, fixed neighboring edge lengths, and projection onto the segmented point cloud. The paper claims consistent improvements over CoTracker3, CDCPD2, and SpatialTrackerV2 in absolute edge-length error, Chamfer distance, and F-scores, and introduces a dataset of 6 objects with 110 minutes / 1,320 motion sequences, including branched deformable objects.
Significance. If the tracking and dataset-quality claims hold, the system would be a valuable, low-cost alternative to marker-based motion capture for deformable-object manipulation research. The dataset scale and inclusion of branched DLOs are genuine contributions, and the optimization formulation is simple and transparent. However, the current evaluation does not establish that the tracked keypoints correspond to consistent physical points on the object: the primary metrics are the same quantities that Eq. (1) directly optimizes, and there is no ground-truth correspondence check. The paper's central claims therefore need additional validation before the dataset can be considered 'high quality' or the tracking 'accurate.'
major comments (3)
- [§III-B, §IV-A.3, Table I] The headline claim of 'consistent improvements in tracking accuracy' is not established by the reported metrics. Eq. (1) minimizes exactly the squared deviation of neighboring keypoint distances from fixed target edge lengths and constrains each keypoint to lie in P_seg. The Abs metric is the RMSE of the first objective term, and CD measures the mean distance to P_seg, which the projection step (Algorithm 1, line 6) drives toward zero by construction. Thus Tables I–III largely measure how well the solver satisfies its own objective, not whether each keypoint index tracks the same material point. No experiment provides marker-based ground truth, synthetic correspondence, or manual re-identification. On 1D objects, keypoints can slide along the rope/wire while preserving edge lengths and point-cloud proximity; on cloth, interior keypoints can drift within the surface. The 'high-quality dat
- [§III-C, §IV] The segmentation step is load-bearing but its failure rate is never quantified. After point-cloud differencing and arm masking, the object is assumed to be the largest coherent DBSCAN cluster (Section III-C). If a table, background object, or arm residual is wrongly included, every keypoint and trajectory in that frame is corrupted. Since the paper claims autonomous operation on 1,320 sequences, it should report per-sequence segmentation success/failure rates, including how often the largest-cluster selection fails and whether such failures are detected rather than silently propagated.
- [Table I, §IV-A.4] The baseline comparison reports a single number per condition with no error bars, confidence intervals, or independent trials. The text states that evaluations comprise 1140 s of rope, 1220 s of branched rope, and 400 s of cloth, but it is unclear whether these are one continuous session per object or multiple sessions. Several improvements are modest (e.g., rope CD 8.04 vs 8.54, rope F<10 74.1 vs 74.9), so 'consistent improvements' needs statistical support. Please report results over multiple independent sequences with variance, or at least clarify the number of sessions and per-session results.
minor comments (6)
- [§I] The introduction still contains placeholder text: 'comprising X objects and totaling X hours of trajectory data.' This should be filled in or removed before publication.
- [§IV-A.4] The phrase 'under the same initialization' is vague. For CoTracker3 and SpatialTrackerV2, how are the initial 2D/3D points chosen? Are the same initial keypoint indices used for all methods? Please specify.
- [Algorithm 1] Line 4 refers to '[30, (8)]' for the edge-length projection but does not reproduce the update. Since the solver is central, please state the projection formula explicitly or include it in an appendix.
- [§IV-A.3] The F-score is described as 'jointly measuring precision and recall' but no formula is given. Define how precision/recall are computed with respect to the point cloud and how F<10 relates to CD.
- [§IV-C] The dataset is described as 'will be released' and 'will be split,' which is appropriate, but the abstract and conclusion state that the paper 'presents' the dataset. Please align the wording with the actual availability status.
- [§II-B] Typo: 'V olumetric' should be 'Volumetric.'
Circularity Check
The reported 'geometric and tracking accuracy' metrics are residuals of Eq. (1)'s own objective and projection constraint; accuracy claims reduce to self-consistency, with no ground-truth correspondence check.
specific steps
-
self definitional
[Section III-B, Eq. (1); Section IV-A.3 Metrics]
"arg min_Xt Σ_{(i,j)∈E} (‖x_{t,i} − x_{t,j}‖ − d_ij)^2 s.t. x_{t,i} ∈ P_seg^t, x_{t,k}=a_{t,k}, ∀k∈A, (1) ... Absolute edge length error (Abs.), reported as RMSE in millimeters, measures the deviation between distances of neighboring keypoints and their target edge lengths"
The Abs. metric is exactly the squared residual minimized in the objective of Eq. (1), averaged over the same edges and the same target lengths d_ij. Therefore the headline 'tracking accuracy' improvement on Abs. reports how well the solver satisfied its own optimization target, not whether keypoints correspond to fixed physical points on the deformable object. Lower Abs. is a property of the optimizer, not an independent accuracy measurement.
-
self definitional
[Algorithm 1 (Section III-D.4); Section IV-A.3 Metrics]
"6: x_l ← arg min_{p∈P_seg} ‖x_l − p‖, ∀l ∉ A ... Chamfer distance (CD), computed as the mean L2 distance between each predicted keypoint and its nearest neighbor on the segmented point cloud"
Algorithm 1 forces every non-anchor keypoint to be the nearest point on the segmented cloud P_seg, and the CD metric is exactly the mean L2 distance from predicted keypoints to P_seg. Thus the main 'geometric accuracy' metric is the residual of the method's own projection constraint. A tracker could place points anywhere on P_seg and score well on CD while keypoint identities drift along a self-similar rope or featureless cloth, because no correspondence ground truth is evaluated.
full rationale
The central quantitative support for 'consistent improvements in both geometric and tracking accuracy' comes from two metrics: absolute edge-length error and Chamfer distance. The first is the objective of Eq. (1), and the second measures the constraint enforced by Algorithm 1's projection step. Both are therefore self-consistency residuals rather than independent measures of physical tracking correctness. The paper never compares its keypoint indices against markers, manual annotations, or synthetic ground truth, so the possibility that keypoints slide along the object while preserving edge lengths and surface proximity is not tested. The baseline comparisons and the released 110-minute dataset provide some independent content, and there is no load-bearing self-citation chain; however, because the headline accuracy claim is supported only by metrics the method is built to minimize, the core evaluation is partially circular. This is a correctness/validity concern as much as a circularity concern, but under the rubric of 'predictions reducing by construction' it warrants a score of 7 rather than a clean bill.
Axiom & Free-Parameter Ledger
free parameters (6)
- Number of keypoints N =
not reported
- Target edge lengths d_ij =
fixed from initialization
- Object classification threshold =
not reported
- DBSCAN parameters =
not reported
- Temporal moving-average window =
not reported
- Number of solver iterations M =
not reported
axioms (4)
- domain assumption The deformable object remains unoccluded throughout manipulation.
- domain assumption Point-cloud differencing isolates the object as the largest coherent cluster after arm masking.
- domain assumption Topology is fixed at initialization; cloth objects are rectangular grids.
- domain assumption Accurate hand-eye calibration and robot-arm simulation are available for arm-point filtering.
Cite this review
Pith. "Pith review of TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects." pith.science (2026). https://pith.science/paper/33QLVLXT
@misc{pith2026260317068,
author = {Pith},
title = {Pith review of: TrackDeform3D: Markerless and Autonomous 3D Keypoint Tracking and Dataset Collection for Deformable Objects},
year = {2026},
howpublished = {\url{https://pith.science/paper/33QLVLXT}},
note = {Machine review of arXiv:2603.17068}
}
read the original abstract
Structured 3D representations such as keypoints and meshes offer compact, expressive descriptions of deformable objects, jointly capturing geometric and topological information useful for downstream tasks such as dynamics modeling and motion planning. However, robustly extracting such representations remains challenging, as current perception methods struggle to handle complex deformations. Moreover, large-scale 3D data collection remains a bottleneck: existing approaches either require prohibitive data collection efforts, such as labor-intensive annotation or expensive motion capture setups, or rely on simplifying assumptions that break down in unstructured environments. As a result, large-scale 3D datasets and benchmarks for deformable objects remain scarce. To address these challenges, this paper presents an affordable and autonomous framework for collecting 3D datasets of deformable objects using only RGB-D cameras. The proposed method identifies 3D keypoints and robustly tracks their trajectories, incorporating motion consistency constraints to produce temporally smooth and geometrically coherent data. TrackDeform3D is evaluated against several state-of-the-art tracking methods across diverse object categories and demonstrates consistent improvements in both geometric and tracking accuracy. Using this framework, this paper presents a high-quality, large-scale dataset consisting of 6 deformable objects, totaling 110 minutes of trajectory data. Project page: https://roahmlab.github.io/trackDeform3D-core-tracking/
Figures
Reference graph
Works this paper leans on
-
[1]
Challenges and outlook in robotic manipulation of deformable objects,
J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, J. Pan, W. Yuan, and M. Gienger, “Challenges and outlook in robotic manipulation of deformable objects,” 2021
2021
-
[2]
A survey on robotic manipulation of deformable objects: Recent advances, open challenges and new frontiers,
F. Gu, Y . Zhou, Z. Wang, S. Jiang, and B. He, “A survey on robotic manipulation of deformable objects: Recent advances, open challenges and new frontiers,” 2023
2023
-
[3]
Cotracker: It is better to track together,
N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker: It is better to track together,” 2024
2024
-
[4]
Tracking cloth deformation: A novel dataset for closing the sim-to- real gap for robotic cloth manipulation learning,
F. Coltraro, J. Borr `as, M. Alberich-Carrami ˜nana, and C. Torras, “Tracking cloth deformation: A novel dataset for closing the sim-to- real gap for robotic cloth manipulation learning,”The International Journal of Robotics Research, vol. 44, no. 9, pp. 1431–1442, 2025
2025
-
[5]
Textureless deformable object tracking with invisible markers,
X. Li, Y . Guo, Y . Tu, Y . Ji, Y . Liu, J. Ye, and C. Zheng, “Textureless deformable object tracking with invisible markers,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 9, pp. 7243– 7254, 2025
2025
-
[6]
Tracking deformable objects with point clouds,
J. Schulman, A. Lee, J. Ho, and P. Abbeel, “Tracking deformable objects with point clouds,” in2013 IEEE International Conference on Robotics and Automation, pp. 1130–1137, 2013
2013
-
[7]
Occlusion-robust deformable object tracking without physics simulation,
C. Chi and D. Berenson, “Occlusion-robust deformable object tracking without physics simulation,” in2019 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), p. 6443–6450, IEEE, Nov. 2019
2019
-
[8]
Track deformable objects from point clouds with structure preserved registration,
T. Tang and M. Tomizuka, “Track deformable objects from point clouds with structure preserved registration,”The International Journal of Robotics Research, vol. 41, no. 6, pp. 599–614, 2022
2022
-
[9]
Tracking partially- occluded deformable objects while enforcing geometric constraints,
Y . Wang, D. McConachie, and D. Berenson, “Tracking partially- occluded deformable objects while enforcing geometric constraints,” 2020
2020
-
[10]
State estimation for deformable objects by point registration and dynamic simulation,
T. Tang, Y . Fan, H.-C. Lin, and M. Tomizuka, “State estimation for deformable objects by point registration and dynamic simulation,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2427–2433, 2017
2017
-
[11]
Differentiable discrete elastic rods for real-time modeling of deformable linear objects,
Y . Chen, Y . Zhang, Z. Brei, T. Zhang, Y . Chen, J. Wu, and R. Va- sudevan, “Differentiable discrete elastic rods for real-time modeling of deformable linear objects,”arXiv preprint arXiv:2406.05931, 2024
Pith/arXiv arXiv 2024
-
[12]
Deft: Differentiable branched discrete elastic rods for modeling furcated dlos in real-time,
Y . Chen, X. Wu, Y . Zong, Y . Chen, A. Li, B. Zhang, and R. Vasudevan, “Deft: Differentiable branched discrete elastic rods for modeling furcated dlos in real-time,”arXiv preprint arXiv:2502.15037, 2025
Pith/arXiv arXiv 2025
-
[13]
Large steps in cloth simulation,
D. Baraff and A. P. Witkin, “Large steps in cloth simulation,”Pro- ceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998
1998
-
[14]
Pokeflex: A real-world dataset of volumetric deformable objects for robotics,
J. Obrist, M. Zamora, H. Zheng, R. Hinchet, F. Ozdemir, J. Zarate, R. K. Katzschmann, and S. Coros, “Pokeflex: A real-world dataset of volumetric deformable objects for robotics,” 2025
2025
-
[15]
Dofs: A real-world 3d deformable object dataset with full spatial information for dynamics model learning,
Z. Zhang, X. Chu, Y . Tang, and K. W. S. Au, “Dofs: A real-world 3d deformable object dataset with full spatial information for dynamics model learning,” 2024
2024
-
[16]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll ´ar, and R. Gir- shick, “Segment anything,” 2023
2023
-
[17]
Sam 2: Segment anything in images and videos,
N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafson,et al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024
Pith/arXiv arXiv 2024
-
[18]
A robust deformable linear object perception pipeline in 3d: From seg- mentation to reconstruction,
S. Zhaole, H. Zhou, L. Nanbo, L. Chen, J. Zhu, and R. B. Fisher, “A robust deformable linear object perception pipeline in 3d: From seg- mentation to reconstruction,”IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 843–850, 2023
2023
-
[19]
Cotracker3: Simpler and better point tracking by pseudo-labelling real videos,
N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi, and C. Rupprecht, “Cotracker3: Simpler and better point tracking by pseudo-labelling real videos,”arXiv preprint arXiv:2410.11831, 2024
Pith/arXiv arXiv 2024
-
[20]
Tapip3d: Tracking any point in persistent 3d geometry,
B. Zhang, L. Ke, A. W. Harley, and K. Fragkiadaki, “Tapip3d: Tracking any point in persistent 3d geometry,”arXiv preprint arXiv:2504.14717, 2025
arXiv 2025
-
[21]
Spatialtrackerv2: 3d point tracking made easy,
Y . Xiao, J. Wang, N. Xue, N. Karaev, Y . Makarov, B. Kang, X. Zhu, H. Bao, Y . Shen, and X. Zhou, “Spatialtrackerv2: 3d point tracking made easy,”arXiv preprint arXiv:2507.12462, 2025
Pith/arXiv arXiv 2025
-
[22]
Understanding human hands in contact at internet scale,
D. Shan, J. Geng, M. Shu, and D. F. Fouhey, “Understanding human hands in contact at internet scale,” 2020
2020
-
[23]
Embedding high-resolution touch across robotic hands enables adaptive human-like grasping,
Z. Zhao, W. Li, Y . Li, T. Liu, B. Li, M. Wang, K. Du, H. Liu, Y . Zhu, Q. Wang, K. Althoefer, and S.-C. Zhu, “Embedding high-resolution touch across robotic hands enables adaptive human-like grasping,” Nature Machine Intelligence, vol. 7, p. 889–900, June 2025
2025
-
[24]
A density-based algorithm for discovering clusters in large spatial databases with noise.,
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise.,” inKDD(E. Simoudis, J. Han, and U. M. Fayyad, eds.), pp. 226–231, AAAI Press, 1996
1996
-
[25]
Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers,
M. Weinmann, B. Jutzi, S. Hinz, and C. Mallet, “Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 105, pp. 286–304, 2015
2015
-
[26]
Topology matching of branched deformable linear objects,
M. Z ¨urn, M. Wnuk, A. Lechler, and A. Verl, “Topology matching of branched deformable linear objects,” in2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7097–7103, IEEE, 2023
2023
-
[27]
The farthest point strategy for progressive image sampling,
Y . Eldar, M. Lindenbaum, M. Porat, and Y . Zeevi, “The farthest point strategy for progressive image sampling,” inProceedings of the 12th IAPR International Conference on Pattern Recognition, V ol. 2 - Conference B: Computer Vision & Image Processing. (Cat. No.94CH3440-5), pp. 93–97 vol.3, 1994
1994
-
[28]
The OpenCV Library,
G. Bradski, “The OpenCV Library,”Dr . Dobb’s Journal of Software Tools, 2000
2000
-
[29]
Clothcap: Seamless 4d clothing capture and retargeting,
G. Pons-Moll, S. Pujades, S. Hu, and M. Black, “Clothcap: Seamless 4d clothing capture and retargeting,”ACM Transactions on Graphics, vol. 36, pp. 1–15, 07 2017
2017
-
[30]
Projective dynamics: Fusing constraint projections for fast simulation,
S. Bouaziz, S. Martin, T. Liu, L. Kavan, and M. Pauly, “Projective dynamics: Fusing constraint projections for fast simulation,”Seminal Graphics Papers: Pushing the Boundaries, V olume 2, 2023
2023
This paper was first reviewed by deepseek-v4-flash on August 2, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.