REVIEW 4 major objections 5 minor 46 references
Leveraging Foundation Models To learn the shape of semi-fluid deformable objects
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A label-free teacher-student pipeline estimates weld-pool keypoints to within 13.4 pixels.
desk verdict Useful proof-of-concept for foundation-model-distilled weld pool keypoints, but the evaluation is too under-specified to justify the headline accuracy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is teacher-student knowledge distillation: DINO (a self-supervised vision transformer that outputs attention maps) proposes prompt points, SAM2 (a prompt-based segmentation foundation model) produces the object mask, and Algorithm 1 turns contour normal-angle changes into a multi-Gaussian heatmap over the keypoint regions. The student is a variational autoencoder with a ResNet encoder that reconstructs the mask and heatmap jointly. The machinery is what makes label-free characterization possible: the teacher generates the ground truth during training, and only the lightweight student is used at inference.
What would settle it
Re-run the student evaluation on frames whose keypoint labels are produced by several independent human annotators (or by an independent tracking method) and compare the mean Euclidean distance to the paper's 15-pixel threshold; if the error exceeds that threshold, the claim of stable keypoint retrieval is not supported.
Extended reading notes
Core claim
The central claim is that the shape of a highly deformable, viscoelastic object like a weld pool can be learned without any prior labeling by distilling the knowledge of foundation models into a small generative student. The teacher pipeline combines DINO's self-supervised attention maps as prompts for SAM2 to obtain a segmentation mask, then Algorithm 1 converts the contour's normal-angle discontinuities into a Gaussian heatmap of keypoint regions. The student, a ResNet-backed variational autoencoder, is trained to reconstruct the mask and heatmap from the raw image. The authors report that this student reaches a mean Euclidean distance of 13.4 pixels to manually labeled keypoints, beating the compared baselines, and that the teacher obtains a mean IoU of 75.26% on the mask.
Load-bearing premise
The load-bearing premise is that the manually labeled keypoints used to measure the 13.4-pixel error are accurate enough, even though the authors themselves call these annotations 'biased and prone to error' in the Discussion.
Editorial extensions
If this is right
- At inference, only the VAE student is needed, so weld-pool shape feedback can in principle run on a lightweight model in a robotic welding loop.
- The DINO-prompting heuristics raise SAM2's mask mIoU from 35.37% to 75.26%, showing that foundation-model segmentation can be adapted to an unseen industrial object without task-specific training.
- The method is model-free with respect to the object's physics, so the same pipeline can be pointed at other semi-fluid or viscoelastic materials without re-engineering the shape model.
- The 13.4-pixel mean Euclidean error beats the active-contour, KCF, SIFT, CPD, and CNN baselines they compared on the same dataset.
Reading between the lines
- Inference: because the teacher's mask quality bounds the contour that generates the keypoint heatmap, improving prompt selection or mask refinement should directly reduce the student's keypoint error, though the paper does not test this coupling.
- Inference: the authors' suggestion to use teacher-generated outputs as evaluation ground truth would make the headline error circular, so an independent annotation protocol is needed before the 13.4-pixel figure can be taken as a true accuracy measure.
- Inference: the same label-free pipeline could in principle transfer to other semi-fluid objects such as glue, dough, or molten materials in industrial settings, but the paper only demonstrates the weld-pool case, so that transfer remains untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a teacher-student framework for characterizing semi-fluid deformable objects, specifically the weld pool in robotic welding. A teacher pipeline combining DINO attention maps and SAM2 segmentation generates a binary mask and a Gaussian heatmap of keypoints; a VAE student with a ResNet backbone is trained to predict this mask and heatmap from raw images. The authors report a mean Euclidean distance (mED) of 13.4 px for student keypoint predictions against manual labels, and a teacher mask mIoU of 75.26%. The claimed contribution is the ability to learn shape keypoints without manually labeled training data, using foundation models as teachers.
Significance. If the quantitative claims hold, the work offers a practical, label-free route to weld-pool keypoint estimation, which could enable feature-based visual servoing in robotic welding. The real-world industrial dataset is a valuable contribution, and the explicit discussion of annotation bias in Section VI is honest and useful. However, the evaluation protocol as described does not yet establish the headline accuracy: the absence of a clear train/test split and the admitted unreliability of the manual labels are both central to the claim. The teacher-side heuristics also lack a held-out validation set. The concept is promising, but the evidence as presented is insufficient to support the stated performance numbers.
major comments (4)
- [Section V (Results) and Table II] The paper does not describe any train/validation/test split for the student evaluation. The student is trained for 100 epochs and then evaluated 'with respect to manually labeled data,' but it is not stated whether the evaluation frames overlap with the training frames. For a generative reconstructor, evaluating on training data could reduce to measuring memorization or reconstruction fidelity rather than generalization to unseen weld-pool images. The 13.4 px mED is therefore not established as a generalization result. Please report the exact number of sequences and frames used for training and for testing, and ensure the test set is sequence-disjoint from the training set.
- [Section VI (Discussion)] The headline 13.4 px error is measured against manual labels that the authors themselves describe as 'biased and prone to error.' The paper then recommends using the teacher-generated labels as ground truth, but those labels are exactly what the student is trained to reproduce, so that alternative would make the evaluation circular. As written, the claimed accuracy rests on an annotation source that the authors discredit in the same paper. To support the quantitative claim, the authors should either provide a more reliable independent evaluation (e.g., multiple annotators with inter-annotator agreement, or downstream task performance) or explicitly characterize the sensitivity of the reported mED to annotation noise.
- [Section V, teacher mIoU paragraph and Figure 6] The teacher mIoU of 75.26% is obtained after applying heuristics (angle threshold λ in Algorithm 1, number of K-means clusters k, DBSCAN parameters, prompt filtering). The text gives no indication that a separate validation set was used to select these parameters; the values appear to be tuned on the same data used for evaluation. This risks overfitting the heuristics to the evaluation set, so the reported mIoU may be optimistic. Please describe the model-selection protocol and report mIoU on a held-out set.
- [Section V, Baselines and Table II] The baseline setup is under-specified. The text mentions comparisons with active contours, KCF, fully connected CNNs, and methods 'without the use of foundation models' [45], but Table II also includes SIFT and VAE rows. It is unclear what the VAE baseline is: is it the same student architecture trained with manual labels, or a different generative model? The baseline VAE mED of 14.55 px is close to the proposed method's 13.4 px, so the claimed improvement over a teacher-free student depends entirely on this comparison. Please specify the training data, architecture, and evaluation protocol for each baseline, and clarify how the standard deviation values in the last column are computed.
minor comments (5)
- [Abstract] There is a typographical error: 'keypoitns' should be 'keypoints.'
- [Section II] The sentence 'the lighter, the fastet to covnerge and the most convenient at learning latent representation are Variational Autoencoders' contains typos and is grammatically awkward; please revise.
- [Algorithm 1] The notation is inconsistent: the line 'θi = ∇Sx ⊗ ∇Sy;' mixes the gradient components, and the update 'F + = fj(Pj; µj, Σj)' does not define the function fj. Please clarify the formulas.
- [Table II] The table header 'Standard Deviation' is ambiguous; it should specify whether the standard deviation is over the four mEDk columns or over the dataset. In addition, the mEDk notation is defined in the text but not used consistently in the table caption.
- [Section IV] There is a typo in 'A subset of the acquired sequences are labele for segmentation task' — 'labele' should be 'labeled.'
Circularity Check
Partial circularity: the paper recommends evaluating the student against teacher-generated GT (which would be self-definitional), and the teacher's 75.26% mIoU is fitted to the evaluation set; the student's 13.4 px error is measured against manual labels and is not internally circular.
-
self definitional
[Section VI (Discussion), Quantitative evaluation]
"We recall that validation images were labeled manually with the possible centers of the heatmap, which makes these annotations biased and prone to error. Since defining accurate labels for these tasks is not possible. To resolve this kind of uncertainty, we recommend adopting the automatically generated GT labels as ground truth for evaluation and not the manually defined GT."
The student is trained to reproduce the teacher's automatically generated GT (mask plus heatmap); Section III states 'We consider the output of the teacher to be the corresponding ground truth' and the teacher's output is 'the ground truth knowledge that is distilled.' If the teacher's own outputs are adopted as evaluation GT, the student's mED compares its predictions to its own training targets. That is self-definitional: a student that memorized the teacher outputs would score perfectly, so the metric would measure reconstruction of the training label rather than generalization. The paper itself recommends this circular evaluation and explicitly says the only non-circular manual labels are 'biased and prone to error.'
-
fitted input called prediction
[Section V (Results), teacher mIoU evaluation]
"after applying some heuristics such as clustering the thresholded mean attention map into different clusters, applying DBSCAN to propose a best fit bounding box and filtering the propmt points increased significantly the values of mIoU. After applying these heuristics, we obtained an mIoU of 75.26%, with an upper quartile of 85%."
The 75.26% mIoU is reported after the same dataset was used to select the DINO prompt heuristics (thresholding, clustering, DBSCAN bounding box, point filtering). No train/validation/test split is described for this selection. The reported teacher mask-retrieval performance is therefore the result of fitting heuristic choices to the evaluation data; calling it the teacher's capacity to retrieve pixel-level information presents a fitted value as a predictive result. This is the 'fitted input called prediction' pattern and inflates the abstract's mIoU claim.
full rationale
The framework is not circular in its training pipeline: the student is trained on teacher-generated heatmaps and masks, which is standard knowledge distillation and requires no manual labels for training. The headline student error of 13.4 px is measured against manual keypoint labels, so it is not by construction equal to the teacher output. However, the paper's own Discussion (Section VI) states that these manual labels are 'biased and prone to error' and recommends switching evaluation to the teacher's automatically generated GT labels. Since the student was trained to reproduce exactly those teacher labels, such an evaluation would be self-definitional: the prediction and ground truth would come from the same generator that produced the training target. This recommended evaluation is a circular step, and the paper's admission undercuts the reliability of the 13.4 px number. Separately, the teacher's reported mIoU of 75.26% (also in the abstract) is obtained after tuning prompt heuristics on the same data, with no described held-out split, so that number is a fitted evaluation statistic rather than an out-of-sample prediction. No load-bearing self-citation chain is present: the only self-citation [45] is used as a baseline comparison. Overall, the central student claim still has some independent content (manual labels), but the proposed validation and the teacher score introduce partial circularity, so the score is 4 rather than 0.
Assumptions & free parameters
free parameters (5)
- angle variation threshold λ =
not reported
- number of K-means clusters k =
not reported (Fig. 3 shows 4)
- DINO attention threshold and number of attention maps =
six attention maps, thresholded at mean attention
- DBSCAN parameters and prompt filtering heuristics =
not reported
- VAE latent dimension and training hyperparameters =
not reported
assumptions (4)
- domain assumption Keypoints defined by normal-angle discontinuities on the contour are the correct semantic features for weld pool control.
- domain assumption DINO attention maps, after thresholding, highlight the weld pool region.
- domain assumption SAM2 with point prompts can segment the molten pool accurately.
- domain assumption Manual keypoint annotations are sufficiently unbiased for evaluation.
Cite this review
Pith. "Pith review of Leveraging Foundation Models To learn the shape of semi-fluid deformable objects." pith.science (2026). https://pith.science/paper/ZMQN4LMU
@misc{pith2026241116802,
author = {Pith},
title = {Pith review of: Leveraging Foundation Models To learn the shape of semi-fluid deformable objects},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZMQN4LMU}},
note = {Machine review of arXiv:2411.16802}
}
read the original abstract
One of the difficulties imposed on the manipulation of deformable objects is their characterization and the detection of representative keypoints for the purpose of manipulation. A keen interest was manifested by researchers in the last decade to characterize and manipulate deformable objects of non-fluid nature, such as clothes and ropes. Even though several propositions were made in the regard of object characterization, however researchers were always confronted with the need of pixel-level information of the object through images to extract relevant information. This usually is accomplished by means of segmentation networks trained on manually labeled data for this purpose. In this paper, we address the subject of characterizing weld pool to define stable features that serve as information for further motion control objectives. We achieve this by employing different pipelines. The first one consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework. And in the second one we leverage foundation models by using them as teachers to characterize the object in the image, without the need of any pre-training and any dataset. The performance of knowledge distillation from foundation models into a smaller generative model shows prominent results in the characterization of deformable objects. The student network was capable of learning to retrieve the keypoitns of the object with an error of 13.4 pixels. And the teacher was evaluated based on its capacities to retrieve pixel level information represented by the object mask, with a mean Intersection Over Union (mIoU) of 75.26%.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[45]
Shape estimation of Viscoelastic fluid materials: an application for weld pool. El Assal, O., Mateo. C.M., CIRON, S., Fofi, D
-
[1]
Xu, Z., He, Z., Wu, J., & Song, S. (2020). Learning 3D Dynamic Scene Representations for Robot Manipulation. Proceedings of Machine Learning Research, 155, 126–142
work page 2020
-
[2]
van der Merwe, M., Lu, Q., Sundaralingam, B., Matak, M., & Hermans, T. (2020). Learning Continuous 3D Reconstructions for Geometrically Aware Grasping. Proceedings - IEEE International Conference on Robotics and Automation, 11516–11522
work page 2020
-
[3]
Garg, R., Vijay Kumar, B. G., Carneiro, G., & Reid, I. (2016). Unsupervised CNN for single view depth estimation: Geometry to the rescue. 9912 LNCS, 740–756
work page 2016
-
[4]
Naseer, M., Khan, S., & Porikli, F. (2019). Indoor Scene Understand- ing in 2.5/3D for Autonomous Agents: A Survey. IEEE Access, 7, 1859–1887
work page 2019
-
[5]
Pan, Z., & Manocha, D. (2016). Motion planning for fluid manipu- lation using simplified dynamics. IEEE International Conference on Intelligent Robots and Systems, 2016-November, 4224–4231
work page 2016
-
[6]
Hou, Y . C., Sahari, K. S. M., & How, D. N. T. (2019). A review on modeling of flexible deformable object for dexterous robotic manipulation. International Journal of Advanced Robotic Systems
work page 2019
-
[7]
Herguedas, R., L ´opez-Nicol´as, G., Arag ¨u´es, R., & Sag ¨u´es, C. (2019). Survey on multi-robot manipulation of deformable objects. IEEE International Conference on Emerging Technologies and Factory Au- tomation, ETFA, 2019-September, 977–984
work page 2019
Show all 46 references
-
[8]
J., & Payeur, P
Nadon, F., Valencia, A. J., & Payeur, P. (2018). Multi-Modal Sensing and Robotic Manipulation of Non-Rigid Objects: A Survey. Robotics 2018, V ol. 7, Page 74, 7(4), 74
2018
-
[9]
M., Corrales, J
Mateo, C. M., Corrales, J. A., & Mezouar, Y . (2020). A manipulation control strategy for granular materials based on a Gaussian mixture model. In Robot 2019: Fourth Iberian Robotics Conference: Advances in Robotics, V olume 2 (pp. 171-183)
2020
-
[10]
Yan, M., Zhu, Y ., Jin, N., & Bohg, J. (2020). Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects. IEEE Robotics and Automation Letters, 5(2), 2372–2379
2020
-
[11]
Borum, A., Matthews, D., & Bretl, T. (2014). State estimation and tracking of deforming planar elastic rods. 2014 IEEE International Conference on Robotics and Automation (ICRA), 4127–4132
2014
-
[12]
E., Borr `as, J., & Aleny `a, G
Tzelepis, G., Aksoy, E. E., Borr `as, J., & Aleny `a, G. (2022). Semantic State Estimation in Cloth Manipulation Tasks
2022
-
[13]
M., Martinez Lastra, J
Malvido Fresnillo, P., Vasudevan, S., Mohammed, W. M., Martinez Lastra, J. L., & Perez Garcia, J. A. (2023). An approach based on machine vision for the identification and shape estimation of deformable linear objects. Mechatronics, 96, 103085
2023
-
[14]
Hu, Z., Hua, L., Qin, X., Ni, M., Ji, F., & Wu, M. (2021). Molten pool behaviors and forming appearance of robotic GMAW on complex surface with various welding positions. Journal of Manufacturing Processes, 64, 1359–1376
2021
-
[15]
Stadler, M., Freton, P., & Gonzalez, J.-J. (2017). Influence of Welding Parameters on the Weld Pool Dimensions and Shape in a TIG Configuration. Applied Sciences, 7(4), 373
2017
-
[16]
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (n.d.). Emerging Properties in Self-Supervised Vision Transformers
-
[17]
Ravi, N., Gabeur, V ., Hu, Y .-T., Hu, R., Ryali, C., Ma, T., Khedr, H., R¨adle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Vasudev Alwala, K., Carion, N., Wu, C.-Y ., Girshick, R., Doll ´ar, P., Feichten- hofer, C., & Fair, M. (n.d.). SAM 2: Segment Anything in Imag...
-
[18]
Gu, F., Zhou, Y ., Wang, Z., Jiang, S., & He, B. (2023). A Survey on Robotic Manipulation of Deformable Objects: Recent Advances, Open Challenges and New Frontiers
2023
-
[19]
Gascuel, M.-P. (n.d.). An Implicit Formulation for Precise Contact Modeling between Flexible Solids
-
[20]
Palli, G. (2020). Model-based Manipulation of Deformable Linear Objects by Multivariate Dynamic Splines. 2020 IEEE Conference on Industrial Cyberphysical Systems (ICPS), 520–525
2020
-
[21]
E., Guler, P., Ficuciello, F., Kragic, D., Siciliano, B., & Wyatt, J
Arriola-Rios, V . E., Guler, P., Ficuciello, F., Kragic, D., Siciliano, B., & Wyatt, J. L. (2020). Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review. Frontiers in Robotics and AI
2020
-
[22]
Rastegarpanah, A., Howard, R., & Stolkin, R. (2022). Tracking linear deformable objects using slicing method. Robotica, 40(4), 1188–1206
2022
-
[23]
Miller, S., van den Berg, J., Fritz, M., Darrell, T., Goldberg, K., & Abbeel, P. (2012). A geometric approach to robotic laundry folding. The International Journal of Robotics Research, 31(2), 249–267
2012
-
[24]
H., Pan, J., Mulcaire, G., & Abbeel, P
Huang, S. H., Pan, J., Mulcaire, G., & Abbeel, P. (2015). Leveraging appearance priors in non-rigid registration, with application to manipu- lation of deformable objects. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 878–885
2015
-
[25]
Tang, T., Fan, Y ., Lin, H.-C., & Tomizuka, M. (2017). State estimation for deformable objects by point registration and dynamic simulation. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2427–2433
2017
-
[26]
Shi, H., Xu, H., Huang, Z., Li, Y ., & Wu, J. (2022). RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks. International Journal of Robotics Research
2022
-
[27]
Lu, J., Jayakumari, A., Richter, F., Li, Y ., & Yip, M. C. (2021). SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction. Proceedings - IEEE International Conference on Robotics and Automation, 2021- May, 4783–4789
2021
-
[28]
Liu, J., Chen, Y ., Dong, Z., Wang, S., Calinon, S., Li, M., & Chen, F. (2022). Robot Cooking with Stir-Fry: Bimanual Non-Prehensile Manipulation of Semi-Fluid Objects. IEEE Robotics and Automation Letters, 7(2), 5159–5166
2022
-
[29]
Yin, H., Varava, A., & Kragic, D. (2021). Modeling, learning, percep- tion, and control methods for deformable object manipulation. Science Robotics, 6(54), 8803
2021
-
[30]
K., Pandey, M., & Rautaray, S
GM, H., Gourisaria, M. K., Pandey, M., & Rautaray, S. S. (2020). A comprehensive survey and analysis of generative models in machine learning. Computer Science Review, 38, 100285
2020
-
[31]
M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, S
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, S. (n.d.). Foundational Models Defining a New Era in Vision: A Survey and Outlook
-
[32]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y . (2020). Generative adver- sarial networks. Communications of the ACM, 63(11), 139–144
2020
-
[33]
G., & Frey, B
Makhzani, A., Shlens, J., Jaitly, N., Brain, G., Openai, I. G., & Frey, B. (2015). Adversarial Autoencoders. Elements of Dimensionality Reduction and Manifold Learning, 577–596
2015
-
[34]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
2021
-
[35]
P., & Welling, M
Kingma, D. P., & Welling, M. (2013). Auto-Encoding Variational Bayes. 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings
2013
-
[36]
Oussidi, A., & Elhassouny, A. (2018). Deep generative models: Survey. 2018 International Conference on Intelligent Systems and Computer Vision (ISCV), 1–8
2018
-
[37]
El-Kaddoury, M., Mahmoudi, A., & Himmi, M. M. (2019). Deep Generative Models for Image Generation: A Practical Comparison Be- tween Variational Autoencoders and Generative Adversarial Networks (pp. 1–8)
2019
-
[38]
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017). Re- thinking Atrous Convolution for Semantic Image Segmentation
2017
-
[39]
Kargar, E., & Kyrki, V . (2021). Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environments
2021
-
[40]
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition (pp. 770–778)
2016
-
[41]
Labelme: Image Polygonal Annotation with Python
Wada, K. Labelme: Image Polygonal Annotation with Python
-
[42]
I., Christensen, K
Liu, J., Fan, Z., Olsen, S. I., Christensen, K. H., & Kristensen, J. K. (2017). Boosting Active Contours for Weld Pool Visual Tracking in Automatic Arc Welding. IEEE Transactions on Automation Science and Engineering, 14(2), 1096–1108
2017
-
[43]
F., Caseiro, R., Martins, P., & Batista, J
Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015). High- Speed Tracking with Kernelized Correlation Filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 583–596
2015
-
[44]
Huo, S., Duan, A., Li, C., Zhou, P., Ma, W., Wang, H., & Navarro- Alarcon, D. (2022). Keypoint-Based Planar Bimanual Shaping of Deformable Linear Objects under Environmental Constraints with Hi- erarchical Action Framework. IEEE Robotics and Automation Letters, 7(2), 5222–5229
2022
-
[46]
Mahasseni, B., Lam, M., & Todorovic, S. (n.d.). Unsupervised Video Summarization with Adversarial LSTM Networks
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.