Self-Supervised Learning for Multimodal Non-Rigid 3D Shape Matching
Pith reviewed 2026-05-24 09:05 UTC · model grok-4.3
The pith
Self-supervised learning matches non-rigid 3D shapes across meshes and point clouds by pairing functional map regularisation with contrastive coupling.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By introducing a self-supervised multimodal learning strategy that combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data, the approach obtains intramodal correspondences for triangle meshes, complete point clouds, and partially observed point clouds, as well as correspondences across these data modalities, achieving state-of-the-art results on several challenging benchmark datasets even in comparison to recent supervised methods and reaching previously unseen cross-dataset generalisation ability.
What carries the argument
mesh-based functional map regularisation combined with a contrastive loss that couples mesh and point cloud data
If this is right
- The method produces intramodal correspondences for triangle meshes, complete point clouds, and partial point clouds.
- It also produces correspondences across mesh and point cloud modalities.
- Performance reaches state-of-the-art levels on multiple benchmarks, including against supervised competitors.
- Cross-dataset generalisation reaches levels not previously observed with comparable methods.
Where Pith is reading between the lines
- Raw point-cloud scans from sensors could be matched at mesh-level quality without first converting them to curated meshes.
- The same coupling idea could be tested on additional modalities such as depth images or volumetric data.
- If the contrastive term proves robust, the framework might reduce reliance on synthetic labeled datasets for training shape matchers.
Load-bearing premise
That combining mesh-based functional map regularisation with a contrastive loss coupling mesh and point cloud data will produce effective correspondences in a fully self-supervised manner without labeled data.
What would settle it
A set of standard benchmarks where the method's correspondence accuracy falls below recent supervised approaches or where cross-dataset generalisation fails to exceed prior self-supervised or supervised baselines.
Figures
read the original abstract
The matching of 3D shapes has been extensively studied for shapes represented as surface meshes, as well as for shapes represented as point clouds. While point clouds are a common representation of raw real-world 3D data (e.g. from laser scanners), meshes encode rich and expressive topological information, but their creation typically requires some form of (often manual) curation. In turn, methods that purely rely on point clouds are unable to meet the matching quality of mesh-based methods that utilise the additional topological structure. In this work we close this gap by introducing a self-supervised multimodal learning strategy that combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data. Our shape matching approach allows to obtain intramodal correspondences for triangle meshes, complete point clouds, and partially observed point clouds, as well as correspondences across these data modalities. We demonstrate that our method achieves state-of-the-art results on several challenging benchmark datasets even in comparison to recent supervised methods, and that our method reaches previously unseen cross-dataset generalisation ability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces a self-supervised multimodal strategy for non-rigid 3D shape matching. It combines mesh-based functional map regularisation with a contrastive loss that couples mesh and point cloud data to obtain correspondences for triangle meshes, complete and partial point clouds, and across modalities. The authors claim state-of-the-art performance on challenging benchmarks, surpassing recent supervised methods, and unprecedented cross-dataset generalisation.
Significance. If the experimental claims hold, the work would be significant for bridging the quality gap between topology-rich mesh matching and raw point-cloud data from sensors via fully self-supervised training. The reported cross-dataset generalisation would be a notable advance over typical supervised approaches that often overfit to specific datasets.
Simulated Author's Rebuttal
We thank the referee for their summary of our manuscript and for noting the potential significance of a self-supervised multimodal approach that bridges mesh-based and point-cloud matching while achieving strong cross-dataset generalization. No specific major comments were provided in the report, so we have no point-by-point responses to offer. The uncertainty in the recommendation appears to stem from the need to verify experimental claims, which we stand behind based on the full paper.
- No specific major comments listed, preventing targeted rebuttals.
- Only the abstract is available here, limiting our ability to address any potential detailed questions on experiments or implementation that might arise from the full manuscript.
Circularity Check
No circularity detectable
full rationale
Only the abstract is available and it contains no equations, derivations, loss formulations, or self-citations. The text describes a high-level approach (mesh-based functional map regularisation combined with contrastive loss) but supplies no technical chain that could be inspected for self-definition, fitted inputs renamed as predictions, or load-bearing self-citations. Consequently no circular step can be quoted or exhibited, and the derivation is treated as self-contained by default.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Scape: shape completion and animation of people
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Se- bastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH
-
[2]
Dpfm: Deep partial functional maps
Souhaib Attaiki, Gautam Pai, and Maks Ovsjanikov. Dpfm: Deep partial functional maps. In International Conference on 3D Vision (3DV), 2021. 2, 3, 4, 7, 8
work page 2021
-
[3]
The wave kernel signature: A quantum mechanical approach to shape analysis
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In ICCV, 2011. 2
work page 2011
-
[4]
Mina: Convex mixed-integer programming for non-rigid shape alignment
Florian Bernard, Zeeshan Khan Suri, and Christian Theobalt. Mina: Convex mixed-integer programming for non-rigid shape alignment. In CVPR, 2020. 2
work page 2020
-
[5]
Faust: Dataset and evaluation for 3d mesh registration
Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In CVPR, 2014. 5
work page 2014
-
[6]
Anisotropic diffusion descriptors
Davide Boscaini, Jonathan Masci, Emanuele Rodol `a, Michael M Bronstein, and Daniel Cremers. Anisotropic diffusion descriptors. In Computer Graphics Forum , vol- ume 35, pages 431–441. Wiley Online Library, 2016. 3
work page 2016
-
[7]
Scale-invariant heat kernel signatures for non-rigid shape recognition
Michael M Bronstein and Iasonas Kokkinos. Scale-invariant heat kernel signatures for non-rigid shape recognition. In CVPR, 2010. 2
work page 2010
-
[8]
Unsupervised deep multi-shape matching
Dongliang Cao and Florian Bernard. Unsupervised deep multi-shape matching. In ECCV, 2022. 1, 2, 3, 4, 5, 6, 7
work page 2022
-
[9]
Fi- nite elements on point based surfaces
Ulrich Clarenz, Martin Rumpf, and Alexandru C Telea. Fi- nite elements on point based surfaces. In PBG, pages 201– 211, 2004. 3
work page 2004
-
[10]
Shrec’16: Partial matching of deformable shapes
Luca Cosmo, Emanuele Rodola, Michael M Bronstein, An- drea Torsello, Daniel Cremers, and Y Sahillioglu. Shrec’16: Partial matching of deformable shapes. Proc. 3DOR , 2(9):12, 2016. 6
work page 2016
-
[11]
Texture transfer during shape transformation
Huong Quynh Dinh, Anthony Yezzi, and Greg Turk. Texture transfer during shape transformation. ACM Transactions on Graphics (ToG), 24(2):289–310, 2005. 1
work page 2005
-
[12]
Deep orientation-aware functional maps: Tackling symme- try issues in shape matching
Nicolas Donati, Etienne Corman, and Maks Ovsjanikov. Deep orientation-aware functional maps: Tackling symme- try issues in shape matching. In CVPR, 2022. 2, 4
work page 2022
-
[13]
Deep geometric functional maps: Robust feature learning for shape correspondence
Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. In CVPR, 2020. 2, 5, 6, 7, 8
work page 2020
-
[14]
Divergence-free shape correspondence by deformation
Marvin Eisenberger, Zorah L ¨ahner, and Daniel Cremers. Divergence-free shape correspondence by deformation. In Computer Graphics Forum, volume 38, pages 1–12. Wiley Online Library, 2019. 2
work page 2019
-
[15]
Smooth shells: Multi-scale shape registration with functional maps
Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registration with functional maps. In CVPR, 2020. 2, 6
work page 2020
-
[16]
Neuromorph: Unsupervised shape interpola- tion and correspondence in one go
Marvin Eisenberger, David Novotny, Gael Kerchenbaum, Patrick Labatut, Natalia Neverova, Daniel Cremers, and An- drea Vedaldi. Neuromorph: Unsupervised shape interpola- tion and correspondence in one go. In CVPR, 2021. 1, 4
work page 2021
-
[17]
Deep shells: Unsupervised shape corre- spondence with optimal transport
Marvin Eisenberger, Aysim Toker, Laura Leal-Taix ´e, and Daniel Cremers. Deep shells: Unsupervised shape corre- spondence with optimal transport. NIPS, 2020. 1, 2, 6
work page 2020
-
[18]
Isometric multi-shape matching
Maolin Gao, Zorah Lahner, Johan Thunberg, Daniel Cre- mers, and Florian Bernard. Isometric multi-shape matching. In CVPR, 2021. 2
work page 2021
-
[19]
3d-coded: 3d cor- respondences by deep deformation
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d-coded: 3d cor- respondences by deep deformation. In ECCV, 2018. 1, 2, 3, 6, 8
work page 2018
-
[20]
Unsupervised cycle- consistent deformation for shape matching
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. Unsupervised cycle- consistent deformation for shape matching. In Computer Graphics Forum. Wiley Online Library, 2019. 1, 3
work page 2019
-
[21]
Unsupervised learning of dense shape correspondence
Oshri Halimi, Or Litany, Emanuele Rodola, Alex M Bron- stein, and Ron Kimmel. Unsupervised learning of dense shape correspondence. In CVPR, 2019. 2, 6
work page 2019
-
[22]
Simulated annealing for 3d shape correspondence
Benjamin Holzschuh, Zorah L ¨ahner, and Daniel Cremers. Simulated annealing for 3d shape correspondence. In 2020 International Conference on 3D Vision (3DV), 2020. 2
work page 2020
-
[23]
Functional map networks for analyzing and exploring large shape col- lections
Qixing Huang, Fan Wang, and Leonidas Guibas. Functional map networks for analyzing and exploring large shape col- lections. ACM Transactions on Graphics (ToG), 33(4):1–11,
-
[24]
Consistent zoomout: Efficient spectral map synchronization
Ruqi Huang, Jing Ren, Peter Wonka, and Maks Ovsjanikov. Consistent zoomout: Efficient spectral map synchronization. In Computer Graphics Forum. Wiley Online Library, 2020. 2
work page 2020
-
[25]
Categorical reparametrization with gumble-softmax
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In ICLR, 2017. 4
work page 2017
-
[26]
Dpc: Unsupervised deep point correspondence via cross and self construction
Itai Lang, Dvir Ginzburg, Shai Avidan, and Dan Raviv. Dpc: Unsupervised deep point correspondence via cross and self construction. In International Conference on 3D Vision (3DV), 2021. 1, 2, 3, 6
work page 2021
-
[27]
The quadratic assignment problem
Eugene L Lawler. The quadratic assignment problem. Man- agement science, 9(4):586–599, 1963. 3
work page 1963
-
[28]
Deep functional maps: Structured prediction for dense shape correspondence
Or Litany, Tal Remez, Emanuele Rodola, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In ICCV, 2017. 1, 2, 6
work page 2017
-
[29]
Fully spectral partial shape matching
Or Litany, Emanuele Rodol `a, Alexander M Bronstein, and Michael M Bronstein. Fully spectral partial shape matching. In Computer Graphics Forum. Wiley Online Library, 2017. 2, 7
work page 2017
-
[30]
WTFM Layer: An Effective Map Ex- tractor for Unsupervised Shape Correspondence
Shengjun Liu, Haojun Xu, Dong-Ming Yan, Ling Hu, Xinru Liu, and Qinsong Li. WTFM Layer: An Effective Map Ex- tractor for Unsupervised Shape Correspondence. Computer Graphics Forum, 2022. 2, 4
work page 2022
-
[31]
Smpl: A skinned multi- person linear model
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. ACM Transactions on Graphics (ToG), 34(6):1–16, 2015. 1
work page 2015
-
[32]
Correspondence learning via linearly- invariant embedding
Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov. Correspondence learning via linearly- invariant embedding. NIPS, 2020. 2, 3, 6
work page 2020
-
[33]
Geodesic convolutional neural net- works on riemannian manifolds
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural net- works on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision work- shops, 2015. 1 9
work page 2015
-
[34]
Shrec 2019: Matching humans with dif- ferent connectivity
Simone Melzi, Riccardo Marin, Emanuele Rodol `a, Umberto Castellani, Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. Shrec 2019: Matching humans with dif- ferent connectivity. In Eurographics Workshop on 3D Object Retrieval, 2019. 1, 5
work page 2019
-
[35]
Zoomout: Spectral upsampling for efficient shape correspondence
Simone Melzi, Jing Ren, Emanuele Rodola, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov. Zoomout: Spectral upsampling for efficient shape correspondence. arXiv preprint arXiv:1904.07865, 2019. 2, 3, 6
-
[36]
Learning latent permutations with gumbel- sinkhorn networks
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel- sinkhorn networks. In ICLR, 2018. 4
work page 2018
-
[37]
Informative descrip- tor preservation via commutativity for shape matching
Dorian Nogneng and Maks Ovsjanikov. Informative descrip- tor preservation via commutativity for shape matching. In Computer Graphics Forum. Wiley Online Library, 2017. 2
work page 2017
-
[38]
Functional maps: a flexible representation of maps between shapes
Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (ToG), 31(4):1–11, 2012. 2, 3
work page 2012
-
[39]
Fast sinkhorn filters: Using matrix scaling for non-rigid shape correspondence with functional maps
Gautam Pai, Jing Ren, Simone Melzi, Peter Wonka, and Maks Ovsjanikov. Fast sinkhorn filters: Using matrix scaling for non-rigid shape correspondence with functional maps. In CVPR, 2021. 3
work page 2021
-
[40]
Computing discrete min- imal surfaces and their conjugates
Ulrich Pinkall and Konrad Polthier. Computing discrete min- imal surfaces and their conjugates. Experimental mathemat- ics, 2(1):15–36, 1993. 3
work page 1993
-
[41]
Pointnet++: Deep hierarchical feature learning on point sets in a metric space
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NIPS, 2017. 2
work page 2017
-
[42]
Structured regularization of functional map compu- tations
Jing Ren, Mikhail Panine, Peter Wonka, and Maks Ovs- janikov. Structured regularization of functional map compu- tations. In Computer Graphics Forum. Wiley Online Library,
-
[43]
Continuous and orientation-preserving correspon- dences via functional maps
Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovs- janikov. Continuous and orientation-preserving correspon- dences via functional maps. ACM Transactions on Graphics (ToG), 37:1–16, 2018. 2, 5, 6
work page 2018
-
[44]
Partial functional cor- respondence
Emanuele Rodol `a, Luca Cosmo, Michael M Bronstein, An- drea Torsello, and Daniel Cremers. Partial functional cor- respondence. In Computer Graphics Forum. Wiley Online Library, 2017. 2, 7
work page 2017
-
[45]
A scalable combinatorial solver for elastic geomet- rically consistent 3d shape matching
Paul Roetzer, Paul Swoboda, Daniel Cremers, and Florian Bernard. A scalable combinatorial solver for elastic geomet- rically consistent 3d shape matching. In CVPR, 2022. 2
work page 2022
-
[46]
Unsupervised deep learning for structured shape matching
Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovs- janikov. Unsupervised deep learning for structured shape matching. In ICCV, 2019. 2, 3, 5, 6
work page 2019
-
[47]
Recent advances in shape correspon- dence
Yusuf Sahillio ˘glu. Recent advances in shape correspon- dence. The Visual Computer, 36(8):1705–1721, 2020. 2
work page 2020
-
[48]
Bending graphs: Hierarchical shape matching using gated optimal transport
Mahdi Saleh, Shun-Cheng Wu, Luca Cosmo, Nassir Navab, Benjamin Busam, and Federico Tombari. Bending graphs: Hierarchical shape matching using gated optimal transport. In CVPR, 2022. 4
work page 2022
-
[49]
Shot: Unique signatures of histograms for surface and tex- ture description
Samuele Salti, Federico Tombari, and Luigi Di Stefano. Shot: Unique signatures of histograms for surface and tex- ture description. Computer Vision and Image Understand- ing, 125:251–264, 2014. 2
work page 2014
-
[50]
Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge. Medi- cal image analy...
work page 2017
-
[51]
Weakly supervised deep functional maps for shape matching
Abhishek Sharma and Maks Ovsjanikov. Weakly supervised deep functional maps for shape matching. NIPS, 2020. 2, 3, 5, 6
work page 2020
-
[52]
arXiv preprint arXiv:2012.00888 (2020)
Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. arXiv preprint arXiv:2012.00888, 2020. 2, 3, 4, 5
-
[53]
A laplacian for nonman- ifold triangle meshes
Nicholas Sharp and Keenan Crane. A laplacian for nonman- ifold triangle meshes. In Computer Graphics Forum, vol- ume 39, pages 69–80. Wiley Online Library, 2020. 3
work page 2020
-
[54]
Concerning nonnegative matrices and doubly stochastic matrices
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343–348, 1967. 4
work page 1967
-
[55]
Implicit field supervision for robust non-rigid shape match- ing
Ramana Sundararaman, Gautam Pai, and Maks Ovsjanikov. Implicit field supervision for robust non-rigid shape match- ing. In ECCV, 2022. 2, 3, 6, 8
work page 2022
-
[56]
Registration of 3d point clouds and meshes: A survey from rigid to nonrigid
Gary KL Tam, Zhi-Quan Cheng, Yu-Kun Lai, Frank C Lang- bein, Yonghuai Liu, David Marshall, Ralph R Martin, Xian- Fang Sun, and Paul L Rosin. Registration of 3d point clouds and meshes: A survey from rigid to nonrigid. IEEE transac- tions on visualization and computer graphics , 19(7):1199– 1217, 2012. 1, 2, 3
work page 2012
-
[57]
A survey on shape correspondence
Oliver Van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. A survey on shape correspondence. In Computer Graphics Forum, 2011. 1, 2
work page 2011
-
[58]
Learning from synthetic humans
Gul Varol, Javier Romero, Xavier Martin, Naureen Mah- mood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 2, 5, 7
work page 2017
-
[59]
Matthias Vestner, Roee Litman, Emanuele Rodola, Alex Bronstein, and Daniel Cremers. Product manifold filter: Non-rigid shape correspondence via kernel density estima- tion in the product space. In CVPR, 2017. 2, 3
work page 2017
-
[60]
Dynamic graph cnn for learning on point clouds
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (ToG), 38(5):1–12, 2019. 2
work page 2019
-
[61]
Geometrically consistent elastic match- ing of 3d shapes: A linear programming solution
Thomas Windheuser, Ulrich Schlickewei, Frank R Schmidt, and Daniel Cremers. Geometrically consistent elastic match- ing of 3d shapes: A linear programming solution. In ICCV,
-
[62]
Pointcontrast: Unsupervised pre- training for 3d point cloud understanding
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. In ECCV, 2020. 5
work page 2020
-
[63]
Corrnet3d: Unsupervised end-to-end learning of dense correspondence for 3d point clouds
Yiming Zeng, Yue Qian, Zhiyu Zhu, Junhui Hou, Hui Yuan, and Ying He. Corrnet3d: Unsupervised end-to-end learning of dense correspondence for 3d point clouds. InCVPR, 2021. 1, 2, 3, 4, 6, 8 10
work page 2021
-
[64]
Next, we explain de- tails on the modifications of our approach for partial shape matching
Supplementary Document In this supplementary document we first provide more implementation details of our method. Next, we explain de- tails on the modifications of our approach for partial shape matching. Afterwards, we provide more ablative experi- ments to demonstrate the advantages of our method. Even- tually, we show additional qualitative results of o...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.