REVIEW 3 major objections 4 minor 95 references
Self-Supervised and Generalizable Tokenization for CLIP-Based 3D Understanding
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that tokenization, not the backbone, is the bottleneck for transferring frozen 2D vision-language models to 3D point clouds, and that superpoint-guided, scale-normalized tokens close the gap without labels.
desk verdict Solid incremental tokenizer paper; the cross-domain table is the real evidence, and the part-segmentation metric concern is likely a non-issue once you check the standard ShapeNetPart protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the superpoint-constrained, scale-normalized token. Superpoints are obtained by cutting the cloud into coherent surface patches with $\ell_0$-cut pursuit over normals and eigenvalue-based descriptors (linearity, planarity, scatterness); a weighted farthest-point sampling balances geometric coverage against uniformity across superpoints; each patch gathers points within a radius $r$ that share the same superpoint label; and relative offsets are divided by $r$, making tokens insensitive to dataset-specific coordinate scales. Self-supervised training runs a teacher-student loop with masked modeling, spatially constrained Sinkhorn k-means that produces pseudo-assignments, and a two-level distillation that aligns 3D tokens with multi-view CLIP features, followed by a superpoint-aware feature propagation to recover point-level predictions. The machinery's job is to make frozen CLIP's 2D patch embeddings compatible with arbitrary 3D geometry.
What would settle it
Run S4Token on ScanNet without using the dataset's provided superpoint annotations, computing superpoints instead with $\ell_0$-cut pursuit from normals and eigenvalue descriptors; if zero-shot mIoU falls from 18.9 back toward the kNN+RPN level of 18.1, the semantic-superpoint prior is not doing the claimed work in annotation-free deployment.
Extended reading notes
Core claim
The central claim is that tokenization, not the transformer backbone, is the bottleneck when adapting VLMs to 3D point clouds, and that the right token design lets a frozen CLIP encoder do open-vocabulary 3D understanding. The paper reports that combining superpoint-based grouping with coordinate scale normalization produces tokens that are semantically informed and scale-invariant, and that this tokenizer, trained self-supervised by masked point modeling, cluster-guided assignment, and cross-modal distillation, transfers from object-level data to real indoor scenes without any fine-tuning. Concretely, S4Token reaches 72.3% mIoU on zero-shot ShapeNetPart part segmentation, outperforming the previous best by 8.4 absolute points, and improves zero-shot ScanNet and S3DIS segmentation by +10.2 and +12.4 mIoU over the kNN tokenizer. In label-efficient fine-tuning of only the tokenizer and task head, it also reaches 85.4% class mIoU and 87.3% instance mIoU on ShapeNetPart and 79.3% mAcc / 72.6% mIoU on S3DIS Area 5.
Load-bearing premise
The load-bearing premise is that automatically cutting a point cloud into coherent surface patches (superpoints) yields patches that match semantic parts in every target domain, and that such patches are available without manual labels; where the cut is wrong for small, fragmented, or noisy clouds, the claimed advantage shrinks to the modest gain from scale normalization alone.
Editorial extensions
If this is right
- If the claim holds, any frozen CLIP-style vision-language model can be given 3D understanding by swapping in this tokenizer, without retraining the backbone.
- Scale normalization plus superpoint grouping should transfer across sensors and scene sizes; the paper reports gains on ScanNet and S3DIS after training only on ShapeNet.
- Open-vocabulary part segmentation reaches 72.3% mIoU on ShapeNetPart with no point-level labels, implying fine-grained 3D part understanding no longer requires annotation.
- The tokenizer is also effective when only tokenizer and task head are fine-tuned, reaching 87.3% instance mIoU on ShapeNetPart and 72.6% mIoU on S3DIS Area 5.
- The two self-supervised objectives are complementary: joint training with both assignment and distillation losses is better than either alone, per the ablation.
Reading between the lines
- Beyond the paper, the same tokenizer could likely be paired with other frozen backbones or with LiDAR-style outdoor clouds, since the radius normalization removes a major source of domain mismatch for scenes of very different scales.
- A testable extension is to make the over-segmentation granularity adaptive to object size; if small-part handling is the limiting factor, this should raise the lowest per-category scores (68.3% for Bag, 69.8% for Earphone) on ShapeNetPart.
- The strongest cross-domain experiment uses ScanNet's provided superpoint annotations rather than superpoints computed from raw geometry, so the strictly annotation-free case still needs the automatic over-segmentation to hold up on real scans.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes S4Token, a 3D point tokenizer designed to replace kNN- or ball-query-based tokenization for frozen CLIP backbones. The tokenizer combines superpoint extraction via ℓ0-cut pursuit, superpoint-constrained weighted farthest-point sampling, and radius-based coordinate normalization, then is pretrained self-supervised with masked point modeling, cluster-guided assignment, and two-level cross-modal distillation to CLIP image features. Experiments cover a full-training tokenizer study on ShapeNetPart, cross-dataset zero-shot semantic segmentation from ShapeNet to ScanNet/S3DIS, zero-shot part segmentation on ShapeNetPart, zero-shot classification on ModelNet40/ScanObjectNN, and ablations. The central cross-domain result is that relative position normalization (RPN) improves the kNN tokenizer by large margins on ScanNet and S3DIS, with additional gains from superpoint grouping; the headline part-segmentation result is 72.3% mIoU, claimed as a +8.4 gain over PartDistill(Pre).
Significance. If the results hold, the paper makes a useful modular contribution: it isolates the tokenizer as a key factor in transferring frozen 2D VLMs to 3D, and the ablation separating kNN, ball query, superpoint grouping, and scale normalization is informative. The consistent cross-dataset gains from RPN are the strongest evidence for the scale-normalization thesis, and the proposed superpoint-level feature propagation is a simple, training-free way to recover dense predictions. The main claims are limited by two concerns: the headline part-segmentation metric is not defined in the comparison table, and the ScanNet cross-dataset experiment relies on dataset-provided superpoints rather than the fully automatic pipeline used elsewhere. The paper does not release code, so reproducibility rests on the written descriptions, which contain at least one formula-level inconsistency in the sampling criterion.
major comments (3)
- [Sec. 4.2 / Table 3 vs. Sec. 4.5 / Table 6] The headline '72.3% mIoU, a +8.4% absolute gain over the previous best' is not verifiable as written because Table 3 never defines whether mIoU is class-level (mIoUC) or instance-level (mIoUI). Section 4.5 reports that the same Ltotal configuration gives mIoUC=55.8 and mIoUI=72.3, so the 72.3 value in Table 3 is the instance-level metric. The paper must state explicitly which protocol applies to S4Token and to each baseline, and should report both metrics if the baselines are ambiguous. If the comparison is actually against class-level mIoUC, the claimed +8.4 gain reverses to an 8.1-point deficit relative to PartDistill(Pre).
- [Sec. 3.2 / Eq. (2)] The inverse-frequency weight w_i is defined as larger for points in smaller superpoints, but the selection criterion multiplies the distance by w_i^{-γ}. For γ>0 this penalizes small superpoints, which contradicts the text's claim that 'As γ → 1, the selection increasingly favors points belonging to smaller superpoints' and the description of Figure B. Either the exponent sign in the criterion or the definition of w_i is wrong; please correct the formula and confirm which variant was actually implemented, since this is the core sampling mechanism of the method.
- [Sec. 4.1 / Table 2] The ScanNet cross-dataset experiment uses ScanNet's provided superpoint annotations instead of running the ℓ0-cut pursuit pipeline that is used for ShapeNet and S3DIS. Because the paper advertises annotation-free generalization, the +10.2 mIoU gain over kNN on ScanNet is not a fully annotation-free comparison. Please either run the same automatic superpoint extraction on ScanNet, or clearly state that deployment relies on dataset-provided superpoints, and quantify the sensitivity of the result to superpoint quality if possible.
minor comments (4)
- [Table 3 / Sec. 4.2] The term 'mIoU' should be defined as mIoUC or mIoUI in the table caption and in the text, matching the notation introduced in Section 4.5.
- [Sec. 3.2 / Eq. (3)] There is a typo in the offset notation in Eq. (3): 'Δptj t' should be written as Δp_j^t = p_j^t - p_t for clarity.
- [Fig. 1 / Sec. 3.4] The legend symbols in Figure 1 (trained, frozen, EMA-updated) are not explained in the main text; please add a full legend in the caption.
- [Abstract] The phrase 'without requiring fine-tuning' is too broad; the tokenizer is pretrained and then used frozen, but the abstract should specify that no fine-tuning is applied on the target dataset rather than implying the method requires no training at all.
Circularity Check
Headline ShapeNetPart number is the paper's own mIoUI relabeled as mIoU, so the +8.4 gain is a metric substitution; the tokenizer derivation itself is otherwise self-contained.
-
self definitional
[Sec 4.2 (Table 3) vs Sec 4.5 (Tables 6-7)]
"Tab. 3 lists 'S4Token (ours) 72.3' under 'mIoU', and Sec 4.2 states 'S4Token achieves 72.3% mIoU, a +8.4% absolute gain over the previous best, i.e., PartDistill(Pre).' Sec 4.5 states 'The joint training strategy achieves the best performance, with a class-level mIoU of 55.8% and an instance-level mIoU of 72.3%.'"
Table 3 never defines its 'mIoU' column. Tables 6-7 define the same Ltotal configuration's 72.3 as mIoUI (instance-level) and 55.8 as mIoUC (class-level). The headline 72.3 is therefore not a new class-level prediction: it is numerically identical to the paper's own instance-level result, relabeled as 'mIoU'. The baselines (PointCLIPv2 49.5, PartDistill 63.9, GeoZe 57.4) use the standard class-level protocol, so the '+8.4 absolute gain' compares mIoUI against mIoUC. Under the comparable class-level metric, S4Token's 55.8 is 8.1 points below PartDistill(Pre), reversing the claimed advantage. The central quantitative 'prediction' thus reduces by definition to a metric-substitution rather than a derived empirical gain.
full rationale
Apart from the headline metric equivocation, the derivation chain is not circular. Eq. (3) normalizes coordinates by a radius r estimated from input centroid spacings; the superpoint grouping is an external graph-cut procedure; and the losses in Eqs. (7)-(9) train against CLIP features without fitting target-domain labels or test-set statistics. The cross-dataset transfers (Table 2), semantic segmentation (Table 4), and classification (Table 5) are self-contained zero-shot evaluations. The self-citations (GeoZe, PointCLIPv2-derived pipelines) are used as baselines or feature-extraction protocols, not as load-bearing uniqueness arguments. The one definitional problem is the ShapeNetPart headline: the 72.3 'mIoU' in Table 3 is the paper's own mIoUI per Tables 6-7, so the claimed +8.4 over class-level baselines is an apples-to-oranges metric substitution. This makes the strongest claimed prediction partially definitional, while the tokenizer method itself has independent content.
Assumptions & free parameters
free parameters (4)
- radius scale alpha =
1
- WFPS exponent gamma =
0.1
- number of clusters K =
24
- distillation weights lambda_l, lambda_g =
0.5, 0.5
assumptions (4)
- domain assumption CLIP features extracted from multi-view renderings following Open3DIS/PointCLIPv2 provide a reliable semantic teacher for 3D superpoints.
- domain assumption l0-cut superpoint segmentation yields regions aligned with semantic parts for all target datasets.
- domain assumption A frozen CLIP ViT can process 3D tokens with relative positional encoding and retain its semantic alignment.
- standard math FPS, ball query, and Sinkhorn clustering behave as expected on irregular point distributions.
Cite this review
Pith. "Pith review of Self-Supervised and Generalizable Tokenization for CLIP-Based 3D Understanding." pith.science (2026). https://pith.science/paper/ERULEXM5
@misc{pith2026250518819,
author = {Pith},
title = {Pith review of: Self-Supervised and Generalizable Tokenization for CLIP-Based 3D Understanding},
year = {2026},
howpublished = {\url{https://pith.science/paper/ERULEXM5}},
note = {Machine review of arXiv:2505.18819}
}
read the original abstract
Vision-language models like CLIP can offer a promising foundation for 3D scene understanding when extended with 3D tokenizers. However, standard approaches, such as k-nearest neighbor or radius-based tokenization, struggle with cross-domain generalization due to sensitivity to dataset-specific spatial scales. We present a universal 3D tokenizer designed for scale-invariant representation learning with a frozen CLIP backbone. We show that combining superpoint-based grouping with coordinate scale normalization consistently outperforms conventional methods through extensive experimental analysis. Specifically, we introduce S4Token, a tokenization pipeline that produces semantically-informed tokens regardless of scene scale. Our tokenizer is trained without annotations using masked point modeling and clustering-based objectives, along with cross-modal distillation to align 3D tokens with 2D multi-view image features. For dense prediction tasks, we propose a superpoint-level feature propagation module to recover point-level detail from sparse tokens.
Figures
Reference graph
Works this paper leans on
-
[1]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pages 4015–4026, 2023. 1
2023
-
[2]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00...
arXiv 2024
-
[3]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 1, 2, 3
2021
-
[4]
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9650–9660, 2021. 1
2021
-
[5]
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Laba...
2023
-
[6]
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. 1
arXiv 2025
-
[7]
Perla: Perceptive 3d language assistant
Guofeng Mei, Wei Lin, Luigi Riz, Yujiao Wu, Fabio Poiesi, and Yiming Wang. Perla: Perceptive 3d language assistant. In CVPR, 2025. 1
2025
-
[8]
Shapesplat: A large-scale dataset of gaussian splats and their self-supervised pretraining
Qi Ma, Yue Li, Bin Ren, Nicu Sebe, Ender Konukoglu, Theo Gevers, Luc Van Gool, and Danda Pani Paudel. Shapesplat: A large-scale dataset of gaussian splats and their self-supervised pretraining. In 3DV, 2024. 1
2024
Show all 95 references
-
[9]
Scenesplat: Gaussian splatting-based scene understanding with vision-language pretraining
Yue Li, Qi Ma, Runyi Yang, Huapeng Li, Mengjiao Ma, Bin Ren, Nikola Popovic, Nicu Sebe, Ender Konukoglu, Theo Gevers, et al. Scenesplat: Gaussian splatting-based scene understanding with vision-language pretraining. arXiv preprint arXiv:2503.18052, 2025. 1
2025 arXiv
-
[10]
Unsupervised deep probabilistic approach for partial point cloud registration
Guofeng Mei, Hao Tang, Xiaoshui Huang, Weijie Wang, Juan Liu, Jian Zhang, Luc Van Gool, and Qiang Wu. Unsupervised deep probabilistic approach for partial point cloud registration. In CVPR, pages 13611–13620, 2023. 1, 7, 8
2023
-
[11]
Pvafn: Point-voxel attention fusion network with multi-pooling enhancing for 3d object detection
Yidi Li, Jiahao Wen, Rui Gong, Bin Ren, Wenhao Li, Chen Cheng, Hong Liu, and Nicu Sebe. Pvafn: Point-voxel attention fusion network with multi-pooling enhancing for 3d object detection. Expert Systems with Applications, page 127608, 2025. 1
2025
-
[12]
Zero-shot point cloud registration
Weijie Wang, Guofeng Mei, Bin Ren, Xiaoshui Huang, Fabio Poiesi, Luc Van Gool, Nicu Sebe, and Bruno Lepri. Zero-shot point cloud registration. arXiv preprint arXiv:2312.03032, 2023. 1
2023 arXiv
-
[13]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017. 1 10
2017
-
[14]
Masked jigsaw puzzle: A versatile position embedding for vision transformers
Bin Ren, Yahui Liu, Yue Song, Wei Bi, Rita Cucchiara, Nicu Sebe, and Wei Wang. Masked jigsaw puzzle: A versatile position embedding for vision transformers. In CVPR, pages 20382– 20391, 2023. 1, 2
2023
-
[15]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. 1
2020
-
[16]
Point transformer
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In CVPR, pages 16259–16268, 2021. 1, 6, 2
2021
-
[17]
Point transformer v2: Grouped vector attention and partition-based pooling
Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. NeurIPS, 35:33330–33342, 2022. 1
2022
-
[18]
Point transformer v3: Simpler faster stronger
Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In CVPR, pages 4840–4851, 2024. 1
2024
-
[19]
Bringing masked autoencoders explicit contrastive properties for point cloud self-supervised learning
Bin Ren, Guofeng Mei, Danda Pani Paudel, Weijie Wang, Yawei Li, Mengyuan Liu, Rita Cucchiara, Luc Van Gool, and Nicu Sebe. Bringing masked autoencoders explicit contrastive properties for point cloud self-supervised learning. In ACCV, 2024. 1, 2, 6, 4
2024
-
[20]
Point-dae: Denoising autoencoders for self-supervised point cloud learning
Yabin Zhang, Jiehong Lin, Ruihuang Li, Kui Jia, and Lei Zhang. Point-dae: Denoising autoencoders for self-supervised point cloud learning. arXiv preprint arXiv:2211.06841, 2022. 1, 4
2022 arXiv
-
[21]
Geomae: Masked geometric target prediction for self-supervised point cloud pre-training
Xiaoyu Tian, Haoxi Ran, Yue Wang, and Hang Zhao. Geomae: Masked geometric target prediction for self-supervised point cloud pre-training. In CVPR, pages 13570–13580, 2023. 1
2023
-
[22]
Masked autoencoders for point cloud self-supervised learning
Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In ECCV, pages 604–621. Springer, 2022. 1, 2, 3, 5, 4
2022
-
[24]
Bootstrap your own latent: A new approach to self-supervised learning
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS,
-
[25]
Momentum contrast for unsupervised visual representation learning
Kaiming He, Haoqi Fan, et al. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729–9738, 2020. 2
2020
-
[26]
Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training. NeurIPS, 35:27061–27074, 2022. 2, 3
2022
-
[27]
Masked autoencoders are scalable vision learners
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 2, 3
2022
-
[28]
Shapenet: An information-rich 3d model repository
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. 2, 6, 7, 8, 3
2015 arXiv
-
[29]
Scannet: Richly-annotated 3d reconstructions of indoor scenes
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pages 5828–5839, 2017. 2, 6, 7, 3, 5
2017
-
[30]
Con- trast with reconstruct: Contrastive 3d representation learning guided by generative pretraining
Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Con- trast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In ICML, pages 28223–28243. PMLR, 2023. 2, 4 11
2023
-
[31]
Frozen clip transformer is an efficient point cloud encoder
Xiaoshui Huang, Zhou Huang, Sheng Li, Wentao Qu, Tong He, Yuenan Hou, Yifan Zuo, and Wanli Ouyang. Frozen clip transformer is an efficient point cloud encoder. In AAAI, volume 38, pages 2382–2390, 2024. 2, 3, 4
2024
-
[32]
Pimae: Point cloud and image interactive masked autoencoders for 3d object detection
Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. In CVPR, pages 5291–5301, 2023. 2, 3
2023
-
[33]
Pointmamba: A simple state space model for point cloud analysis
Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. In NeurIPS,
-
[34]
Pix4point: image pretrained standard transformers for 3d point cloud understanding
Guocheng Qian, Abdullah Hamdi, Xingdi Zhang, and Bernard Ghanem. Pix4point: image pretrained standard transformers for 3d point cloud understanding. In 3DV, pages 1280–1290. IEEE, 2024. 2, 4
2024
-
[35]
Can we solve 3d vision tasks starting from a 2d vision transformer? arXiv preprint arXiv:2209.07026, 2022
Yi Wang, Zhiwen Fan, Tianlong Chen, Hehe Fan, and Zhangyang Wang. Can we solve 3d vision tasks starting from a 2d vision transformer? arXiv preprint arXiv:2209.07026, 2022. 2
2022 arXiv
-
[36]
Pointcontrast: Unsupervised pre-training for 3d point cloud understanding
Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 5...
2020
-
[37]
Spatio-temporal self-supervised representation learning for 3d point clouds
Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In CVPR, pages 6535–6545, 2021. 2
2021
-
[38]
Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds
Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In CVPR, 2020. 2
2020
-
[39]
Point discrimi- native learning for data-efficient 3d point cloud analysis
Fayao Liu, Guosheng Lin, Chuan-Sheng Foo, Chaitanya K Joshi, and Jie Lin. Point discrimi- native learning for data-efficient 3d point cloud analysis. In 3DV, pages 42–51. IEEE, 2022. 2
2022
-
[40]
Data augmentation-free unsupervised learning for 3d point cloud understanding
Guofeng Mei, Cristiano Saltori, Fabio Poiesi, Jian Zhang, Elisa Ricci, Nicu Sebe, and Qiang Wu. Data augmentation-free unsupervised learning for 3d point cloud understanding. BMVC,
-
[41]
Unsupervised point cloud representation learning by clustering and neural rendering
Guofeng Mei, Cristiano Saltori, Elisa Ricci, Nicu Sebe, Qiang Wu, Jian Zhang, and Fabio Poiesi. Unsupervised point cloud representation learning by clustering and neural rendering. IJCV, pages 1–19, 2024. 2
2024
-
[42]
Pointclustering: Unsupervised point cloud pre-training using transformation invariance in clustering
Fuchen Long, Ting Yao, Zhaofan Qiu, Lusong Li, and Tao Mei. Pointclustering: Unsupervised point cloud pre-training using transformation invariance in clustering. In CVPR, pages 21824– 21834, 2023. 2, 8
2023
-
[43]
Gd-mae: generative decoder for mae pre-training on lidar point clouds
Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: generative decoder for mae pre-training on lidar point clouds. In CVPR, pages 9403–9414, 2023. 3
2023
-
[44]
Pointgpt: Auto- regressively generative pre-training from point clouds
Guangyan Chen, Meiling Wang, Yi Yang, Kai Yu, Li Yuan, and Yufeng Yue. Pointgpt: Auto- regressively generative pre-training from point clouds. NeurIPS, 36, 2024. 3, 4
2024
-
[45]
Point cloud pre-training with diffusion models
Xiao Zheng, Xiaoshui Huang, Guofeng Mei, Yuenan Hou, Zhaoyang Lyu, Bo Dai, Wanli Ouyang, and Yongshun Gong. Point cloud pre-training with diffusion models. In CVPR, pages 22935–22945, 2024. 3
2024
-
[46]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840–6851, 2020. 3
2020
-
[47]
Spatio-temporal graph diffusion for text-driven human motion generation
Chang Liu, Mengyi Zhao, Bin Ren, Mengyuan Liu, Nicu Sebe, et al. Spatio-temporal graph diffusion for text-driven human motion generation. In BMVC, pages 722–729, 2023. 3 12
2023
-
[48]
Denoising diffusion probabilistic models for action-conditioned 3d motion generation
Mengyi Zhao, Mengyuan Liu, Bin Ren, Shuling Dai, and Nicu Sebe. Denoising diffusion probabilistic models for action-conditioned 3d motion generation. In ICASSP, pages 4225–4229. IEEE, 2024. 3
2024
-
[49]
Efficiently modeling long sequences with structured state spaces
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022. 3
2022
-
[50]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 3
2021 arXiv
-
[51]
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, pages 2790–2799. PMLR, 2019. 3
2019
-
[52]
Adaptformer: Adapting vision transformers for scalable visual recognition
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS, 35:16664–16678, 2022. 3
2022
-
[53]
Visual prompt tuning
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 3
2022
-
[54]
Exploring sparse visual prompt for domain adaptive dense prediction
Senqiao Yang, Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, Yulu Gan, Zehui Chen, and Shanghang Zhang. Exploring sparse visual prompt for domain adaptive dense prediction. In AAAI, volume 38, pages 16334–16342, 2024. 3
2024
-
[55]
Point-peft: Parameter-efficient fine-tuning for 3d pre-trained models
Yiwen Tang, Ray Zhang, Zoey Guo, Xianzheng Ma, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Point-peft: Parameter-efficient fine-tuning for 3d pre-trained models. In AAAI, volume 38, pages 5171–5179, 2024. 3
2024
-
[56]
Gaprompt: Geometry-aware point cloud prompt for 3d vision model
Zixiang Ai, Zichen Liu, Yuanhang Lei, Zhenyu Cui, Xu Zou, and Jiahuan Zhou. Gaprompt: Geometry-aware point cloud prompt for 3d vision model. ICML, 2025. 3
2025
-
[57]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,
-
[58]
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 3
1907 arXiv
-
[59]
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 3
2019
-
[60]
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347–10357. PMLR, 2021. 3
2021
-
[61]
Ast: Audio spectrogram transformer
Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778, 2021. 3
2021 arXiv
-
[62]
Ssast: Self-supervised audio spectrogram transformer
Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In AAAI, volume 36, pages 10699–10709, 2022. 3
2022
-
[63]
Imagebind: One embedding space to bind them all
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, pages 15180–15190, 2023. 3
2023
-
[64]
Pointclip: Point cloud understanding by clip
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, pages 8552–8562,
-
[65]
Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In CVPR, pages 2639–2650, 2023. 3, 6, 7, 5
2023
-
[66]
P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting
Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. NeurIPS, 35:14388–14402,
-
[67]
Image2point: 3d point-cloud understand- ing with 2d image pretrained models
Chenfeng Xu, Shijia Yang, Tomer Galanti, Bichen Wu, Xiangyu Yue, Bohan Zhai, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understand- ing with 2d image pretrained models. In European Conference on Computer Vision, pages 638–656. Spri...
2022
-
[68]
Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:2212.08320, 2022
Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? arXiv preprint arXiv:2212.08320, 2022. 3, 4
2022 arXiv
-
[69]
Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In CVPR, pages 1179–1189, 2023. 3
2023
-
[70]
Point cloud gan
Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, and Ruslan Salakhutdinov. Point cloud gan. CoRR, abs/1810.05795, 2018. 4
2018 arXiv
-
[71]
Large-scale point cloud semantic segmentation with superpoint graphs
Loic Landrieu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In CVPR, pages 4558–4567, 2018. 4
2018
-
[72]
Open3d: A modern library for 3d data processing
Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arXiv:1801.09847, 2018. 4
2018 arXiv
-
[73]
Pointnext: Revisiting pointnet++ with improved training and scaling strategies
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. NeurIPS, 35:23192–23204, 2022. 4
2022
-
[74]
Investigating self-supervised methods for label-efficient learning
Srinivasa Rao Nandam, Sara Atito, Zhenhua Feng, Josef Kittler, and Muhammed Awais. Investigating self-supervised methods for label-efficient learning. IJCV, pages 1–16, 2025. 5
2025
-
[75]
Sinkhorn distances: Lightspeed computation of optimal transport
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 26,
-
[76]
Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance
Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In CVPR, pages 4018–4028, 2024. 6, 3
2024
-
[77]
Pointclip v2: Adapting clip for powerful 3d open-world learning
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. NeurIPS, 2022. 6, 7, 8, 9
2022
-
[78]
Adamw and super-convergence is now the fastest way to train neural nets
Sylvain Gugger and Jeremy Howard. Adamw and super-convergence is now the fastest way to train neural nets. last accessed, 19, 2018. 6
2018
-
[79]
Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese
Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016. 6, 7, 4
2016
-
[80]
Partdistill: 3d shape part segmentation by vision-language model distillation
Ardian Umam, Cheng-Kun Yang, Min-Hung Chen, Jen-Hui Chuang, and Yen-Yu Lin. Partdistill: 3d shape part segmentation by vision-language model distillation. In CVPR, pages 3470–3479,
-
[81]
Maskclip: Masked self-distillation advances contrastive language-image pretraining
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. In CVPR, pages 10995–11005, 2023. 8
2023
-
[82]
Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip
Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In ICCVW, pages 2048–2059, 2023. 8 14
2023
-
[83]
Openscene: 3d scene understanding with open vocabularies
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, pages 815–824, 2023. 8
2023
-
[84]
Geometrically-driven aggregation for zero-shot 3d point cloud understanding
Guofeng Mei, Luigi Riz, Yiming Wang, and Fabio Poiesi. Geometrically-driven aggregation for zero-shot 3d point cloud understanding. In CVPR, pages 27896–27905, 2024. 8, 9
2024
-
[85]
Cus3d: Clip-based unsupervised 3d segmentation via object-level denoise
Fuyang Yu, Runze Tian, Zhen Wang, Xiaochuan Wang, and Xiaohui Liang. Cus3d: Clip-based unsupervised 3d segmentation via object-level denoise. In ICME, pages 1–6. IEEE, 2024. 8
2024
-
[86]
3d shapenets: A deep representation for volumetric shapes
Zhirong Wu, Shuran Song, et al. 3d shapenets: A deep representation for volumetric shapes. In CVPR, pages 1912–1920, 2015. 8
1912
-
[87]
Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data
Mikaela Angelina Uy, Quang-Hieu Pham, et al. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In ICCV, pages 1588–1597,
-
[88]
Vconv-dae: Deep volumetric shape learning without object labels
Abhishek Sharma, Oliver Grau, and Mario Fritz. Vconv-dae: Deep volumetric shape learning without object labels. In ECCV, pages 236–250, 2016. 9
2016
-
[89]
Pcp-mae: Learning to predict centers for point masked autoencoders
Xiangdong Zhang, Shaofeng Zhang, and Junchi Yan. Pcp-mae: Learning to predict centers for point masked autoencoders. arXiv preprint arXiv:2408.08753, 2024. 2, 3, 4
2024 arXiv
-
[90]
Point-bert: Pre- training 3d point cloud transformers with masked point modeling
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre- training 3d point cloud transformers with masked point modeling. InCVPR, pages 19313–19322,
-
[91]
Masked discrimination for self-supervised learning on point clouds
Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrimination for self-supervised learning on point clouds. In ECCV, pages 657–675. Springer, 2022. 4
2022
-
[92]
Masked surfel prediction for self-supervised point cloud learning
Yabin Zhang, Jiehong Lin, Chenhang He, Yongwei Chen, Kui Jia, and Lei Zhang. Masked surfel prediction for self-supervised point cloud learning. arXiv preprint arXiv:2207.03111,
-
[93]
Towards compact 3d representations via point feature enhancement masked autoencoders
Yaohua Zha, Huizhen Ji, Jinmin Li, Rongsheng Li, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Towards compact 3d representations via point feature enhancement masked autoencoders. In AAAI, volume 38, pages 6962–6970, 2024. 4
2024
-
[94]
Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding
Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In CVPR, pages 9902–9912, 2022. 4
2022
-
[95]
A scalable active framework for region annotation in 3d shape collections
Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM TOG, 35:1–12, 2016. 4, 5
2016
-
[96]
Unsupervised point cloud pre-training via occlusion completion
Hanchen Wang, Qi Liu, Xiangyu Yue, Joan Lasenby, and Matt J Kusner. Unsupervised point cloud pre-training via occlusion completion. In CVPR, pages 9782–9792, 2021. 4 15 Contents 1 Introduction 1 2 Related work 2 3 S4Token 3 3.1 Preliminaries . . . . . . . . . . . . . . . . . ....
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.