Pith. sign in

REVIEW 3 major objections 3 minor 72 references

Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PointSD claims that a frozen Stable Diffusion model can teach 3D point cloud backbones useful semantics, replacing text conditioning with point-cloud tokens and aligning 3D features to SD's image features.

desk verdict PointSD is a solid empirical paper on using frozen Stable Diffusion as a teacher for 3D point cloud SSL; the layer-ablation table contradicts its own text, and the conditioning mechanism is under-evidenced, but the core result survives. read the letter →

arxiv 2507.09102 v1 pith:G7KNTRYS submitted 2025-07-12 cs.CV

Yiyang Chen , Shanshan Zhao , Lunhao Duan , Changxing Ding , Dacheng Tao This is my paper
classification cs.CV
keywords pointcloudself-supervisedlearningStableDiffusiontext-to-imagemodelscross-attentionconditioningfeaturealignment3Drepresentationpre-trainingpoint-to-imagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a large frozen text-to-image diffusion model, Stable Diffusion, can be a powerful teacher for 3D point cloud pre-training. It proposes PointSD, which first turns Stable Diffusion into a point-to-image generator by swapping the text encoder for a 3D encoder, then uses the resulting image features as alignment targets for a 3D backbone. If the claim is right, 3D models can inherit semantic knowledge from an image model trained on web-scale data without needing any labels, and can surpass prior self-supervised methods that train their own 3D diffusion models on small 3D datasets. The reported numbers, including 95.18% on ScanObjectNN OBJ-BG and 90.08% on PB-T50-RS, support that claim.

What carries the argument

Two mechanisms carry the argument. The first is the cross-attention substitution: point-cloud tokens $H = g(P)$ replace text features as the keys and values in Stable Diffusion's cross-attention layers, so the UNet's denoising is conditioned on 3D content rather than language. Cross-attention is the mechanism by which the image generator's intermediate features attend to conditioning tokens, and in this design those tokens come from a 3D encoder instead of a text encoder. The second is the alignment objective $L_{\text{alignment}} = \lVert s(R_P) - R_X \rVert_2^2$, which forces the 3D backbone's projected representation to match SD's down-sampling-layer features extracted from a noise-free rendered image conditioned on the same point cloud. The paper also couples these with a patch-mixing point-cloud augmentation and a stitched-image counterpart so the aligned pairs are harder and more diverse.

What would settle it

Take the same point cloud at the same viewpoint and generate two images from it using different random seeds; do the same for two different point clouds at the same viewpoint. If the point-cloud conditioning is load-bearing, the images from different objects should differ more than the two images of the same object. This distance comparison would settle whether the Stable Diffusion features used for alignment actually encode 3D shape, and the paper only provides qualitative examples of the generated images.

Watch

Extended reading notes

Core claim

PointSD's central claim is that a frozen Stable Diffusion model, trained on web-scale text-image pairs, can serve as a teacher for 3D self-supervised learning. The method first converts the text-to-image generator into a point-to-image generator: a 3D encoder outputs tokens that are fed through Stable Diffusion's cross-attention layers as the condition, and only the 3D encoder and the key/value projection matrices are trained while the rest of the UNet stays frozen. In the second stage, a noise-free rendered image of the same object is passed through the same conditioned UNet, and the 3D backbone is trained so that its projected features match the SD features taken from the down-sampling layers, using an L2 alignment loss. The paper reports that this recipe pushes ScanObjectNN accuracy to 95.18% on OBJ-BG, 93.63% on OBJ-ONLY, and 90.08% on PB-T50-RS, and that it beats the prior diffusion-based point-cloud pre-training method in all three settings while matching a label-supervised cross-modal method with roughly half the parameters.

Load-bearing premise

The load-bearing premise is that letting Stable Diffusion's image generator look at point-cloud tokens through its attention mechanism actually produces image features that reflect 3D shape, even though Stable Diffusion was trained on text-conditioned natural images; if that conditioning is noisy or weak, the alignment targets in the second stage carry little useful 3D information.

Editorial extensions

If this is right

  • A 3D backbone pre-trained this way outperforms single-modal generative pre-training methods, including masked autoencoders and 3D diffusion denoising, across all three ScanObjectNN settings.
  • The alignment transfer is not limited to classification: the pre-trained backbone improves few-shot accuracy on ModelNet40, part segmentation, and lifts a 3D detection backbone on an indoor-scene benchmark from 37.9 to 42.4 AP50.
  • The point-cloud condition is what makes the features useful: ablations show 90.08% with point-cloud conditioning versus 89.42% with class-text conditioning and 89.31% with no conditioning.
  • SD's down-sampling layers carry the most semantic signal for 3D learning; using them as alignment targets gives 90.08% versus 87.12% for middle-layer and 86.78% for up-sampling-layer features.
  • The second-stage alignment is needed on top of denoising: high-time-step denoising alone reaches 88.45%, and adding feature alignment raises accuracy to 90.08%.
  • Editorial inference: because PointSD only relies on cross-attention conditioning, the same two-stage recipe could plausibly work with any large frozen conditional image generator, not just Stable Diffusion; the paper's pre-trained-model ablation compares image encoders rather than other diffusion backbones, so this remains untested.
  • Editorial inference: the method's dependence on rendered views means its success may partly reflect a fixed rendering style; a direct test is to render the same 3D objects with varied lighting, camera height, or background and measure whether downstream accuracy degrades as the rendering distribution changes.
  • Editorial inference: the authors attribute their weaker part-segmentation results to object-level alignment and show in the supplementary that adding a masked-point reconstruction loss improves classification; an untested follow-up is whether that same mix also closes the part-segmentation gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. PointSD proposes a two-stage self-supervised pretraining method for point clouds that leverages a frozen Stable Diffusion v1.5 model. In Stage I, a 3D encoder replaces the text encoder, and its output tokens condition the SD UNet through cross-attention layers while only the K/V projection matrices are trained, yielding a point-to-image diffusion model. In Stage II, noise-free rendered images are passed through SD with point-cloud tokens as the condition, and a 3D backbone plus projector is trained to align backbone features with the resulting SD features via an L2 loss. The paper also introduces a mixing augmentation for point clouds and stitched images. Experiments cover ScanObjectNN classification, ModelNet40 classification and few-shot learning, ShapeNetPart part segmentation, ScanNetV2 detection, SemanticKITTI segmentation in the supplementary, and several ablations over conditions, augmentation, SD layers, pre-trained models, and training stages.

Significance. If the claims hold, the paper provides a useful demonstration that a large frozen text-to-image diffusion model can transfer semantic knowledge to 3D point cloud representations through a trainable conditioning adapter in a label-free manner. The scope of evaluation is broad, the code is publicly released, and the method improves consistently over a Transformer baseline and over PointDif in the reported tables. The central mechanism, however, rests on the assumption that point-cloud tokens actively steer the Stable Diffusion features that serve as alignment targets, and this assumption is not directly measured; in addition, the paper contains a direct contradiction about which SD layer produces the best features. These issues must be addressed before the contribution can be considered established.

major comments (3)
  1. [Sec. 4.3 / Table 7] Table 7 reports 'up' as the best SD layer (90.08) and 'down' as the worst (86.78), while the text in Sec. 4.3 states that the best performance is achieved by the down-sampling layers and the worst by the up-sampling layers; the same down-sampling claim appears in Sec. 3.2, where Eq. (5) defines R_X as coming from the 'last down-sampling layer'. Because the headline PB-T50-RS result is 90.08, the value in the 'up' row, the reader cannot tell which configuration actually produced the main results. This contradiction must be resolved, and the main experiments must be clearly attributed to the correct layer or re-run accordingly.
  2. [Sec. 3.2, Stage II; Table 5] The central claim that point-cloud tokens H actively control the SD features used for alignment is not directly verified. Stage II extracts R_X from a noise-free rendered image with H as the condition, but the only quantitative evidence that H matters is the 0.77-point gap between 'pc' and 'zero' in Table 5, supported only by qualitative generations in Fig. 4. Since the same rendered image is used in both cases, R_X could be dominated by image content, which would make the Stage I point-to-image construction superfluous. Please add a direct fidelity test, such as measuring the feature-level distance between R_X extracted with H versus a zero or random condition, or showing that cross-attention maps localize the point-cloud structure; without such evidence the proposed mechanism is not established.
  3. [Sec. 4.2 / Table 1] The comparison in Table 1 mixes different fine-tuning protocols. The text states that recent works use rotation augmentation and 2048 points and that the authors follow this protocol, whereas older rows such as PointDif, Point-BERT, and Point-MAE are cited with their published numbers, which were obtained with different settings (typically 1024 points and no rotation). The claimed consistent improvement over PointDif is therefore not established under an identical protocol. Please re-run the baselines under the same fine-tuning settings as Ours, or clearly mark the protocol change and report its effect on accuracy.
minor comments (3)
  1. [Fig. 1] The caption and the surrounding discussion of low versus high time steps could be clearer about which end of the time-step range corresponds to higher noise; the current wording is easy to misread.
  2. [Table 7] The row labels 'down', 'mid', and 'up' should be defined in the caption with the exact layer indices used for feature extraction, so that the reported choice is reproducible.
  3. [Fig. 3 / Fig. 5] The t-SNE visualizations are qualitative; please either specify the subset of ShapeNet categories used in Fig. 3 or report a quantitative cluster-quality metric in addition to the plots.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PointSD is an empirically evaluated two-stage teacher-student framework whose results are measured against external benchmarks.

full rationale

The paper makes no first-principles derivation; its central claim is an empirical claim about a pre-training recipe. Stage I trains a 3D encoder g and cross-attention projections so point-cloud tokens H condition a frozen SD UNet via cross-attention (Eqs. 3-4 and 8). Stage II freezes g and the SD model and aligns a separate 3D backbone f to SD features R_X extracted from noise-free rendered images conditioned on H (Eqs. 5 and 9). The alignment target is produced by a frozen teacher, not by the student backbone itself, so this is a standard teacher-student/self-distillation loop rather than a circular reduction. Downstream numbers on ScanObjectNN, ModelNet40, ShapeNetPart, ScanNetV2, and SemanticKITTI are external benchmarks; no benchmark number is fitted or constructed from the method's own equations, and the reported gains are measured rather than derived. Citations to PointDif and VPD are external works and do not overlap with the present authorship, so no self-citation chain is load-bearing. One reporting inconsistency, which is a correctness/mechanism concern rather than a circularity issue: Section 3.2 and Section 4.3 state that down-sampling SD layers give the best features, while Table 7 reports 'up'=90.08 versus 'down'=86.78, so the reader cannot tell which layer configuration produced the headline results. Likewise, the conditioning fidelity of the point-cloud tokens is only supported qualitatively (Fig. 4) and by the modest 'pc' versus 'zero' ablation gap in Table 5; this is weak mechanistic evidence but not a circular reduction. No step in the paper reduces a claimed result to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on domain assumptions about cross-modal transfer (point clouds to rendered images to SD features) and a small set of hand-selected hyperparameters. There are no new physical entities or fitted scientific constants. The most fragile entries are the assumption that frozen SD can be conditioned by point-cloud tokens and the hand choice of the down-sampling layer, which Table 7 appears to contradict.

free parameters (4)
  • Stage I diffusion timestep range = [500, 1000]
    Hand-chosen and ablated in Table 9; it controls how much low-level detail the denoising loss sees during Stage I.
  • Projector block count = 3
    Selected by ablation (Table 10) for mapping 3D features into the SD feature space.
  • Augmentation probability = 0.5
    Eqs. 8 and 9 use alpha ~ U(0,1) with augmented samples when alpha > 0.5; this 50% mix is a manual trade-off to avoid over-augmentation.
  • SD feature extraction layer = last down-sampling layer
    Chosen because the text claims down-sampling layers are most semantic, but Table 7 shows the up-sampling layer yields higher accuracy, making this hand choice internally inconsistent.
assumptions (4)
  • standard math The LDM denoising objective (Eq. 1) and the cross-attention update rule (Eq. 4) are valid as formulated.
    Standard diffusion and cross-attention results cited from Ho et al. [22] and Rombach et al. [41], used without modification.
  • domain assumption Rendered images from ULIP faithfully represent the 3D object's shape and semantics.
    Both stages pair point clouds with these renderings; if the renderings miss discriminative geometry, the alignment targets are uninformative (Sec. 4.1).
  • domain assumption Frozen SD can be successfully re-purposed to point-cloud conditioning by training only W_k, W_v and the 3D encoder.
    Stage I relies on this transfer; the paper validates it only through qualitative generation (Fig. 4) and later downstream accuracy.
  • domain assumption Noise-free images passed through SD yield semantically meaningful features.
    Stage II inherits VPD's premise; the authors extend it to point-cloud-conditioned rendering without a dedicated measurement of feature quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning." pith.science (2026). https://pith.science/paper/G7KNTRYS

@misc{pith2026250709102,
  author       = {Pith},
  title        = {Pith review of: Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7KNTRYS}},
  note         = {Machine review of arXiv:2507.09102}
}
read the original abstract

Diffusion-based models, widely used in text-to-image generation, have proven effective in 2D representation learning. Recently, this framework has been extended to 3D self-supervised learning by constructing a conditional point generator for enhancing 3D representations. However, its performance remains constrained by the 3D diffusion model, which is trained on the available 3D datasets with limited size. We hypothesize that the robust capabilities of text-to-image diffusion models, particularly Stable Diffusion (SD), which is trained on large-scale datasets, can help overcome these limitations. To investigate this hypothesis, we propose PointSD, a framework that leverages the SD model for 3D self-supervised learning. By replacing the SD model's text encoder with a 3D encoder, we train a point-to-image diffusion model that allows point clouds to guide the denoising of rendered noisy images. With the trained point-to-image diffusion model, we use noise-free images as the input and point clouds as the condition to extract SD features. Next, we train a 3D backbone by aligning its features with these SD features, thereby facilitating direct semantic learning. Comprehensive experiments on downstream point cloud tasks and ablation studies demonstrate that the SD model can enhance point cloud self-supervised learning. Code is publicly available at https://github.com/wdttt/PointSD.

Figures

Figures reproduced from arXiv: 2507.09102 by the authors.

Figure 1
Figure 1. Illustration of the forward process and the reverse pro [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of our PointSD. In the first stage, the 3D encoder extracts features to serve as the condition for guiding the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualization on a subset of the categories from [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Illustration of augmented training samples construction. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 4
Figure 4. Figure 4: Visualization of point-to-image generation results. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: t-SNE visualization on ModelNet40 and ScanObjectNN [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

72 extracted references · 50 canonical work pages

  1. [1]

    Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding

    Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Ro- drigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9902–9912, 2022. 1, 2, 3

    work page 2022

  2. [2]

    Se- mantickitti: A dataset for semantic scene understanding of lidar sequences

    Jens Behley, Martin Garbade, Andres Milioto, Jan Quen- zel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Se- mantickitti: A dataset for semantic scene understanding of lidar sequences. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 9297–9307,

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 3

    arXiv 2023

  4. [4]

    Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 1, 7

    work page 2021

  6. [6]

    Shapenet: An information-rich 3d model repository.arXiv preprint arXiv:1512.03012, 2015

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository.arXiv preprint arXiv:1512.03012, 2015. 5, 8

    arXiv 2015

  7. [7]

    Pimae: Point cloud and image interactive masked autoencoders for 3d object detection

    Anthony Chen, Kevin Zhang, Renrui Zhang, Zihan Wang, Yuheng Lu, Yandong Guo, and Shanghang Zhang. Pimae: Point cloud and image interactive masked autoencoders for 3d object detection. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 5291–5301, 2023. 2

    work page 2023

  8. [8]

    Videocrafter1: Open diffusion models for high-quality video generation, 2023

    Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter1: Open diffusion models for high-quality video generation, 2023. 3

    2023

Show all 72 references
  1. [9]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 1

    2020

  2. [10]

    Pointmixup: Augmentation for point clouds

    Yunlu Chen, Vincent Tao Hu, Efstratios Gavves, Thomas Mensink, Pascal Mettes, Pengwan Yang, and Cees GM Snoek. Pointmixup: Augmentation for point clouds. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 33...

    2020

  3. [11]

    Text-to-image diffusion mod- els are zero shot classifiers.Advances in Neural Information Processing Systems, 36, 2024

    Kevin Clark and Priyank Jaini. Text-to-image diffusion mod- els are zero shot classifiers.Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  4. [12]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 7

    2017

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018. 1, 2

    2018 arXiv

  6. [14]

    Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jian- jian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transform- ers help 3d representation learning? InThe Eleventh Inter- national Conference on Learning Representati...

  7. [15]

    Revisiting point cloud shape classification with a simple and effective baseline

    Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng. Revisiting point cloud shape classification with a simple and effective baseline. InProceedings of the 38th In- ternational Conference on Machine Learning, pages 3809–

  8. [16]

    3d semantic segmentation with submani- fold sparse convolutional networks

    Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submani- fold sparse convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 9224–9232, 2018. 12

    2018

  9. [17]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neura...

    2020

  10. [18]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023. 3

    2023 arXiv

  11. [19]

    Joint-mae: 2d-3d joint masked au- toencoders for 3d point cloud pre-training.arXiv preprint arXiv:2302.14007, 2023

    Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzhi Li, and Pheng-Ann Heng. Joint-mae: 2d-3d joint masked au- toencoders for 3d point cloud pre-training.arXiv preprint arXiv:2302.14007, 2023. 2, 5, 6

    2023 arXiv

  12. [20]

    Mvtn: Multi-view transformation network for 3d shape recognition

    Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, 2021. 5

    2021

  13. [21]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000– 16009, 2022. 1, 2, 7

    2022

  14. [22]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 2, 3

    2020

  15. [23]

    Regu- larization strategy for point cloud via rigidly mixed sample

    Dogyoon Lee, Jaeha Lee, Junhyeop Lee, Hyeongmin Lee, Minhyeok Lee, Sungmin Woo, and Sangyoun Lee. Regu- larization strategy for point cloud via rigidly mixed sample. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15900–15909, 2021. 4

    2021

  16. [24]

    Your diffusion model is secretly a zero-shot classifier

    Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 2206–2217,

  17. [25]

    Pointcnn: Convolution on x-transformed points

    Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. InAdvances in Neural Information Processing Sys- tems. Curran Associates, Inc., 2018. 5

    2018

  18. [26]

    Masked discrim- ination for self-supervised learning on point clouds.Pro- ceedings of the European Conference on Computer Vision (ECCV), 2022

    Haotian Liu, Mu Cai, and Yong Jae Lee. Masked discrim- ination for self-supervised learning on point clouds.Pro- ceedings of the European Conference on Computer Vision (ECCV), 2022. 5, 6, 7

    2022

  19. [27]

    Zero-1-to-3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to-3: Zero-shot one image to 3d object. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023. 3

    2023

  20. [28]

    Wonder3d: Sin- gle image to 3d using cross-domain diffusion.arXiv preprint arXiv:2310.15008, 2023

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Sin- gle image to 3d using cross-domain diffusion.arXiv preprint arXiv:2310.15008, 2023. 3

    arXiv

  21. [29]

    Sgdr: Stochas- tic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983, 2016. 5

    2016 arXiv

  22. [30]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  23. [31]

    Rethinking network design and local geometry in point cloud: A simple residual mlp framework.arXiv preprint arXiv:2202.07123, 2022

    Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework.arXiv preprint arXiv:2202.07123, 2022. 1, 5, 6

    2022 arXiv

  24. [32]

    An end-to- end transformer model for 3d object detection

    Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to- end transformer model for 3d object detection. InProceed- ings of the IEEE/CVF international conference on computer vision, pages 2906–2917, 2021. 6, 7

    2021

  25. [33]

    Masked autoencoders for point cloud self-supervised learning

    Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. InEuropean conference on computer vision, pages 604–621. Springer, 2022. 1, 2, 3, 5, 6

    2022

  26. [34]

    Diffusion autoen- coders: Toward a meaningful and decodable representation

    Konpat Preechakul, Nattanat Chatthee, Suttisak Wizad- wongsa, and Supasorn Suwajanakorn. Diffusion autoen- coders: Toward a meaningful and decodable representation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022. 2

    2022

  27. [35]

    Pointnet: Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660,

  28. [36]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space.Advances in neural information processing systems, 30, 2017. 1, 5

    2017

  29. [37]

    Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining

    Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. InInternational Conference on Machine Learn- ing (ICML), 2023. 2, 5, 6, 12, 13

    2023

  30. [38]

    Pointnext: Revisiting pointnet++ with improved training and scaling strategies

    Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. InAdvances in Neural In- formation Processing Systems, pages 23192–23204. Curran Associates,...

    2022

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  32. [40]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 1

    2020

  33. [41]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 3, 7

    2022

  34. [42]

    Self-supervised few-shot learning on point clouds.Advances in Neural Information Processing Systems, 33:7212–7221, 2020

    Charu Sharma and Manohar Kaul. Self-supervised few-shot learning on point clouds.Advances in Neural Information Processing Systems, 33:7212–7221, 2020. 6

    2020

  35. [43]

    Mvdream: Multi-view diffusion for 3d gen- eration.arXiv preprint arXiv:2308.16512, 2023

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d gen- eration.arXiv preprint arXiv:2308.16512, 2023. 3

    2023 arXiv

  36. [44]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2, 3

    2015

  37. [45]

    Epmf: Efficient perception-aware multi-sensor fusion for 3d semantic seg- mentation.IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 46(12):8258–8273, 2024

    Mingkui Tan, Zhuangwei Zhuang, Sitao Chen, Rong Li, Kui Jia, Qicheng Wang, and Yuanqing Li. Epmf: Efficient perception-aware multi-sensor fusion for 3d semantic seg- mentation.IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 46(12):8258–8273, 2024. 1

    2024

  38. [46]

    Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion

    Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. arXiv preprint arXiv:2308.12469, 2023. 2, 3

    2023 arXiv

  39. [47]

    Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data

    Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. InProceedings of the IEEE/CVF international conference on computer vision, pages 158...

    2019

  40. [48]

    Visualizing data using t-sne.Journal of machine learning research, 9 (11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9 (11), 2008. 8, 12

    2008

  41. [49]

    Dynamic graph cnn for learning on point clouds.ACM Transactions on Graphics (tog), 38(5):1–12, 2019

    Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds.ACM Transactions on Graphics (tog), 38(5):1–12, 2019. 1, 5, 6

    2019

  42. [50]

    Pointpatchmix: Point cloud mixing with patch scoring

    Yi Wang, Jiaze Wang, Jinpeng Li, Zixu Zhao, Guangyong Chen, Anfeng Liu, and Pheng Ann Heng. Pointpatchmix: Point cloud mixing with patch scoring. InProceedings of the AAAI Conference on Artificial Intelligence, pages 5686– 5694, 2024. 4

    2024

  43. [51]

    P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting

    Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Ji- wen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. InAdvances in Neural Information Processing Systems, pages 14388– 14402. Curran Associates, Inc., 2022. 5

    2022

  44. [52]

    Take-a-photo: 3d-to-2d generative pre-training of point cloud models

    Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Ji- wen Lu. Take-a-photo: 3d-to-2d generative pre-training of point cloud models. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5640–5650,

  45. [53]

    3d shapenets: A deep representation for volumetric shapes

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Lin- guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015. 6, 12

    1912

  46. [54]

    Pointcontrast: Unsupervised pre- training for 3d point cloud understanding

    Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre- training for 3d point cloud understanding. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 5...

    2020

  47. [55]

    Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding

    Le Xue, Mingfei Gao, Chen Xing, Roberto Mart ´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. InProceedings of the IEEE/CVF Conference on...

    2023

  48. [56]

    Ulip-2: Towards scalable multimodal pre-training for 3d understanding

    Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Jun- nan Li, Roberto Mart´ın-Mart´ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. InProceed- ings of the IEEE/CVF Conference on Computer V...

    2024

  49. [57]

    Gd-mae: gen- erative decoder for mae pre-training on lidar point clouds

    Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: gen- erative decoder for mae pre-training on lidar point clouds. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9403–9414, 2023. 2

    2023

  50. [58]

    Dis- diff: Unsupervised disentanglement of diffusion probabilis- tic models.arXiv preprint arXiv:2301.13721, 2023

    Tao Yang, Yuwang Wang, Yan Lv, and Nanning Zheng. Dis- diff: Unsupervised disentanglement of diffusion probabilis- tic models.arXiv preprint arXiv:2301.13721, 2023. 2

    2023 arXiv

  51. [59]

    Diffusion model with cross attention as an inductive bias for disentanglement

    Tao Yang, Cuiling Lan, Yan Lu, et al. Diffusion model with cross attention as an inductive bias for disentanglement. arXiv preprint arXiv:2402.09712, 2024. 2

    2024 arXiv

  52. [60]

    A scalable active framework for region annotation in 3d shape collections.ACM Transactions on Graphics (ToG), 35(6):1–12, 2016

    Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Shef- fer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections.ACM Transactions on Graphics (ToG), 35(6):1–12, 2016. 7

    2016

  53. [61]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19313–19322, 2022. 1, 2, 5, 6, 7, 12, 13

    2022

  54. [62]

    Exploring diffusion time- steps for unsupervised representation learning.arXiv preprint arXiv:2401.11430, 2024

    Zhongqi Yue, Jiankun Wang, Qianru Sun, Lei Ji, Eric I Chang, Hanwang Zhang, et al. Exploring diffusion time- steps for unsupervised representation learning.arXiv preprint arXiv:2401.11430, 2024. 2

    2024 arXiv

  55. [63]

    Towards compact 3d representations via point feature enhancement masked au- toencoders.arXiv preprint arXiv:2312.10726, 2023

    Yaohua Zha, Huizhen Ji, Jinmin Li, Rongsheng Li, Tao Dai, Bin Chen, Zhi Wang, and Shu-Tao Xia. Towards compact 3d representations via point feature enhancement masked au- toencoders.arXiv preprint arXiv:2312.10726, 2023. 2, 5, 6, 13

    2023 arXiv

  56. [64]

    Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.Advances in neural information processing sys- tems, 35:27061–27074, 2022

    Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, and Hongsheng Li. Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.Advances in neural information processing sys- tems, 35:27061–27074, 2022. 3, 5, 6

    2022

  57. [65]

    Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders

    Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, and Hong- sheng Li. Learning 3d representations from 2d pre-trained models via image-to-point masked autoencoders. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21769–21780, 2023. 2, 5, 6, 12

    2023

  58. [66]

    Self-supervised pretraining of 3d features on any point-cloud

    Zaiwei Zhang, Rohit Girdhar, Armand Joulin, and Ishan Misra. Self-supervised pretraining of 3d features on any point-cloud. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10252–10263, 2021. 1, 2, 3

    2021

  59. [67]

    Unsupervised rep- resentation learning from pre-trained diffusion probabilistic models.Advances in Neural Information Processing Sys- tems, 35:22117–22130, 2022

    Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised rep- resentation learning from pre-trained diffusion probabilistic models.Advances in Neural Information Processing Sys- tems, 35:22117–22130, 2022. 2

    2022

  60. [68]

    Unleashing text-to-image diffu- sion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffu- sion models for visual perception. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5729–5739, 2023. 2, 3, 4

    2023

  61. [69]

    Point cloud pre-training with diffusion models

    Xiao Zheng, Xiaoshui Huang, Guofeng Mei, Yuenan Hou, Zhaoyang Lyu, Bo Dai, Wanli Ouyang, and Yongshun Gong. Point cloud pre-training with diffusion models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22935–22945, 2024. 1,...

    2024

  62. [70]

    As shown in Tab

    Additional Experiments Projector for Feature Alignment.We conduct experi- ments to investigate how the number of transformer blocks in the projector affects performance. As shown in Tab. 10, when the number of blocks is set to 3, our method achieves the best performance, sugge...

  63. [71]

    Visualization and Illustration Visualization of Point-to-image Generation.As shown in Fig. 4, we visualize the point cloud and the corresponding rendered image in the left two columns, respectively, and the results generated with different seeds are shown in the five right col...

  64. [72]

    Illustration of the Augmentation Strategy.We show our augmentation strategy in Fig

    ScanObjectNN PB-T50-RS is a real-world dataset con- taining background noise, while our model is pre-trained on synthetic data, making it harder for the model to sep- arate different classes of samples in feature space without fine-tuning. Illustration of the Augmentation Stra...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.