Pith. sign in

REVIEW 4 major objections 5 minor 51 references

XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read XMask3D claims that aligning 3D point cloud features to the vision-language space at the level of segmentation masks, using a diffusion-model mask generator conditioned on 3D geometry, improves open-vocabulary 3D semantic segmentation…

desk verdict Genuinely new diffusion-conditioning idea and one clean ablation, but the headline results are confounded by an unreported test-time MaskCLIP ensemble that baselines don't get. read the letter →

arxiv 2411.13243 v1 pith:DA2HHSLE submitted 2024-11-20 cs.CV cs.AI

Ziyi Wang , Yanbo Wang , Xumin Yu , Jie Zhou , Jiwen Lu This is my paper · ORCID
classification cs.CVcs.AI
keywords openvocabulary3Dsemanticsegmentationpointcloudmask-levelfeaturealignmentdiffusionmodeldenoisingUNetCLIPembeddingspacecross-modalfusionzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that open-vocabulary 3D semantic segmentation gets better when the 3D model is aligned to the shared 2D-text embedding space not globally or point-by-point, but mask-by-mask. To do this, XMask3D adds a 2D branch built from a pre-trained text-to-image diffusion model's denoising UNet, which produces class-agnostic masks with strong open-vocabulary semantics. The global 3D point cloud feature is injected as the text condition into that frozen UNet, making the generated masks geometry-aware. The masks are then back-projected onto the 3D points to pool 3D mask embeddings, which are pulled toward CLIP mask features from the 2D branch by a cosine contrastive loss. If this works as claimed, mask-level reasoning is a recipe for fine-grained open-vocabulary 3D segmentation that any point cloud segmentation backbone can adopt.

What carries the argument

The load-bearing object is the cross-modal mask itself: a class-agnostic binary mask generated by the 2D branch, back-projected onto the 3D points, with a pooled embedding computed on both sides. The 2D branch is the denoising UNet of a pre-trained text-to-image diffusion model, kept frozen, plus a Mask2Former-style mask generator, conditioned not on a caption but on the global 3D feature through an Implicit 3D Captioner. This condition couples the two modalities: because the frozen UNet only recognizes the 2D-text embedding space, the gradient through the mask loss forces the global 3D feature toward that space. The mask-level contrastive loss is the mechanism that transfers open-vocabulary semantics into the 3D branch while preserving geometry.

What would settle it

On the ScanNet B12/N7 benchmark, train the same XMask3D pipeline but replace the mask-level target $G_{\text{CLIP}}$ with shuffled or random CLIP mask features; if the 3D branch still gains roughly 11.7 mIoU on novel categories, the mask-level alignment is not the mechanism doing the work, whereas if the gain disappears, the alignment is doing what the paper claims.

Watch

Extended reading notes

Core claim

The paper's central claim is that aligning 3D geometry features to the shared vision-language embedding space at the granularity of segmentation masks, rather than at scene, patch, or point granularity, is the right level of contrastive supervision for open-vocabulary 3D semantic segmentation. The 2D branch is a mask generator built on the denoising UNet of a pre-trained text-to-image diffusion model; instead of using an image caption as the text condition, an Implicit 3D Captioner converts the global 3D feature into the condition, so the frozen diffusion model pulls 3D geometry toward the 2D-text space while producing geometry-aware masks. These 2D masks are back-projected to 3D, and each mask's pooled 3D embedding is matched to the corresponding CLIP mask embedding via the cosine loss $L_{\text{mask}}=\frac{1}{M}\sum_{i=1}^{M}\left(1-\frac{G_{3d}^{i}\cdot(G_{\text{CLIP}}^{i})^{\mathsf{T}}}{\|G_{3d}^{i}\|\|G_{\text{CLIP}}^{i}\|}\right)$. A fusion block concatenates the complementary 2D mask embeddings with 3D point features on points with correspondences. The paper reports that this mask-level regularization raises novel-category accuracy of the 3D branch by 11.7 mIoU on the B12/N7 ScanNet benchmark and that the fused output beats both branches.

Load-bearing premise

The pipeline depends on the 2D branch's own predicted masks being good enough that the CLIP features computed from those masks are a trustworthy teaching signal; when the 2D masks are imperfect, the mask-level loss trains the 3D branch to copy the 2D branch's errors.

Editorial extensions

If this is right

  • Any point cloud segmentation encoder-decoder can be dropped into the 3D branch and gain open-vocabulary ability, because XMask3D only changes how the 3D features are regularized and fused.
  • On the long-tail ScanNet200 benchmarks, novel-class mIoU improves by roughly 70 percent over PLA, the largest relative gain the paper reports.
  • The fused output beats both the 2D-only and 3D-only branch outputs on both base and novel categories in the B12/N7 ablation, so the two modalities are complementary rather than redundant.
  • Introducing the mask-level loss after an initial warm-up at epoch 50 improves the 3D branch's novel mIoU by 11.7 on B12/N7, implying that mask quality at the start of training matters.
  • The same pipeline can be repurposed for instance or panoptic 3D segmentation by swapping the 3D backbone for an instance or panoptic model, as the authors state in the limitations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: because the diffusion UNet is frozen and only the mask generator and the 3D-to-2D condition are trained, improvements in text-to-image diffusion models should transfer to XMask3D's 3D open-vocabulary ability without retraining the 3D branch.
  • The reported failure cases, such as shower curtain versus curtain and a large picture versus wall, suggest that injecting full-scene context, not just the global 3D feature from one view's point cloud, might fix categories that differ mainly by environment.
  • An ablation that corrupts or shuffles the target CLIP mask features would separate how much of the 11.7 point gain comes from the mask-level alignment itself versus from the 3D-conditioned diffusion distillation, which the paper does not report.
  • A lighter 2D mask generator would test whether the diffusion UNet's dense vision-language features are the source of the gain or just a convenient carrier for it; the authors flag computational cost as a limitation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. XMask3D proposes a mask-level cross-modal alignment framework for open-vocabulary 3D semantic segmentation. The method has three main components: (i) a 2D mask generator built on a frozen diffusion UNet whose text condition is replaced by an implicit embedding from the 3D encoder (Section 3.2); (ii) a mask-level cosine contrastive loss that pulls 3D mask-pooled features toward CLIP mask features obtained with MaskCLIP from the predicted 2D masks (Section 3.3, Eq. (6)); and (iii) a fusion block that concatenates 2D pseudo mask features with 3D features (Section 3.4). Experiments on ScanNet20, ScanNet200, and S3DIS report gains over PLA and OpenScene, with an ablation showing +11.7 novel mIoU for the 3D branch on ScanNet B12/N7 when the mask loss is enabled.

Significance. The proposed mask-level alignment is a reasonable and timely idea, and the diffusion-based 2D generator with an implicit 3D condition is novel. The paper ships code and gives detailed per-class results and failure cases. However, the headline comparisons are confounded by an auxiliary test-time ensemble with MaskCLIP logits that is described only in Appendix A.4, and the strongest concurrent method, OV3D, outperforms XMask3D on ScanNet20. The internal ablation is genuine evidence for the training-time mechanism, but the central claim of state-of-the-art open-vocabulary performance is not established by the current experiments.

major comments (4)
  1. [A.4, Eqs. (16)-(17)] The final results in Tables 1 and 2 are obtained with p_final proportional to p^λ p_aux^(1-λ), where p_aux is a softmax over MaskCLIP features GCLIP and text embeddings. This inference-time auxiliary classifier is not applied to PLA, OpenScene, or OV3D in the comparisons, so the reported gains (e.g., ScanNet200 B170/N30 hIoU 18.0 vs. 11.4/14.2) do not isolate the proposed cross-modal mask reasoning. Please report all methods without p_aux, or apply the equivalent MaskCLIP classifier to the baselines, and state the value of λ. Please also state whether the fused numbers in Table 3b use Eq. (16); if they do, the +1.6 fused improvement from Lmask is not attributable to the mask loss alone.
  2. [Table 1 and Section 4.2] The claim that XMask3D 'outperforms previous methods across most benchmarks' is not supported on ScanNet20, where OV3D achieves higher hIoU on all three splits (72.4/68.5/64.8 vs. 70.0/61.7/55.7). Since OV3D is the strongest published open-vocabulary 3D segmentation method, the paper should either add OV3D results on ScanNet200 and S3DIS or explicitly qualify the comparison. Reporting only qualitative orthogonality is insufficient for a state-of-the-art claim.
  3. [Section 3.3 and A.1, Eq. (9)] The regularization target GCLIP is computed with MaskCLIP from masks predicted by the same 2D branch. If the 2D masks are inaccurate, the contrastive loss in Eq. (6) can reinforce 2D branch errors into the 3D features. The paper does not analyze this circular dependence. Please add an experiment using ground-truth masks or a corrupted-mask control to quantify the sensitivity of Lmask to mask quality, and report the overlap between predicted masks and object regions.
  4. [Section 4.1 and Tables 1-2] All results appear to be from a single seed, and no error bars or standard deviations are reported. This matters because several comparisons are close (e.g., S3DIS B6/N6 hIoU 44.9 vs. 44.2 for OpenScene, and ScanNet200 B150/N50 hIoU 15.5 vs. 15.2). Please report mean and standard deviation over at least three runs, or justify why a single run is sufficient.
minor comments (5)
  1. [Eq. (2)] The product notation 'Qt k=1 αk' should be a product symbol, and the noise schedule α1,...,αT should be defined before use.
  2. [Table 4(a)] The category partitions are garbled: for example, 'toiletcounter' in the B15/N4 row is missing a comma, and the base/novel columns are not aligned in the rendered table.
  3. [Section 4.2] The sentence 'the OpenScene results are derived from the implementation of UniM-OV3D' is confusing because UniM-OV3D is not otherwise compared; please clarify how these numbers were obtained and whether they match the original OpenScene paper.
  4. [Figure 2] The labels 'CLIP I/T Feat.' and 'Mask Feat.' are hard to parse; please use explicit labels such as 'CLIP image/text features' and 'mask embeddings'.
  5. [A.1, Eq. (10)] The symbol M is used both for the number of masks and for the attention mask matrix; rename the matrix (e.g., A) to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the mask-level regularization target is a frozen CLIP feature, and the central claim is supported by an independent ablation; the MaskCLIP test-time ensemble is an evaluation confound rather than a circular step.

full rationale

Walking the derivation chain, Lmask in Eq. 6 aligns 3D mask embeddings G3d to GCLIP, which is computed by MaskCLIP from a frozen CLIP model and the 2D branch's predicted masks (Eq. 9). This is a distillation target, not a quantity fitted to the held-out novel categories, so the open-vocabulary evaluation is not forced by construction. The final inference in Eqs. 16-17 does combine the model logits with paux derived from GCLIP, meaning Tables 1-2 do not fully isolate the proposed mask-reasoning contribution from a test-time MaskCLIP ensemble. However, this is an attribution or ablation-confounding issue, not a circular equivalence: paux is generated by a frozen CLIP teacher and is not a fitted parameter, and the paper's own ablation in Table 3b shows that adding Lmask independently raises the 3D branch novel mIoU from 14.0 to 25.7 (+11.7), providing non-circular evidence for the central mechanism. The paper also explicitly acknowledges that mask quality can be suboptimal at the start of training ('the quality of mask prediction at the onset of training may be suboptimal'), which is a self-dependence caveat about bootstrapping, but it does not make any equation reduce to its own input. No load-bearing self-citation chain appears: the cited bases (ODISE, MaskCLIP, PLA) are external prior work. Therefore the derivation is self-contained and no circularity is found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several inherited empirical assumptions about pre-trained diffusion and CLIP models, plus per-benchmark hyperparameter tuning. No new physical entities are introduced; the only new component, the Implicit 3D Captioner, is a learned network with no external falsifiable handle. The largest unquantified influence is the choice of loss weights, which are tuned separately for each benchmark.

free parameters (6)
  • omega_mask (mask regularization weight) = 0.5/0.5/1/2/2/1/1 per benchmark
    Set differently for ScanNet B15/N4, B12/N7, B10/N9, ScanNet200 B170/N30, B150/N50, S3DIS B8/N4, B6/N6; selected by tuning per dataset (Appendix A.4).
  • omega_bi (binary head weight) = 16/12/8/48/32/20/15 per benchmark
    Tuned per benchmark, same source as omega_mask in Appendix A.4.
  • shared loss weights = omega_seg=4, omega_3d_view=1, omega_2d_view=4, omega_fuse_view=1.5
    Chosen hyperparameters shared across all benchmarks; no sensitivity analysis is reported.
  • lambda (auxiliary logit balance) = not reported
    Fixed balancing factor in Eq. 16 for combining CLIP mask auxiliary logits; value not specified in the paper.
  • mask regularization warm-up epoch = 50
    Mask-level loss is introduced after epoch 50 based on the observation that early masks are suboptimal (Section 4.1).
  • tau (learnable temperature) = learned
    Learnable temperature in Eq. 15 for supervised segmentation losses; not fixed.
assumptions (5)
  • domain assumption Cross-attention maps and intermediate features of the pre-trained denoising UNet correlate with text concepts and can serve as the basis for accurate open-vocabulary masks.
    Section 3.2 design insights, relying on Prompt-to-Prompt and ODISE; not re-derived in this paper.
  • domain assumption Masked attention with frozen CLIP weights yields faithful CLIP embeddings for image regions given a predicted mask.
    A.1, Eqs. 9-11; relies on MaskCLIP. If MaskCLIP embeddings are biased, the mask regularization target is biased.
  • domain assumption The learned implicit 3D embedding can be injected into the frozen diffusion UNet in place of a text embedding while preserving semantic controllability.
    Section 3.2, Eq. 4; assumes MLP(f3d) is in-distribution for the pre-trained UNet text condition.
  • standard math Back-projection of 2D masks to 3D via camera intrinsics and view projection is correct and complete for visible points.
    Section 3.1; assumes calibrated cameras and occlusion-free visibility for the N' projected points.
  • domain assumption Scene captions from ViT-GPT2 and CLIP text embeddings are informative enough to align view-level features.
    A.2; inherited from PLA [11]. Weak captions would weaken the view-level regularization.
invented entities (1)
  • Implicit 3D Captioner
    purpose: Learned MLP mapping global 3D feature f3d to an implicit condition embedding injected into the frozen diffusion UNet, enabling geometry-aware mask generation.
    New learnable component introduced in Section 3.2; it has no falsifiable handle outside this paper and its value is only internal to the pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation." pith.science (2026). https://pith.science/paper/DA2HHSLE

@misc{pith2026241113243,
  author       = {Pith},
  title        = {Pith review of: XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DA2HHSLE}},
  note         = {Machine review of arXiv:2411.13243}
}
read the original abstract

Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https://github.com/wangzy22/XMask3D.

Figures

Figures reproduced from arXiv: 2411.13243 by the authors.

Figure 1
Figure 1. The overall framework of XMask3D. The 3D model with only coarse 3D-2D-text alignment struggles to segment novel categories with accurate boundaries. We propose to incorporate a 2D open mask generator conditioned on global 3D geometry features to create geometry-aware segmentation masks of novel categories. Then we apply fine-grained mask-level regularization on 3D features, thereby enhancing the dense open vocabular… view at source ↗
Figure 2
Figure 2. The detailed architecture of XMask3D. We introduce an auxiliary 2D branch, which utilizes global point cloud features as conditional input to generate open vocabulary masks. The contour of the mask is utilized for regularization at the mask level on 3D features, and the embeddings of the mask are fused with the 3D features to enhance cross-modal complementarity. implemented using any off-the-shelf architectures such… view at source ↗
Figure 3
Figure 3. Visualization Comparisons between XMask3D and Previous Methods. We compare XMask3D with PLA [11] and OpenScene [34] on the novel categories table, bookshelf, chair and bed. The regions corresponding to the novel categories are highlighted in red boxes. We also analyze the effects of mask regularization through visualizations in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: In the first group, the 2D branch exhibits unsatisfactory results of the base category [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 4
Figure 4. Figure 4: Visualization Results of Ablations. The first and second groups show results from ScanNet B15/N4 and B12/N7 benchmark, respectively. In each group, the first and second rows display segmentation results without and with the proposed mask regularization. The last three …
Figure 5
Figure 5. Figure 5: Complete Visualization Comparisons. We show comprehensive comparison between XMask3D fused/2D branch/3D branch outputs and previous methods (PLA [11]/OpenScene [34]). The figure corresponds to [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Visualization Results of Ablations. The first and second rows display segmentation results without and with the proposed mask regularization. The last three columns compare the outputs from the intermediate 2D and 3D branches with the final fusion block. set ωmask = 0.…
Figure 7
Figure 7. Figure 7: Failure cases of XMask3D. We focus on shower curtain, picture, and sink novel categories in each line, respectively. B.3 Illustrations of Ablations We present supplementary illustrations of ablation studies pertaining to mask regularization and cross-modality fusion in…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 24 canonical work pages

  1. [1]

    3d semantic parsing of large-scale indoor spaces

    Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016

    2016

  2. [2]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021

    2021

  3. [3]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022

    work page 2022

  4. [4]

    Per-pixel classification is not all you need for semantic segmentation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021

    work page 2021

  5. [5]

    Transductive zero-shot learning for 3d point cloud classification

    Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Transductive zero-shot learning for 3d point cloud classification. In WACV, 2020

    work page 2020

  6. [6]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303.11797, 2023

    arXiv 2023

  7. [7]

    4d spatio-temporal convnets: Minkowski convolutional neural networks

    Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, 2019

    work page 2019

  8. [8]

    Spconv: Spatially sparse convolution library

    Spconv Contributors. Spconv: Spatially sparse convolution library. https://github.com/ traveller59/spconv, 2022

    2022

Show all 51 references
  1. [9]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017

    2017

  2. [10]

    Decoupling zero-shot semantic segmen- tation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmen- tation. In CVPR, pages 11583–11592, 2022

    2022

  3. [11]

    Pla: Language-driven open-vocabulary 3d scene understanding

    Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023

    2023

  4. [12]

    Open-vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskclip. In ICML, 2023

    2023

  5. [13]

    Scaling open-vocabulary image segmen- tation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmen- tation with image-level labels. In ECCV, 2022

    2022

  6. [14]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arX...

    2023 arXiv

  7. [15]

    Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models

    Huy Ha and Shuran Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. arXiv preprint arXiv:2207.11514, 2022

    2022 arXiv

  8. [16]

    Unim-ov3d: Uni-modality open- vocabulary 3d scene understanding with fine-grained feature representation

    Qingdong He, Jinlong Peng, Zhengkai Jiang, Kai Wu, Xiaozhong Ji, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Mingang Chen, and Yunsheng Wu. Unim-ov3d: Uni-modality open- vocabulary 3d scene understanding with fine-grained feature representation. arXiv preprint arXiv:2401.11395, 2024

    2024 arXiv

  9. [17]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  10. [18]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33, 2020

    2020

  11. [19]

    Clip2point: Transfer clip to point cloud classification with image-depth pre-training

    Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, 2023

    2023

  12. [20]

    Open-vocabulary 3d semantic segmentation with foundation models

    Li Jiang, Shaoshuai Shi, and Bernt Schiele. Open-vocabulary 3d semantic segmentation with foundation models. In CVPR, pages 21284–21294, 2024

    2024

  13. [21]

    Diffusion models for zero-shot open-vocabulary segmentation

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprint arXiv:2306.09316, 2023. 15

    2023 arXiv

  14. [22]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, 2023

    2023

  15. [23]

    Language- driven semantic segmentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. arXiv preprint arXiv:2201.03546, 2022

    2022 arXiv

  16. [24]

    Tagclip: Improving discrimination ability of open-vocabulary semantic segmentation

    Jingyao Li, Pengguang Chen, Shengju Qian, and Jiaya Jia. Tagclip: Improving discrimination ability of open-vocabulary semantic segmentation. arXiv preprint arXiv:2304.07547, 2023

    2023 arXiv

  17. [25]

    Guiding text-to-image diffusion model towards grounded generation

    Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. In ICCV, 2023

    2023

  18. [26]

    Magic3d: High-resolution text-to-3d content creation

    Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023

    2023

  19. [27]

    3d open-vocabulary segmentation with foundation models

    Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. 3d open-vocabulary segmentation with foundation models. arXiv preprint arXiv:2305.14093, 2023

    2023 arXiv

  20. [28]

    Segment any point cloud sequences by distilling vision foundation models

    Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. NeurIPS, 36, 2023

    2023

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  22. [30]

    Open-vocabulary semantic segmentation with frozen vision-language models

    Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulary semantic segmentation with frozen vision-language models. arXiv preprint arXiv:2210.15138, 2022

    2022 arXiv

  23. [31]

    Generative zero-shot learning for semantic segmentation of 3d point clouds

    Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 3DV, 2021

    2021

  24. [32]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021

    2021

  25. [33]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  26. [34]

    Openscene: 3d scene understanding with open vocabularies

    Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023

    2023

  27. [35]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2022

    2022

  28. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  29. [37]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022

    2022

  30. [38]

    Language-grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. In ECCV, 2022

    2022

  31. [39]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35, 2022

    2022

  32. [40]

    Dreamgaussian: Generative gaussian splatting for efficient 3d content creation

    Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2023

    2023

  33. [41]

    Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models

    Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In ICCV, 2023

    2023

  34. [42]

    Point transformer v3: Simpler, faster, stronger

    Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster, stronger. arXiv preprint arXiv:2312.10035, 2023. 16

    2023 arXiv

  35. [43]

    Point transformer v2: Grouped vector attention and partition-based pooling

    Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. NeurIPS, 2022

    2022

  36. [44]

    3d open-vocabulary panoptic segmentation with 2d-3d vision-language distillation.arXiv preprint arXiv:2401.02402, 2024

    Zihao Xiao, Longlong Jing, Shangxuan Wu, Alex Zihao Zhu, Jingwei Ji, Chiyu Max Jiang, Wei- Chih Hung, Thomas Funkhouser, Weicheng Kuo, Anelia Angelova, et al. 3d open-vocabulary panoptic segmentation with 2d-3d vision-language distillation.arXiv preprint arXiv:2401.02402, 2024

    2024 arXiv

  37. [45]

    Open-vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023

    2023

  38. [46]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, 2023

    2023

  39. [47]

    Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding

    Jihan Yang, Runyu Ding, Weipeng Deng, Zhe Wang, and Xiaojuan Qi. Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding. arXiv preprint arXiv:2304.00962, 2023

    2023 arXiv

  40. [48]

    Vit-gpt2 image captioning

    ydshieh. Vit-gpt2 image captioning. https://huggingface.co/nlpconnect/ vit-gpt2-image-captioning

  41. [49]

    Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip

    Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In ICCV, 2023

    2023

  42. [50]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023

    2023

  43. [51]

    Point transformer

    Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In ICCV, 2021. 17

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.