Pith. sign in

REVIEW 4 major objections 6 minor 74 references

Any-to-3D Generation via Hybrid Diffusion Supervision

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A unified framework, XBind, generates 3D objects from text, image, or audio prompts using shared multimodal embeddings as diffusion conditions.

desk verdict A plausible first stab at unified any-to-3D with audio, but the reported SOTA margins are largely circular because the evaluation metrics overlap with the training objective. read the letter →

arxiv 2411.14715 v1 pith:C5M5H2XI submitted 2024-11-22 cs.CV

Yijun Fan , Yiwei Ma , Jiayi Ji , Xiaoshuai Sun , Rongrong Ji This is my paper · ORCID
classification cs.CV
keywords any-to-3Dgenerationmultimodaldiffusionmodelsscoredistillationsamplinghybridsupervisionmodalitysimilarityloss3Dobjectaudio-to-3D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that 3D object generation can be made truly multimodal: one framework, XBind, takes a text prompt, an image, or even an audio clip and produces a textured 3D object aligned with it, without retraining when the input modality changes. The key move is to treat embeddings from a shared multimodal encoder (ImageBind) as the conditioning signal for two frozen diffusion models, one 2D and one 3D-aware, and to add a Modality Similarity loss that ties the prompt embedding to CLIP embeddings of rendered views. The paper reports that this hybrid supervision, run through a coarse-to-fine three-phase pipeline from NeRF to DMTet, yields higher CLIP-R, CLIP-I, and ARCC scores than its constructed baselines and than established text-to-3D methods. If correct, this removes the need to convert prompts between modalities before generating 3D content and makes audio-to-3D possible in a single pipeline.

What carries the argument

The central mechanism is the Modality Similarity (MS) loss combined with Hybrid Diffusion Supervision. The MS loss is a cosine-similarity objective between the ImageBind embedding of the prompt and CLIP image embeddings of augmented rendered views, giving any modality a direct gradient path into the 3D representation. Hybrid Diffusion Supervision adds pixel-level planar losses (augmented SDS and CDS from the 2D diffusion model) and spatial-level stereoscopic losses (3D SDS and a reference-view loss from the 3D-aware diffusion model), so the framework gets both open-world imagination and view consistency. The three-phase schedule, which starts with a low-resolution NeRF and refines geometry and then texture with DMTet (a hybrid surface representation), is the optimization scaffold that deploys these losses at increasing resolution.

What would settle it

Measure the distribution gap between ImageBind embeddings and the CLIP image embeddings that Stable Diffusion v2-1-unclip was trained on using a paired-image set; if the two spaces are far apart, the paper's direct conditioning of the 2D diffusion model by ImageBind vectors cannot be doing the work it claims.

Watch

Extended reading notes

Core claim

The central claim is that a single 3D-generation pipeline can be conditioned on any modality by using a shared embedding space. XBind encodes the input with ImageBind, feeds the resulting embedding into both a 2D diffusion model (Stable Diffusion v2-1-unclip) and a 3D-aware diffusion model (Zero-1-to-3), and optimizes a NeRF-to-DMTet representation under hybrid diffusion supervision. A new Modality Similarity loss ties the modality embedding to CLIP embeddings of augmented rendered views, which the paper says is what lets arbitrary modalities actually steer the 3D object. On the reported benchmarks, XBind reaches CLIP-R 0.8050 on the DreamFusion text gallery, compared with 0.7319 for ProlificDreamer, CLIP-I 0.8554 for image prompts, and ARCC 0.4860 for audio prompts; the ablations attribute the gains to the stereoscopic supervision, planar supervision, three-phase schedule, and MS loss.

Load-bearing premise

The load-bearing premise is that ImageBind embeddings can be fed directly into Stable Diffusion v2-1-unclip as if they were CLIP image embeddings, even though the two embedding spaces come from separately trained models and the paper describes no projection or fine-tuning to reconcile them.

Editorial extensions

If this is right

  • A single trained pipeline can switch among text, image, and audio prompts without retraining or explicit modality conversion.
  • Audio-to-3D generation becomes practical without first converting sound to text or images, avoiding information loss.
  • Combining 2D and 3D diffusion supervision suppresses view-inconsistent artifacts such as the multi-face Janus problem.
  • The coarse-to-fine NeRF-then-DMTet schedule yields high-resolution textured meshes rather than low-fidelity radiance fields.
  • Prompt alignment, quantified by CLIP-R, CLIP-I, and ARCC, is higher than the constructed baselines and than the compared text-to-3D methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any new modality later added to a shared embedding space, such as depth or thermal sensing, should become a 3D prompt with no change to the 3D pipeline.
  • Substituting a different aligned encoder for ImageBind would test whether the recipe depends on the shared-space property or on ImageBind's specific training; if the metrics transfer, the method generalizes as a recipe.
  • Extending the same hybrid supervision to full 3D scenes, beyond single objects, is a natural next step that the paper's stated limitations leave open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes XBind, a unified framework for generating 3D objects from prompts of arbitrary modality (text, image, audio). The method encodes the prompt with ImageBind into a shared embedding space, conditions a 2D diffusion model (Stable Diffusion v2-1-unclip) and a 3D-aware diffusion model (Zero-1-to-3) on that embedding, and optimizes a NeRF/DMTet representation in three phases (coarse NeRF, DMTet geometry, DMTet texture). The principal new components are the Modality Similarity (MS) loss (Eq. 2), which maximizes the cosine similarity between the ImageBind prompt embedding and CLIP image embeddings of rendered and augmented views; Hybrid Diffusion Supervision combining planar 2D losses with stereoscopic 3D losses; and the three-phase coarse-to-fine optimization schedule. The paper claims that XBind outperforms three constructed baselines and four SOTA text-to-3D methods on CLIP-R, CLIP-I, and a newly proposed ARCC metric (Tables I and II), and it presents qualitative comparisons and an ablation study (Figs. 3-7).

Significance. XBind targets a genuinely useful gap: a single pipeline that accepts text, image, and audio prompts for 3D object generation without an explicit modality-to-text or modality-to-image conversion step, and the qualitative results in Figs. 3-5 and the module-level ablation in Fig. 7 suggest that the framework can produce plausible, view-consistent objects. The MS loss is a reasonable mechanism for tightening prompt-render alignment, and the combination of ImageBind, Stable unCLIP, and Zero-1-to-3 priors inside one optimization loop is a sensible engineering idea with clear presentation. These strengths are real and should be credited. However, the quantitative evidence is not currently independent: CLIP-R, CLIP-I, and ARCC all overlap with the MS training objective, there are no error bars, significance tests, or repeated seeds, and the ablation is qualitative only. The significance of the paper is therefore conditional on a substantially reworked evaluation using human studies or held-out embedding spaces that were not part of the training objective.

major comments (4)
  1. [Sec. V-A.4, Eq. (2), Tables I-II] The three evaluation metrics overlap directly with the MS loss that XBind optimizes, so Tables I and II do not provide independent evidence of superiority. CLIP-R measures CLIP cosine similarity between the text prompt and rendered views, while the MS loss in Eq. (2) maximizes cos(C_m, E(x)) with E being the CLIP image encoder; consequently the text-to-3D margin (XBind 0.8050/0.8167 vs. ProlificDreamer 0.7319) partly reflects the fact that XBind explicitly optimizes a CLIP-similarity term that the SOTA methods do not. For image-to-3D, CLIP-I measures similarity to a reference image whose ImageBind embedding is the anchor C_m of the MS loss, and for audio-to-3D the proposed ARCC metric (Eq. 14) computes cos(C(A), C(I)) with the same ImageBind encoder that produces the conditioning embedding C_m, making the audio scores in Table I a near-direct evaluation of whether the MS loss was active. The baselines in Table I also omit the MS loss, and Section VI (Limitations) does not acknowledge this metric-objective overlap. The revision must add independent evaluation: human preference ratings, prompt-render similarity measured with an embedding model not used in the loss, and ideally multi-view consistency or geometry metrics, with repeated-seed runs and significance tests.
  2. [Sec. IV-A and footnote 1 (p. 4)] The conditioning path from ImageBind to the 2D diffusion model is underspecified and possibly invalid as stated. The paper says XBind conditions Stable Diffusion v2-1-unclip on ImageBind embeddings, but v2-1-unclip was trained to accept CLIP image embeddings of a specific dimension, whereas ImageBind embeddings come from a different encoder family; the paper describes no projection, linear layer, fine-tuning, or normalization step that reconciles the two spaces. The entire modality-conditional pipeline rests on this unstated compatibility assumption, and without it the method cannot be reproduced or even verified. Please specify the exact conditioning mechanism (embedding dimension, normalization, learned projection, or a fine-tuned unCLIP adapter) and validate it empirically; an ablation comparing raw ImageBind conditioning, a learned projection, and genuine CLIP embeddings would demonstrate that the claimed cross-modal pre-alignment is what makes any-to-3D work.
  3. [Tables I-II and Sec. V-B through V-F] The quantitative evaluation lacks statistical hygiene and internal consistency. No error bars, number of seeds, or significance tests are reported; the ablation study (Fig. 7) is qualitative only, leaving the individual contributions of the MS loss, hybrid supervision, and the three-phase schedule unquantified. In addition, XBind's CLIP-R score is 0.8050 in Table I but 0.8167 in Table II, and the paper does not state whether the prompt sets differ or why the numbers change. The ARCC metric (Eq. 14) is used to evaluate the method on the very objective it is trained on; even setting the circularity aside, a newly proposed evaluation metric should be validated against human judgments before being used as a headline result.
  4. [Sec. IV (Eqs. 2, 5, 8, 9, 11-13) and Algorithm 1] Several load-bearing hyperparameters are never given: the MS loss weights omega_g, omega_l, omega_z in Eq. (2); the hybrid-supervision weights omega_ms, omega_cds, omega_img, omega_nz, omega_3d, omega_rgb, and omega_mask in Eqs. (5), (8), (9), and (11)-(13); the augmentation weights W_i in Eq. (2); and the per-phase iteration counts N_phase-1, N_phase-2, and N_phase-3 in Algorithm 1. Without these values the framework cannot be reproduced, and the sensitivity of the reported results to these choices cannot be assessed. Please report the full configuration (or place it in a supplement) and state whether the same schedule is used for all three input modalities.
minor comments (6)
  1. [Sec. IV-A, Eq. (2)] The definition of W_i and the 'avg' operation over rendered images is ambiguous; please clarify whether the loss is averaged over a random subset of viewpoints per iteration and confirm explicitly that gradients flow through the differentiable renderer into the NeRF/DMTet parameters.
  2. [Table II] The SOTA results are obtained from threestudio; please state the resolution, number of iterations, and per-method compute budgets, since CLIP-R can be sensitive to training length and viewpoint coverage.
  3. [Fig. 3] The image-to-3D examples are described as style-preserving generation rather than reconstruction, but the criterion for when an output counts as aligned with the input image is never defined; a human-rated measure of style or layout alignment would make this claim concrete.
  4. [References] Reference [47] duplicates reference [8] (both cite Rombach et al.); the duplicate should be merged or removed.
  5. [Abstract and Sec. VII] The claim that XBind is 'the first method to generate 3D objects from any modality prompts' should be supported by a broader related-work search or softened, since the survey in Section II does not discuss possible audio-to-3D or unified multimodal-to-3D attempts in detail.
  6. [Throughout] Minor formatting issues include the inconsistent spacing of 'DMT ET' and the boxed notation in Algorithm 1; these should be cleaned up in revision.

Circularity Check

2 steps flagged · score 7.0 of 10

Quantitative SOTA margins are largely circular: the MS loss optimizes the same CLIP/ImageBind cosine similarities used by CLIP-R, CLIP-I, and ARCC, so Tables I and II do not independently establish superiority.

  1. fitted input called prediction [Section IV-A Eq. (2); Section V-A.4]
    "we compute the MS loss between the embedding of the input modality prompt and those of the rendered images as follows: LMS = ωgLms-g + ωlLms-l + ωzLms-z, Lms-g = Σi {Wi × avg [cos(Cm, E(G(x)))]}, ... where Cm represents the embedding of the modality prompt encoded by the multimodal-aligned encoder, E denotes the CLIP image encoder"

    CLIP-R computes the CLIP text-image similarity between each text prompt and the rendered views, and Cm for a text prompt is the same text embedding fed into the MS loss; E is the same CLIP image encoder. For image prompts, CLIP-I measures the reference-image-to-rendered CLIP similarity, again the quantity maximized by Eq. (2). Since the SOTA text-to-3D methods and the constructed baselines in Tables I and II are not trained with this term, their lower scores partly reflect that the evaluation metric is contained in the training loss. The margin is therefore not independent evidence of superior modality alignment.

  2. fitted input called prediction [Section V-A.4 Eq. (14); Section IV-A Eq. (2)]
    "For audio-to-3D, due to the lack of suitable evaluation metrics, we propose the Audio-Rendering Cosine Consistency (ARCC) metric: ARCC(A, I) = cos(C(A), C(I)), (14) where A and I represent the audio prompts and rendered images, respectively, and C denotes the multimodal-aligned encoder."

    For an audio prompt, the same ImageBind audio embedding is used as Cm in the MS loss and as C(A) in ARCC, and the rendered-image side is scored in the same aligned visual embedding space that ImageBind binds to that audio modality. Thus ARCC measures how well XBind maximized the very audio-image similarity that the MS loss and the frozen ImageBind space define, rather than an external or held-out measure of audio-3D alignment. The reported ARCC superiority over baselines largely reflects that only XBind optimizes this objective.

full rationale

The paper's technical contributions—hybrid diffusion supervision, three-phase optimization, and the any-modality pipeline—are independent of the metrics and are supported by qualitative figures and ablations. However, the headline quantitative claim rests on CLIP-R, CLIP-I, and ARCC, and each of these metrics is the same cosine-similarity objective trained in Eq. (2): Cm (from ImageBind/CLIP) is matched against rendered-view embeddings from the CLIP image encoder, while CLIP-R, CLIP-I, and ARCC score rendered views against the same prompt-embedding space. The constructed baselines and SOTA text-to-3D systems do not include the MS loss, so Tables I and II substantially re-demonstrate the effect of adding the MS loss to the reported metrics. No self-citation chain or imported uniqueness theorem is involved; the circularity is the loss/metric overlap. Score 7 reflects that the quantitative comparison is substantially circular while the qualitative and ablative evidence retains independent value.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The framework depends on pretrained models (ImageBind, Stable unCLIP, Zero-1-to-3) plus a set of unlisted loss weights and iteration counts. The ARCC metric is a self-referential evaluation tool, and the compatibility of ImageBind embeddings with Stable unCLIP conditioning is assumed without a described mechanism.

free parameters (3)
  • MS loss weights omega_g, omega_l, omega_z = not reported
    Eq. 2 introduces three weighting parameters for global, local, and normalize augmentations; no values or schedules are provided.
  • Hybrid supervision weights (omega_ms, omega_cds, omega_img, omega_nz, omega_3d, omega_rgb, omega_mask) = not reported
    Eqs. 5, 8, 9, 11, and 12 define weighted combinations of losses, but none of the weights are listed in the implementation details.
  • Phase iteration counts N_phase-1, N_phase-2, N_phase-3 = not reported
    Algorithm 1 requires per-phase iteration counts; without them, the coarse-to-fine schedule cannot be replicated exactly.
assumptions (3)
  • domain assumption ImageBind embeddings can be used directly as conditioning for Stable Diffusion v2-1-unclip, despite that model being trained on CLIP image embeddings.
    The paper's method section and footnote 1 state the 2D model accepts CLIP image embeddings, but no projection or fine-tuning is described to bridge from ImageBind embeddings.
  • domain assumption Score distillation gradients (SDS, CDS) are valid for optimizing NeRF and DMTet representations in all three phases.
    The method relies on SDS (Eq. 4, 5) and CDS (Eq. 3) gradients to update 3D parameters; this is a standard but unproven-for-this-setting assumption inherited from prior work.
  • domain assumption Cosine similarity in CLIP/ImageBind embedding spaces is a valid proxy for prompt-3D alignment.
    Both the MS loss and the evaluation metrics are defined as embedding cosine similarity, so the validity of the reported quality hinges on this association.
invented entities (1)
  • Audio-Rendering Cosine Consistency (ARCC) metric
    purpose: Evaluation metric for audio-to-3D alignment.
    ARCC (Eq. 14) computes cosine similarity between audio and rendered-image embeddings using the same multimodal-aligned encoder family used in the training pipeline, so it does not provide an independent test of quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Any-to-3D Generation via Hybrid Diffusion Supervision." pith.science (2026). https://pith.science/paper/C5M5H2XI

@misc{pith2026241114715,
  author       = {Pith},
  title        = {Pith review of: Any-to-3D Generation via Hybrid Diffusion Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5M5H2XI}},
  note         = {Machine review of arXiv:2411.14715}
}
read the original abstract

Recent progress in 3D object generation has been fueled by the strong priors offered by diffusion models. However, existing models are tailored to specific tasks, accommodating only one modality at a time and necessitating retraining to change modalities. Given an image-to-3D model and a text prompt, a naive approach is to convert text prompts to images and then use the image-to-3D model for generation. This approach is both time-consuming and labor-intensive, resulting in unavoidable information loss during modality conversion. To address this, we introduce XBind, a unified framework for any-to-3D generation using cross-modal pre-alignment techniques. XBind integrates an multimodal-aligned encoder with pre-trained diffusion models to generate 3D objects from any modalities, including text, images, and audio. We subsequently present a novel loss function, termed Modality Similarity (MS) Loss, which aligns the embeddings of the modality prompts and the rendered images, facilitating improved alignment of the 3D objects with multiple modalities. Additionally, Hybrid Diffusion Supervision combined with a Three-Phase Optimization process improves the quality of the generated 3D objects. Extensive experiments showcase XBind's broad generation capabilities in any-to-3D scenarios. To our knowledge, this is the first method to generate 3D objects from any modality prompts. Project page: https://zeroooooooow1440.github.io/.

Figures

Figures reproduced from arXiv: 2411.14715 by the authors.

Figure 1
Figure 1. Comparison of various methods for Any-to-3D generation: (a) Utilizing separate expert models for Any-to-3D generation. (b) Simply concatenating [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Examples generated by XBind. The first row represents text-to-3D, the second row represents image-to-3D with the image prompt input located at [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Qualitative comparison with baselines. The first row represents text-to-3D, the second row represents image-to-3D, and the third row represents [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Qualitative comparison with SOTA methods in the text-to-3D domain. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Results of using different 2D diffusion model supervision losses in [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Ablation study of XBind. The first row represents text-to-3D, the second row represents image-to-3D, and the third row represents audio-to-3D. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 30 canonical work pages

  1. [1]

    A comprehensive survey on 3d content generation,

    J. Liu, X. Huang, T. Huang, L. Chen, Y . Hou, S. Tang, Z. Liu, W. Ouyang, W. Zuo, J. Jiang et al. , “A comprehensive survey on 3d content generation,” arXiv preprint arXiv:2402.01166 , 2024

    arXiv 2024

  2. [2]

    Mvdream: Multi- view diffusion for 3d generation,

    Y . Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang, “Mvdream: Multi- view diffusion for 3d generation,” arXiv preprint arXiv:2308.16512 , 2023

    arXiv 2023

  3. [3]

    Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior,

    J. Sun, B. Zhang, R. Shao, L. Wang, W. Liu, Z. Xie, and Y . Liu, “Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior,” arXiv preprint arXiv:2310.16818 , 2023

    arXiv 2023

  4. [4]

    Crm: Single image to 3d textured mesh with convolutional reconstruction model,

    Z. Wang, Y . Wang, Y . Chen, C. Xiang, S. Chen, D. Yu, C. Li, H. Su, and J. Zhu, “Crm: Single image to 3d textured mesh with convolutional reconstruction model,” arXiv preprint arXiv:2403.05034 , 2024

    arXiv 2024

  5. [5]

    Scalable diffusion models with transformers,

    W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 4195–4205

    2023

  6. [6]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 . Springer, 2015, pp. 234–241

    2015

  7. [7]

    Hierarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, vol. 1, no. 2, p. 3, 2022

    arXiv 2022

  8. [8]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

Show all 74 references
  1. [9]

    Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,

    R. Chen, Y . Chen, N. Jiao, and K. Jia, “Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 246–22 256

    2023

  2. [10]

    Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation,

    H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich, “Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 12 619–12 629

    2023

  3. [11]

    Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior,

    Z. Wu, P. Zhou, X. Yi, X. Yuan, and H. Zhang, “Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior,” 2024

    2024

  4. [12]

    Score distillation sampling with learned manifold corrective,

    T. Alldieck, N. Kolotouros, and C. Sminchisescu, “Score distillation sampling with learned manifold corrective,” 2024

    2024

  5. [13]

    Taming mode collapse in score distillation for text-to-3d generation,

    P. Wang, D. Xu, Z. Fan, D. Wang, S. Mohan, F. Iandola, R. Ranjan, Y . Li, Q. Liu, Z. Wang, and V . Chandra, “Taming mode collapse in score distillation for text-to-3d generation,” arXiv preprint: 2401.00909, 2024

    2024 arXiv

  6. [14]

    Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,

    Z. Wang, C. Lu, Y . Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  7. [15]

    Text2mesh: Text-driven neural stylization for meshes,

    O. Michel, R. Bar-On, R. Liu, S. Benaim, and R. Hanocka, “Text2mesh: Text-driven neural stylization for meshes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 492–13 502

    2022

  8. [16]

    Cad: Photorealistic 3d generation via adversarial dis- tillation,

    Z. Wan, D. Paschalidou, I. Huang, H. Liu, B. Shen, X. Xiang, J. Liao, and L. Guibas, “Cad: Photorealistic 3d generation via adversarial dis- tillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 10 194–10 207

    2024

  9. [17]

    Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,

    J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” arXiv preprint arXiv:2309.16653, 2023

    2023 arXiv

  10. [18]

    Make-it-3d: High-fidelity 3d creation from a single image with diffu- sion prior,

    J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen, “Make-it-3d: High-fidelity 3d creation from a single image with diffu- sion prior,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 22 819–22 829

    2023

  11. [19]

    Realfusion: 360deg reconstruction of any object from a single image,

    L. Melas-Kyriazi, I. Laina, C. Rupprecht, and A. Vedaldi, “Realfusion: 360deg reconstruction of any object from a single image,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 8446–8455

    2023

  12. [20]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180–15 190

    2023

  13. [21]

    Any-to-any generation via composable diffusion,

    Z. Tang, Z. Yang, C. Zhu, M. Zeng, and M. Bansal, “Any-to-any generation via composable diffusion,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  14. [22]

    Dreamfusion: Text- to-3d using 2d diffusion,

    B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “Dreamfusion: Text- to-3d using 2d diffusion,” 2022

    2022

  15. [23]

    Zero-1-to-3: Zero-shot one image to 3d object,

    R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. V on- drick, “Zero-1-to-3: Zero-shot one image to 3d object,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 9298–9309

    2023

  16. [24]

    Mip-nerf: A multiscale representation for anti- aliasing neural radiance fields,

    J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti- aliasing neural radiance fields,” in Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , 2021, pp. 5855–5864

    2021

  17. [25]

    Instant neural graphics primitives with a multiresolution hash encoding,

    T. M ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM transactions on graphics (TOG), vol. 41, no. 4, pp. 1–15, 2022

    2022

  18. [26]

    Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthe- sis,

    T. Shen, J. Gao, K. Yin, M.-Y . Liu, and S. Fidler, “Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthe- sis,” Advances in Neural Information Processing Systems , vol. 34, pp. 6087–6101, 2021. 13

    2021

  19. [27]

    Diffusion models: A comprehensive survey of methods and applications,

    L. Yang, Z. Zhang, Y . Song, S. Hong, R. Xu, Y . Zhao, W. Zhang, B. Cui, and M.-H. Yang, “Diffusion models: A comprehensive survey of methods and applications,” ACM Computing Surveys , vol. 56, no. 4, pp. 1–39, 2023

    2023

  20. [28]

    Score-based generative modeling through stochastic differ- ential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differ- ential equations,” arXiv preprint arXiv:2011.13456 , 2020

    2011 arXiv

  21. [29]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , vol. 33, pp. 6840– 6851, 2020

    2020

  22. [30]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  23. [31]

    Vector quantized diffusion model for text-to-image synthesis,

    S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo, “Vector quantized diffusion model for text-to-image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 696–10 706

    2022

  24. [32]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in neural information processing systems , vol. 34, pp. 8780–8794, 2021

    2021

  25. [33]

    Improving diffusion-based image synthesis with context pre- diction,

    L. Yang, J. Liu, S. Hong, Z. Zhang, Z. Huang, Z. Cai, W. Zhang, and B. Cui, “Improving diffusion-based image synthesis with context pre- diction,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  26. [34]

    Shifted diffusion for text-to-image generation,

    Y . Zhou, B. Liu, Y . Zhu, X. Yang, C. Chen, and J. Xu, “Shifted diffusion for text-to-image generation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 10 157–10 166

    2023

  27. [35]

    Text2video-zero: Text-to-image diffusion models are zero-shot video generators,

    L. Khachatryan, A. Movsisyan, V . Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi, “Text2video-zero: Text-to-image diffusion models are zero-shot video generators,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 954–15 964

    2023

  28. [36]

    Videofusion: Decomposed diffusion models for high-quality video generation,

    Z. Luo, D. Chen, Y . Zhang, Y . Huang, L. Wang, Y . Shen, D. Zhao, J. Zhou, and T. Tan, “Videofusion: Decomposed diffusion models for high-quality video generation,” in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2023, pp. 10 209–10 218

    2023

  29. [37]

    Vidm: Video implicit diffusion models,

    K. Mei and V . Patel, “Vidm: Video implicit diffusion models,” in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 8, 2023, pp. 9117–9125

    2023

  30. [38]

    Lavie: High-quality video generation with cascaded latent diffusion models,

    Y . Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y . Wang, C. Yang, Y . He, J. Yu, P. Yanget al., “Lavie: High-quality video generation with cascaded latent diffusion models,” arXiv preprint arXiv:2309.15103 , 2023

    2023 arXiv

  31. [39]

    Prodiff: Progressive fast diffusion model for high-quality text-to-speech,

    R. Huang, Z. Zhao, H. Liu, J. Liu, C. Cui, and Y . Ren, “Prodiff: Progressive fast diffusion model for high-quality text-to-speech,” in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 2595–2605

    2022

  32. [40]

    Taming diffusion models for audio-driven co-speech gesture generation,

    L. Zhu, X. Liu, X. Liu, R. Qian, Z. Liu, and L. Yu, “Taming diffusion models for audio-driven co-speech gesture generation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 544–10 553

    2023

  33. [41]

    Speech enhancement and dereverberation with diffusion-based genera- tive models,

    J. Richter, S. Welker, J.-M. Lemercier, B. Lay, and T. Gerkmann, “Speech enhancement and dereverberation with diffusion-based genera- tive models,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2351–2364, 2023

    2023

  34. [42]

    Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,

    J.-M. Lemercier, J. Richter, S. Welker, and T. Gerkmann, “Storm: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023

    2023

  35. [43]

    Hyperfields: Towards zero-shot generation of nerfs from text,

    S. Babu, R. Liu, A. Zhou, M. Maire, G. Shakhnarovich, and R. Hanocka, “Hyperfields: Towards zero-shot generation of nerfs from text,” arXiv preprint arXiv:2310.17075, 2023

    2023 arXiv

  36. [44]

    Texfusion: Synthesiz- ing 3d textures with text-guided image diffusion models,

    T. Cao, K. Kreis, S. Fidler, N. Sharp, and K. Yin, “Texfusion: Synthesiz- ing 3d textures with text-guided image diffusion models,” inProceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4169–4181

    2023

  37. [45]

    Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching,

    Y . Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y . Chen, “Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 6517–6526

    2024

  38. [46]

    Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors,

    D. Z. Chen, H. Li, H.-Y . Lee, S. Tulyakov, and M. Nießner, “Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 21 081–21 091

    2024

  39. [47]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2022, pp. 10 684–10 695

    2022

  40. [48]

    Syncdreamer: Learning to generate multiview-consistent images from a single-view image,

    Y . Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang, “Syncdreamer: Learning to generate multiview-consistent images from a single-view image,” arXiv preprint arXiv:2309.03453 , vol. 15, 2023

    2023 arXiv

  41. [49]

    Objaverse: A universe of annotated 3d objects,

    M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi, “Objaverse: A universe of annotated 3d objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 13 142–13 153

    2023

  42. [50]

    Objaverse-xl: A uni- verse of 10m+ 3d objects,

    M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V . V oleti, S. Y . Gadreet al., “Objaverse-xl: A uni- verse of 10m+ 3d objects,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  43. [51]

    Omnivore: A single model for many visual modalities,

    R. Girdhar, M. Singh, N. Ravi, L. Van Der Maaten, A. Joulin, and I. Misra, “Omnivore: A single model for many visual modalities,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 102–16 112

    2022

  44. [52]

    Polyvit: Co-training vision transformers on images, videos and audio,

    V . Likhosherstov, A. Arnab, K. Choromanski, M. Lucic, Y . Tay, A. Weller, and M. Dehghani, “Polyvit: Co-training vision transformers on images, videos and audio,” arXiv preprint arXiv:2111.12993 , 2021

    2021 arXiv

  45. [53]

    Look, listen and learn,

    R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in Proceed- ings of the IEEE international conference on computer vision , 2017, pp. 609–617

    2017

  46. [54]

    Omnimae: Single model masked pretraining on images and videos,

    R. Girdhar, A. El-Nouby, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Omnimae: Single model masked pretraining on images and videos,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 10 406–10 417

    2023

  47. [55]

    Audio-visual instance discrimination with cross-modal agreement,

    P. Morgado, N. Vasconcelos, and I. Misra, “Audio-visual instance discrimination with cross-modal agreement,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 475–12 486

    2021

  48. [56]

    Contrastive multiview coding,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16 . Springer, 2020, pp. 776–794

    2020

  49. [57]

    Bevt: Bert pretraining of video transformers,

    R. Wang, D. Chen, Z. Wu, Y . Chen, X. Dai, M. Liu, Y .-G. Jiang, L. Zhou, and L. Yuan, “Bevt: Bert pretraining of video transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 14 733–14 743

    2022

  50. [58]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  51. [59]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  52. [60]

    Unifying vision-and-language tasks via text generation,

    J. Cho, J. Lei, H. Tan, and M. Bansal, “Unifying vision-and-language tasks via text generation,” in International Conference on Machine Learning. PMLR, 2021, pp. 1931–1942

    2021

  53. [61]

    Merlot: Multimodal neural script knowledge models,

    R. Zellers, X. Lu, J. Hessel, Y . Yu, J. S. Park, J. Cao, A. Farhadi, and Y . Choi, “Merlot: Multimodal neural script knowledge models,” Advances in neural information processing systems, vol. 34, pp. 23 634– 23 651, 2021

    2021

  54. [62]

    Multimodal few-shot learning with frozen language models,

    M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill, “Multimodal few-shot learning with frozen language models,” Advances in Neural Information Processing Systems , vol. 34, pp. 200– 212, 2021

    2021

  55. [63]

    Vilam: A vision-language model with enhanced visual grounding and generalization capability,

    X. Yang, L. Xu, H. Li, and S. Zhang, “Vilam: A vision-language model with enhanced visual grounding and generalization capability,” arXiv preprint arXiv:2311.12327, 2023

    2023 arXiv

  56. [64]

    Tvlt: Textless vision- language transformer,

    Z. Tang, J. Cho, Y . Nie, and M. Bansal, “Tvlt: Textless vision- language transformer,” Advances in neural information processing sys- tems, vol. 35, pp. 9617–9632, 2022

    2022

  57. [65]

    i-code: An integrative and composable multimodal learning framework,

    Z. Yang, Y . Fang, C. Zhu, R. Pryzant, D. Chen, Y . Shi, Y . Xu, Y . Qian, M. Gao, Y .-L. Chen et al. , “i-code: An integrative and composable multimodal learning framework,” inProceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 9, 2023, pp. 10 880–10 890

    2023

  58. [66]

    Merlot reserve: Neural script knowledge through vision and language and sound,

    R. Zellers, J. Lu, X. Lu, Y . Yu, Y . Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y . Choi, “Merlot reserve: Neural script knowledge through vision and language and sound,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022,...

    2022

  59. [67]

    Codi-2: In-context interleaved and interactive any-to-any generation,

    Z. Tang, Z. Yang, M. Khademi, Y . Liu, C. Zhu, and M. Bansal, “Codi-2: In-context interleaved and interactive any-to-any generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 425–27 434

    2024

  60. [68]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” in ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  61. [69]

    Audioldm: Text-to-audio generation with latent diffusion models,

    H. Liu, Z. Chen, Y . Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley, “Audioldm: Text-to-audio generation with latent diffusion models,” arXiv preprint arXiv:2301.12503 , 2023

    2023 arXiv

  62. [70]

    Clipcap: Clip prefix for image captioning,

    R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,” arXiv preprint arXiv:2111.09734 , 2021

    2021 arXiv

  63. [71]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  64. [72]

    Hifa: High-fidelity text-to- 3d generation with advanced diffusion guidance,

    J. Zhu, P. Zhuang, and S. Koyejo, “Hifa: High-fidelity text-to- 3d generation with advanced diffusion guidance,” arXiv preprint arXiv:2305.18766, 2023

    2023 arXiv

  65. [73]

    Magic3d: High-resolution text-to- 3d content creation,

    C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y . Liu, and T.-Y . Lin, “Magic3d: High-resolution text-to- 3d content creation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 300–309

    2023

  66. [74]

    threestudio: A unified framework for 3d content generation,

    Y .-C. Guo, Y .-T. Liu, R. Shao, C. Laforte, V . V oleti, G. Luo, C.-H. Chen, Z.-X. Zou, C. Wang, Y .-P. Cao, and S.-H. Zhang, “threestudio: A unified framework for 3d content generation,” https://github.com/ threestudio-project/threestudio, 2023

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.