Pith. sign in

REVIEW 3 major objections 5 minor 52 references

A vision-language model's Yes/No verdict becomes a differentiable signal that steers text-to-3D generation toward semantic and geometric correctness.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:37 UTC pith:WFBHR2RX

load-bearing objection A genuinely useful training-free idea with solid optimization-based results; the feed-forward guidance claims need stronger evidence before they carry the weight the paper puts on them. the 3 major comments →

arxiv 2511.14271 v2 pith:WFBHR2RX submitted 2025-11-18 cs.CV

Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation

Weimin Bai , Yubo Li , Weijian Luo , Zeqiang Lai , Yequan Wang , Wenzheng Chen , He Sun This is my paper
classification cs.CV
keywords text-to-3D generationvision-language modelsdifferentiable rewardscore distillation samplingtest-time guidancespatial reasoninggeometric consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a frozen vision-language model can act as a general, differentiable critic for text-to-3D generation, provided its binary verdict is converted into a scalar signal. The core idea is to ask the model whether multi-view renderings of a 3D object satisfy two criteria at once—content match with the prompt and geometric soundness across views—and then use the difference between the Yes and No log-odds as the reward. The authors show the same signal works in two different settings: as an extra objective inside score-distillation optimization, and as a test-time gradient that nudges the sampling loop of native 3D diffusion models. If the claim holds, one language-grounded signal can fix both fine-grained semantic omissions and severe spatial assembly errors without retraining the generator.

Core claim

On the paper's own terms, the discovery is that a vision-language model constrained to answer 'Yes' or 'No' yields a usable, differentiable reward for 3D generation: rVLM = z_yes − z_no, the difference between the two output logits. The reward is computed over multiple rendered views of the 3D representation with a dual-query prompt that simultaneously asks whether the object matches the text and whether it is geometrically consistent across views. Because the whole chain—3D parameters, differentiable renderer, VLM, scalar—is differentiable, backpropagation gives a gradient that steers the representation. The same signal is used both as a reward term combined with score distillation sampling

What carries the argument

The central object is the dual-query critic signal rVLM = z_yes − z_no. A vision-language model is prompted with a structured question containing two criteria—Content Match, meaning the object corresponds to the text description, and Geometric Quality, meaning the object looks sound and consistent across views with no Janus faces, broken surfaces, or intersecting geometry—and is forced to answer only 'Yes' or 'No'. The difference of the output logits makes the entire mapping from 3D parameters to reward differentiable, so gradients can flow back through the renderer into the representation. In optimization-based use, this reward is added to the score-distillation loss with an annealing weigh

Load-bearing premise

The load-bearing premise is that the gradient of the frozen VLM's Yes-minus-No score with respect to a partially denoised 3D latent is numerically stable and consistently points toward a better shape at every sampling step; if that gradient is unreliable at high-noise steps, the test-time guidance cannot be what corrects the geometry.

What would settle it

On a fixed prompt and a fixed native 3D backbone, run the full 50-step sampling and replace the VLM gradient at the first ten steps with a random vector of the same norm; if the final assets are still coherent, those early gradients are not causing the correction. Or compute the gradient norm and direction at early steps for a known part-assembly failure: near-zero norm or directions that flip relative to the sampler's own noise would indicate the guidance term is not the mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single frozen VLM reward can be added to an optimization-based pipeline and improve text alignment, 3D plausibility, texture, geometry, and coherence metrics at once.
  • Native 3D feed-forward models can be corrected at test time, with no retraining, by injecting the VLM gradient during their sampling loop.
  • Dropping either the geometry criterion or the multi-view input brings back classic artifacts such as Janus faces, floating parts, and fractured surfaces, so both are load-bearing design choices.
  • The same reward formula transfers across two very different paradigms—thousands of optimization steps or a 50-step denoiser—suggesting it is a general critic rather than a pipeline-specific patch.
  • Because the reward comes from language-grounded reasoning, it responds to single-concept prompt changes such as color, action, or spatial relation, which contrastive text encoders often miss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the same dual-query log-odds recipe could transfer to other generative domains with differentiable renderings, such as video, 4D, or multi-object scenes, because the critic never needs 3D-specific training.
  • Our inference: splitting the two criteria into separate critic heads would likely make the gradient more interpretable and let users trade off semantic strictness against geometric strictness independently; the paper lists this as a natural next step.
  • Our inference: the test-time guidance probably depends on gradient reliability at high-noise steps; gating or annealing the guidance by noise level is a cheap change that would either extend or expose the limits of the mechanism.
  • Our inference: for very long prompts the reward may saturate and underuse the VLM's visual detail, a point the paper's own limitation section notes; hierarchical queries that first lock coarse structure and then refine details could extend the approach.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VLM3D, a framework that repurposes a pretrained vision-language model (VLM) as a differentiable critic for text-to-3D generation. The core signal is the log-odds rVLM = z_yes − z_no obtained from a dual-query prompt that asks the VLM to judge both content match and geometric quality from multi-view renderings. This signal is used in two settings: (1) as a reward term added to SDS-style optimization, and (2) as a test-time guidance term injected into the denoising loop of native 3D feed-forward models such as Hunyuan3D. The paper reports state-of-the-art results on the GPTEval3D benchmark (Table 1) and qualitative/quantitative corrections of spatial errors in feed-forward generation (Table 2, Figs. 1, 4), with ablations showing the importance of the geometric query and multi-view inputs.

Significance. If the claims hold, VLM3D is a simple, parameter-free way to inject both semantic and spatial knowledge into diverse 3D generation pipelines. The strength of the idea is that it uses off-the-shelf VLM log-odds with no trained reward model, and the ablations in Fig. 6 provide controlled evidence that the geometry query and multi-view input matter. The manuscript is clearly written and the qualitative results are compelling. However, the quantitative support is currently uneven: the GPTEval3D comparison changes the judge relative to the original benchmark without clarifying whether all baselines were re-scored, no confidence intervals are provided for Elo ratings, and the feed-forward evaluation uses an undefined metric on a handful of selected cases. The central idea is plausible, but the evidence presented here is not yet sufficient for the strong claims made.

major comments (3)
  1. [§5.1, Table 1 and Footnote 1] The headline claim that VLM3D outperforms all baselines on every metric depends on a common evaluation protocol. The footnote says only that GPT-4V was deprecated and GPT-4o-mini was used; it does not state whether every baseline row was re-scored under this new judge or whether numbers are copied from prior publications. If the baseline scores come from different judges, the comparison is not valid. Additionally, Elo scores are reported as point estimates with no variance. With 110 prompts, even large Elo gaps may be within noise. Please report the exact protocol, per-prompt win rates, bootstrapped confidence intervals, or a paired significance test.
  2. [§5.2, Table 2] The quantitative support for the feed-forward guidance paradigm is inadequate. The 'Geo.' metric is never defined — the text only refers to 'a comprehensive Geometry score' — and the details for FID/CLIP-FID computation are missing. More importantly, the evaluation set consists only of the selected cases shown in Figs. 1 and 4, which is a handful of curated examples. This makes the claim of 'correcting severe spatial errors' essentially anecdotal. Please define the Geo. metric, describe the rendering and evaluation pipeline, report results over a larger pre-specified set of prompts with spatial/part-assembly challenges, and include error bars or statistical tests.
  3. [§4.3.2, Eq. (10)] The test-time guidance formula is only described 'conceptually'. The actual computation graph for ∇_{z_t} r_VLM is unspecified: which differentiable renderer is used, how the gradient is backpropagated through the frozen 7B VLM and the DiT model, and how the gradient is scaled or clipped. No evidence is provided that the gradient is a reliable semantic/spatial direction at high-noise early denoising steps, where the rendered images are far from valid 3D objects. This is load-bearing for the second paradigm. Please provide implementation details and report the gradient norm/SNR over t, an ablation of λ_TTG, and a quantitative evaluation on a non-selected set of prompts.
minor comments (5)
  1. [§4.3.2, Eq. (10)] Typo: 'guidace gradient' should be 'guidance gradient'.
  2. [§5.2, Qualitative Comparison] Typo: 'VM3D-guided' should be 'VLM3D-guided'.
  3. [Table 2] The arrow directions in the header (CLIP-D↓, FID↓, CLIP-FID↓, Geo.↓) are not standard and should be clarified; also define CLIP-D and state which views/renders are used for FID and CLIP-FID.
  4. [References] References [2] and [3] both cite the Qwen2.5-VL technical report, and [43] and [44] both cite the same GPT-4V evaluation paper. Please consolidate to avoid duplicate citations.
  5. [Footnote 1] The footnote states that GPT-4V has been deprecated in GPTEval3D; consider citing the current benchmark version and clarify whether the original GPTEval3D prompt set is used unchanged.

Circularity Check

0 steps flagged

No significant circularity: rVLM is a parameter-free log-odds reward from a frozen VLM, and the main benchmark uses a separate evaluator and established prompt set.

full rationale

The central derivation chain is Eq. 8 (rVLM = log P(Yes|y,X) − log P(No|y,X) = z_yes − z_no), followed by Eq. 9 (L_total = L_SDS − λ_VLM r_VLM) and Eq. 10 (z_{t−1} = Sampler(z_t, t) + λ_TTG ∇_{z_t} r_VLM). No parameter is fit to the target benchmark: the reward is extracted from frozen Qwen2.5-VL log-odds, and the GPTEval3D evaluation is performed by GPT-4o-mini using the benchmark's established pairwise Elo protocol on its 110 prompts. The dual-query prompt is a design choice, not an inverse of the evaluation metric. The ablations in Sec. 5.3 (removing the geometry query, using single-view input) directly test the critic components' contributions, which is the right kind of control for a non-circular claim. The only self-citation is in Sec. 4.3.2: 'As in [4, 15], this test-time guidance approach requires no model re-training.' Reference [4] shares authors with this paper, but it is paired with external [15] and supports only the peripheral, non-load-bearing claim that test-time guidance can be applied without retraining; the central result does not reduce to that citation. The manuscript's own limitation paragraph (Sec. 6: VLM3D 'still misses finer details such as the nurse's lifted leg and outstretched arm') further confirms that the reported gains are not definitional. Weaknesses such as the undefined 'Geo.' metric in Table 2 and the absence of gradient-reliability analysis for Eq. 10 are correctness/evidence concerns, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on the empirical effectiveness of a frozen VLM's binary log-odds as a differentiable reward, plus the stability of backpropagated gradients through a 7B VLM and a 3D diffusion sampler. The paper provides ablations supporting the query design, but several hyperparameters (λ_VLM, λ_TTG, N) are unreported, and the evaluator (GPT-4o-mini) differs from the original benchmark protocol.

free parameters (3)
  • λ_VLM (reward weight) = annealing schedule; exact values unspecified
    Controls balance between SDS loss and VLM reward (Eq. 9); authors describe a high-to-low schedule but do not give the schedule; all optimization results depend on it.
  • λ_TTG (test-time guidance scale) = not specified
    Gradient scale in Eq. 10; no value given; controls how strongly the VLM critic steers the 3D diffusion sampler.
  • N (number of rendered views) = not specified
    Sizes the multi-view input in Eq. 5; the ablation says multi-view is critical, but the exact N used in experiments is not stated.
axioms (4)
  • domain assumption A frozen VLM's binary Yes/No log-odds form a meaningful, differentiable semantic-spatial critic (Eqs. 7-8).
    Core of the method, Sec 4.1; supported only empirically; VLM logits are not calibrated and gradient quality is unverified.
  • domain assumption Qwen2.5-VL can reason about 3D consistency from a set of 2D renderings when prompted with the dual-query.
    Relies on claimed spatial capabilities of VLMs (Sec 2.3); the geometric-quality query presupposes this ability.
  • domain assumption GPT-4o-mini pairwise Elo on GPTEval3D is a valid proxy for human judgment.
    Table 1 uses this judge; original benchmark used GPT-4V; the paper does not re-validate the new judge against human ratings.
  • domain assumption The native-3D diffusion latent can be decoded, rendered, and backpropagated through at every sampling step (Eq. 10).
    Test-time guidance requires the current prediction to be differentiable; stability across the 50 steps is not analyzed.

pith-pipeline@v1.3.0-alltime-deepseek · 12325 in / 14466 out tokens · 131783 ms · 2026-08-03T21:37:39.254873+00:00 · methodology

0 comments
read the original abstract

Text-to-3D generation has advanced rapidly, yet state-of-the-art models, encompassing both optimization-based and feed-forward architectures, still face two fundamental limitations. First, they struggle with coarse semantic alignment, often failing to capture fine-grained prompt details. Second, they lack robust 3D spatial understanding, leading to geometric inconsistencies and catastrophic failures in part assembly and spatial relationships. To address these challenges, we propose VLM3D, a general framework that repurposes large vision-language models (VLMs) as powerful, differentiable semantic and spatial critics. Our core contribution is a dual-query critic signal derived from the VLM's Yes or No log-odds, which assesses both semantic fidelity and geometric coherence. We demonstrate the generality of this guidance signal across two distinct paradigms: (1) As a reward objective for optimization-based pipelines, VLM3D significantly outperforms existing methods on standard benchmarks. (2) As a test-time guidance module for feed-forward pipelines, it actively steers the iterative sampling process of SOTA native 3D models to correct severe spatial errors. VLM3D establishes a principled and generalizable path to inject the VLM's rich, language-grounded understanding of both semantics and space into diverse 3D generative pipelines.

Figures

Figures reproduced from arXiv: 2511.14271 by He Sun, Weijian Luo, Weimin Bai, Wenzheng Chen, Yequan Wang, Yubo Li, Zeqiang Lai.

Figure 1
Figure 1. Figure 1: Reproducing the “Embracing Peace” Statue with VLM3D. We challenge VLM3D’s dual paradigms with San Diego’s iconic monument. Top (Optimization-based): Given a long text description, baseline MVDream [34] suffers a catastrophic semantic failure, omitting the nurse entirely. Our VLM3D critic successfully recovers both figures and their signature pose. Key details are highlighted in red. Bottom (Feed-forward mo… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of VLM3D as a General Critic Framework. Our core contribution, a differentiable dual-query rVLM, acts as a versatile critic for 3D generation. It can be applied in two distinct paradigms: (1) As a Reward Objective: It is integrated into optimization-based pipelines (e.g., SDS [26]), replacing 2D priors with rich semantic and spatial reward. (2) As a Test-Time Guidance Module: It is used to guide t… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of VLM3D with optimization-based baselines. VLM3D outperforms these methods [46, 52] in semantic fidelity while retaining high perceptual quality. Although baseline methods achieve good texture and detail—via differentiable preference rewards or non-differentiable optimization—they often miss fine-grained concepts (highlighted in red) that VLM3D captures accurately. satility in two primary frame… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of VLM3D with feed-forward baselines. We present a qualitative comparison with SOTA native 3D models [12, 50]. Baselines exhibit significant failures, generating incomplete geometry, disconnected parts or distorted shapes. Our VLM3D, integrated as a test-time guidance module to the [12] pipeline, successfully corrects these severe spatial faults and produces coherent 3D assets. 5.1. VLM Reward f… view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity Analysis to Text Perturbations. We compare VLM3D and MVDream on pairs of prompts that differ by a single concept (highlighted in red). VLM3D accurately changes clothing color (first row), and updates spatial relations (second row), demonstrating its better semantic understanding than baselines. SDS [26] VLM3D (w/o Geometry Query) VLM3D (w/ Single View Input) VLM3D* Stable Diffusion [32] An oran… view at source ↗
Figure 6
Figure 6. Figure 6: Ablation of Geometric Query and Multi-View Input. We assess the impact of (a) removing the explicit geometry-consistency query from the VLM prompt and (b) using a single view instead of multi-view images. Omitting either component degrades 3D qual￾ity—leading to Janus-face artifacts, floating parts, and fractured surfaces. Each row uses a different diffusion backbone: the top employs Stable Diffusion 2-1 [… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 25 linked inside Pith

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Pith/arXiv arXiv 2023

  2. [3]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. Details Qwen2.5-VL architecture and capabilities

    Pith/arXiv arXiv 2025

  3. [4]

    Blind inversion using latent diffusion priors.arXiv preprint arXiv:2407.01027, 2024

    Weimin Bai, Siyi Chen, Wenzheng Chen, and He Sun. Blind inversion using latent diffusion priors.arXiv preprint arXiv:2407.01027, 2024

    Pith/arXiv arXiv 2024

  4. [5]

    Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.arXiv preprint arXiv:2106.08254, 2021

    Pith/arXiv arXiv 2021

  5. [6]

    Spatialvlm: Endowing vision–language models with spatial reasoning capabilities

    Boyuan Chen et al. Spatialvlm: Endowing vision–language models with spatial reasoning capabilities. InCVPR, 2024

    2024

  6. [7]

    Fan- tasia3d: Disentangling geometry and appearance for high- quality text-to-3d content creation

    Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fan- tasia3d: Disentangling geometry and appearance for high- quality text-to-3d content creation. InICCV, pages 22246– 22256, 2023

    2023

  7. [8]

    Spatialrgpt: Grounded spatial reasoning in vision–language models

    An-Chieh Cheng et al. Spatialrgpt: Grounded spatial reasoning in vision–language models. InarXiv preprint arXiv:2406.01584, 2024

    Pith/arXiv arXiv 2024

  8. [9]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. InNeurIPS, pages 6840–6851, 2020

    2020

  9. [10]

    Debi- asing scores and prompts of 2d diffusion for robust text-to-3d generation.arXiv preprint arXiv:2303.15413, 2023

    Susung Hong, Donghoon Ahn, and Seungryong Kim. Debi- asing scores and prompts of 2d diffusion for robust text-to-3d generation.arXiv preprint arXiv:2303.15413, 2023

    Pith/arXiv arXiv 2023

  10. [11]

    Lrm: Large reconstruction model for single image to 3d

    Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023

    Pith/arXiv arXiv 2023

  11. [12]

    Hunyuan3d 2.1: From images to high- fidelity 3d assets with production-ready pbr material.arXiv preprint arXiv:2506.15442, 2025

    Team Hunyuan3D, Shuhui Yang, Mingxin Yang, Yifei Feng, Xin Huang, Sheng Zhang, Zebin He, Di Luo, Haolin Liu, Yunfei Zhao, et al. Hunyuan3d 2.1: From images to high- fidelity 3d assets with production-ready pbr material.arXiv preprint arXiv:2506.15442, 2025

    Pith/arXiv arXiv 2025

  12. [13]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. PMLR, 2021

    2021

  13. [14]

    3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 2023

    2023

  14. [15]

    Fast diffusion em: a diffusion model for blind inverse problems with application to deconvolution

    Charles Laroche, Andrés Almansa, and Eva Coupete. Fast diffusion em: a diffusion model for blind inverse problems with application to deconvolution. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5271–5281, 2024

    2024

  15. [16]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  16. [17]

    Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.arXiv preprint arXiv:2311.06214, 2023

    Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model.arXiv preprint arXiv:2311.06214, 2023

    Pith/arXiv arXiv 2023

  17. [18]

    On erro- neous agreements of clip image embeddings.arXiv preprint arXiv:2411.05195, 2024

    Siting Li, Pang Wei Koh, and Simon Shaolei Du. On erro- neous agreements of clip image embeddings.arXiv preprint arXiv:2411.05195, 2024

    Pith/arXiv arXiv 2024

  18. [19]

    Magic3d: High-resolution text-to- 3d content creation

    Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to- 3d content creation. InCVPR, pages 300–309, 2023

    2023

  19. [20]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  20. [21]

    Latent-nerf for shape-guided generation of 3d shapes and textures.arXiv preprint arXiv:2211.07600, 2022

    Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures.arXiv preprint arXiv:2211.07600, 2022

    Pith/arXiv arXiv 2022

  21. [22]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthe- sis. InECCV, 2020

    2020

  22. [23]

    Deep geometric moments promote shape consistency in text- to-3d generation

    Utkarsh Nath, Rajeev Goel, Eun Som Jeon, Changhoon Kim, Kyle Min, Yezhou Yang, Yingzhen Yang, and Pavan Turaga. Deep geometric moments promote shape consistency in text- to-3d generation. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4331–4341. IEEE, 2025

    2025

  23. [24]

    Triplet- clip: Improving compositional reasoning of clip via synthetic vision-language negatives.Advances in neural information processing systems, 37:32731–32760, 2024

    Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Triplet- clip: Improving compositional reasoning of clip via synthetic vision-language negatives.Advances in neural information processing systems, 37:32731–32760, 2024

    2024

  24. [25]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023. 9

    2023

  25. [26]

    Barron, and Ben Mildenhall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. InICLR, 2023. Introduces Score Distillation Sampling (SDS)

    2023

  26. [27]

    Refining clip’s spatial awareness: A visual-centric perspective.arXiv preprint arXiv:2504.02328, 2025

    Congpei Qiu, Yanhao Wu, Wei Ke, Xiuxiu Bai, and Tong Zhang. Refining clip’s spatial awareness: A visual-centric perspective.arXiv preprint arXiv:2504.02328, 2025

    Pith/arXiv arXiv 2025

  27. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  28. [29]

    Learning transferable visual mod- els from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    Alec Radford et al. Learning transferable visual mod- els from natural language supervision.arXiv preprint arXiv:2103.00020, 2021

    Pith/arXiv arXiv 2021

  29. [30]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  30. [31]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InInternational Confer- ence on Machine Learning, pages 8821–8831. PMLR, 2021

    2021

  31. [32]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  32. [33]

    Retrieval-augmented score distillation for text-to-3d genera- tion.arXiv preprint arXiv:2402.02972, 2024

    Junyoung Seo, Susung Hong, Wooseok Jang, Inès Hyeonsu Kim, Minseop Kwak, Doyup Lee, and Seungryong Kim. Retrieval-augmented score distillation for text-to-3d genera- tion.arXiv preprint arXiv:2402.02972, 2024

    Pith/arXiv arXiv 2024

  33. [34]

    Mvdream: Multi-view diffusion for 3d gener- ation.arXiv preprint arXiv:2308.16512, 2023

    Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d gener- ation.arXiv preprint arXiv:2308.16512, 2023

    Pith/arXiv arXiv 2023

  34. [35]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational confer- ence on machine learning, pages 2256–2265. pmlr, 2015

    2015

  35. [36]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    Pith/arXiv arXiv 2011

  36. [37]

    Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chun- yuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu- Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf.arXiv:2309.14525, 2023

    Pith/arXiv arXiv 2023

  37. [38]

    Dreamgaussian: Generative gaussian splatting for effi- cient 3d content creation.arXiv preprint arXiv:2309.16653, 2023

    Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for effi- cient 3d content creation.arXiv preprint arXiv:2309.16653, 2023

    Pith/arXiv arXiv 2023

  38. [39]

    Sclip: Rethink- ing self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethink- ing self-attention for dense vision-language inference. In European Conference on Computer Vision, pages 315–332. Springer, 2024

    2024

  39. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  40. [41]

    Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion.arXiv preprint arXiv:2305.16213, 2023

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion.arXiv preprint arXiv:2305.16213, 2023

    Pith/arXiv arXiv 2023

  41. [42]

    Janus problem

    Zecheng Wang et al. Taming mode collapse in score distil- lation for text-to-3d generation. InCVPR, 2024. Analyzes geometric inconsistency (“Janus problem”) in SDS

    2024

  42. [43]

    Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation

    Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. InCVPR, 2024

    2024

  43. [44]

    Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation

    Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. InCVPR, pages 22227–22238, 2024

    2024

  44. [45]

    Grm: Large gaussian reconstruction model for efficient 3d recon- struction and generation

    Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d recon- struction and generation. InEuropean Conference on Com- puter Vision, pages 1–20. Springer, 2024

    2024

  45. [46]

    Dream- reward: Text-to-3d generation with human preference

    Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yueqi Duan, and Jun Zhu. Dream- reward: Text-to-3d generation with human preference. In European Conference on Computer Vision, pages 259–276. Springer, 2024

    2024

  46. [47]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Pith/arXiv arXiv 2022

  47. [48]

    Understanding and mitigating compositional issues in text-to-image generative models.arXiv preprint arXiv:2406.07844, 2024

    Arman Zarei, Keivan Rezaei, Samyadeep Basu, Mehrdad Saberi, Mazda Moayeri, Priyatham Kattakinda, and So- heil Feizi. Understanding and mitigating compositional issues in text-to-image generative models.arXiv preprint arXiv:2406.07844, 2024

    Pith/arXiv arXiv 2024

  48. [49]

    From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei, Junhui Chen, Yu-Jie Yuan, Xinyue Cai, Guowei Huang, et al. From flatland to space: Teaching vision-language models to perceive and reason in 3d.arXiv preprint arXiv:2503.22976, 2025

    arXiv 2025

  49. [50]

    Clay: A controllable large-scale generative model for creating high- quality 3d assets.ACM Transactions on Graphics (TOG), 43 (4):1–20, 2024

    Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high- quality 3d assets.ACM Transactions on Graphics (TOG), 43 (4):1–20, 2024

    2024

  50. [51]

    Hunyuan3d 2.0: Scaling diffusion mod- els for high resolution textured 3d assets generation.arXiv preprint arXiv:2501.12202, 2025

    Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3d 2.0: Scaling diffusion mod- els for high resolution textured 3d assets generation.arXiv preprint arXiv:2501.12202, 2025

    Pith/arXiv arXiv 2025

  51. [52]

    Dreamdpo: Aligning text-to-3d generation with human preferences via direct preference optimization

    Zhenglin Zhou, Xiaobo Xia, Fan Ma, Hehe Fan, Yi Yang, and Tat-Seng Chua. Dreamdpo: Aligning text-to-3d generation with human preferences via direct preference optimization. arXiv preprint arXiv:2502.04370, 2025. 10

    Pith/arXiv arXiv 2025

  52. [53]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023. 11

    Pith/arXiv arXiv 2023