{"id":"72d29a92-1c09-40bb-b09a-0355d34f11b8","arxiv_id":"2411.14715","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"XBind generates 3D objects from text, image, or audio prompts using ImageBind aligned embeddings and hybrid 2D/3D diffusion supervision.","lead":"This paper introduces XBind, a system that generates a 3D object from a text, image, or audio prompt by feeding aligned embeddings into off-the-shelf 2D and 3D diffusion models. It claims to be the first unified any-to-3D generator, but its main quantitative evidence is weakened by evaluation metrics that mirror the training objective.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA margins are largely circular: the MS loss (Eq. 2) optimizes the same CLIP/ImageBind cosine similarity used by CLIP-R and ARCC, so Tables I and II do not independently establish superiority.","rationale":"The single most load-bearing condition for the central claim is that the quantitative evaluation be independent of the proposed loss. Eq. 2 explicitly maximizes cosine similarities between the prompt embedding and CLIP embeddings of rendered views, and Section V-A.4 measures alignment with CLIP-R and ARCC using the same kinds of cosine similarities over the same prompts and rendered views. The baselines and SOTA methods do not include MS loss, so the comparison in Tables I and II is structurally biased even before considering the embedding-space compatibility question. The reader's weakest_assumption focuses on the ImageBind-to-Stable-unCLIP conditioning mismatch, which is a real implementation concern; however, the metric-objective overlap is more decisive because it infects the central quantitative claims regardless of any projection or compatibility fix. The paper's qualitative results and the ablation study do provide some evidence of a functioning pipeline, so the concern is about the strength of the SOTA claim rather than the existence of the system. If the proposed ablation with MS loss disabled preserves the reported margins, the circularity concern would be refuted and the quantitative claim would be substantially strengthened. Since this reinforces the reader's rejection rather than moving it to a different verdict, the verdict should remain unchanged.","tokens_in":18483,"tokens_out":7354,"duration_ms":77503,"concrete_test":"Run XBind with the MS loss term removed (set omega_ms = 0 in Eq. 9) while keeping hybrid diffusion supervision, three-phase optimization, and all other hyperparameters identical; re-render from the same 120 viewpoints and recompute CLIP-R, CLIP-I, and ARCC on the same prompt set. If XBind's margins over ProlificDreamer (CLIP-R 0.8167 vs 0.7319) and over the baselines in Table I collapse or reverse, the reported superiority is an artifact of directly optimizing the evaluation metric. A complementary check is a blind human-preference comparison between XBind and ProlificDreamer on the same text prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantitative claim is undercut by a direct overlap between its proposed training objective and its evaluation metrics. Section IV-A defines the MS loss (Eq. 2) as maximizing cosine similarity between the ImageBind embedding C_m of the input prompt and CLIP image embeddings E(x) of rendered/augmented views. The metrics in Section V-A.4 are CLIP-R (CLIP text-image similarity between prompt and rendered images), CLIP-I (CLIP similarity between reference and rendered images), and ARCC (ImageBind cosine similarity between audio and rendered images). For text-to-3D, CLIP-R is effectively the objective being optimized; for audio-to-3D, ARCC uses the same ImageBind encoder and the same prompt embedding that supplies C_m in the MS loss, so it is a close relative of the training signal. The constructed baselines and the SOTA text-to-3D methods do not optimize this term, so the margins in Tables I and II partly measure whether XBind's own loss was included. The qualitative figures and the Fig. 7 ablation suggest the pipeline can produce plausible objects, but the quantitative 'outperforms SOTA' claim is not independent evidence as submitted. The Limitations section does not address this metric-objective overlap, and no human study or evaluation using a held-out embedding space is reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes XBind, a unified framework for generating 3D objects from prompts of arbitrary modality (text, image, audio). The method encodes the prompt with ImageBind into a shared embedding space, conditions a 2D diffusion model (Stable Diffusion v2-1-unclip) and a 3D-aware diffusion model (Zero-1-to-3) on that embedding, and optimizes a NeRF/DMTet representation in three phases (coarse NeRF, DMTet geometry, DMTet texture). The principal new components are the Modality Similarity (MS) loss (Eq. 2), which maximizes the cosine similarity between the ImageBind prompt embedding and CLIP image embeddings of rendered and augmented views; Hybrid Diffusion Supervision combining planar 2D losses with stereoscopic 3D losses; and the three-phase coarse-to-fine optimization schedule. The paper claims that XBind outperforms three constructed baselines and four SOTA text-to-3D methods on CLIP-R, CLIP-I, and a newly proposed ARCC metric (Tables I and II), and it presents qualitative comparisons and an ablation study (Figs. 3-7).","tokens_in":18822,"tokens_out":12094,"duration_ms":151334,"significance":"XBind targets a genuinely useful gap: a single pipeline that accepts text, image, and audio prompts for 3D object generation without an explicit modality-to-text or modality-to-image conversion step, and the qualitative results in Figs. 3-5 and the module-level ablation in Fig. 7 suggest that the framework can produce plausible, view-consistent objects. The MS loss is a reasonable mechanism for tightening prompt-render alignment, and the combination of ImageBind, Stable unCLIP, and Zero-1-to-3 priors inside one optimization loop is a sensible engineering idea with clear presentation. These strengths are real and should be credited. However, the quantitative evidence is not currently independent: CLIP-R, CLIP-I, and ARCC all overlap with the MS training objective, there are no error bars, significance tests, or repeated seeds, and the ablation is qualitative only. The significance of the paper is therefore conditional on a substantially reworked evaluation using human studies or held-out embedding spaces that were not part of the training objective.","major_comments":[{"comment":"The three evaluation metrics overlap directly with the MS loss that XBind optimizes, so Tables I and II do not provide independent evidence of superiority. CLIP-R measures CLIP cosine similarity between the text prompt and rendered views, while the MS loss in Eq. (2) maximizes cos(C_m, E(x)) with E being the CLIP image encoder; consequently the text-to-3D margin (XBind 0.8050/0.8167 vs. ProlificDreamer 0.7319) partly reflects the fact that XBind explicitly optimizes a CLIP-similarity term that the SOTA methods do not. For image-to-3D, CLIP-I measures similarity to a reference image whose ImageBind embedding is the anchor C_m of the MS loss, and for audio-to-3D the proposed ARCC metric (Eq. 14) computes cos(C(A), C(I)) with the same ImageBind encoder that produces the conditioning embedding C_m, making the audio scores in Table I a near-direct evaluation of whether the MS loss was active. The baselines in Table I also omit the MS loss, and Section VI (Limitations) does not acknowledge this metric-objective overlap. The revision must add independent evaluation: human preference ratings, prompt-render similarity measured with an embedding model not used in the loss, and ideally multi-view consistency or geometry metrics, with repeated-seed runs and significance tests.","section":"Sec. V-A.4, Eq. (2), Tables I-II"},{"comment":"The conditioning path from ImageBind to the 2D diffusion model is underspecified and possibly invalid as stated. The paper says XBind conditions Stable Diffusion v2-1-unclip on ImageBind embeddings, but v2-1-unclip was trained to accept CLIP image embeddings of a specific dimension, whereas ImageBind embeddings come from a different encoder family; the paper describes no projection, linear layer, fine-tuning, or normalization step that reconciles the two spaces. The entire modality-conditional pipeline rests on this unstated compatibility assumption, and without it the method cannot be reproduced or even verified. Please specify the exact conditioning mechanism (embedding dimension, normalization, learned projection, or a fine-tuned unCLIP adapter) and validate it empirically; an ablation comparing raw ImageBind conditioning, a learned projection, and genuine CLIP embeddings would demonstrate that the claimed cross-modal pre-alignment is what makes any-to-3D work.","section":"Sec. IV-A and footnote 1 (p. 4)"},{"comment":"The quantitative evaluation lacks statistical hygiene and internal consistency. No error bars, number of seeds, or significance tests are reported; the ablation study (Fig. 7) is qualitative only, leaving the individual contributions of the MS loss, hybrid supervision, and the three-phase schedule unquantified. In addition, XBind's CLIP-R score is 0.8050 in Table I but 0.8167 in Table II, and the paper does not state whether the prompt sets differ or why the numbers change. The ARCC metric (Eq. 14) is used to evaluate the method on the very objective it is trained on; even setting the circularity aside, a newly proposed evaluation metric should be validated against human judgments before being used as a headline result.","section":"Tables I-II and Sec. V-B through V-F"},{"comment":"Several load-bearing hyperparameters are never given: the MS loss weights omega_g, omega_l, omega_z in Eq. (2); the hybrid-supervision weights omega_ms, omega_cds, omega_img, omega_nz, omega_3d, omega_rgb, and omega_mask in Eqs. (5), (8), (9), and (11)-(13); the augmentation weights W_i in Eq. (2); and the per-phase iteration counts N_phase-1, N_phase-2, and N_phase-3 in Algorithm 1. Without these values the framework cannot be reproduced, and the sensitivity of the reported results to these choices cannot be assessed. Please report the full configuration (or place it in a supplement) and state whether the same schedule is used for all three input modalities.","section":"Sec. IV (Eqs. 2, 5, 8, 9, 11-13) and Algorithm 1"}],"minor_comments":[{"comment":"The definition of W_i and the 'avg' operation over rendered images is ambiguous; please clarify whether the loss is averaged over a random subset of viewpoints per iteration and confirm explicitly that gradients flow through the differentiable renderer into the NeRF/DMTet parameters.","section":"Sec. IV-A, Eq. (2)"},{"comment":"The SOTA results are obtained from threestudio; please state the resolution, number of iterations, and per-method compute budgets, since CLIP-R can be sensitive to training length and viewpoint coverage.","section":"Table II"},{"comment":"The image-to-3D examples are described as style-preserving generation rather than reconstruction, but the criterion for when an output counts as aligned with the input image is never defined; a human-rated measure of style or layout alignment would make this claim concrete.","section":"Fig. 3"},{"comment":"Reference [47] duplicates reference [8] (both cite Rombach et al.); the duplicate should be merged or removed.","section":"References"},{"comment":"The claim that XBind is 'the first method to generate 3D objects from any modality prompts' should be supported by a broader related-work search or softened, since the survey in Section II does not discuss possible audio-to-3D or unified multimodal-to-3D attempts in detail.","section":"Abstract and Sec. VII"},{"comment":"Minor formatting issues include the inconsistent spacing of 'DMT ET' and the boxed notation in Algorithm 1; these should be cleaned up in revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the overlap between the MS training objective and the CLIP/ImageBind-based evaluation metrics; as submitted, the 'outperforms SOTA' claim is not quantitatively defensible. I chose major_revision rather than reject because the problems are fixable within the scope of a revision: the authors can add human studies and held-out embedding-space metrics, and they can specify or add a projection for the ImageBind-to-unCLIP conditioning path. If a revision cannot provide independent quantitative evaluation, the paper should not be accepted. I would also ask the handling editor to have the novelty claim checked against the broader multimodal-to-3D literature, and to require the missing hyperparameters for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first paper I have seen that does audio-conditioned 3D object generation inside a unified any-to-3D pipeline, and the qualitative results are genuinely interesting. But the quantitative claims should not be trusted as submitted: the MS loss in Eq. 2 directly maximizes cosine similarity between the prompt embedding and rendered views, and the CLIP-R and ARCC metrics measure almost exactly that same quantity. Tables I and II therefore partly say “XBind wins because its loss is the metric.” The stress-test note holds up on reading; the Limitations section never addresses this overlap.\n\nWhat is new and good: using ImageBind as a shared condition encoder is a natural choice, and the three-phase NeRF-to-DMTet schedule with hybrid 2D/3D supervision is a sensible engineering combination. The ablation in Fig. 7 gives visual evidence that each component matters, and the audio-to-3D examples look coherent, not like noise. That is a real contribution, even if the evaluation is not.\n\nThe soft spots are serious. For text-to-3D, CLIP-R is nearly the MS loss itself; for audio-to-3D, ARCC uses the same ImageBind encoder and the same prompt embedding that supplies C_m in Eq. 2. The baselines do not get the benefit of this loss, so the reported margins are not independent evidence. There are also no error bars or significance tests, most loss weights are unreported, and no code or data is released. The reader’s concern about embedding compatibility is also valid: ImageBind embeddings are fed directly into Stable Diffusion v2-1-unclip, which expects CLIP image embeddings, and the paper describes no projection or fine-tuning. That is an unstated compatibility assumption. It might work in practice because both spaces are somewhat aligned, but it needs to be argued or ablated, not assumed.\n\nNone of this makes the core idea incoherent; it makes the evidence weaker than claimed. This paper is for people working on 3D generation who want to see whether unified multimodal conditioning is feasible, and for anyone teaching evaluation pitfalls in generative model papers. A serious referee should see it, but the current version would need major revision: independent or held-out metrics, a human evaluation, error bars, and either a demonstrated embedding compatibility or a learned projection. I would not cite it for performance numbers; I might cite it as the first audio-to-3D attempt once the evaluation is cleaned up.","headline":"A plausible first stab at unified any-to-3D with audio, but the reported SOTA margins are largely circular because the evaluation metrics overlap with the training objective.","tokens_in":19378,"tokens_out":2540,"would_cite":false,"duration_ms":25579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified framework, XBind, generates 3D objects from text, image, or audio prompts using shared multimodal embeddings as diffusion conditions.","keywords":["any-to-3D generation","multimodal generation","diffusion models","score distillation sampling","hybrid diffusion supervision","modality similarity loss","3D object generation","audio-to-3D"],"falsifier":"Measure the distribution gap between ImageBind embeddings and the CLIP image embeddings that Stable Diffusion v2-1-unclip was trained on using a paired-image set; if the two spaces are far apart, the paper's direct conditioning of the 2D diffusion model by ImageBind vectors cannot be doing the work it claims.","tokens_in":18300,"feed_emoji":"🧊","tokens_out":12956,"duration_ms":108601,"temperature":0.7,"pith_summary":"This paper claims that 3D object generation can be made truly multimodal: one framework, XBind, takes a text prompt, an image, or even an audio clip and produces a textured 3D object aligned with it, without retraining when the input modality changes. The key move is to treat embeddings from a shared multimodal encoder (ImageBind) as the conditioning signal for two frozen diffusion models, one 2D and one 3D-aware, and to add a Modality Similarity loss that ties the prompt embedding to CLIP embeddings of rendered views. The paper reports that this hybrid supervision, run through a coarse-to-fine three-phase pipeline from NeRF to DMTet, yields higher CLIP-R, CLIP-I, and ARCC scores than its constructed baselines and than established text-to-3D methods. If correct, this removes the need to convert prompts between modalities before generating 3D content and makes audio-to-3D possible in a single pipeline.","feed_headline":"One framework turns text, image, or audio into 3D","feed_subtitle":"Shared multimodal embeddings condition 2D and 3D diffusion, beating text-to-3D baselines on prompt alignment.","key_machinery":"The central mechanism is the Modality Similarity (MS) loss combined with Hybrid Diffusion Supervision. The MS loss is a cosine-similarity objective between the ImageBind embedding of the prompt and CLIP image embeddings of augmented rendered views, giving any modality a direct gradient path into the 3D representation. Hybrid Diffusion Supervision adds pixel-level planar losses (augmented SDS and CDS from the 2D diffusion model) and spatial-level stereoscopic losses (3D SDS and a reference-view loss from the 3D-aware diffusion model), so the framework gets both open-world imagination and view consistency. The three-phase schedule, which starts with a low-resolution NeRF and refines geometry and then texture with DMTet (a hybrid surface representation), is the optimization scaffold that deploys these losses at increasing resolution.","core_discovery":"The central claim is that a single 3D-generation pipeline can be conditioned on any modality by using a shared embedding space. XBind encodes the input with ImageBind, feeds the resulting embedding into both a 2D diffusion model (Stable Diffusion v2-1-unclip) and a 3D-aware diffusion model (Zero-1-to-3), and optimizes a NeRF-to-DMTet representation under hybrid diffusion supervision. A new Modality Similarity loss ties the modality embedding to CLIP embeddings of augmented rendered views, which the paper says is what lets arbitrary modalities actually steer the 3D object. On the reported benchmarks, XBind reaches CLIP-R 0.8050 on the DreamFusion text gallery, compared with 0.7319 for ProlificDreamer, CLIP-I 0.8554 for image prompts, and ARCC 0.4860 for audio prompts; the ablations attribute the gains to the stereoscopic supervision, planar supervision, three-phase schedule, and MS loss.","pith_inferences":["Any new modality later added to a shared embedding space, such as depth or thermal sensing, should become a 3D prompt with no change to the 3D pipeline.","Substituting a different aligned encoder for ImageBind would test whether the recipe depends on the shared-space property or on ImageBind's specific training; if the metrics transfer, the method generalizes as a recipe.","Extending the same hybrid supervision to full 3D scenes, beyond single objects, is a natural next step that the paper's stated limitations leave open."],"forward_implications":["A single trained pipeline can switch among text, image, and audio prompts without retraining or explicit modality conversion.","Audio-to-3D generation becomes practical without first converting sound to text or images, avoiding information loss.","Combining 2D and 3D diffusion supervision suppresses view-inconsistent artifacts such as the multi-face Janus problem.","The coarse-to-fine NeRF-then-DMTet schedule yields high-resolution textured meshes rather than low-fidelity radiance fields.","Prompt alignment, quantified by CLIP-R, CLIP-I, and ARCC, is higher than the constructed baselines and than the compared text-to-3D methods."],"supporting_citations":[{"why":"Supplies the shared embedding space that binds text, image, and audio, which XBind uses as its modality prompt encoder.","marker":"[20]"},{"why":"Provides the 2D diffusion model variant that accepts CLIP image embeddings, enabling ImageBind embeddings to condition planar supervision.","marker":"[47]"},{"why":"Supplies the 3D-aware diffusion prior used for spatial-level stereoscopic supervision and reference-view consistency.","marker":"[23]"},{"why":"Defines the Consistency Distillation Sampling loss used for geometry refinement in Phase II.","marker":"[11]"},{"why":"Defines SDS loss and the DreamFusion prompt gallery used for text-to-3D evaluation and baselines.","marker":"[22]"},{"why":"Supplies the CLIP image encoder used in the MS loss and in the CLIP-R and CLIP-I metrics.","marker":"[30]"},{"why":"Supplies the augmented SDS loss in image space and the z-variance regularization for NeRF.","marker":"[72]"},{"why":"Supplies DMTet, the hybrid surface representation optimized in Phases II and III.","marker":"[26]"}],"fun_headline_variants":["XBind: one framework for text, image, and audio to 3D","Hybrid diffusion supervision unifies any-to-3D generation","First any-to-3D model: text, image, audio in, 3D out","Any modality becomes a 3D object with XBind's MS loss","XBind turns any prompt modality into a 3D object"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ImageBind embeddings can be fed directly into Stable Diffusion v2-1-unclip as if they were CLIP image embeddings, even though the two embedding spaces come from separately trained models and the paper describes no projection or fine-tuning to reconcile them.","fun_headline_variants_meta":{"raw":{"variants":["XBind: one framework for text, image, and audio to 3D","Hybrid diffusion supervision unifies any-to-3D generation","First any-to-3D model: text, image, audio in, 3D out","Any modality becomes a 3D object with XBind's MS loss","XBind turns any prompt modality into a 3D object"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000624,"raw_usage":{"total_tokens":2919,"prompt_tokens":1002,"completion_tokens":1917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":1818}},"tokens_in":618,"tokens_out":1917,"duration_ms":14326,"temperature":1.0,"reasoning_tokens":1818,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:58:41.435476+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the distribution gap between ImageBind embeddings and the CLIP image embeddings that Stable Diffusion v2-1-unclip was trained on using a paired-image set; if the two spaces are far apart, the paper's direct conditioning of the 2D diffusion model by ImageBind vectors cannot be doing the work it claims.","supporting_citations":[{"cited_title":"Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior,","cited_arxiv_id":null,"evidence_quote":"Defines the Consistency Distillation Sampling loss used for geometry refinement in Phase II."},{"cited_title":"Dreamfusion: Text- to-3d using 2d diffusion,","cited_arxiv_id":null,"evidence_quote":"Defines SDS loss and the DreamFusion prompt gallery used for text-to-3D evaluation and baselines."},{"cited_title":"Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthe- sis,","cited_arxiv_id":null,"evidence_quote":"Supplies DMTet, the hybrid surface representation optimized in Phases II and III."}],"review_version":1}