REVIEW 4 major objections 5 minor 1 cited by
LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read An iterative prompt-refinement loop can make frozen text-to-3D models produce physically conforming, novel car designs, improving a combined physics-and-domain score by up to 106.7% without retraining.
desk verdict Promising prompt-refinement recipe for physics-aware text-to-3D, but Equation (3) has a sign error that contradicts the reported results and needs to be resolved before the paper is publishable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the three-term black-box objective of Equation (3), minimized over prompts: $f_{\mathrm{physical}} + f_{\mathrm{domain}} - \beta F_{\mathrm{novelty}}$. Here $f_{\mathrm{physical}}$ is the drag coefficient from an OpenFOAM CFD simulation, normalized to the range [0,1]; $f_{\mathrm{domain}}$ is a temperature-softened BLIP-2 cosine similarity between orthographic renders and the concept "car"; and $F_{\mathrm{novelty}}$ is the minimum over reference cars of pixel-level plus EfficientNet feature-map differences on masked orthographic renders, with weight $\beta = e^{-\hat{\mu}/\hat{\sigma}}$ derived from the reference set's drag statistics. The framework alternates sampling $N$ prompts, generating and scoring artifacts, and selecting the $N$ best prompt-score pairs as exemplars, so memory stays constant while the LLM receives in-context directional feedback. Orthographic physical light simulation (via OPTIX) is the shared visual representation that makes both domain recognition and novelty comparison distortion-free; switching to perspective projection degrades DPAR gains by 3.38% to 52.21%.
What would settle it
Re-run the same 40-step loop on the car task, then re-evaluate all final candidates with a high-fidelity CFD mesh or a different drag code; if the prompts picked by the proxy do not yield lower drag than the unrefined baseline, the physical score is gameable. A second decisive test is to remove $F_{\mathrm{novelty}}$ from Equation (3) while keeping the drag and domain terms; if DPAR still improves or malformed cars appear among the exemplars, the claimed need for the full three-term objective is not supported.
Extended reading notes
Core claim
The paper's central claim is that physical conformance can be injected into frozen text-to-3D pipelines by steering the prompt distribution rather than retraining the generator. LLM-to-Phy3D runs an online loop: an LLM proposes prompts of the form "A Car in the shape of ...", the text-to-3D model (Shap-E or Trellis) turns each into a mesh, and each mesh is scored by Equation (3), combining normalized CFD drag, a BLIP-2 domain-alignment score, and a geometric-novelty score computed on orthographic renders against a reference set. The prompts that produced the best-scoring cars are kept as exemplars for the next round, so the LLM learns which of its own wordings translate into physically better designs. On the DPAR metric, the expected ratio of domain alignment to drag, the refined pipeline beats the same LLM-to-3D model without refinement for all four LLMs and both generators, with improvements from 4.46% to 106.74%. The ablations support the three-term design: drag alone rewards malformed and fragmented cars, domain alone yields near-copies, and novelty alone favors impractical shapes.
Load-bearing premise
The load-bearing premise is that the three-term score in Equation (3), combining drag, domain match, and geometric novelty, correctly identifies cars that are genuinely physically conforming rather than shapes that merely game the proxy; the paper's own ablation shows drag alone can be gamed by fragmented cars, so the full objective must keep excluding them.
Editorial extensions
If this is right
- Applying the loop to GPT-4o Mini, GPT-3.5, Gemini 2.0 Flash Lite, and Mistral 3.1 Small improves DPAR over the unrefined baseline with both Shap-E and Trellis, from +4.46% to +106.74%.
- Using orthographic instead of perspective camera projection improves DPAR by +3.38% to +52.21%, showing that geometric distortion in the visual evaluation materially changes which designs are found.
- The full three-term objective is necessary: drag-only search favors malformed or fragmented cars, domain-only search produces near-copies of reference cars, and novelty-only search favors impractical shapes.
- Prompt exemplars give the LLM enough directional feedback that GPT-4o Mini and GPT-3.5 converge in under 10 iterations, so the approach works with constant memory and a fixed number of evaluations per step.
- The inherited weaknesses of the base models remain: outputs can be fragmented, non-watertight, or exhibit Janus-face artifacts, and such cases are handled by regeneration and bounded scores.
Reading between the lines
- An untested extension: replace the drag simulator with another physical evaluator, such as lift, structural stress, or energy dissipation, and the same loop should push the LLM toward that target because only the scorer changes.
- Because the loop touches the generator only through prompts and rendered images, it should also work with proprietary or closed text-to-3D services, provided the evaluators can view the outputs.
- The reported margins come from a single automotive setup with one reference set and no repeated runs; a stress test with different seeds, reference cars, and target domains would show how stable the gains are.
- The novelty term is anchored to the reference set through $\beta$ and the minimum over references, so the kind and radicalness of discovered designs will depend on that set; changing it would change what counts as novel.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes LLM-to-Phy3D, an online black-box refinement loop that iteratively prompts an LLM to generate text-to-3D objects, evaluates them with a CFD drag simulator, a BLIP2-based domain alignment measure, and a geometric novelty metric over orthographic renders, and feeds the best-scoring prompts back as exemplars. The central claim is that this loop improves physical conformity and novelty over conventional LLM-to-3D pipelines, with reported DPAR improvements of 4.46% to 106.74% across four LLMs and two text-to-3D generators. Ablation studies show that each objective term alone is insufficient, and the paper argues that the combined objective is necessary.
Significance. If the reported results hold, the framework offers a practical way to inject physics and domain knowledge into frozen text-to-3D models, which is a relevant contribution for engineering design. The use of external non-differentiable simulators (OpenFOAM) and a fixed domain VLM avoids the circularity of fitting to a self-generated target; the orthographic rendering pipeline is a reasonable design choice; and the authors are explicit about inherited failure modes such as non-watertight meshes and Janus artifacts. However, the formal objective as written is internally inconsistent with the reported behavior, and the empirical evidence rests on single runs without error bars, so the central claim is not yet established.
major comments (4)
- [Section 3.1, Eq. (3); Section 3.2(b), Eq. (5)] The second term of the objective is written as +fdomain(g(x), S), and Eq. (5) defines fdomain as the softmax of the BLIP2 similarity to S over the similarity to ¬S, so fdomain is high for in-domain objects. Minimizing Eq. (3) therefore rewards low fdomain, i.e., artifacts that do not resemble the target domain. This contradicts the sentence immediately after Eq. (3) that artifacts less resembling S are penalized, and it is inconsistent with the ablation in Figure 5(b), where optimizing 'fdomain only' produces cars that 'visually highly resemble the reference cars.' Every DPAR improvement in Table 1 is attributed to this objective, so the formal method, as written, cannot produce the reported behavior. The authors should correct the sign (e.g., use -fdomain or +(1-fdomain)) and revise the surrounding text to state the intended direction consistently.
- [Section 4.1 and Table 1; Appendix A.3] All results come from a single run with fixed seeds (Appendix A.3: 'the same random seeds are used throughout the iterations'), with no standard deviation, confidence intervals, or repeated trials. The headline improvements (4.46% to 106.74%) and the exact 1.0000 DPAR entries for GPT-4o Mini are therefore not established as robust. Please report multiple independent runs with different seeds and error bars, and explain why the GPT-4o Mini DPAR saturates exactly at 1.0000 for both Shap-E and Trellis.
- [Section 4.1, DPAR definition] DPAR is defined as E_{x∈D}[ fdomain(g(x),S) / fphysical(x) ], a ratio of two normalized scores that are both in [0,1]. The rating is not bounded above, so a single object with fphysical near its lower bound can dominate the expectation, and the metric conflates improvements in physical performance with improvements in domain alignment. For a fair and interpretable comparison, please report the two components separately (or justify the ratio with a bounded and physically meaningful formulation) and clarify the behavior when fphysical approaches zero.
- [Section 4.1; Table 2] The related work and Table 2 acknowledge prior black-box engineering design optimization methods (Rios et al. [2023], Wong et al. [2024], Xu et al. [2025]), but the experimental section compares only against plain LLM-to-3D baselines, not against these existing optimizers. Without such comparisons, the specific contribution of the selection-pressure mechanism and the combined visual/physics surrogate relative to prior black-box approaches cannot be assessed. Please add comparisons to these methods or explain why they are not applicable in this setting.
minor comments (5)
- [Section 3.2(b), Eq. (5)] The temperature hyperparameter Γ is mentioned in the main text as set to 0.01, but its role in the softmax is not explained until the supplementary material; please define it clearly in the main text.
- [Section 3.2(c), Eqs. (6)-(7)] The notation switches from lowercase fnovelty(x,y) in Eq. (6) to uppercase Fnovelty(x,Y) in Eq. (7); please make the distinction between per-reference and set-level novelty explicit and consistent.
- [Section 4.1] The normalization bounds a=-1.0 and b=1.0 are introduced for fphysical, but it is not stated whether these correspond to the realistic range of drag coefficients for the test scenario; please clarify the mapping between the raw CFD output and the normalized score.
- [Appendix A.2] The heading 'Failure Scenarios and Mitgation Approaches' contains a typo; it should read 'Mitigation.'
- [Figures 1, 4, and 11-18] The rendered images are small and the drag values are not annotated on the figures; consider labeling the physical performance directly on the images so the claimed improvement is visually verifiable.
Circularity Check
No significant circularity: the DPAR gains are measured against external CFD, VLM, and rendering components, not derived from the paper's own definitions or fitted parameters.
full rationale
No circular step is exhibited. The optimizer in Eq. (3) combines fphysical from OpenFOAM CFD, fdomain from a BLIP2 VLM, and Fnovelty from EfficientNet features and OPTIX orthographic renderings; these components are external to the LLM and to the reported DPAR metric, and none is defined in terms of the claimed result. The hyperparameter beta is fitted from the reference set, but it only weights the novelty term, which is not part of the DPAR evaluation, so the Table 1 improvements are not a fitted quantity renamed as a prediction. The paper does cite the authors' prior PREDO line (Rios et al. 2023, Wong et al. 2024) as motivation and in Table 2, but the central empirical comparison is against non-optimizing LLM-to-3D baselines and does not depend on those citations being true, so this is a positioning weakness rather than load-bearing self-citation. One non-circular correctness concern should be noted separately: Eq. (3) as written minimizes +fdomain, while fdomain in Eq. (5) is high for target-domain objects; the surrounding text and the Figure 5 ablations describe the opposite behavior. This is an internal sign inconsistency that would invalidate the formal objective as stated, but it is a mathematical correctness issue, not a reduction of the claimed prediction to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- beta (novelty weight) =
beta = exp(-mu/sigma) from reference set
- Gamma (domain temperature) =
0.01
- a and b (normalization bounds) =
a=-1.0, b=1.0
assumptions (6)
- standard math Navier-Stokes and continuity equations govern the incompressible flow used to compute drag.
- domain assumption OpenFOAM CFD simulation is a valid surrogate for aerodynamic drag of generated 3D vehicles.
- domain assumption BLIP2 cosine similarity on rendered views measures target-domain alignment.
- domain assumption Pixel and EfficientNet feature differences on orthographic renders measure meaningful geometric novelty.
- domain assumption In-context exemplars of prompt-score pairs improve LLM prompt generation.
- domain assumption Orthographic rendering preserves surface topology relevant for novelty comparisons.
Cite this review
Pith. "Pith review of LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs." pith.science (2026). https://pith.science/paper/FLLCTSLU
@misc{pith2026250611148,
author = {Pith},
title = {Pith review of: LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FLLCTSLU}},
note = {Machine review of arXiv:2506.11148}
}
read the original abstract
The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.
Figures
Figures from the paper (14 more)
Forward citations
Cited by 1 Pith paper
-
GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement
GReFEM shows MLLMs zero-shot isolate load-activated geometric features for volumetric mesh refinement with higher precision than matched-budget geometric heuristics.
Reference graph
Works this paper leans on
- [2024]
-
[2025]
doi: 10.1109/TPAMI.2024.3522305. T. Bolkart, T. Li, and M. J. Black. Instant multi-view head capture through learnable registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 768–779, June
arXiv 2024
-
[1]
ALLMO.ai. A comprehensive list of Large Language Model knowledge cut off dates - ALLMO: Boost Your Brand’s Visibility in AI Search — allmo.ai. https://www.allmo.ai/articles/ list-of-large-language-model-cut-off-dates . [Accessed 14-05-2025]. C. Banerjee, K. Nguyen, C. Fookes, and K. George. Physics-informed computer vision: A review and perspectives. ACM ...
work page 2025
-
[5]
URL https://api.semanticscholar.org/ CorpusID:277150679. 10 N. Fabian, K. Moreland, D. Thompson, A. C. Bauer, P. Marion, B. Gevecik, M. Rasquin, and K. E. Jansen. The paraview coprocessing library: A scalable, general purpose in situ visualization library. In 2011 IEEE symposium on large data analysis and visualization, pages 89–96. IEEE,
work page 2011
-
[7]
J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730–19742. PMLR, 2023a. M. Li, Y . Duan, J. Zhou, and J. Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF conference...
-
[8]
semanticscholar.org/CorpusID:267334657
URL https://api. semanticscholar.org/CorpusID:267334657. D. Liu, J. Zhang, A.-D. Dinh, E. Park, S. Zhang, A. Mian, M. Shah, and C. Xu. Generative physical ai in vision: A survey. arXiv preprint arXiv:2501.10928,
-
[12]
Z. Wang, J. Lorraine, Y . Wang, H. Su, J. Zhu, S. Fidler, and X. Zeng. Llama-mesh: Unifying 3d mesh generation with language models. arXiv preprint arXiv:2411.09595,
- [14]
Show all 18 references
-
[15]
URL https: //ojs.aaai.org/index.php/AAAI/article/view/20188
doi: 10.1609/aaai.v36i3.20188. URL https: //ojs.aaai.org/index.php/AAAI/article/view/20188. Q. Xu, J. Liu, M. Wong, C. Chen, and Y . Ong. Looks great, functions better: Physics compliance text-to-3d shape generation. In International Joint Conference on Neural Networks (IJCNN). IEEE,
-
[16]
Zhang, L
Y . Zhang, L. Wang, C. Zou, T. Wu, and R. Ma. Diff3ds: Generating view-consistent 3d sketch via differentiable curve rendering. ArXiv, abs/2405.15305,
-
[17]
semanticscholar.org/CorpusID:270045697
URL https://api. semanticscholar.org/CorpusID:270045697. 12 A Technical Appendices and Supplementary Material A.1 Background Much interest in computer vision, computer graphics, and continuum dynamics is on the boundary between mediums. In physics, particularly in solid or flu...
2001
-
[18]
The instruc- tions highlighted in green provide contextual information on the role, the search task the LLM needs to solve, and the operating constraints it must follow
✗ ✓ LLM-to-Phy3D (Ours) ✓ ✓ 17 Figure 9: Example of meta prompt designed for aerodynamic design optimization task. The instruc- tions highlighted in green provide contextual information on the role, the search task the LLM needs to solve, and the operating constraints it must ...
2019
-
[1997]
Shimabucoro, S
L. Shimabucoro, S. Ruder, J. Kreutzer, M. Fadaee, and S. Hooker. Llm see, llm do: Guiding data generation to target non-differentiable objectives. arXiv preprint arXiv:2407.01490,
-
[1998]
M. Wong, T. Rios, S. Menzel, and Y . S. Ong. Prompt evolutionary design optimization with generative shape and vision-language models. In 2024 IEEE Congress on Evolutionary Computation (CEC), pages 1–8. IEEE,
2024
-
[2000]
T. Rios, S. Menzel, and B. Sendhoff. Large language and text-to-3d models for engineering design optimization. In 2023 IEEE Symposium Series on Computational Intelligence (SSCI) , pages 1704–1711. IEEE,
2023
-
[2001]
Nie, C.-A
A. Nie, C.-A. Cheng, A. Kolobov, and A. Swaminathan. Importance of directional feedback for llm-based optimizers. In NeurIPS 2023 Foundation Models for Decision Making Workshop,
2023
-
[2018]
Dutta, M
A. Dutta, M. Zheng, Z. Gao, B. Planche, A. Choudhuri, T. Chen, A. K. Roy-Chowdhury, and Z. Wu. Chrome: Clothed human reconstruction with occlusion-resilience and multiview-consistency from a single image. ArXiv, abs/2503.15671,
-
[2022]
doi: 10.1145/3554727
ISSN 0360-0300. doi: 10.1145/3554727. URL https://doi.org/10.1145/3554727. P. Dutre, P. Bekaert, and K. Bala. Advanced global illumination. AK Peters/CRC Press,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.