{"id":"268e6b7d-dd89-4a81-a640-b2a6ba14eb83","arxiv_id":"2506.01853","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ShapeLLM-Omni unifies text, image, and 3D generation and understanding in one autoregressive LLM using discrete 3D tokens and a new 3D-Alpaca training dataset.","lead":"This paper builds a single AI system that can understand 3D objects and create them from text or images, by turning 3D shapes into discrete tokens and training a large language model on a new 3D dataset. It is a step toward AI assistants that work naturally with 3D content, with applications in gaming, design, and robotics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 3D-editing capability rests on an unvalidated image-mediated data pipeline; no identity or edit-success metric is reported, so the editing leg of the central claim is not established.","rationale":"The strongest claim is a four-capability unified system. Three capabilities (text-to-3D, image-to-3D, 3D captioning) have quantitative evaluations, though generation results are consistently weaker than Trellis and captioning metrics are mixed. The fourth capability, 3D editing, has no quantitative evaluation at all. Its training signal is produced by a two-model pipeline (ChatGPT-4o edits a 2D render; Trellis reconstructs 3D) with no reported quality filtering beyond 'filtering out erroneous cases' in Section 3.4 step 5. No metric checks whether identity is preserved or whether the edit appears in the reconstructed mesh. This is exactly the reader's weakest assumption, and I agree it is the most load-bearing: if these pairs are noisy, the model learns incorrect mappings and the central claim's editing leg fails. The paper's own Limitation section concedes the editing data is scarce, which is honest but does not replace an evaluation. The proposed test would settle the concern: validate a random sample of the pairs with human annotators and then run a held-out editing evaluation. Since the reader already assigned CONDITIONAL due to this and other issues, my stress-test does not change the verdict.","tokens_in":19894,"tokens_out":7476,"duration_ms":80813,"concrete_test":"Render 100 randomly sampled editing pairs from the 70k (Section 3.4 step 6) before/after meshes from four fixed viewpoints. Have three independent annotators label each pair for (i) identity preservation and (ii) presence of the requested edit. If more than 20% of pairs fail either criterion, the editing ground truth is too noisy to train the claimed capability. Separately, on 50 held-out prompts, apply ShapeLLM-Omni's edit and score the same two criteria against a no-edit baseline; the editing claim is unsupported if edit success does not significantly exceed baseline while identity is preserved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weak point is the 3D-editing half of the claimed four-capability system. In Section 3.4, steps (5)-(6), editing pairs are created by rendering the original asset to a frontal image, having ChatGPT-4o edit that image, and reconstructing the edited image with Trellis. The resulting mesh pair is treated as ground truth for the editing instruction. The paper gives no consistency metric for these pairs: no identity-preservation score, no edit-success rate, and no measure of how often Trellis reconstructs something semantically unrelated or geometrically unstable. The only editing evaluation in Section 4 is qualitative (Figure 6), and Appendix A.2 shows selected dataset examples rather than an independent test. If a substantial fraction of the 70k pairs preserve neither identity nor the requested edit, the model is trained on spurious input-output mappings, and the claim that it supports 3D editing is not supported by evidence. The paper's own Limitation section says 70k pairs are 'far too few' for ChatGPT-4o-level editing, but even basic editing capability is unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ShapeLLM-Omni, a Qwen-2.5-VL-Instruct-7B model extended with a discrete 3D token vocabulary from a VQVAE that maps 64^3 voxel grids to 1024 tokens. The model is trained in a fully autoregressive next-token prediction setting on 3D-Alpaca, a newly constructed corpus containing text-to-3D, image-to-3D, 3D captioning, and language-driven 3D editing instances, together with UltraChat for language preservation. The authors report that the model retains language capabilities, achieves generation quality below Trellis but above most baselines on Toys4K metrics, is competitive with PointLLM on captioning, and qualitatively supports 3D editing. The main contributions are the unified framework and the large 3D instruction dataset.","tokens_in":20044,"tokens_out":8903,"duration_ms":74845,"significance":"If the empirical claims are confirmed, this is a useful step toward native 3D multimodal LLMs: the early fusion of discrete 3D tokens with text and images inside a single autoregressive model is a natural and scalable design, and the 3D-Alpaca corpus (2.56M samples, 3.46B tokens) is a substantial community resource. The authors are commendably transparent that Trellis, the model that supplies the tokenizer and decoder, outperforms the proposed system on all generation metrics. However, the paper's headline claim of a four-capability system is only partially supported: the editing leg lacks any quantitative validation, and several quantitative statements are overstated relative to the tables. These issues are local and addressable.","major_comments":[{"comment":"The claimed 3D editing capability is not established. In Section 3.4, steps (5)–(6), each editing pair is produced by rendering a 3D asset to a frontal image, asking ChatGPT-4o to edit the image, and reconstructing the edited image with Trellis; no identity-preservation score, edit-success rate, or reconstruction-consistency metric is reported for the 70k pairs. The only evaluation in Section 4.3 is the qualitative Figure 6, and Appendix A.2 shows dataset examples, not an independent test. Because editing is one of the four central capabilities claimed in Section 3.2 and the paper's own Limitation states that 70k pairs are far too few for ChatGPT-4o-level editing, this claim needs either quantitative validation (e.g., held-out edit alignment and identity metrics, human study) or a substantially weakened formulation.","section":"§3.4, §4.3"},{"comment":"No train/test separation is reported for the 3D understanding evaluation. 3D-Alpaca is built from Objaverse-XL/Objaverse plus other 3D datasets, and Table 5 evaluates captioning on PointLLM's curated Objaverse test set. The paper must state explicitly that the test assets were excluded from 3D-Alpaca; without that statement, the captioning numbers in Table 5 may be inflated by train/test overlap. The same clarification is needed for the Toys4K generation evaluation in Table 4, since the paper does not state that Toys4K test prompts/assets are disjoint from the 3D-Alpaca training collection.","section":"§4.2, Table 5; §3.4"},{"comment":"Two quantitative claims in the text are not supported by the reported numbers. First, \"our generation results outperform all baseline methods except for Trellis\" is contradicted by Table 4: on image-to-3D CLIP score, SAR3D obtains 84.70 while ShapeLLM-Omni obtains 84.5; moreover, Trellis is better than the proposed model on every generation metric, not merely in an unspecified way. Second, Table 5's caption \"our model achieves better performance on 3D understanding/caption tasks\" is misleading: PointLLM-13B* is higher on Sentence-BERT (50.15 vs. 48.34) and SimCSE (50.83 vs. 49.72), and the text in Section 4.2 calling the model \"second only to PointLLM\" is therefore not accurate across metrics. These overstatements should be corrected.","section":"§4.2, Tables 4 and 5"},{"comment":"Several dataset and training numbers are internally inconsistent. The Introduction states 62k paired 3D meshes for editing, Section 3.4 reports 70k valid editing samples, and Table 2 reports 420k items for 3D-Edit; the relationship between pairs and dialogue items should be stated. The Introduction also gives 710k text/image-3D pairs while Section 3.4 and Table 2 use 712k. For training, Section 4.1 reports a learning-rate decay from 5e-5 to 5e-6 and 15 epochs, whereas Appendix B.1 reports a constant 1e-5 learning rate and 60k iterations; the schedule and stopping criterion should be reconciled.","section":"§3.4, Table 2; §4.1 and Appendix B.1"}],"minor_comments":[{"comment":"The SIQA row for Chameleon-7B appears as \"5740.4\", which is either a formatting error or a corrupted value; the bold/underline markers in this table are also applied inconsistently.","section":"Table 3"},{"comment":"The column header \"BLUE-1\" should read \"BLEU-1\", and the duplicate references [22] and [23] should be merged.","section":"Table 5"},{"comment":"Figure 10 has the caption \"Some cases of 3D editing result from our method\" but it shows 3D-to-caption results; this caption appears to have been copied from Figure 6 and should be corrected.","section":"Figure 10"},{"comment":"The description of channel concatenation (\"4096 tokens with 8 channels into 1024 tokens with 32 channels\") should specify how neighboring tokens are grouped, for example by reshaping the 16^3 latent grid before concatenation, so the operation is unambiguous.","section":"§3.3"},{"comment":"The abstract contains a sentence fragment: \"Finally, by performing instruction-based training of the Qwen-2.5-vl-7B-Instruct model on the 3D-Alpaca dataset.\" should be completed or merged with the following sentence.","section":"Abstract"},{"comment":"The captioning ground truth is produced by Qwen-2.5-VL-Instruct, the same model family being fine-tuned; this self-distillation choice should be acknowledged as a limitation, since it may bias the teacher-student relationship and the diversity of captions.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the multimodal and 3D generation community. The main risk is not novelty but support: the editing claim needs quantitative evidence, and the data-split question must be resolved. If the authors can add editing metrics, clarify train/test separation, and correct the overstatements in Sections 4.2 and 4.3, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real engineering contribution with a new large 3D instruction dataset, but the paper's central 'unified 3D editing' claim is not yet backed by evidence. The editing capability is built on a data pipeline that is never validated, and there is no quantitative editing evaluation at all.\n\nWhat is actually new: the 3D-Alpaca corpus (2.56M dialogues, 3.46B tokens across generation, captioning, editing, plus UltraChat) is a substantial artifact and likely the most useful part of the paper. The model itself is a sensible combination: Qwen-2.5-VL-7B backbone, discrete 3D tokens from a Trellis-style VQVAE, trained with next-token prediction. That it handles text, image, and 3D input/output in one autoregressive model, including editing and understanding, appears to be genuinely new compared to SAR3D, Trellis, PointLLM, and LLaMA-Mesh. The language-capability retention numbers are also decent.\n\nThe soft spots are in proportion. The biggest is 3D editing: Section 3.4 builds 62k-70k editing pairs by rendering a mesh, asking ChatGPT-4o to edit the image, then reconstructing with Trellis. There is no identity-preservation score, no edit-success rate, no check that Trellis returned something semantically consistent. The only editing evaluation is qualitative (Figure 6). The paper itself admits the 70k pairs are 'far too few' for ChatGPT-4o-level editing, but even basic editing capability is unquantified. That is load-bearing for the four-capability headline.\n\nSecond, the generation results are consistently weaker than Trellis on every metric in Table 4, yet the qualitative section says the generation quality 'matches that of TRELLIS.' The quantitative table is the more reliable signal; the text should not overstate it. Third, there are no error bars anywhere, and no code or data is released despite the GitHub link, so the 3D-Alpaca dataset cannot yet be used by others. Fourth, the captioning results are mixed: lexical metrics favor this model, but Sentence-BERT and SimCSE favor PointLLM, so 'strong 3D understanding' is an overstatement.\n\nThe stress-test note holds up; I don't think it is a manufactured flaw. The editing leg is the weakest part. The rest of the paper is plausible and honestly written; the limitations section is candid.\n\nWho should read this: people working on 3D LLMs, multimodal tokenization, or instruction datasets. The dataset alone makes it worth refereeing, but the editing claims need either a quantitative benchmark or a toned-down claim. I would send it to review but ask for major revision before acceptance.","headline":"Solid engineering with a valuable new 3D dataset, but the editing claim is unvalidated and the generation numbers are weaker than the prose admits.","tokens_in":173,"tokens_out":2231,"would_cite":true,"duration_ms":30435,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper unifies text-to-3D, image-to-3D, 3D understanding, and 3D editing in one autoregressive model.","keywords":["3D native multimodal language model","3D VQVAE","discrete 3D tokens","text-to-3D generation","image-to-3D generation","3D object captioning","text-based 3D editing","3D-Alpaca dataset"],"falsifier":"Take a random sample of the 70k editing pairs, reconstruct each edited mesh, and have raters check two things: whether the requested change is present and whether unchanged parts of the object still match the original. If a substantial fraction of pairs fail either check, the editing supervision is corrupted and the model's editing behavior on held-out prompts would be expected to degrade in proportion.","tokens_in":19609,"feed_emoji":"🧊","tokens_out":8142,"duration_ms":73336,"temperature":0.7,"pith_summary":"ShapeLLM-Omni claims that a single autoregressive language model can treat 3D as a native token modality, so that text, images, and 3D assets can appear in any order in the input and the model can answer with text or new 3D tokens. The paper reports this as the first unified framework covering text-to-3D, image-to-3D, 3D understanding, and 3D editing in one system. To make that feasible it compresses each 3D mesh into 1024 discrete tokens with an 8192-entry codebook and builds a 2.56M-sample instruction corpus (3.46B tokens) named 3D-Alpaca covering generation, captioning, and editing. The value of the claim is a path to natural-language interactive 3D content creation, where the same model that understands a mesh can also change it or generate a new one from a sentence or a picture.","feed_headline":"One 7B model now generates, captions, and edits 3D","feed_subtitle":"Text, image, and 3D tokens share one autoregressive model; the 3D-Alpaca corpus makes it trainable.","key_machinery":"The load-bearing object is the 3D VQVAE with its 8192-entry codebook. It encodes a $64^3$ voxel grid into a $16^3$ latent grid, serializes that grid into 4096 tokens, concatenates every four neighboring tokens along the channel dimension to obtain 1024 tokens with 32 channels, and quantizes them into the codebook. Those 1024 tokens are what the LLM learns to read and write, so a mesh is just another token sequence in the same transformer. A second load-bearing mechanism is the 3D-Alpaca corpus; for editing it is built by an image-mediated pipeline that renders an asset, asks ChatGPT-4o to edit the rendered image from a text instruction, and reconstructs the edited image back into a 3D mesh with Trellis, yielding 70k training pairs after filtering.","core_discovery":"On its own terms, the paper's central discovery is that 3D can be folded into a multimodal LLM's next-token prediction loop without a task-specific head: a 3D VQVAE turns a $64^{3}$ voxel grid into 1024 discrete tokens, these tokens are added to the LLM vocabulary, and after instruction tuning on 3D-Alpaca the single model handles text-to-3D, image-to-3D, 3D captioning, and instruction-based 3D editing. The same 1024 tokens serve as both the input representation for understanding and the output representation for generation, letting the model read an existing mesh, respond to a prompt, and emit an edited or new mesh in one continuous token stream. The paper reports language benchmarks close to its 7B backbone, captioning scores near a dedicated PointLLM model, and generation quality below but close to Trellis, which it uses as the performance upper bound.","pith_inferences":["If the central claim holds, the image-mediated editing pipeline could be reused to mint cheap training pairs for new categories, making 3D-editing supervision scalable.","The paper does not explore chains like caption-then-edit; a direct test of the native claim would be whether the model can compose those steps in one session.","A testable extension is an ablation of editing-pair count (for example 5k versus 70k pairs) to see whether editing quality is bounded by data quantity or by the fidelity of the automatically generated pairs.","Because images stay continuous while 3D is discrete, the architecture is asymmetric, so one prediction is that the model's image-conditioned 3D reasoning should be strong while image generation itself would require a separate image tokenizer."],"forward_implications":["A single model can interleave text, images, and 3D tokens, so 3D generation can use multi-turn conversational context instead of a fixed one-shot prompt.","Instruction-based 3D editing becomes an autoregressive continuation problem: the model reads the current mesh tokens plus the edit request and emits the edited mesh tokens, supporting iterative changes in one system.","The 3D-Alpaca corpus, with 2.56M samples and 3.46B tokens covering generation, captioning, and editing, is a reusable instruction dataset for training other or larger backbones.","The 1024-token voxel representation fits 3D within the context windows of existing LLMs while staying accurate enough for downstream reconstruction via a rectified-flow voxel-to-mesh refiner."],"supporting_citations":[{"why":"Supplies the 3D U-Net VAE architecture, the 64^3/16^3 voxel representation, the 8192-entry codebook design, and the rectified-flow voxel-to-mesh reconstructor used to build editing pairs and decode outputs.","marker":"[91]"},{"why":"Provides the pretrained Qwen-2.5-VL-7B backbone whose visual encoder is frozen and whose vocabulary is extended with 3D tokens.","marker":"[4]"},{"why":"Performs the image-editing, asset classification, and editing-prompt generation steps in the 3D-Alpaca pipeline.","marker":"[34]"},{"why":"Supplies the category distribution and a large pool of 3D assets used to select the training subset.","marker":"[22]"},{"why":"Provides the 3D captioning benchmark, evaluation metrics, and prompt template used to assess 3D-to-caption performance.","marker":"[94]"},{"why":"Defines the prior unified autoregressive mesh-and-language framework that this work extends by making 3D a discrete token modality with a VQVAE.","marker":"[82]"},{"why":"Provides the mixed-modality early-fusion design and the token-serving approach that motivates concatenating 4096 tokens into 1024.","marker":"[71]"},{"why":"Serves as the autoregressive 3D VQVAE baseline compared on generation and understanding tasks.","marker":"[14]"},{"why":"Adds text-only multi-turn dialogue data so the fine-tuned model retains general conversational ability.","marker":"[24]"}],"fun_headline_variants":["ShapeLLM-Omni: one 7B model does 3D generation and understanding","3D tokens join LLM: natural generation and comprehension","Native 3D multimodal LLM without task-specific heads","VQVAE tokens make 3D a first-class LLM citizen","One autoregressive model now speaks 3D fluently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The editing capability stands on the unverified assumption that the auto-generated training pairs—ChatGPT-4o's image edits followed by Trellis 3D reconstruction—actually realize the intended edits while preserving the rest of the object; the paper gives no consistency metric for these pairs and no quantitative 3D-editing evaluation.","fun_headline_variants_meta":{"raw":{"variants":["ShapeLLM-Omni: one 7B model does 3D generation and understanding","3D tokens join LLM: natural generation and comprehension","Native 3D multimodal LLM without task-specific heads","VQVAE tokens make 3D a first-class LLM citizen","One autoregressive model now speaks 3D fluently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1540,"prompt_tokens":990,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":457}},"tokens_in":606,"tokens_out":550,"duration_ms":6061,"temperature":1.0,"reasoning_tokens":457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:32:13.491556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 70k editing pairs, reconstruct each edited mesh, and have raters check two things: whether the requested change is present and whether unchanged parts of the object still match the original. If a substantial fraction of pairs fail either check, the editing supervision is corrupted and the model's editing behavior on held-out prompts would be expected to degrade in proportion.","supporting_citations":[{"cited_title":"Pointllm: Empow- ering large language models to understand point clouds","cited_arxiv_id":null,"evidence_quote":"Provides the 3D captioning benchmark, evaluation metrics, and prompt template used to assess 3D-to-caption performance."}],"review_version":1}