{"id":"4acb4653-9f1f-4635-b65b-ef7ebc54fba7","arxiv_id":"2505.24245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A conditional 3D generation framework that combines masked autoencoding and diffusion in token space, with prefix learning and reconstruction-guided sampling, reports state-of-the-art results on ShapeNet and Objaverse.","lead":"LTM3D generates 3D shapes from an image or a sentence by combining a masked autoencoder with a diffusion model in a latent token space, then adds a reconstruction-guided sampling step to refine the result. The authors report better fidelity and geometric accuracy than Michelangelo and a reimplemented CLAY on ShapeNet and Objaverse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The state-of-the-art claim rests on an unvalidated self-reimplementation of CLAY; without a check against the official model or equivalent reference baseline, Tables 1, 2, and 4 do not yet establish SOTA.","rationale":"The framework itself is coherent: the MAR-style decomposition of the joint conditional distribution, the prefix-learning alignment during generation, and the reconstruction-guided sampling are plausible and the paper honestly discloses the separate-models limitation and provides substantial implementation detail. The decisive issue is external validity of the SOTA comparison. The paper's own Section 4.2 explains that CLAY was reimplemented by the authors because its code and training data are unavailable, and no official CLAY result is reproduced to validate that reimplementation. Combined with the absence of error bars, the small numeric margins in Tables 1, 2, and 4 cannot be distinguished from an under-trained or differently tuned baseline. This is exactly the weakest assumption the reader identified, and it supports the existing CONDITIONAL verdict rather than a stronger one. The concrete test is therefore to compare LTM3D against the official CLAY or against a released, independently validated reference baseline under the same metric protocol; if the reproduced CLAY is not at parity with such a reference, the state-of-the-art claims need to be revised.","tokens_in":14485,"tokens_out":13276,"duration_ms":165723,"concrete_test":"Obtain the official CLAY model outputs or checkpoints (or use a released reference implementation validated against a published CLAY result) and run both CLAY and LTM3D on the same ShapeNet split with the same DISN renders and metric code, reporting CD, EMD, F-Score, and P-FID over at least three seeds. If the reference CLAY's metrics differ from the reproduced baseline's by more than the seed-to-seed variance, the comparisons in Tables 1, 2, and 4 must be re-run before the state-of-the-art claim can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states that because CLAY's code and training data are unavailable, the authors 'reproduce its results by implementing a 540M-parameter diffusion model for CLAY,' using the same VAE and training hours as LTM3D. The SOTA claims in Tables 1, 2, and 4 therefore depend on this reimplementation being an accurate stand-in for the official CLAY. A 540M diffusion model has many unspecified design choices (conditioning injection, noise schedule, training data mixture, augmentation, checkpoint selection), and no official weights or evaluation outputs are available to check parity. The reported margins are also small and no error bars or repeated-seed statistics are given; for example, in Table 1 the image-conditioned CD difference is 0.0058 vs. 0.0063, and in Table 2 the text-conditioned CD difference is 0.0280 vs. 0.0365. If the official CLAY is stronger than this reimplementation, the apparent advantages could shrink or reverse, so the central contribution claim is not yet externally secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LTM3D proposes a token-space conditional 3D shape generation framework that combines an auto-regressive masked autoencoder with a per-token diffusion denoiser, inspired by MAR. It introduces Prefix Learning to align image/text condition tokens with shape latent tokens during generation, and a Latent Token Reconstruction module with Reconstruction-Guided Sampling to stabilize autoregressive token sampling. The framework is evaluated on image- and text-conditioned SDF generation on ShapeNet and Objaverse, with additional experiments on 3D Gaussian Splatting, point cloud, and mesh representations. The paper claims state-of-the-art results against Michelangelo and a CLAY baseline, and provides ablations of the two proposed modules.","tokens_in":14684,"tokens_out":3747,"duration_ms":47481,"significance":"If confirmed, LTM3D is a well-designed contribution to conditional 3D generation: it provides a unified token-space formulation that supports multiple output representations, includes a sensible decoupling of conditioning from shape encoding, and reports extensive comparisons on standard benchmarks with both geometric and semantic metrics. The proposed Prefix Learning and Reconstruction-Guided Sampling are clearly motivated and ablated. The manuscript is honest about limitations, including the need for per-representation models and the suboptimal results of joint multi-representation training. However, the central state-of-the-art claim depends on an unvalidated reimplementation of CLAY and on single-run metric comparisons with small margins; these issues need to be addressed before the claim is externally secured.","major_comments":[{"comment":"The central claim of outperforming CLAY rests on a 540M-parameter reimplementation of CLAY because the official code and data are unavailable. The manuscript gives no evidence that this reimplementation matches the official CLAY's behavior, and the reported margins against it are small and sometimes direction-dependent (e.g., in Table 4, LTM3D's EMD is worse than the reproduced CLAY's, and in Tables 1 and 2 the P-IS is lower). As written, the SOTA claim is not externally validated; please either compare against the official CLAY model or its published metrics, provide a detailed parity check, and report what design choices (conditioning injection, noise schedule, training data mixture, checkpoint selection) were made in the reimplementation.","section":"Section 4.2, Tables 1, 2, 4"},{"comment":"All main results and ablations are reported as single runs without error bars or repeated-seed statistics, yet several decisive comparisons are close: Table 1 shows CD of 0.0058 vs. 0.0063 and Table 2 shows CD of 0.0280 vs. 0.0365. With single runs it is impossible to assess whether these differences are statistically significant. Please provide means and standard deviations over at least three runs (or equivalent confidence measures) for the main tables, and ideally for the ablations in Tables 5-7.","section":"Section 4.2, Tables 1-7"},{"comment":"The Reconstruction-Guided Sampling schedule is described inconsistently. Eq. (5) defines the blend as xF = (1 - alpha_i)*xhat + alpha_i*xS; Supplementary B.5 states alpha_i = 0.1 for i <= 30, while Supplementary D states that sampled tokens are blended with reconstructed tokens using a linear weight alpha = 0.9. The roles of alpha in the two statements appear contradictory, and the relation between the fusion 'step' (Table 6) and 'ratio' (Table 7) is not defined. Please clarify the exact schedule and explicitly state which setting was used to produce the final numbers.","section":"Eqs. (4)-(5), Supplementary B.5 and D"},{"comment":"The ablation tables show only point estimates and the differences are often within what one would expect from sampling noise (e.g., Table 7 shows CD varying only from 0.0054 to 0.0053, and F-Score from 0.2600 to 0.2612 across fusion ratios). The claim that Reconstruction-Guided Sampling reduces uncertainty would be more convincing with a direct measurement of output variance across repeated samples (e.g., per-prompt variance of CD or ULIP), rather than point estimates of average metrics.","section":"Section 4.3, Tables 5-7"}],"minor_comments":[{"comment":"The F-Score entries for LTM3D are written as '0.1108(8)' and '0.1109(4)'; the parenthetical digits are unexplained and appear to be typographical artifacts.","section":"Table 2"},{"comment":"The paper claims a general framework for multi-representation generation, but Section 5 states that a separate model is required for each representation, and Supplementary F shows that joint training with multiple representations yields poor results. Please soften the 'general' claim or make the per-representation requirement explicit in the abstract and introduction.","section":"Section 5 and Supplementary F"},{"comment":"The definition of P-FID is imprecise: 'calculated between the generated shapes and ground truth shapes on the test dataset' does not specify the feature extractor or the number of samples used. Since P-FID is a distributional metric, details of the point sets and backbone matter for reproducibility.","section":"Section 4.1"},{"comment":"The schematic uses several visual tokens (? marks, 'learnable tokens', 'prefix tokens', 'recon. shape tokens') without a legend; the caption should define each token type and the meaning of 'Only in generation' and the alpha blend notation.","section":"Figure 2"},{"comment":"The factorization p(X|T) = product p(x_i | x_<i, T) is standard, but the claim that optimizing this product avoids the exponential complexity of joint-distribution modeling should be justified more carefully, since the condition x_<i still encodes the full prefix; a brief reference to why MAE random shuffling suffices would help.","section":"Section 3.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the framework is technically sound. The main risk is that the SOTA claim is anchored to an author-built CLAY baseline with no external validation; this is fixable within revision by adding official-baseline comparisons or a rigorous parity report and by running repeated-seed evaluations. I do not see evidence of circularity or fabrication, and the authors are transparent about limitations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is basically MAR [14] transplanted into 3D latent token space, plus two add-ons (Prefix Learning and Latent Token Reconstruction/RGS). That is a reasonable and useful thing to do, and the experiments are extensive enough to show the combination works. But the headline claim—state-of-the-art on image- and text-conditioned 3D generation—does not hold up as reported, because the CLAY baseline is a self-reimplementation that nobody can validate. Your stress-test note lands.\n\nWhat deserves credit: the multi-representation story (SDF, point cloud, 3DGS, mesh in one framework, even if one model per representation), the RGS idea of blending reconstructed tokens during autoregressive sampling to stabilize early steps, and the honest limitations section—they admit joint training across representations fails because the latent spaces are unaligned. The ablations on prefix learning and fusion steps are informative, and the consistent (if small) gains from RGS on CD/EMD/F-Score are believable.\n\nThe soft spots are proportionate to how much the SOTA claim matters. First, the CLAY reproduction: Section 4.2 says CLAY's code and training data are unavailable, so they implemented a 540M diffusion model themselves. That leaves a lot of room for differences—conditioning injection, noise schedule, data mixture, checkpoint selection. No official weights to check parity. The reported margins are small (CD 0.0058 vs 0.0063 on image; 0.0280 vs 0.0365 on text), and there are no error bars. Second, no code or data released; the ethics statement says they 'plan to share' but the manuscript gives no artifact. Third, the RGS ablations show the optimal fusion step and ratio are chosen on the test set, which is mild but worth noting. The P-IS diversity trade-off is acknowledged, not hidden.\n\nNone of this is fatal to the method. The architecture is coherent, the evaluation is broad, and the paper tells you where it falls short. What it needs is a strongly validated baseline (official CLAY or another released reference) and uncertainty estimates. If the authors supply that, I think the central claims will stand.\n\nI would send this to peer review. It is a solid, incremental-but-useful contribution to the 3D generation subfield, and the baseline issue is fixable. I'd ask the reviewers to require the authors to check against official CLAY or release code for the reimplementation.","headline":"A sensible MAR-style autoregressive diffusion framework for 3D generation, with solid experiments, but the state-of-the-art claim leans on an unverifiable self-reimplementation of CLAY.","tokens_in":15218,"tokens_out":1941,"would_cite":false,"duration_ms":24546,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LTM3D combines auto-regressive and diffusion modeling to generate 3D shapes from images and text, outperforming prior state-of-the-art methods on ShapeNet and Objaverse.","keywords":["conditional 3D generation","latent token modeling","auto-regressive diffusion","masked autoencoder","prefix learning","reconstruction-guided sampling","multi-representation 3D","ShapeNet"],"falsifier":"Run the same evaluation protocol with the official CLAY model (released weights or a faithful reproduction) and compare CD, EMD, F-Score, and P-FID on the same ShapeNet and Objaverse splits; if official CLAY matches or exceeds LTM3D, the paper's state-of-the-art claim fails. A second test: disable Reconstruction-Guided Sampling entirely (set $\\alpha_i=1$ for all steps) and measure whether the metric gap persists, isolating the module's contribution.","tokens_in":14283,"feed_emoji":"🧊","tokens_out":6129,"duration_ms":60641,"temperature":0.7,"pith_summary":"The paper introduces LTM3D, a framework for conditional 3D shape generation that combines auto-regressive token dependency modeling with diffusion-based distribution modeling. The central claim is that factorizing the joint conditional distribution over shape latent tokens into per-token conditional probabilities, learned by a masked auto-encoder and an MLP-based denoiser, captures both global structure and fine detail better than pure diffusion or pure auto-regressive models. Two additional components—Prefix Learning, which aligns condition tokens with shape tokens during generation, and a Latent Token Reconstruction module with Reconstruction-Guided Sampling, which reduces token uncertainty—improve prompt fidelity and structural accuracy. The paper reports state-of-the-art results on image- and text-conditioned generation on ShapeNet and Objaverse across SDF, point cloud, mesh, and 3D Gaussian Splatting representations. If correct, LTM3D offers a general token-space recipe for conditional 3D generation that is decoupled from the shape encoder and adaptable to new condition modalities.","feed_headline":"Auto-regressive diffusion tops 3D shape generation benchmarks","feed_subtitle":"One token-space design handles images, text, and four 3D formats with a single backbone.","key_machinery":"The central machinery is the conditional distribution modeling backbone, which combines a Masked Auto-Encoder and an MLP-based DenoiseNet in the style of masked autoregressive diffusion. The MAE computes dependency vectors $z_i = \\text{MAE}(x_{<i}, T)$ for each token given the preceding tokens and condition, and the DenoiseNet predicts the noise $\\epsilon$ for token $x_i$ at diffusion time step $t$, optimized by $\\mathbb{E}_{\\epsilon,t}[\\|\\epsilon - \\epsilon_\\theta(x_i|t,z_i)\\|^2]$. Prefix Learning generates prefix tokens $T = \\text{FF}(\\text{CrossAttn}(Q,C)+Q)$ from learnable queries $Q$ and condition tokens $C$, aligning the condition to the shape token space during generation. Latent Token Reconstruction produces reconstructed tokens $\\hat{X} = \\text{SelfAttn}(\\text{CrossAttn}(Q_S,C)+Q_S)$, which are linearly blended with sampled tokens during the first 30 sampling steps to guide the MAE conditioning.","core_discovery":"LTM3D claims that the joint conditional distribution $p(X|T)$ over shape latent tokens can be approximated by the product $\\prod_i p(x_i | x_{<i}, T)$, where each per-token conditional distribution is computed by a masked auto-encoder that encodes the preceding tokens and the condition into $z_i$, and a diffusion denoiser that models $p(x_i | z_i)$ through a noise-prediction objective. This factorization avoids the exponential cost of modeling the full joint distribution while retaining inter-token dependencies. The paper further proposes Prefix Learning to project image or text condition tokens into prefix tokens during generation, and a Latent Token Reconstruction module that reconstructs shape tokens from conditions and blends them with sampled tokens early in autoregressive sampling. On image- and text-conditioned generation tasks, the framework outperforms Michelangelo and the authors' reproduced CLAY on Chamfer Distance, EMD, F-Score, and P-FID, and also improves cross-view consistency.","pith_inferences":["Because the state-of-the-art claim largely hinges on a self-implemented 540M-parameter CLAY, a direct comparison against the official CLAY release (if it becomes available) would be a stronger test of that claim; the actual gap could be smaller or larger than reported.","The paper's finding that joint training across representations produces low-quality shapes suggests that a unified latent space across different 3D encoders is a key open bottleneck; future work could pursue shared codebooks or contrastive alignment of latent spaces.","The MAE-based factorization with random shuffling is a 3D analogue of masked autoregressive diffusion for images; the same design might benefit other 3D-conditioned tasks such as reconstruction, completion, or part-based generation.","The reported ULIP gains on image-conditioned ShapeNet are modest while geometric metrics (CD, EMD, F-Score) improve substantially, which may indicate that faithfulness to geometry is easier to achieve than fine-grained semantic alignment with conditions."],"forward_implications":["The framework can be applied to any 3D representation with a suitable token encoder, removing the need for separate generative backbones per representation.","Decoupling condition alignment from shape encoding means new condition modalities, such as audio, can be added without retraining the shape encoder.","Reconstruction-Guided Sampling offers a general way to improve prompt fidelity in autoregressive-diffusion hybrids, potentially transferable to other token-space generative models.","The factorization approach addresses the exponential complexity of joint distribution modeling over long token sequences, making it feasible to scale to larger 3D datasets and longer token sequences.","State-of-the-art results on both image- and text-conditioned generation suggest that the autoregressive-diffusion hybrid is a competitive paradigm for 3D generation."],"supporting_citations":[{"why":"Supplies the masked autoregressive diffusion framework that LTM3D adapts for 3D latent token modeling.","marker":"[14]"},{"why":"Provides the masked autoencoder architecture used to learn inter-token dependencies.","marker":"[8]"},{"why":"Provides the VAE that encodes SDF shapes into the latent token space.","marker":"[36]"},{"why":"The main diffusion-based baseline for image- and text-conditioned generation that LTM3D must outperform.","marker":"[40]"},{"why":"The other diffusion baseline; its official code and data are unavailable, so the authors reproduce it.","marker":"[39]"},{"why":"Provides the 3D Gaussian representation and volume VAE used for 3DGS shape encoding.","marker":"[37]"},{"why":"The ULIP model used to compute fidelity scores between generated shapes and condition prompts.","marker":"[33]"},{"why":"Source of the P-FID and P-IS metrics and the point cloud upsampling model.","marker":"[20]"},{"why":"ShapeNet is the main dataset for image- and text-conditioned generation benchmarks.","marker":"[3]"},{"why":"Objaverse is the larger-scale dataset used for additional image-conditioned comparisons.","marker":"[7]"}],"fun_headline_variants":["LTM3D fuses diffusion and autoregression for 3D shapes","Token-space model beats baselines on 3D shape fidelity","One model, four 3D formats: LTM3D leads in accuracy","Autoregressive diffusion framework excels at conditional 3D","Latent token reconstruction sharpens text-to-3D and image-to-3D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported state-of-the-art comparison with CLAY rests on the authors' own reimplementation of CLAY (540M parameters, same VAE and training hours), since CLAY's code and data are unavailable; if that reimplementation is weaker than the official model, the central 'outperforms existing methods' claim would be overstated.","fun_headline_variants_meta":{"raw":{"variants":["LTM3D fuses diffusion and autoregression for 3D shapes","Token-space model beats baselines on 3D shape fidelity","One model, four 3D formats: LTM3D leads in accuracy","Autoregressive diffusion framework excels at conditional 3D","Latent token reconstruction sharpens text-to-3D and image-to-3D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2757,"prompt_tokens":947,"completion_tokens":1810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1713}},"tokens_in":563,"tokens_out":1810,"duration_ms":16157,"temperature":1.0,"reasoning_tokens":1713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:27:29.141542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same evaluation protocol with the official CLAY model (released weights or a faithful reproduction) and compare CD, EMD, F-Score, and P-FID on the same ShapeNet and Objaverse splits; if official CLAY matches or exceeds LTM3D, the paper's state-of-the-art claim fails. A second test: disable Reconstruction-Guided Sampling entirely (set $\\alpha_i=1$ for all steps) and measure whether the metric gap persists, isolating the module's contribution.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Provides the masked autoencoder architecture used to learn inter-token dependencies."},{"cited_title":"3dshape2vecset: A 3d shape representation for neu- ral fields and generative diffusion models.ACM Transactions on Graphics (TOG), 42(4):1–16, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the VAE that encodes SDF shapes into the latent token space."},{"cited_title":"Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"The main diffusion-based baseline for image- and text-conditioned generation that LTM3D must outperform."},{"cited_title":"Clay: A controllable large-scale generative model for creat- ing high-quality 3d assets.ACM Transactions on Graphics (TOG), 43(4):1–20, 2024","cited_arxiv_id":null,"evidence_quote":"The other diffusion baseline; its official code and data are unavailable, so the authors reproduce it."}],"review_version":1}