{"id":"62cd939b-2eec-4651-801f-4ea155b11e39","arxiv_id":"2501.09347","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A transformer-based model reconstructs 3D objects from unposed monocular videos, trained with score distillation and iterative diffusion-based pseudo-view augmentation.","lead":"UVRM is a neural network that builds a 3D model of an object from a short video without being told where the camera was. It uses a pre-trained diffusion model to supervise the reconstruction, and the authors show it works on synthetic objects and real hydrant videos.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SDS loss in Eq. (7) is asserted to preserve relative camera poses up to a global affine, but the paper itself concedes this only holds if the sampled pose distribution matches the unknown true distribution; if it fails, the pose-free pipeline may distill the reference-conditioned diffusion…","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing step: the SDS loss's claimed preservation of relative camera poses is unproved and is explicitly conditional on a pose-distribution match that real videos do not provide. My reading of the manuscript confirms this is the most sensitive point in the argument. If Eq. (7) does not preserve relative poses, the weak-supervision signal degenerates into a single-image-conditional plausibility prior, and the self-supervision loop cannot repair the gap because it uses the same diffusion model to generate its pseudo-labels. The paper's own Limitations section reinforces the conditional nature of the contribution by stating that the method has not yet been scaled to a large reconstruction model. This concern does not require rejecting the paper: the proposed training pipeline may still work, and the architecture is reasonable, but the evidence presented (small-scale experiments, no code, no error bars) does not yet support the strong pose-free scalability claim. Since the reader's verdict was already CONDITIONAL, my stress-test does not change the verdict; it sharpens the specific condition that should be checked before the central claim is accepted.","tokens_in":13287,"tokens_out":9380,"duration_ms":137571,"concrete_test":"Use the ground-truth poses in G-Objaverse (available but withheld during normal training) as a diagnostic oracle: train the full UVRM twice, once with the paper's fixed-orbit SDS sampler in Eq. (8) and once with SDS poses drawn from the empirical distribution of the training videos' true camera poses, keeping all other hyperparameters identical. Evaluate both on the same held-out views with ground-truth poses. If the oracle-distribution model achieves materially higher PSNR/SSIM, then the SDS signal only works when P(pr) matches P(pgt), a condition unavailable for real unposed videos, and the central claim is weakened. If the two variants are comparable, the fixed orbit already supplies the needed supervision and the concern is retired.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the SDS loss (Eq. 7) to supply multi-view supervision from unposed video. Section 3.3 asserts, without derivation, that Eq. 7 matches the distribution of rendered views to the ground-truth view distribution up to a global affine transform, and then concedes that a perfect match is only theoretically guaranteed when the sampled pose distribution equals the unknown true distribution, which is hardly the case for real videos. This is the load-bearing step. With a fixed orbit sampler (Eq. 8) and conditioning on a single reference frame, SDS only makes each rendered view plausible under the reference-conditioned diffusion prior; it imposes no coupling between rendered views and the actual unseen frames of the video. The pseudo-view augmentation (Eqs. 9-11) then generates its training targets with the same diffusion model used in the SDS loss, so the pixel-wise loss in Eq. 12 can entrench prior hallucinations rather than video-specific geometry. The reported experiments (20 objects, up to 128 in Fig. A1) and the gain over Zero123-XL do not distinguish reconstructing the video object from distilling the diffusion prior into a consistent 3D shape. Until the distribution-matching property is either proved for the actual training distribution or replaced by a supervision term that directly touches real video frames, the pose-free training claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UVRM, a 3D reconstruction model that takes an arbitrary number of monocular video frames without camera poses and outputs a triplane NeRF representation. To avoid pose annotations during training, the method combines score distillation sampling (SDS) conditioned on a reference frame with an iterative pseudo-view augmentation procedure in which a pretrained diffusion model is used to denoise the model's own renderings, which then serve as pseudo ground-truth views for a pixel-wise loss. The model is a transformer that compresses per-frame VAE latents into three learnable tokens, which condition a style-based triplane synthesizer. Experiments are reported on 20 objects from G-Objaverse Food (plus a 128-object scaling study) and on real CO3D Hydrant videos, comparing against Nope-NeRF and Zero123-XL.","tokens_in":13670,"tokens_out":4517,"duration_ms":42982,"significance":"If the central claim holds, UVRM would be a meaningful step toward scaling 3D reconstruction training to large unposed video corpora, removing the pose-estimation bottleneck that currently constrains LRM training. The architectural idea of pose-invariant latent alignment and the two-stage weak/self-supervision pipeline are interesting and potentially influential. However, the evidence in the paper is thin: the experiments use only a handful of synthetic objects plus one real-world category, the quantitative tables report single aggregate numbers without variance, no pose-supervised oracle baseline is included, and the key theoretical property of the SDS loss is asserted rather than proved. The method's reliance on the same diffusion prior for both SDS supervision and pseudo-view generation raises a circularity concern that the current experiments do not resolve.","major_comments":[{"comment":"The self-supervision term in Eq. (12) is written as MSE(g_Θ(p'_i;V), I^A_i) + λ·LPIPS(g_Θ(p'_i;V), I^A_i), where I^A_i is defined in Eq. (9) as (g_Θ(p'_i;V), p'_i). Thus the two arguments to MSE and LPIPS are identical, making those terms identically zero and the described pseudo-view supervision vacuous. The target should presumably be the diffusion-augmented view (I^A_i)' from Eq. (11), not the original rendering. This is a load-bearing error in the training objective as written; as it stands, Eq. (12) does not implement the analysis-by-synthesis augmentation described in the text.","section":"Eq. (12), Sec. 3.3"},{"comment":"The paper asserts that Eq. (7) matches the distribution of rendered views to the ground-truth view distribution up to a global affine camera transformation, without proof. The text immediately concedes that a perfect match is only theoretically guaranteed when the sampled pose distribution equals the unknown true distribution, which the fixed orbit sampler of Eq. (8) cannot ensure for real videos. Since the SDS term is conditioned only on a single reference image and uses a fixed orbit, it does not directly enforce agreement with the unseen frames of the input video. This property is the load-bearing justification for pose-free training; the manuscript needs either a formal argument or a concrete empirical diagnostic (e.g., comparing relative pose distributions of rendered views with those of the input video) to establish that the supervision is not merely making individual views plausible under the reference-conditioned diffusion prior.","section":"Sec. 3.3, Eq. (7)"},{"comment":"The pseudoview augmentation (Eqs. 9–11) uses the same pretrained diffusion model D_φ that provides the SDS gradients in Eq. (7). Consequently, the pseudo ground-truth views are generated by the same prior that supervises the reconstruction, so the training objective may largely distill the diffusion prior into a consistent 3D shape rather than reconstructing the specific object in the video. Table 1 and Table A1 compare against Zero123-XL and Nope-NeRF, but they do not include a pose-supervised UVRM upper bound, nor do the real-data experiments report quantitative metrics on held-out CO3D frames. Without such comparisons, the results do not disentangle genuine reconstruction of the input video from prior distillation.","section":"Sec. 3.3, Sec. 4"},{"comment":"The experiments are reported at a very small scale: Table 1 uses 20 objects only, the main real-data evaluation is qualitative on CO3D Hydrant, and Table A1 gives a Nope-NeRF success rate of 22% without defining the success criterion or clarifying how failures are incorporated into the reported PSNR/SSIM. Additionally, the Discussion in Sec. 3.3 states that UVRM trained with camera poses converges faster and achieves higher quality, but no experiment in the paper substantiates this claim. The authors should add a pose-supervised oracle baseline, report variance across objects, describe the success criterion, and provide quantitative real-video results.","section":"Sec. 4.2, Sec. 4.4, Table A1"}],"minor_comments":[{"comment":"The notation in Eq. (10) is circular: I^A_{i,t} appears on both sides. It should read I^A_{i,t} = √α_t I^A_i + √(1−α_t)ε, or similar, to denote the noisy version of the rendered image.","section":"Eq. (10)"},{"comment":"There are several typos: 'loss fuction' in Sec. 3.1, 'we replacing' in Sec. 3.3, and 'Consequentially' in Sec. 3.3. Also 'V AE' in the Fig. 2 caption should be 'VAE'.","section":"Sec. 3.1"},{"comment":"References [22] and [23] both cite Raffel et al. (T5) and appear to be the same work; one should be removed or the second should be a distinct citation.","section":"References"},{"comment":"The Limitations paragraph states that the method has not yet been scaled to a large reconstruction model, yet the title and abstract emphasize 'Scalable.' The scaling claim is only demonstrated up to 128 objects, so the authors should temper the language or report results at a larger scale.","section":"Sec. 5, Appendix A.1"},{"comment":"The runtime comparison would benefit from reporting the number of training steps and the resolution/gpu type uniform across methods; currently the single-object UVRM time (990 min/obj) is slower than Nope-NeRF (380 min/obj), which is fine but could be explained more clearly.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a proof-of-concept with a promising idea, but the manuscript needs careful revision. The typo in Eq. (12) is not a small slip: it makes the self-supervision term vacuous as written. The SDS distribution-matching claim and the circularity of using the same diffusion model for both supervision and pseudo-targets are central concerns. The experiment section is also too thin for a claim of scalability. However, the core direction is worth pursuing, and none of the issues are irreparable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about UVRM is that it is the first attempt I know of to train a feed-forward multi-view reconstruction model on unposed video, using SDS as weak supervision plus iterative pseudo-view augmentation from the same diffusion model. That combination is new relative to Nope-NeRF (per-scene optimization) and Zero123 (single-image to 3D). The paper is honest about what it does and does not do: it explicitly says the method is not yet scaled to a large reconstruction model, and calls SDS a weak supervision because the pose-matching property only holds if the sampled camera distribution matches the unknown true one.\n\nWhat the paper does well: the architecture is clean—a transformer aggregates arbitrary-length video tokens into three latent tokens, decoded into a triplane via a style-based synthesizer. The ablations in Table 1 and Figure 7 show both the SDS loss and the iterative augmentation contribute: without SDS the views lose consistency, without augmentation the details suffer. The CO3D hydrant results show the method works on real video, at least qualitatively. The authors also report Nope-NeRF's failure modes and include an honest limitations paragraph.\n\nThe soft spots are real. The central assumption—that SDS preserves relative camera poses up to a global affine—is asserted in Section 3.3 without derivation, and the paper concedes it only holds when the sampled pose distribution equals the ground-truth distribution, which is unlikely for real videos. The pseudo-views that provide the pixel-wise loss are generated by the same diffusion model used in the SDS loss, so the training can partly entrench the diffusion prior rather than reconstruct video-specific geometry. The experiments are small: 20 objects for the main results, 128 in an appendix, and no error bars or code. There is also no comparison to a pose-supervised version of the same model, which would quantify what pose-free training actually costs. The Nope-NeRF comparison mixes a per-scene optimizer with a generalizable model, so the runtime story is not apples-to-apples.\n\nDespite these caveats, the paper deserves a serious referee. The idea is novel, the writing is clear, and the authors are upfront about limits. The main weakness is not a contradiction; it is a missing proof and missing scale. A good referee could push for stronger supervision from real frames, oracle pose-supervised baselines, and variance estimates. I would not cite this as a scalable 3D reconstruction model yet, but I would cite it as a proof-of-concept for pose-free training. Bring it to the reading group; it will generate a lively discussion.","headline":"A genuinely new pose-free training pipeline for 3D reconstruction, worth a serious referee, but the central SDS property is asserted without proof and the experiments are still proof-of-concept scale.","tokens_in":14152,"tokens_out":2713,"would_cite":true,"duration_ms":27859,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UVRM trains a feed-forward 3D reconstruction model on monocular videos without any pose annotations, using a transformer to build pose-invariant latent features and a diffusion-based score distillation and pseudo-view augmentation…","keywords":["3D reconstruction","pose-free training","monocular video","tri-plane representation","score distillation sampling","large reconstruction model","neural radiance field","diffusion model"],"falsifier":"Train UVRM on a video whose true camera trajectory is deliberately far from the pose sampler in Eq. (8), such as all frames from a narrow frontal arc at fixed elevation, and check whether reconstruction collapses; or render a trained model's views, run a standard structure-from-motion pose estimator on them, and measure whether the estimated relative poses match the video's true relative poses up to a global affine transform.","tokens_in":13106,"feed_emoji":"🎥","tokens_out":8187,"duration_ms":70171,"temperature":0.7,"pith_summary":"The paper asks whether a 3D reconstruction model can be trained directly from monocular videos without any camera pose annotations, and its answer, with a system called UVRM, is yes. UVRM uses a transformer to compress an arbitrary number of video frames into a fixed set of pose-invariant latent tokens, which a style-based synthesizer decodes into a tri-plane neural radiance field. Training replaces pose-annotated render losses with score distillation sampling plus iteratively generated pseudo-views produced by a pretrained diffusion model, so no ground-truth camera poses are needed. Experiments on synthetic and real video collections show plausible object reconstructions, and the method is more robust than a per-object pose-free NeRF baseline.","feed_headline":"Unposed videos now train 3D reconstruction models","feed_subtitle":"UVRM drops pose annotations, using a transformer and diffusion guidance to reconstruct objects from raw video.","key_machinery":"Three components carry the argument. The latent alignment encoder is a T5-style transformer with three learnable query tokens; it compresses an arbitrarily long sequence of VAE-encoded frames into a fixed three-token 3D latent, learning viewpoint invariance implicitly. The tri-plane synthesizer is a StyleGAN-style convolutional generator whose layers are modulated by the 3D latent through adaptive instance normalization, producing a three-axis-aligned feature grid that an MLP decodes to color and density for volumetric rendering. The training objective is the pose-free core: SDS gradients from a pretrained image-to-3D diffusion model with randomly sampled camera poses give weak supervision, while an iterative augmentation loop renders views from the current model, adds noise, lets the same diffusion model denoise them conditioned on a reference frame, and trains with MSE and LPIPS losses. The two signals are complementary, with SDS maintaining multi-view consistency and augmentation recovering high-frequency detail.","core_discovery":"UVRM establishes that a feed-forward 3D reconstruction model can be trained and evaluated on monocular video without any pose information. The central claim is that a transformer, by attending over VAE-encoded frames, learns a pose-invariant latent representation of the object, and a style-based synthesizer turns this latent into a tri-plane field from which arbitrary novel views render. To supervise this without poses, the paper combines score distillation sampling, which matches the distribution of rendered views to the true view distribution up to a global affine transform, with an iterative analysis-by-synthesis step that uses the same diffusion model to turn rendered images into pseudo-views for pixel-wise losses. The result is a pose-free training pipeline that scales to multiple objects and produces stable 360-degree reconstructions where per-object pose-free NeRF methods fail.","pith_inferences":["If the distribution-matching property of SDS holds for general video, the pipeline could in principle train on arbitrary internet video; the paper's experiments are confined to small curated collections, so large-scale scaling is a testable next step the authors do not perform.","The global affine ambiguity in the SDS match means reconstructed objects live in an arbitrary canonical coordinate frame, and applications needing metric scale would require an extra calibration signal.","The iterative augmentation schedule, moving from diffusion-guided coarse views to photometric fine views, resembles a curriculum that could transfer to other neural representations such as Gaussian splatting.","A direct diagnostic - running a standard structure-from-motion pose estimator on UVRM's rendered views and comparing relative poses to ground truth - would show whether the model learns genuine relative geometry or only plausible appearance."],"forward_implications":["Pose-free training removes the need for structure-from-motion or pose annotations, making large-scale 2D video corpora usable for 3D reconstruction training.","Reconstruction time amortizes across objects: average per-object cost decreases as the number of objects in a collection grows, since a single forward pass handles each object.","The SDS plus pseudo-view augmentation recipe supplies a self-supervised training objective that does not depend on ground-truth poses, depth, or calibrated multi-view data.","Starting from one reference frame, the same pose-free training objective can propagate a 2D edit into the reconstructed 3D representation."],"supporting_citations":[{"why":"Zero-1-to-3, the pretrained image-to-3D diffusion model that supplies the SDS loss gradients and pseudo-view denoising.","marker":"[17]"},{"why":"DreamFusion, origin of the score distillation sampling objective used as weak supervision.","marker":"[21]"},{"why":"Nope-NeRF, the per-object pose-free NeRF baseline that UVRM compares against and outperforms in robustness.","marker":"[2]"},{"why":"NeRF, the neural radiance field framework underlying the volumetric rendering.","marker":"[19]"},{"why":"The tri-plane representation adopted as the neural 3D output of the synthesizer.","marker":"[3]"},{"why":"StyleGAN, whose style-based generator architecture with AdaIN decodes the tri-plane.","marker":"[12]"},{"why":"CO3D, the real-world video dataset used to evaluate pose-free reconstruction.","marker":"[24]"},{"why":"G-Objaverse Food subset, the synthetic dataset used for ablations and quantitative comparisons.","marker":"[44]"},{"why":"Objaverse, the underlying object dataset from which G-Objaverse is derived.","marker":"[7]"}],"fun_headline_variants":["3D reconstruction without camera poses: UVRM","UVRM trains 3D models from raw videos, no poses needed","Pose-free 3D reconstruction from monocular video","UVRM: 3D from unposed videos, no camera poses","Unposed video to 3D: UVRM scales without pose labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the score distillation loss, with its randomly sampled camera poses, still matches the true camera distribution closely enough to preserve relative geometry; the paper concedes this is only guaranteed when the sampled pose distribution equals the unknown ground-truth distribution, which is unlikely for real videos.","fun_headline_variants_meta":{"raw":{"variants":["3D reconstruction without camera poses: UVRM","UVRM trains 3D models from raw videos, no poses needed","Pose-free 3D reconstruction from monocular video","UVRM: 3D from unposed videos, no camera poses","Unposed video to 3D: UVRM scales without pose labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2964,"prompt_tokens":948,"completion_tokens":2016,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1926}},"tokens_in":564,"tokens_out":2016,"duration_ms":14300,"temperature":1.0,"reasoning_tokens":1926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:06:19.003671+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train UVRM on a video whose true camera trajectory is deliberately far from the pose sampler in Eq. (8), such as all frames from a narrow frontal arc at fixed elevation, and check whether reconstruction collapses; or render a trained model's views, run a standard structure-from-motion pose estimator on them, and measure whether the estimated relative poses match the video's true relative poses up to a global affine transform.","supporting_citations":[{"cited_title":"Zero-1-to-3: Zero-shot one image to 3d object, 2023","cited_arxiv_id":null,"evidence_quote":"Zero-1-to-3, the pretrained image-to-3D diffusion model that supplies the SDS loss gradients and pseudo-view denoising."},{"cited_title":"Nope-nerf: Optimising neu- ral radiance field with no pose prior","cited_arxiv_id":null,"evidence_quote":"Nope-NeRF, the per-object pose-free NeRF baseline that UVRM compares against and outperforms in robustness."},{"cited_title":"Efficient geometry-aware 3d generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"The tri-plane representation adopted as the neural 3D output of the synthesizer."},{"cited_title":"A style-based generator architecture for generative adversarial networks","cited_arxiv_id":null,"evidence_quote":"StyleGAN, whose style-based generator architecture with AdaIN decodes the tri-plane."},{"cited_title":"Com- mon objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction","cited_arxiv_id":null,"evidence_quote":"CO3D, the real-world video dataset used to evaluate pose-free reconstruction."},{"cited_title":"a cupcake with whipped cream","cited_arxiv_id":null,"evidence_quote":"G-Objaverse Food subset, the synthetic dataset used for ablations and quantitative comparisons."}],"review_version":1}