{"id":"3fbcc5ae-d2ba-47d8-ad12-656ef4513de8","arxiv_id":"2506.10980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A masked fine-tuning strategy adapts large reconstruction models to perform feed-forward 3D scene inpainting in 0.4 seconds with competitive state-of-the-art quality.","lead":"Researchers built a 3D scene inpainting model that runs in 0.4 seconds, about 1000 times faster than previous optimization-based methods. It turns a single 2D inpainted image into a complete, consistent 3D scene that can be viewed from new angles, a key step toward interactive VR/AR editing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model is only ever trained with real, geometrically consistent reference content (Sec. 3.2), but at inference the reference is a 2D-inpainted hallucination; the paper never tests this distribution shift, so the 3D-geometry part of the headline claim rests on an unmeasured transfer assumption.","rationale":"Reader's weakest assumption identifies exactly the same soft spot, and my read agrees. The central claim has two parts: speed and quality. The speed part is well supported by architecture (feed-forward transformer from GS-LRM) and the 0.4s number is plausible; even with hardware caveats and no code release, the order-of-magnitude speedup is not in serious doubt. The quality part is where the argument is least secure. The self-supervised scheme is clever: by gray-masking non-reference views while keeping the reference intact, the model is trained to reconstruct the reference's content in masked regions from context. But this creates a train/test gap: training references are always real image patches with coherent 3D structure, while inference references are generated by a 2D diffusion model. The paper's ablations are thorough regarding mask types and encoding, and the qualitative results are suggestive, but no experiment varies the reference distribution or measures geometric consistency of the inpainted region beyond image-space metrics. Table 3 is a good sanity check for reference adherence, but a ground-truth reference still has true geometry, so it cannot test the actual failure mode. I do not see an internal inconsistency or a reason to reject; the concern is an unmeasured generalization assumption, which is exactly what a conditional acceptance should hinge on. The proposed test (feeding a geometrically invalid reference and measuring novel-view quality/depth discontinuity) would settle whether the network has learned general reference propagation or appearance-prior-dependent copying. Verdict remains conditional.","tokens_in":13034,"tokens_out":8030,"duration_ms":95800,"concrete_test":"Use the SPIn-NeRF benchmark and, for each scene, replace only the masked region of the Flux-inpainted reference with a random texture patch taken from another scene (same mask shape), so the reference is deliberately geometrically inconsistent with the scene; keep all other inputs identical. Render the inpainted novel views and compare M-LPIPS and C-FID against the standard Flux-reference run, and also compute the average depth-map discontinuity across the inpaint boundary from the predicted Gaussians. If the metrics degrade substantially or the boundary depth becomes discontinuous, the model relies on real-appearance priors rather than a general propagation rule; if performance holds, the transfer concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Load-bearing concern: Sec. 3.2 trains InstaInpaint by keeping the reference frame completely intact and masking only the other input views. The reference region whose appearance must be propagated is always a real scene patch (an object, a surface, or a random image region) with physically consistent depth and texture. At test time, that same region is the output of a 2D diffusion inpainter (Flux, [44]), which contains no true 3D geometry and may violate scene scale, surface orientation, or occlusion. The paper assumes the network has learned a general reference-propagation rule ('speculate geometry' from context), but all evidence is consistent with a weaker alternative: the network has learned appearance-to-geometry priors from natural images, so it works when the reference looks like a real object/surface and silently produces implausible geometry when the reference is a diffusion hallucination. The main experiments use a single diffusion model for all reference-based methods and evaluate only image metrics (LPIPS/FID/KID, M-LPIPS/C-FID); no depth consistency or multi-view geometry metric is reported for the inpainted volume. Table 3 uses a ground-truth image as reference, which tests adherence to a real-geometry reference and therefore cannot close the gap. The limitations paragraph (Sec. 5) mentions dynamic scenes and sparse view coverage but not this reference-distribution shift. If the transfer fails, the headline 'state-of-the-art performance' on 16 benchmark scenes does not establish the generalizable 3D inpainting capability claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"InstaInpaint proposes a feed-forward large reconstruction model for 3D scene inpainting. Given posed input views, multi-view masks, and one 2D-inpainted reference view, the model predicts per-pixel 3D Gaussians and renders the completed scene in about 0.4 s, compared with optimization-based baselines that take tens of minutes to hours. The model is first trained as a GS-LRM reconstruction model and then masked-finetuned on DL3DV video clips; the authors generate three kinds of training masks (video object masks, depth-warped geometric masks, and random masks) and ablate mask encoding and sampling. Experiments on SPIn-NeRF and LLFF compare against NeRF/3DGS inpainting methods and LRM-based baselines using LPIPS, FID, KID and border variants; additional qualitative results show object insertion and multi-region inpainting. The paper's central claims are a 1000x speed-up with state-of-the-art quality and learned geometric consistency for inpainted regions.","tokens_in":13343,"tokens_out":8071,"duration_ms":90815,"significance":"If the claims hold, the paper makes a strong practical contribution: it converts 3D scene inpainting from a per-scene optimization problem into a sub-second feed-forward inference problem, which is directly relevant for interactive VR/AR editing. The self-supervised masked-finetuning scheme is a sensible way to avoid collecting paired before/after 3D data, and the mask-generation taxonomy (object/geometric/random) is a useful analysis. The empirical study is broad: two standard benchmarks, optimization-based and LRM-based baselines, and ablations on mask types, mask encoding, and sampling probabilities. However, the significance is conditional on two things the paper does not yet establish: that the model transfers from real reference views at training to hallucinated diffusion reference views at test time, and that the reported metric differences are statistically reliable. The paper does not claim code release, and the method has no machine-checked derivations; its value rests on the experimental evidence.","major_comments":[{"comment":"The central transfer assumption is unmeasured. During masked finetuning, the reference view is always a real, geometrically consistent frame of the same video clip (Eq. 2 keeps I_ref intact), and all photometric losses are computed against real held-out views. At test time the reference is the output of a 2D diffusion inpainter (Flux), which has no true 3D geometry and can violate scale, orientation, and occlusion. The reported benchmarks (Tables 1, 2, 4, 5) contain only image-space metrics; no depth, point-cloud, or multi-view geometric consistency metric is reported for the inpainted volume, and Table 3 uses a ground-truth reference, which cannot probe this distribution shift. The limitations paragraph in Section 5 mentions dynamic scenes and sparse coverage but not this shift. I would need either (a) a direct test of geometry transfer (e.g., rendering depth/point clouds of the inpainted region from novel views and comparing them with the reference's implied geometry, for real vs. diffusion references), or (b) training/evaluation on references with intentionally inconsistent geometry, before accepting the claim that the model learned to identify geometry from the surrounding context for hallucinated content.","section":"Sec. 3.2 (Masked Finetuning), Eq. (2); Tables 1-3"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported. Some of the headline differences are small: in Table 2, LPIPS is 0.4122 for MVInpainter+LRM versus 0.4147 for InstaInpaint, and several ablations in Table 4 differ by less than 1 FID point (e.g., 84.535 vs. 85.645). Without variance estimates it is not possible to know whether the state-of-the-art and ablation conclusions reflect systematic improvements or noise. The GScream column in Table 1 is also missing on the LLFF side ('- -') without a footnote, which makes the benchmark comparison incomplete. Please add multiple seeds or confidence intervals, and either fill in or explain the missing GScream LLFF numbers.","section":"Tables 1, 2, 4, 5"},{"comment":"The claim that object masks improve geometric consistency for inserted instances is supported only by qualitative images (the tilted traffic cone and twisted minion in Fig. 11) and by roughly equal quantitative scores. Since geometric consistency is one of the paper's explicit design goals, this should be measured (e.g., depth-map consistency across novel views, or a chamfer/point-cloud metric in the masked region). Without such a metric, the design choice is not quantitatively validated.","section":"Sec. 4.2 / Table 4 / Fig. 11"}],"minor_comments":[{"comment":"The GScream LLFF entries are missing without explanation; please add the numbers or a footnote stating why they are unavailable.","section":"Table 1"},{"comment":"The text says 'Insection-over-Union' and should read 'intersection-over-union'; Eq. (3) would also benefit from an explicit statement of the homogeneous-coordinate conversion when applying the inverse extrinsic.","section":"Section 3.3"},{"comment":"There are several typos: 'freeforwardly' in Section 1, 'MVInapinter' in the Figure 7 caption, 'Boarder Impact' and 'Mechenisms' in Appendix D, and 'PNSR' in Table 3; reference [15] also has a garbled author string ('Sanja annd Williams').","section":"Throughout"},{"comment":"The phrase 'quartile frames' is not the right description for the frame selection example (1st, 5th, 10th, 15th of 15 frames); please rephrase to something like 'four evenly spaced frames'.","section":"Section 3.2"},{"comment":"The mask-sampling probabilities in Table 5 appear to be selected on the same evaluation datasets used for the final numbers; please state whether a validation split was used to avoid selection bias.","section":"Section 4 / Table 5"},{"comment":"The Limitations paragraph should explicitly acknowledge the reference-distribution shift between training (real reference views) and inference (diffusion-inpainted references) and note that no geometric metric is currently reported.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the runtime result is clearly valuable. My recommendation is driven by the unmeasured reference-distribution shift and the lack of error bars, both of which are addressable in revision. If the authors supply a geometric consistency evaluation on diffusion references and variance estimates, I would be inclined to accept. There is no indication of misconduct; the missing GScream LLFF numbers and typos are presentation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper to know about: InstaInpaint makes 3D scene inpainting feed-forward, taking about 0.4s instead of hours, by fine-tuning GS-LRM with a masked self-supervised scheme. That speed claim is real and it is the main contribution. The quality is competitive on the two standard benchmarks, beating optimization-based methods on the distribution metrics while matching the best LRM baseline on LPIPS.\n\nWhat is actually new: the masked fine-tuning recipe. The idea is concurrent with MaskedLRM, but the authors adapt it to real-world scene-level inpainting, introduce three mask types (object, geometric, random) to avoid object bias, and show multi-region and insertion support. The ablations on mask type, encoding, and sampling probabilities are thorough. They are also honest about the concurrent work, which I appreciate.\n\nSoft spots, in proportion. The evaluation is thin: 16 scenes total, no error bars or multiple seeds, and the LLFF column for GScream is missing. The runtime claim is not verifiable without code, which is not released. The bigger conceptual concern is the distribution shift: the model is trained with a reference view showing real, geometrically consistent content, and at inference the reference is a 2D diffusion inpaint. The stress-test note worries this is unmeasured and load-bearing. I think it is a fair concern but not a fatal one. The paper does show qualitative object-insertion results where the reference is a diffusion hallucination, and the model produces plausible geometry in novel views. That is direct evidence, albeit qualitative and only two examples. What is missing is a quantitative geometry-consistency metric (e.g., depth or multi-view reprojection error) on the inpainted region, and a more systematic test varying the diffusion reference. I would like to see that before fully trusting the generality claim.\n\nOverall: this is a solid systems paper with a genuine advance in latency. It deserves a serious referee and likely acceptance after revision, with extra experiments on the reference shift and error bars. I would bring it to reading group.\n\nRecommendation: send to peer review.","headline":"Fast feed-forward 3D inpainting with real speed gains and honest ablations; the reference-distribution shift is a real gap but not fatal.","tokens_in":13900,"tokens_out":3295,"would_cite":true,"duration_ms":37052,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single feed-forward network can inpaint 3D scenes in 0.4 seconds, matching the quality of optimization methods that take hours.","keywords":["3D scene inpainting","large reconstruction model","Gaussian splatting","feed-forward editing","masked fine-tuning","reference-based inpainting","novel view synthesis","object insertion"],"falsifier":"Take a trained InstaInpaint model and feed it a reference image whose inpainted content strongly contradicts the surrounding geometry—for example, a small object drawn floating in mid-air with no depth cue. If the model propagates that appearance into novel views without correcting the geometry, the novel-view renders will show the object unattached to the scene, which would reveal that the model copies reference texture rather than inferring scene-consistent geometry. A quantitative version: measure the disparity or depth discontinuity between the inpainted region and its surroundings in the predicted Gaussians and compare it with the true scene geometry.","tokens_in":12845,"feed_emoji":"⚡","tokens_out":4989,"duration_ms":51453,"temperature":0.7,"pith_summary":"This paper tries to establish that 3D scene inpainting—repairing or completing a reconstructed scene after removing or inserting objects—can be executed as a single feed-forward pass rather than a slow per-scene optimization. The proposed model, InstaInpaint, takes four posed images, multi-view masks, and one 2D-inpainted reference view, and outputs a 3D Gaussian splatting scene in about 0.4 seconds. On two standard benchmarks this speed is achieved while matching or beating optimization-based methods that take 40 minutes to 15 hours. If the claim holds, real-time interactive editing of reconstructed scenes becomes practical, and 3D inpainting stops being a bottleneck for VR/AR editing workflows.","feed_headline":"3D scene inpainting now takes 0.4 seconds, not hours","feed_subtitle":"A masked large reconstruction model edits 3D scenes in one feed-forward pass while matching optimization-based quality.","key_machinery":"The masked large reconstruction model (masked LRM): a ViT-based transformer with Plücker-ray conditioning that consumes four posed images and predicts per-pixel 3D Gaussian parameters. During masked fine-tuning, the reference view is encoded intact, while the other three input views have masked regions replaced with gray pixels and concatenated with binary masks, so the model learns to copy appearance from the reference while inferring geometry from context. The output tokens decode into pixel-aligned Gaussians that are merged into one scene and supervised by photometric losses on held-out novel views. The three mask-generation strategies—object masks from a video segmentation model, geometric masks obtained by depth-warping random ovals from the reference view, and random image masks—are the mechanism that prevents object bias and teaches the model to handle arbitrary edited regions.","core_discovery":"The central discovery is that an LRM pretrained for reconstruction can be repurposed into an instant 3D inpainter by masked fine-tuning on large-scale real-world video data, without needing before-and-after 3D training pairs. The model treats the single untouched reference view as the source of appearance for the gray-masked regions in the other input views, and learns to infer geometry for those regions from surrounding context. Three generated mask types—video-segmentation object masks, depth-warped geometric masks, and random image masks—force the model to generalize beyond simple object removal. The authors argue that the design choice of masking only the non-reference views, not the reference view, is essential for keeping the reference tokens in the same data distribution as reconstructed tokens.","pith_inferences":["Editorially, the same masked-finetuning recipe could turn other reconstruction-first models into editors for relighting or material changes, since the reference view need only supply a new appearance for the masked region.","Editorially, the measured FID/KID numbers inherit the ceiling of the 2D diffusion inpainter that produces the reference; swapping in a stronger or multi-view-consistent 2D generator should lift 3D output quality without retraining.","Editorially, the declared limitation on dynamic scenes and sparse four-view coverage implies a stress test: fast-moving objects or wide baseline captures would likely break the feed-forward geometry assumption, so practical deployment would need a fallback optimization stage."],"forward_implications":["Interactive editing of reconstructed scenes becomes feasible: object removal, object insertion, and multi-region inpainting all run in one 0.4-second pass instead of minutes-to-hours of per-scene optimization.","The same architecture can switch between removal and insertion by swapping the 2D inpainting model that produces the reference image, including text-driven diffusion inpainting.","Since the model is a feed-forward LRM, it can be trained on large-scale real-world video datasets without expensive annotated before/after 3D pairs, so further scaling is relatively cheap.","Scene editing and novel-view synthesis are unified into a single network, so the edited scene can be rendered immediately without a separate reconstruction step."],"supporting_citations":[{"why":"Supplies the GS-LRM architecture—a ViT transformer with Plücker-ray conditioning and per-pixel Gaussian decoding—that InstaInpaint adapts and fine-tunes.","marker":"[7]"},{"why":"Provides the DL3DV-10k real-world video dataset with camera poses used both for stage-one reconstruction training and for masked fine-tuning.","marker":"[9]"},{"why":"The video segmentation model that generates multi-view-consistent object masks for training.","marker":"[11]"},{"why":"Defines the original Large Reconstruction Model concept that establishes single-image-to-3D reconstruction, extended here to multi-view scene inpainting.","marker":"[6]"},{"why":"MVInpainter, a multi-view 2D inpainting baseline that, combined with an LRM, is the direct comparison for the proposed single-stage approach.","marker":"[10]"},{"why":"SPIn-NeRF, which supplies one of the two evaluation benchmarks and one of the main optimization-based baselines.","marker":"[26]"},{"why":"InFusion, the strongest reference-based 3D-GS baseline that InstaInpaint compares against in runtime and quality.","marker":"[35]"},{"why":"LLFF, the second evaluation benchmark with SPIn-NeRF-annotated masks.","marker":"[41]"}],"fun_headline_variants":["Feed-forward 3D scene inpainting in 0.4s","Masked LRM delivers 0.4s 3D scene inpainting","3D inpainting in 0.4s with masked LRM","0.4s feed-forward 3D inpainting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model is trained with the reference view showing the true original appearance of the scene, but at test time the reference is a 2D-inpainted image that can contain geometry the model has never seen; the claim rests on the transfer from real appearance to hallucinated appearance being good enough.","fun_headline_variants_meta":{"raw":{"variants":["Feed-forward 3D scene inpainting in 0.4s","Masked LRM delivers 0.4s 3D scene inpainting","3D inpainting in 0.4s with masked LRM","0.4s feed-forward 3D inpainting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2553,"prompt_tokens":930,"completion_tokens":1623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1541}},"tokens_in":546,"tokens_out":1623,"duration_ms":12236,"temperature":1.0,"reasoning_tokens":1541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:12:42.976566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained InstaInpaint model and feed it a reference image whose inpainted content strongly contradicts the surrounding geometry—for example, a small object drawn floating in mid-air with no depth cue. If the model propagates that appearance into novel views without correcting the geometry, the novel-view renders will show the object unattached to the scene, which would reveal that the model copies reference texture rather than inferring scene-consistent geometry. A quantitative version: measure the disparity or depth discontinuity between the inpainted region and its surroundings in the predicted Gaussians and compare it with the true scene geometry.","supporting_citations":[{"cited_title":"Gs-lrm: Large reconstruction model for 3d gaussian splatting","cited_arxiv_id":null,"evidence_quote":"Supplies the GS-LRM architecture—a ViT transformer with Plücker-ray conditioning and per-pixel Gaussian decoding—that InstaInpaint adapts and fine-tunes."},{"cited_title":"Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision","cited_arxiv_id":null,"evidence_quote":"Provides the DL3DV-10k real-world video dataset with camera poses used both for stage-one reconstruction training and for masked fine-tuning."},{"cited_title":"SAM 2: Segment anything in images and videos","cited_arxiv_id":null,"evidence_quote":"The video segmentation model that generates multi-view-consistent object masks for training."},{"cited_title":"LRM: Large reconstruction model for single image to 3d","cited_arxiv_id":null,"evidence_quote":"Defines the original Large Reconstruction Model concept that establishes single-image-to-3D reconstruction, extended here to multi-view scene inpainting."},{"cited_title":"Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing","cited_arxiv_id":null,"evidence_quote":"MVInpainter, a multi-view 2D inpainting baseline that, combined with an LRM, is the direct comparison for the proposed single-stage approach."},{"cited_title":"Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields","cited_arxiv_id":null,"evidence_quote":"SPIn-NeRF, which supplies one of the two evaluation benchmarks and one of the main optimization-based baselines."},{"cited_title":"Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar","cited_arxiv_id":null,"evidence_quote":"LLFF, the second evaluation benchmark with SPIn-NeRF-annotated masks."}],"review_version":1}