{"id":"e75d401c-43a8-4b0d-91bf-d5f91c34b049","arxiv_id":"2411.15551","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"GB-NeRF improves NeRF inpainting by fine-tuning a diffusion model on RGB-plus-normal image pairs and replacing standard score distillation with a two-term balanced loss.","lead":"This paper introduces GB-NeRF, a method for removing objects from 3D scenes captured as NeRFs by teaching a diffusion model to understand surface shapes and using a simplified distillation loss. It reports better perceptual quality and geometric consistency than prior NeRF inpainting methods on standard benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Geometric gains in Table 1 hinge on an undefined and likely out-of-distribution normal rendering: Eq. 10 feeds NeRF-rendered normals to a diffusion model fine-tuned on DIODE normals, with no formula or distribution check.","rationale":"The reader's weakest assumption captures the key risk. The central claim has two pillars: BSD for stable appearance guidance and the fine-tuned RGB-plus-normal diffusion model for geometry. Of these, the geometric pillar is the less supported: Eq. 10 is the only place where the normal prior enters, and it is not reproducible as written. The distribution shift concern is concrete: DIODE normals are produced from RGB-D scans and have a particular range and invalid-pixel structure, while NeRF-rendered normals from an unconstrained density field can be arbitrarily different, especially in masked regions. If the normal prior operates out-of-distribution, the D-PSNR and D-FID improvements are not evidence of better geometry. This does not make the paper hopeless: the appearance SOTA claim (FID 67.587 vs 72.616, BRISQUE 17.496 vs 23.562) may still hold, and the issue is addressable by documenting the normal rendering and reporting a distribution check. The reader's CONDITIONAL verdict remains appropriate. I also note the BSD equations omit the mask and inpainting conditioning that the fine-tuning section implies, and the ablation table is non-monotonic, but these are secondary to the normal-distribution issue.","tokens_in":11531,"tokens_out":10212,"duration_ms":98974,"concrete_test":"Using the released code, extract the exact normal rendering routine used in Eq. 10. Render normal maps for the 8 SPIn-NeRF scenes at evaluation viewpoints and compute the FID between these rendered normals and a held-out set of DIODE normals, separately for masked and unmasked regions, and compare with DIODE train/val FID. If the rendered normals are far outside the DIODE distribution, the geometric prior in Eq. 10 is out-of-distribution and the D-PSNR/D-FID gains in Table 1 are not evidence of improved geometry. Optionally, rerun with an alternative normal rendering convention (e.g., depth-gradient vs density-gradient normals); if the geometric advantage over MVIP-NeRF reverses, the claim depends on an unspecified choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline SOTA claim is supported mainly by geometric metrics (D-PSNR 14.157 vs 13.914; D-FID 150.473 vs 172.127) and by qualitative geometry improvements. Both rest on Eq. 10, the BSD loss on a 'normal map rendered from the NeRF scene' (Sec. 4.2.3). For Eq. 10 to provide meaningful supervision, the rendered normal map n must be a valid input to the fine-tuned Stable Diffusion model: its distribution must overlap the DIODE normal maps used in Sec. 4.1. The paper never defines n. It gives no formula for rendering normals from a NeRF (density-gradient normals, depth-derived normals, camera-space vs world-space convention), no handling of empty/low-density regions where normals are undefined, and no description of how the mask is incorporated. DIODE normals come from a specific RGB-D sensor pipeline, have their own range and invalid-pixel statistics, and are encoded by a VAE trained on natural RGB images. During inpainting, the NeRF's masked geometry is initially unconstrained, so rendered normals can contain near-zero densities, floating geometry, and high-frequency noise that are far outside the DIODE distribution. If the normal prior is applied out-of-distribution, the reported D-PSNR and D-FID improvements may be an artifact of the diffusion model imposing smoothness on arbitrary normal-valued inputs, not evidence of accurate geometry. This is also a reproducibility blocker: without the normal rendering formula, Eq. 10 cannot be independently implemented or checked.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GB-NeRF, a NeRF inpainting method for removing objects from multi-view scenes. It fine-tunes a latent diffusion model with LoRA on DIODE RGB-normal pairs augmented with BLIP captions, so that the model learns both appearance and geometric priors. The authors also introduce Balanced Score Distillation (BSD), which removes the unconditional noise-prediction term from Classifier Score Distillation (CSD), leaving only positive- and negative-prompt terms. The method is evaluated on subsets of SPIn-NeRF and LLFF, comparing against SPIn-NeRF+LaMa, SPIn-NeRF+SDS, and MVIP-NeRF. The paper claims state-of-the-art appearance fidelity and geometric consistency, supported by quantitative gains in SSIM, FID, NIMA, BRISQUE, D-FID, and D-PSNR, plus qualitative visual comparisons.","tokens_in":11916,"tokens_out":5602,"duration_ms":45936,"significance":"If substantiated, the work is a worthwhile contribution: it demonstrates that LoRA-based fine-tuning on RGB-normal pairs can inject geometric priors into a pretrained diffusion model, and that a two-term score distillation reduces optimization variance and computational cost relative to SDS and CSD. The paper provides a public code link, ablates its two main components, and reports a broad set of metrics. However, the current evidence for the state-of-the-art claim is weakened by an underspecified geometric loss, a potential distribution shift in the normal-map input, and evaluation choices that limit the strength of the empirical conclusions.","major_comments":[{"comment":"The normal map n used in the geometric BSD loss is never defined. The paper gives no rendering formula (e.g., normalized negative density gradient or depth-derived normals), no convention for camera-space versus world-space normals, no handling of empty or low-density regions, and no description of how the mask is applied to the rendered normal map. Without this, Eq. (10) is not implementable, and the reported geometric gains (D-PSNR, D-FID in Table 1) are not reproducible. Please specify the exact normal rendering procedure and the preprocessing steps used before the latent encoder.","section":"Sec. 4.2.3, Eq. (10)"},{"comment":"The fine-tuned diffusion model is trained on DIODE normal maps, but at inference it receives NeRF-rendered normals from the masked region, where geometry is initially unconstrained. The resulting distribution (sensor-specific value ranges, invalid pixels, density noise, floating geometry) may be far outside the DIODE distribution, so the geometric prior could be imposing arbitrary smoothness rather than accurate geometry. Please provide evidence that the rendered-normal distribution overlaps the training distribution (e.g., histograms, FID between rendered and DIODE normals, or a synthetic experiment with known ground-truth geometry), and state how invalid normals are handled.","section":"Sec. 4.1 and Eq. (10)"},{"comment":"The paper selects 8 of the 10 SPIn-NeRF scenes and a 4-scene LLFF subset without explaining the selection criteria or the evaluation views used. Because the headline claim rests on Table 1 and Table 2, and several metric gaps are small (e.g., SSIM 0.4266 vs 0.4208, PSNR 19.489 vs 19.813), the lack of per-scene results, error bars, or significance tests makes it difficult to judge whether the improvements are consistent. Please report per-scene numbers, justify the subset, and add variance or significance information.","section":"Sec. 5, Datasets"},{"comment":"BSD is designed by setting omega3=0 based on an ablation on the same SPIn-NeRF benchmark used for the final comparison, and the remaining weights (omega1=7.5, omega2=6.5, omega1=1.5, omega2=0.5, lambda1=0.1, lambda2=lambda3=0.0001, LoRA rank 32) are tuned on that same data. This makes the method's advantage less clearly a consequence of the principle rather than of benchmark-specific tuning. Please include sensitivity curves for the key hyperparameters and, if possible, validate on a held-out scene set or additional datasets.","section":"Sec. 4.2.1 and Table 2"},{"comment":"The paper claims BSD outperforms CSD, but Table 1 does not include a CSD baseline under the same fine-tuned diffusion backbone; the only comparison is to MVIP-NeRF's SDS-based pipeline. A direct CSD ablation with the same LoRA weights would be needed to support the superiority claim over CSD.","section":"Sec. 5.1, Baselines"}],"minor_comments":[{"comment":"The dataset name is spelled 'SPIN-NeRF' in the table caption while the text uses 'SPIn-NeRF'; please unify the spelling.","section":"Table 1"},{"comment":"The quantitative relationship between omega3 and the metrics is not given; only qualitative images are shown. Reporting quantitative values would strengthen the motivation for setting omega3=0.","section":"Fig. 3"},{"comment":"The paper argues that PSNR is unreliable for inpainting evaluation, yet it still reports D-PSNR as a positive result. Please provide the same argument for why D-PSNR is reliable when it supports the paper's claims.","section":"Sec. 5, Metrics"},{"comment":"The unmasked depth loss Lg_unma is included in the final loss, but the paper does not state whether ground-truth depth is available for both datasets; please clarify how Lg_unma is computed for scenes without depth.","section":"Sec. 4.2.3, Eq. (11)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint with a practical contribution, but the missing normal rendering formula and dataset-selection details make the state-of-the-art claim difficult to verify. The scope seems suitable for a computer vision journal; however, the authors should also clarify the relationship to MVIP-NeRF, since the baseline is the closest competitor and the architectural overlap is high."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely incremental but useful paper. The new bits are the LoRA fine-tuning of Stable Diffusion on DIODE RGB+normal pairs with BLIP captions, and BSD, which is just CSD with the unconditional term removed (their Eq. 7 with omega3=0). That is a special case, but the paper's empirical study of omega3 is honest and useful, and dropping one network evaluation per step is a small real saving. The fine-tuning strategy appears well motivated, the ablations support it, the writing is clear, and the code is released. The appearance improvements (FID, NIMA, BRISQUE) do not depend on the questionable geometry path, so they are reasonably solid.\n\nThe soft spots are real. The stress-test is on target: Eq. 10 feeds a rendered normal map n into the diffusion model, but the paper never defines how n is computed from the NeRF. No formula, no convention (camera vs world space, density-gradient vs depth-derived), no handling of empty regions. That is load-bearing because the geometric gains (D-PSNR, D-FID) are the main support for the SOTA claim. And the distribution-shift worry is legitimate: DIODE normals come from a depth sensor, while an inpainting NeRF's masked geometry starts unconstrained, so the rendered normals can be far out of distribution. The reported D-FID improvement could partly be the diffusion model imposing smoothness on arbitrary inputs. This is fixable, but as written it is a reproducibility blocker.\n\nThe evaluation also has standard weaknesses: 8 of 10 SPIn-NeRF scenes and only 4 LLFF scenes without explanation, no error bars or significance tests, and several metric gaps are small (SSIM 0.4266 vs 0.4208, D-PSNR 14.157 vs 13.914). BSD's hyperparameters are tuned on the same benchmarks used for the final comparison, so there is some circularity, though that is common in this area.\n\nOverall: this is an honest, incremental contribution for the NeRF-inpainting niche. The appearance claim is plausible; the geometric claim is under-supported until the normal rendering is specified and validated. I would send it to peer review. A serious referee can ask for the missing formula, a distribution analysis of rendered normals, error bars, and full-dataset results. If those come back clean, the paper is citable. I would not desk-reject it.","headline":"A solid, modest NeRF-inpainting paper whose appearance gains are credible but whose geometry claims rest on an undocumented normal-rendering step that a referee should push to fix.","tokens_in":688,"tokens_out":654,"would_cite":false,"duration_ms":26469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that NeRF inpainting improves when the diffusion prior is fine-tuned on paired RGB images and normal maps and when score distillation drops its unconditional noise term, yielding a balanced two-prompt gradient called…","keywords":["NeRF inpainting","Balanced Score Distillation","geometric prior","normal map","diffusion model","LoRA","object removal","score distillation sampling"],"falsifier":"Render a trained GB-NeRF's normal maps on held-out views and compute the FID between those rendered normals and DIODE normal maps; if the distance is large, the geometric prior is being applied out-of-distribution. Alternatively, during masked-region distillation, replace the rendered normal map with the ground-truth normal map: if D-FID does not improve further, the geometric gains come from RGB supervision rather than from the normal channel.","tokens_in":11370,"feed_emoji":"🎨","tokens_out":4300,"duration_ms":38311,"temperature":0.7,"pith_summary":"This paper argues that NeRF inpainting fails when the 2D diffusion prior ignores geometry and when score distillation injects random noise. It proposes GB-NeRF, which fine-tunes a diffusion model with LoRA on paired RGB images and normal maps from DIODE, so the model learns to complete both appearance and surface orientation. It then replaces the distracting unconditional noise term in score distillation with Balanced Score Distillation (BSD), a two-term gradient that balances a positive prompt against a negative prompt. On the SPIn-NeRF and LLFF object-removal benchmarks, the authors report that GB-NeRF beats SPIn-NeRF and MVIP-NeRF on most perceptual and geometric metrics, though not on PSNR, which they argue is unreliable for inpainting.","feed_headline":"Normal-aware diffusion prior sharpens NeRF object removal","feed_subtitle":"Fine-tuning on RGB-normal pairs plus Balanced Score Distillation improves geometric and perceptual quality in NeRF inpainting.","key_machinery":"The load-bearing objects are Balanced Score Distillation (BSD) and the RGB-normal fine-tuning pipeline. BSD is the gradient $\\delta = \\omega_1 \\epsilon_\\phi(x_t; y, t) - \\omega_2 \\epsilon_\\phi(x_t; y_{\\mathrm{neg}}, t)$, where $\\epsilon_\\phi$ are noise predictions from the diffusion model conditioned on positive and negative text prompts and the unconditional term is dropped; the same form is applied to normal maps in the geometric loss. The fine-tuning stage trains the U-Net and text encoder with LoRA on DIODE RGB-normal pairs, using BLIP-generated captions prepended with modality identifiers ('RGB image' or 'normal map'), so the model retains its appearance prior while learning to complete structurally accurate normal maps.","core_discovery":"GB-NeRF's central claim is that a diffusion prior fine-tuned to generate normal maps together with RGB images, supervised by a Balanced Score Distillation loss, yields state-of-the-art NeRF inpainting in both appearance and geometry. The gradient used for masked regions is $\\delta_{\\mathrm{BSD}} = \\omega_1 \\epsilon_\\phi(x_t; y, t) - \\omega_2 \\epsilon_\\phi(x_t; y_{\\mathrm{neg}}, t)$, applied to rendered RGB images and to rendered normal maps alike, with the unconditional noise-prediction term removed. Removing that term makes the supervision more deterministic in occluded areas, and combining this with LoRA fine-tuning on RGB-normal pairs improves perceptual quality and geometric consistency relative to baseline NeRF inpainting methods.","pith_inferences":["Editorial inference: BSD is effectively a directional classifier-free guidance in prompt space, so the same positive-minus-negative gradient could be applied to video inpainting or 3D Gaussian splatting, where the negative prompt controls what content the model avoids hallucinating.","Editorial inference: the claimed geometric gains depend on rendered normal maps being in-distribution for the DIODE-fine-tuned model; a direct check would be to compute an FID score between rendered normal maps and DIODE normal maps on held-out scenes.","Editorial inference: because PSNR drops while perceptual metrics rise, the method is trading pixel-level fidelity for plausible structure; applications that need exact geometry, such as industrial or medical scan completion, may value the PSNR loss more than this paper does.","Editorial inference: the paper's ablation suggests that normal-map-only fine-tuning hurts performance while RGB-plus-normal fine-tuning helps, implying the geometry prior is only useful when appearance knowledge is preserved; a testable follow-up is to vary the LoRA rank to find the minimal capacity needed for the geometric prior."],"forward_implications":["BSD requires only two diffusion evaluations per optimization step instead of three, reducing computation and eliminating the unconditional-weight hyperparameter whose nonzero values cause blurry or artifact-laden inpaintings.","The fine-tuned diffusion model can complete normal maps as well as RGB images, so geometric supervision in masked regions no longer depends solely on the weak normal-map priors of a pretrained diffusion model.","The reported results show that on SPIn-NeRF and LLFF, GB-NeRF improves SSIM, FID, NIMA, BRISQUE, D-PSNR and D-FID over MVIP-NeRF and SPIn-NeRF variants, at the cost of slightly lower PSNR, which the paper attributes to the ill-posed nature of inpainting.","Ablation results indicate the two components are complementary: LoRA fine-tuning chiefly lowers D-FID, BSD chiefly improves perceptual metrics, and combining them yields the best overall balance.","Since BSD and the RGB-normal fine-tuning are formulated for any differentiable 3D generator, the same pipeline can be dropped into other NeRF-based scene-editing tasks without changing the loss structure."],"supporting_citations":[{"why":"MVIP-NeRF is the primary baseline that applies appearance and geometric score distillation to NeRF inpainting; GB-NeRF builds on it by adding fine-tuning and BSD.","marker":"[1]"},{"why":"DreamFusion's SDS provides the original score distillation gradient that BSD modifies by removing the random noise and unconditional terms.","marker":"[22]"},{"why":"CSD supplies the classifier score distillation formulation with positive, unconditional, and negative noise predictions, which BSD simplifies into two terms.","marker":"[37]"},{"why":"SPIn-NeRF provides the base NeRF inpainting pipeline, the perceptual-loss setup, and the SPIn-NeRF benchmark dataset used for evaluation.","marker":"[18]"},{"why":"DIODE contributes the paired RGB and normal-map images used to fine-tune the diffusion model for geometric priors.","marker":"[32]"},{"why":"BLIP generates captions for fine-tuning prompts, enabling the diffusion model to keep its text understanding while learning normal-map generation.","marker":"[12]"},{"why":"LoRA provides the low-rank adaptation mechanism that allows fine-tuning the U-Net and text encoder without destroying the pretrained Stable Diffusion weights.","marker":"[10]"},{"why":"The task-prompt inpainting training method is followed for the self-supervised inpainting loss used during diffusion fine-tuning on RGB and normal maps.","marker":"[40]"}],"fun_headline_variants":["GB-NeRF's BSD sharpens NeRF inpainting geometry","Normal diffusion prior wins for NeRF inpainting","Balanced score distillation: better NeRF object removal","NeRF inpainting elevates with normal-aware diffusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The geometric gains rest on the assumption that a normal map rendered from the NeRF lies in the same distribution as the DIODE normal maps used to fine-tune the diffusion model, and the paper does not give the rendering formula or quantify the distribution shift.","fun_headline_variants_meta":{"raw":{"variants":["GB-NeRF's BSD sharpens NeRF inpainting geometry","Normal diffusion prior wins for NeRF inpainting","Balanced score distillation: better NeRF object removal","NeRF inpainting elevates with normal-aware diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1423,"prompt_tokens":891,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":507,"tokens_out":532,"duration_ms":5224,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:10:16.013983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a trained GB-NeRF's normal maps on held-out views and compute the FID between those rendered normals and DIODE normal maps; if the distance is large, the geometric prior is being applied out-of-distribution. Alternatively, during masked-region distillation, replace the rendered normal map with the ground-truth normal map: if D-FID does not improve further, the geometric gains come from RGB supervision rather than from the normal channel.","supporting_citations":[{"cited_title":"Mvip- nerf: Multi-view 3d inpainting on nerf scenes via diffusion prior","cited_arxiv_id":null,"evidence_quote":"MVIP-NeRF is the primary baseline that applies appearance and geometric score distillation to NeRF inpainting; GB-NeRF builds on it by adding fine-tuning and BSD."},{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"LoRA provides the low-rank adaptation mechanism that allows fine-tuning the U-Net and text encoder without destroying the pretrained Stable Diffusion weights."},{"cited_title":"A task is worth one word: Learning with task prompts for high-quality versatile image inpainting, 2023","cited_arxiv_id":null,"evidence_quote":"The task-prompt inpainting training method is followed for the self-supervised inpainting loss used during diffusion fine-tuning on RGB and normal maps."}],"review_version":1}