{"id":"a7f8c1d7-1826-4fdd-9e2d-707d14e6fa5c","arxiv_id":"2412.05881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A diffusion inpainting model conditioned on a second viewpoint of the same scene produces 3D-consistent fills for occluded regions without 3D supervision.","lead":"This paper proposes InConDiff, an image inpainting model that fills masked regions using a second photo of the same scene as visual context. Because the second view often shows what is hidden behind an occluder, the method aims to keep inpainted content 3D-consistent without explicit 3D supervision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation protocol never defines the reference image for PSNR/LPIPS/SSIM, and no metric operationalizes '3D consistency', so the central claim of 3D-consistent outperformance is not quantitatively supported as reported.","rationale":"The paper proposes a plausible mechanism: conditioning a diffusion denoiser on a second viewpoint through cross-attention, along with RePaint-style resampling, can inject geometric context during inpainting. The architecture follows CroCo, the training objective in Eq. (7) is straightforward, and the qualitative examples do suggest that the in-context image influences the output. I read the paper in good faith and do not see an internal contradiction that would refute the method. However, the central claim has two empirical components: (1) InConDiff outperforms state-of-the-art methods, and (2) it ensures 3D consistency. Both components rest on the evaluation in Section 4, and that evaluation is underspecified at the exact point that matters. The reader's weakest assumption correctly identifies the undefined ground-truth reference for the 2D metrics; this is load-bearing because if the reference is the original occluded image, the metrics reward methods that preserve the occlusion, which is the opposite of the task. I extend the concern to the absence of any direct 3D-consistency metric: even with a well-defined reference, PSNR/LPIPS/SSIM are 2D similarity scores and cannot by themselves certify geometric consistency with the scene or with the additional viewpoint. Table 4's internally inconsistent PSNR row and the unspecified Laplace schedule are additional reproducibility issues, but they are secondary. Because the mechanism remains plausible and the flaw is in the evidence rather than the architecture, a conditional verdict is appropriate: the paper should be accepted only if the authors specify and ideally release the evaluation protocol, report metrics against a proper occluder-free reference, and include at least one multi-view geometric consistency check. This matches the reader's conditional verdict, so I recommend no change.","tokens_in":11445,"tokens_out":5150,"duration_ms":54085,"concrete_test":"Ask the authors for the exact evaluation script or metric protocol, then recompute the headline numbers on HM3D (which provides meshes and camera poses) using three references: (a) the original image with the occluder, (b) the in-context image warped into the target viewpoint using known depth and pose, restricted to masked pixels visible in both views, and (c) a manually cleaned occluder-free background. If the reported PSNR/LPIPS/SSIM gains over SD-XL disappear under (b) or (c), the 'outperforms SOTA' claim fails. Additionally compute a geometric consistency score, such as mean reprojection error or depth error in the masked region against the warped reference; if no such metric is reported, the '3D consistency' claim remains untested.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 ('Evaluation Metrics') defines PSNR, LPIPS, and SSIM only by their textbook meanings and never states what ground truth the masked pixels are compared against. This matters directly for the central claim: the method's purpose is to remove occluders (cars, pedestrians, poles), so the original unmasked image still contains the very content the model is asked to erase. If metrics are computed against that image, correct removal is penalized and InConDiff's apparent advantage over SD/SD-XL is uninterpretable. No alternative reference is specified: there is no statement that scores are computed against a second viewpoint warped into the target view, a manually cleaned background, or a reconstructed occluder-free image. The word '3D consistency' is likewise never operationalized: Tables 1-4 report only 2D image-similarity metrics, and no geometric, multi-view, or depth-based consistency measure appears anywhere. Thus the headline claims that the method 'ensures 3D consistency' and 'outperforms the state-of-art methods' hinge on an evaluation whose reference and whose very quantity of interest are undefined. This does not refute the underlying mechanism, but it does mean the quantitative evidence cannot currently bear the weight of the abstract's claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InConDiff, an image-inpainting method that extends an unconditional DiT-based diffusion model by injecting a second image of the same scene as in-context guidance during denoising. The authors argue that the cross-view information acts as a 3D prior, allowing harmonized and 3D-consistent inpainting without explicit 3D supervision. They describe a post-conditioning scheme with resampling and a Laplace noise schedule, and report quantitative results on HM3D, MegaDepth, StreetView, and WalkingTour, comparing against RePaint, Stable Diffusion, and Stable Diffusion XL. The central claim is that InConDiff outperforms the state of the art and \"ensures the 3D consistency\" of inpainted regions.","tokens_in":11678,"tokens_out":4200,"duration_ms":40644,"significance":"If the claims were fully supported, the method would be a simple and practical way to inject geometric awareness into 2D diffusion inpainting, with potential applications in object removal, view synthesis, and robotics. The architectural modification is clean and the paper provides code, ablations on mask ratio and resampling schedule, and a transparent comparison to several baselines. However, the evidence as presented is not yet load-bearing: the evaluation never defines the reference image for PSNR/LPIPS/SSIM, no metric operationalizes \"3D consistency\", and key numerical results appear internally inconsistent. These gaps directly affect the abstract's claim of 3D-consistent outperformance, so the paper requires substantial revision before the quantitative conclusions can be accepted.","major_comments":[{"comment":"The reference image for PSNR, LPIPS, and SSIM is never specified. The task is to remove occluders (cars, pedestrians, poles), so the original unmasked image still contains the very content to be erased; computing metrics against that image would penalize correct removal. If the metrics are computed against a second viewpoint warped into the target view, a manually cleaned background, or a reconstructed occluder-free image, that protocol is absent. The paper must state the exact reference and the region over which metrics are averaged, and report results separately for the masked area if relevant. Without this, the reported advantage over SD/SD-XL is uninterpretable.","section":"Section 4 (Evaluation Metrics); Tables 1-3"},{"comment":"No metric operationalizes \"3D consistency\", which is the paper's central claim. Tables 1-4 report only 2D image-similarity metrics. The authors should add a quantitative geometric measure, for example: (i) multi-view reprojection error between the inpainted image and a second view, using known depth/pose; (ii) photometric or feature-metric consistency after warping the inpainted region to the second view; (iii) depth-map consistency with a pretrained monocular depth estimator before and after inpainting; or (iv) a downstream geometric task such as relative pose estimation accuracy on inpainted images. Without such a metric, the phrase \"ensures the 3D consistency\" is not verified by the reported experiments.","section":"Abstract; Section 4; Conclusion"},{"comment":"The PSNR values for Njumps=1 are 18.7 regardless of Nsteps (250, 500, 1000), and LPIPS and SSIM are also identical across those columns (0.26 and 0.83 respectively). This suggests either a copy-paste or alignment error in the table, or that the number of denoising steps has no effect on output quality when no resampling is used, which would be surprising for a DDPM. The row should be rechecked and corrected. Additionally, no variance or confidence intervals are reported for any table, so the significance of small differences from baselines (e.g., MegaDepth semantic SSIM 0.86 vs 0.88) cannot be assessed.","section":"Table 4"},{"comment":"The comparison protocol is underspecified. The paper does not state the number of test images per dataset, how semantic and random masks are generated (beyond the random-mask description), whether the exact same masks are used for all compared methods, or whether the Stable Diffusion and SD-XL checkpoints are fine-tuned on the target datasets. These details are necessary to interpret the quantitative claims, especially because the baselines are general-purpose models and may not have been adapted to the dataset-specific distribution. The authors should provide the full evaluation setup, ideally in the main text or a clearly marked supplementary document.","section":"Section 4.1 (Comparison to State-of-the-Art)"}],"minor_comments":[{"comment":"The phrase \"We prove its efficiency of InConDiff\" is ungrammatical and should read \"We demonstrate the effectiveness of InConDiff\" or similar.","section":"Section 1 (Contributions)"},{"comment":"The notation for the noise schedule is inconsistent: Equation (8) uses \\hat\\alpha_t while Equations (1)-(2) use \\bar\\alpha_t. Please unify the notation.","section":"Section 3.3 (Conditioning DDPMs on the known region)"},{"comment":"The caption says \"semantic (top) and random (top) masks\" where the second \"top\" should presumably be \"bottom\".","section":"Figure 6 caption"},{"comment":"There is a typo in the reference to Sohl-Dickstein et al.; the name is written as \"SohlDickstein\" without a space.","section":"Section 2 (Related Work)"},{"comment":"The term \"in-context\" is used throughout but never formally defined. Since \"in-context learning\" has a well-known meaning in large language models, the authors should clarify that here it refers to conditioning on an additional image as input to the denoising network.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting problem, and the core idea is plausible. However, the evaluation is currently not adequate to support the abstract's claims. The missing reference definition for the image-quality metrics and the absence of any 3D-consistency metric are load-bearing issues that require new experiments or at least a careful re-analysis of existing ones. The internal inconsistency in Table 4 also needs correction. I would advise inviting a revision that addresses these points, and I would be willing to review the revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is solid: condition an unconditional diffusion inpainter on a second viewpoint of the same scene, using a DiT with cross-attention between noised and clean tokens, and train without 3D supervision. That is a real extension beyond RePaint's single-image resampling and CroCo's representation-learning objective, and the paper is honest about its limitations, including failures in crowded scenes. The code release is a plus, and the ablation on mask ratio and resampling settings shows some thought about what matters.\n\nThat said, the stress-test concern lands directly. Section 4 never says what the PSNR/LPIPS/SSIM scores are computed against. For semantic masks that remove cars and pedestrians, the original unmasked image contains exactly the content the model is asked to erase. If that is the reference, the metrics penalize correct removal and the reported advantage over SD-XL is uninterpretable. No alternative reference is described—no warped second view, no cleaned background, nothing. And '3D consistency' is never operationalized: the tables only report 2D image-similarity metrics, with no multi-view or depth-based check. The qualitative figures look plausible, but the central quantitative claim is hanging without a defined measurement.\n\nOther soft spots are minor by comparison. Table 4 has odd repetitions across step and jump settings that look like a typo or a saturated setting. The Laplace schedule is mentioned as adopted but its parameters are not given. The comparison to only RePaint, SD, and SD-XL is thin for an 'outperforms state-of-the-art' claim, and the WalkingTour results are per-scene with no aggregate. None of these refute the mechanism, though. The method is sensible, the training setup is reasonable, and the idea deserves follow-up.\n\nI would send this to peer review rather than desk reject—the novelty is real and the flaws are fixable. But a serious reviewer should push hard for a defined evaluation protocol, a real 3D-consistency metric, and error bars before the numbers can be trusted. For a reading group, it is useful as a case study in how evaluation gaps can undermine an otherwise promising method.","headline":"The second-viewpoint conditioning idea is genuinely useful, but the evaluation never defines what the metrics are measured against, so the '3D-consistent outperformance' claim is not yet supported.","tokens_in":12225,"tokens_out":2790,"would_cite":true,"duration_ms":28631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a diffusion model on paired views lets it inpaint occluded regions that stay consistent with the scene's 3D structure — without 3D supervision.","keywords":["image inpainting","diffusion models","3D consistency","in-context guidance","multi-view training","unconditional DDPM","cross-view completion","noise schedule resampling"],"falsifier":"Open Section 4 and the authors' released code and check which image is used as the reference for PSNR, LPIPS, and SSIM. If the scores compare against the original masked image that still contains the occluder, the metrics penalize exactly the desired removal; on HM3D, one could render clean background images with known geometry and recompute the numbers, which would settle whether the advantage over Stable Diffusion XL reflects true 3D consistency.","tokens_in":11208,"feed_emoji":"🧩","tokens_out":5946,"duration_ms":53600,"temperature":0.7,"pith_summary":"The paper tries to establish that an unconditional diffusion model, trained only on pairs of images showing the same scene, learns enough 3D structure to repaint masked occluders in a geometrically consistent way. The claim is that injecting a clean second viewpoint of the scene into the denoising process acts as in-context guidance, harmonizing masked and unmasked regions while keeping the result coherent with the scene's actual geometry. If true, this gives a route to 3D-aware inpainting that needs no 3D labels, no depth maps, and no multi-view geometry at inference — just a second picture of the same place. The authors report that their method, InConDiff, beats RePaint, Stable Diffusion, and Stable Diffusion XL on four datasets with both semantic and random masks.","feed_headline":"A second photo makes diffusion inpainting 3D-consistent","feed_subtitle":"Training on image pairs lets a denoiser fill occlusions consistent with scene geometry, no 3D labels needed.","key_machinery":"The key machinery is in-context guidance via cross-view completion: a Diffusion Transformer whose decoder cross-attends to clean image patches from a second view of the same scene, trained with the standard denoising objective $L_2 = \\mathbb{E}_{t,x_0,\\epsilon}\\|\\epsilon - g_\\theta(x_t,t,x')\\|^2$. The second view provides the inductive bias that lets a 2D denoiser absorb 3D priors. This is paired with RePaint-style post-conditioning of the reverse process, where known pixels are resampled from the forward noising distribution while masked pixels come from the model, and a Laplace noise schedule that reduces the number of resampling jumps needed for boundary harmonization.","core_discovery":"The central discovery is that 3D consistency is not something a diffusion model has to be given; it can be induced by training the denoiser on pairs of views. InConDiff modifies an unconditional DDPM so that the denoising network sees, in addition to the noisy image being denoised, a clean image from an alternative viewpoint of the same scene. The decoder cross-attends to the clean view's tokens while predicting the noise, letting it compare the two views and recover geometric cues during 2D denoising. At inference, a mask is applied post hoc, and a resampling schedule with noise jumps harmonizes the boundary between generated and known pixels. The authors argue that this yields semantically coherent and 3D-consistent inpaintings without explicit 3D supervision.","pith_inferences":["If the method scales to video, the same in-context principle suggests that using several frames rather than one additional image could remove moving objects consistently across time — the paper itself names multi-view extension as future work.","The metric protocol is an open question: the paper never states whether PSNR, LPIPS, and SSIM are computed against the original image with the occluder or against a clean reference, so the reported gains should be read with that ambiguity in mind.","A natural testable extension is single-image inpainting by retrieving a second view from a visual database, making the in-context image a retrieved prior rather than a required input."],"forward_implications":["If the claim holds, inpainting systems can be made 3D-aware by training on any paired-view corpus, including photo collections with known camera poses, rather than requiring synthetic 3D data or depth supervision.","Post-hoc masking means the same trained model can handle novel mask shapes, including masks that cover a large fraction of the image, without retraining on masks.","The reported performance gap over Stable Diffusion XL suggests that scene-specific geometry, not just large-scale 2D priors, is the limiting factor for consistent content filling.","A faster Laplace-based resampling schedule cuts the inference-time overhead of harmonization, making the approach more practical for real use."],"supporting_citations":[{"why":"Provides the denoising objective and reverse-process parametrization that the method is built on.","marker":"[9]"},{"why":"Supplies the resampling and post-conditioning mechanism that lets an unconditional model perform mask-based inpainting.","marker":"[17]"},{"why":"Supplies the Diffusion Transformer backbone that makes injecting a second image into the denoiser straightforward.","marker":"[22]"},{"why":"Supplies the cross-view completion architecture and pair-quality scoring used to extract training pairs and design the network.","marker":"[29]"},{"why":"Supplies DINOv2, used to detect and mask occlusion classes for the semantic-mask experiments.","marker":"[20]"},{"why":"Supplies the Laplace noise schedule that reduces the number of resampling jumps needed for harmonization.","marker":"[7]"}],"fun_headline_variants":["Diffusion inpainting gains 3D consistency from a second view","Inpainting with a pair of photos: 3D consistency without 3D labels","Two-view diffusion: 3D-consistent inpainting with no 3D data","Cross-view diffusion fills occlusions with 3D-aware results","A second viewpoint guides diffusion to 3D-consistent inpainting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumption that a second viewpoint of the same scene injects enough 3D structure into an otherwise 2D denoiser to make inpaintings geometrically consistent, and that the reported metrics compare against a ground truth the paper never actually defines.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion inpainting gains 3D consistency from a second view","Inpainting with a pair of photos: 3D consistency without 3D labels","Two-view diffusion: 3D-consistent inpainting with no 3D data","Cross-view diffusion fills occlusions with 3D-aware results","A second viewpoint guides diffusion to 3D-consistent inpainting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3136,"prompt_tokens":848,"completion_tokens":2288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2186}},"tokens_in":464,"tokens_out":2288,"duration_ms":14636,"temperature":1.0,"reasoning_tokens":2186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:14:47.786903+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open Section 4 and the authors' released code and check which image is used as the reference for PSNR, LPIPS, and SSIM. If the scores compare against the original masked image that still contains the occluder, the metrics penalize exactly the desired removal; on HM3D, one could render clean background images with known geometry and recompute the numbers, which would settle whether the advantage over Stable Diffusion XL reflects true 3D consistency.","supporting_citations":[{"cited_title":"Denoising diffusion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides the denoising objective and reverse-process parametrization that the method is built on."},{"cited_title":"Repaint: Inpainting using denoising diffusion probabilistic mod- els","cited_arxiv_id":null,"evidence_quote":"Supplies the resampling and post-conditioning mechanism that lets an unconditional model perform mask-based inpainting."},{"cited_title":"Scalablediffusionmod- els with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the Diffusion Transformer backbone that makes injecting a second image into the denoiser straightforward."},{"cited_title":"Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-view completion architecture and pair-quality scoring used to extract training pairs and design the network."}],"review_version":1}