{"id":"fe0b2dea-fd36-423f-a664-00db3483cd27","arxiv_id":"2501.03173","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MObI jointly inpaints camera and lidar views of driving scenes, inserting objects from a single reference image at a user-specified 3D bounding box.","lead":"MObI inserts objects from a single reference photograph into both the camera image and the lidar point cloud of a driving scene, at a user-specified 3D bounding box. It couples reference-guided diffusion inpainting with 3D box conditioning so synthetic test cases can be generated across sensors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lidar realism metrics in Table 2 may be dominated by the VAE reconstruction baseline, not by the diffusion model, so the claimed multimodal improvement needs a control that isolates generation quality.","rationale":"The reader's identified weakest assumption is the use of the same 200-image test set for checkpoint selection and final evaluation. That is a legitimate correctness concern about FID-based model selection bias, but it mainly affects the magnitude and fairness of the comparison against PbE on camera FID/LPIPS and is fixable by re-selection on a validation set. In this pass, a more load-bearing gap appears in the lidar evaluation: the paper's central novelty over PbE is joint multimodal generation, yet its lidar realism metric (D-LPIPS/I-LPIPS) is computed on the output of a diffusion model whose range decoder has been specially fine-tuned for reconstruction. Because Table 1 demonstrates that the autoencoder alone can reconstruct object mask patches with very low error, the reported lidar numbers in Table 2 (e.g., D-LPIPS 0.129 in reinsertion) are not enough to establish that the diffusion process, rather than the autoencoder, is responsible for the lidar realism. This does not make the paper's core construction false; the detection-based geometric check in Sec. 3.3 provides independent evidence of correct 3D placement, and the camera results remain meaningful. However, the claim that MObI jointly generates realistic lidar object appearance is under-supported as currently written. Because a concrete, inexpensive control exists (autoencoder-only reconstruction on the same test set) and would resolve the ambiguity, the verdict remains CONDITIONAL rather than requiring rejection. I disagree with the reader that the test-set model selection is the single most load-bearing concern; it affects headline FID comparisons but the central novelty of the paper is the lidar modality, where the evaluation design has a more direct hole.","tokens_in":19223,"tokens_out":1826,"duration_ms":16346,"concrete_test":"Run the same D-LPIPS and I-LPIPS computation (Sec. 3.2) on reconstructions obtained by encoding and decoding the original object-centric range view without any diffusion sampling — i.e., pass the ground-truth object patch through E^(R) and D^(R) and compare against the original patch. If this autoencoder-only baseline already achieves D-LPIPS close to the reported 0.129/0.132 and I-LPIPS close to 0.148/0.153, then the diffusion model's marginal contribution to lidar realism is small and the headline multimodal claim should be re-scoped. Also report the corresponding metrics for the same 200 test objects used in Table 2, not a separate reconstruction set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that one diffusion model jointly generates camera and lidar objects with high realism and coherence. The novelty over PbE is the lidar modality, so the lidar realism numbers must actually measure the generation. In Section 3.2, lidar realism is computed as D-LPIPS and I-LPIPS between the full diffusion output patch (after the range decoder) and the original range-image patch. But Table 1 shows the adapted range autoencoder alone already reconstructs object masks with median depth error around 0.037 m and intensity MSE around 2.0 on the same range-view representation. Without a reported two-component decomposition separating VAE reconstruction error from diffusion sampling error, the excellent lidar LPIPS scores in Table 2 could be dominated by the frozen/autoencoder part, while the diffusion contribution to lidar object appearance remains unmeasured. The paper explicitly states the lidar VAE was fine-tuned for 'better object reconstruction' (Sec. 2.1) and reports that PbE-style full-image realism metrics are unavailable for lidar (Sec. 3.2), so this is a genuine gap in the evaluation chain. If the diffusion head contributes little beyond the autoencoder, the claimed multimodal joint generation is weakened, though the camera results and the detector check on camera-lidar fusion would still stand.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents MObI, a latent diffusion framework that jointly inpaints camera images and lidar range views, conditioned on a single reference RGB image, a 3D bounding box, and scene context. The method extends Paint-by-Example with a projected 3D bounding box encoder and a gated cross-modal attention module, and adapts a Stable Diffusion VAE to the lidar range-view modality through normalization and fine-tuning. Experiments on nuScenes cover object reinsertion and replacement, with camera realism metrics (FID, LPIPS, CLIP-I), proposed lidar realism metrics (D-LPIPS, I-LPIPS), ablations of the bounding box and cross-attention components, and a downstream 3D object detection evaluation with BEVFusion on reinserted objects. The central claims are that a single diffusion model can generate coherent camera and lidar object appearances, that 3D bounding box conditioning improves spatial alignment and realism, and that the resulting counterfactual multimodal scenes are usable by an off-the-shelf detector.","tokens_in":19511,"tokens_out":3478,"duration_ms":32125,"significance":"The problem is well motivated: multimodal counterfactual data for autonomous driving testing requires controllable and realistic object insertion, and existing methods either lack 3D control or do not jointly handle camera and lidar. MObI's combination of reference-based inpainting, 3D box conditioning, and joint camera-lidar generation is a reasonable contribution, and the detection evaluation on reinserted objects is a direct and valuable test of the conditioning mechanism. If the quantitative realism claims are robust, the method would be a useful baseline for future multimodal inpainting work. However, as detailed below, the lidar realism numbers and the FID-based checkpoint selection currently leave the strength of the quantitative claims in question. The paper is empirical and does not suffer from circular reasoning; the self-citation [15] is not load-bearing.","major_comments":[{"comment":"The final model is selected by FID on a test set of 200 pre-selected images (Sec. 2.4), and Table 2 reports FID on reinsertion and replacement evaluation sets that appear to be the same or an overlapping set. This selection procedure makes the reported FID improvements over PbE optimistically biased, because the model was chosen to minimize FID on the evaluation data. The paper should either use a separate validation set for checkpoint selection, or report Table 2 on a held-out set that was not used for model selection, and state explicitly which images were used in each step.","section":"Sec. 2.4 and Table 2"},{"comment":"The lidar realism metrics D-LPIPS and I-LPIPS are computed on the full diffusion output after the range decoder, compared against the original range-view patch. Table 1 shows that the adapted range autoencoder alone already reconstructs object masks with a median depth error of 0.037 m and intensity MSE of 2.009, meaning the VAE component contributes substantially to the final range appearance. Without a control that isolates the VAE reconstruction error from the diffusion sampling error (for example, reporting the same LPIPS distances for the VAE reconstruction of the ground-truth object patch, with and without the diffusion step), the excellent lidar scores in Table 2 cannot be attributed to the diffusion model, and the central claim of joint camera-lidar generation is not quantitatively established for lidar.","section":"Sec. 3.2 and Table 1"},{"comment":"The description of the ground-truth patches used for FID and LPIPS is ambiguous: 'extended patches around the object from the final composited images, compared to the real patches' does not specify how the extended region is defined, how the patch coordinates align between the composited output and the real scene, or how the tracked-reference setting (where the reference comes from a different timestamp and the object may differ in pose or lighting) maps to a 'ground-truth patch'. This ambiguity affects the reproducibility of Table 2 and the fairness of the comparison with PbE and copy&paste, since all methods are evaluated on the same patches only if the patch construction is well-defined and consistent.","section":"Sec. 3.2, FID/LPIPS definitions"},{"comment":"The realism results are reported as point estimates without confidence intervals or significance tests. Given that the evaluation uses only 200 objects and the claims are comparative (MObI vs. PbE; bbox conditioning vs. not), the paper should include bootstrap confidence intervals over objects for the main realism metrics, so the reader can judge whether the differences are meaningful rather than noise.","section":"Table 2 and Sec. 3.2"}],"minor_comments":[{"comment":"Typo: 'bonding box' should be 'bounding box'.","section":"Sec. 2.1"},{"comment":"The cross-domain reference definition contains an apparent typo: '(rainy(Id) ≠ rainy(Id) or night(Is) ≠ night(Id))' should presumably be '(rainy(Is) ≠ rainy(Id) or night(Is) ≠ night(Id))'.","section":"Sec. 3.1"},{"comment":"The text says 'strong coherence (lightning, weather conditions, occlusions, etc.)'; 'lightning' should be 'lighting'.","section":"Sec. 3.1 qualitative results"},{"comment":"The captions contain the typo 'seperate' instead of 'separate'.","section":"Fig. S1 and S2 captions"},{"comment":"The caption of Fig. 6 says 'Left:' twice; the second occurrence should be 'Right:'.","section":"Sec. 3.3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the evaluation methodology needs tightening before the quantitative claims can be trusted. The FID-based checkpoint selection issue and the missing VAE-vs-diffusion decomposition for lidar realism are the two load-bearing weaknesses; both are fixable within the paper's scope. I do not see a circularity problem, and the detection experiment is a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper earns a real referee. The method extends Paint-by-Example to jointly generate camera and lidar object patches conditioned on a 3D bounding box from a single RGB reference, and as far as I can tell from the cited work, that combination is new. The detection experiment is the strongest part: they reinsert objects into nuScenes frames with a tracked reference and show an off-the-shelf BEVFusion detector finds them with only modest errors. That is a direct test of the 3D box conditioning and it largely holds up.\n\nThey also do useful engineering: the lidar VAE adaptation is ablated step by step (Table 1), the cross-modal gated attention is a reasonable adaptation of ControlNet-style gating, and the limitations section is honest about out-of-domain failures and background artefacts. No formal theory here, but none is needed.\n\nSoft spots are all in the quantitative realism claims. First, Sec 2.4 says the final checkpoint is chosen by FID on a test set of 200 pre-selected images, and the same set is then used to report Table 2. That inflates the comparison against PbE. It is fixable: hold out a separate selection set, or report both. Second, the lidar realism metrics are D-LPIPS and I-LPIPS computed on the diffusion output after the range decoder, but without a control that runs the range autoencoder alone on the same object patch. Given that Table 1 shows the fine-tuned VAE already reconstructs object masks with median depth error 0.037 m, a large fraction of the 'good' lidar LPIPS could be the autoencoder, not the diffusion model. That gap needs an ablation, not a rewrite. Third, the replacement LPIPS/FID ground truth is ambiguous: comparing the inpainted patch to the original patch is not a meaningful target when the reference is a different object in a different domain. I'd ask them to state explicitly what the reference patch is and whether they expect similarity or realism.\n\nNone of this sinks the method. The qualitative results are consistent with the detection numbers, and the concept is useful for counterfactual scene generation. I'd send it to review with the request that the authors either redo the checkpoint selection or show the eval doesn't change, and add the lidar VAE control and a clearer GT definition.","headline":"Genuinely new joint camera-lidar object inpainting with a solid geometry check, but the realism numbers need a cleaner evaluation before you should trust them.","tokens_in":20038,"tokens_out":3032,"would_cite":true,"duration_ms":28020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MObI's central claim is that a single diffusion model can insert objects into both camera images and lidar range views at a 3D location given by a bounding box, and that this joint conditioning yields realistic, scene-consistent results.","keywords":["multimodal object inpainting","diffusion models","camera-lidar generation","3D bounding box conditioning","range view generation","autonomous driving data","reference-based inpainting","object insertion"],"falsifier":"The cleanest check is to reselect the model checkpoint on a held-out validation split disjoint from the 200-image test set and recompute the realism metrics; if the method no longer beats the reference-based baseline on FID and LPIPS, the realism claim is falsified. A second check would be removing the bounding box token at inference and measuring whether detection alignment drops, which would test whether the box conditioning is actually doing the geometric work.","tokens_in":19048,"feed_emoji":"🚗","tokens_out":7470,"duration_ms":62980,"temperature":0.7,"pith_summary":"This paper introduces MObI, a diffusion-based method for inpainting objects into driving scenes jointly in camera images and lidar range views. The central claim is that a single model, conditioned on one reference RGB image, a projected 3D bounding box, and scene context, can generate a semantically consistent object in both modalities at a specified location. If true, this gives a controllable way to create counterfactual driving scenes, such as replacing a car with a pedestrian at the same 3D position, without collecting new real-world data. The authors argue that conditioning on the 3D box, rather than only a 2D edit mask, is what makes placement and scale accurate, and they support this with realism metrics and detection experiments showing an off-the-shelf detector still finds the reinserted objects.","feed_headline":"One diffusion model writes objects into both camera and lidar feeds","feed_subtitle":"Single reference image and 3D box control where the object lands in each sensor view.","key_machinery":"The carrying mechanism is a latent diffusion model inherited from reference-based inpainting and fine-tuned so that camera images and lidar range views are encoded into a shared latent space and denoised together. Three conditioning paths are added: Fourier-encoded projected 3D bounding box tokens processed by a small MLP; a gated cross-attention adapter that injects box and reference tokens into the denoising U-Net with zero-initialised gates; and modality-dependent cross-attention layers that let camera tokens attend to lidar tokens and vice versa within the same batch. A key engineering piece is the lidar range-view autoencoder: a pretrained image VAE adapted through average-pooled downsampling, intensity and depth normalisation, and fine-tuned input and output residual blocks, which lets the image-based diffusion machinery operate on lidar range views.","core_discovery":"MObI extends reference-based image inpainting so that one diffusion model generates camera and lidar outputs jointly from a single reference image and a 3D bounding box. The model is trained in a self-supervised way by masking out the object region and predicting the full scene conditioned on the reference, the box, and both modalities' contexts. At inference it inserts the generated object patch back into the original camera image and lidar range view, and the paper reports that on a 200-object nuScenes validation set the 512-resolution model beats the reference-based baseline [65] on FID, LPIPS, and CLIP-I for camera reinsertion, while also producing low lidar depth and intensity perceptual distances. Detection results with an off-the-shelf camera-lidar detector show that reinserted objects remain detectable with small translation, scale, and orientation errors, which the paper reads as evidence that the 3D box conditioning keeps geometry accurate.","pith_inferences":["A natural extension the paper leaves implicit is explicit multi-frame consistency: conditioning on all boxes in a scene or enforcing temporal coherence across frames could prevent the background-object modifications that currently happen when an edit mask overlaps another object.","If the realism results hold under a clean checkpoint-selection protocol, MObI would offer a low-cost way to generate hard negative and counterfactual traffic scenes for testing detector robustness, including weather and lighting shifts that are rare in real logs.","The bounding-box conditioning idea could transfer to modalities beyond camera and lidar, such as radar or occupancy grids, since the conditioning tokens are modality-agnostic and only the encoder and decoder would need adaptation."],"forward_implications":["Object insertion and replacement are possible with a single reference photo and a 3D box, so the same framework covers reinsertion, cross-domain replacement across weather and time-of-day, novel-view synthesis, and object deletion via an empty reference.","Because the model is conditioned on a projected 3D bounding box rather than only a mask, the generated object inherits the box's position, scale, and depth, which is what makes an off-the-shelf camera-lidar detector still detect the reinserted object.","Generating camera and lidar jointly with cross-modal attention yields better camera realism than camera-only inpainting at the same resolution in the paper's tables, and the low lidar depth and intensity perceptual distances establish an initial baseline for lidar object inpainting metrics.","The approach can apply to classes not seen during training, though the paper reports lower quality and occasional reverts to familiar classes for very out-of-domain references."],"supporting_citations":[{"why":"Supplies the pretrained diffusion inpainting model, training procedure, and evaluation baseline that the method extends.","marker":"[65]"},{"why":"Provides the zoom-in crop and viewport transformation strategy for centering objects in the camera input.","marker":"[5]"},{"why":"Supplies the Fourier-embedding approach for encoding projected 3D bounding boxes as conditioning tokens.","marker":"[10]"},{"why":"Provides the nuScenes dataset from which training objects, reference images, and evaluation selections are drawn.","marker":"[3]"},{"why":"Provides the pretrained latent diffusion and autoencoder weights that are fine-tuned for camera and lidar generation.","marker":"[43]"},{"why":"Encodes the reference image into the CLIP feature that guides object appearance during inpainting.","marker":"[41]"},{"why":"Is the off-the-shelf camera-lidar detector used to measure whether reinserted objects remain geometrically detectable.","marker":"[34]"},{"why":"Supplies the zero-initialised gating mechanism used by the bounding box adapter and cross-modal attention.","marker":"[71]"},{"why":"Provides the interleaved gated cross-attention pattern used to inject new conditioning modalities.","marker":"[1]"}],"fun_headline_variants":["Single diffusion model fills camera and lidar from one image","One 3D box guides camera-lidar object inpainting","Diffusion inpainting for both camera and lidar at once","Joint camera-lidar inpainting from a single reference image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains over the baseline assume that choosing the final checkpoint by FID on the same 200-image test set that is then used to report FID and LPIPS does not bias the comparison; if checkpoint selection has overfit this set, the improvements would shrink or vanish on a truly unseen set.","fun_headline_variants_meta":{"raw":{"variants":["Single diffusion model fills camera and lidar from one image","One 3D box guides camera-lidar object inpainting","Diffusion inpainting for both camera and lidar at once","Joint camera-lidar inpainting from a single reference image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1620,"prompt_tokens":896,"completion_tokens":724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":512,"tokens_out":724,"duration_ms":6463,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:57.253382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The cleanest check is to reselect the model checkpoint on a held-out validation split disjoint from the 200-image test set and recompute the realism metrics; if the method no longer beats the reference-based baseline on FID and LPIPS, the realism claim is falsified. A second check would be removing the bounding box token at inference and measuring whether detection alignment drops, which would test whether the box conditioning is actually doing the geometric work.","supporting_citations":[{"cited_title":"nuscenes: A multi- modal dataset for autonomous driving","cited_arxiv_id":null,"evidence_quote":"Provides the nuScenes dataset from which training objects, reference images, and evaluation selections are drawn."},{"cited_title":"Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation","cited_arxiv_id":null,"evidence_quote":"Is the off-the-shelf camera-lidar detector used to measure whether reinserted objects remain geometrically detectable."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-initialised gating mechanism used by the bounding box adapter and cross-modal attention."}],"review_version":1}