{"id":"a2234ad6-b627-4618-bd7a-6368fc458d2f","arxiv_id":"2507.23058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A single reference image guides a diffusion model to insert coherent objects into camera-plus-lidar driving scenes and to insert mammographic anomalies into new scans.","lead":"This dissertation presents two reference-guided diffusion inpainting methods: MObI inserts objects into camera and lidar driving scenes from a single RGB image and a 3D bounding box, and AnydoorMed inserts breast anomalies into mammograms from a reference lesion. A smart generalist should read it because controllable synthetic counterfactuals could stress-test autonomous-driving and medical diagnostic models without expensive real-world data collection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference-fidelity claim is not supported: both methods condition only on frozen global encodings plus a box, with no identity or fine-structure loss, and the paper's own failures (lost microcalcifications, horse-to-car) show the central 'realistic and controllable insertion' claim holds only…","rationale":"The reader's weakest assumption identifies the fragile premise that a single frozen reference encoding plus a bounding box carries enough information for identity, orientation, scale, and fine structure. My reading agrees and sharpens it: the architecture provides no direct supervision tying the generated patch to the reference, and AnydoorMed's high-frequency detail path is fed from the context image rather than the reference, so fine attributes like microcalcifications have no dedicated conditioning channel. The paper's own failure figures demonstrate exactly this failure mode. This does not refute the methods' ability to generate coarse, in-distribution, perceptually plausible insertions; the reported FID/LPIPS improvements over PbE and AnyDoor support that weaker claim. But it does mean the abstract's strong claims of 'controllable' and 'impressive detail preservation' are not supported at the level of object identity or clinical micro-structure. A task-specific fidelity test, such as calcification recall or a same-class identity-swap experiment, would settle whether the central claim holds. Since the reader already recommends conditional acceptance with narrowed claims, my concern does not change the verdict: the paper should be published only with claims trimmed to coarse semantic consistency and with explicit acknowledgement that fine reference detail is not reliably preserved.","tokens_in":43932,"tokens_out":8487,"duration_ms":110512,"concrete_test":"On the 426 VinDr-Mammo validation positives used in Table 3.1, compute a calcification-level recall: segment individual microcalcifications (or use expert annotations) in each reference crop and count what fraction appear in the reinserted region; report recall separately for suspicious-calcification cases. If calcification recall is below roughly 80% in a meaningful fraction of cases, the 'structural integrity' and 'detail preservation' claims fail for the most safety-relevant attribute. For MObI, run the analogous identity test: reinsert the same object under the same 3D box with references of two different car models and measure CLIP-I/DINOv2 similarity of the inpainted region; if the outputs are near-identical, the reference is not actually conditioning identity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that a single reference encoding and a bounding box specify the inserted object's identity, orientation, scale, and fine structure. In MObI, the reference is a frozen CLIP token (Section 2.3.3); in AnydoorMed, it is a frozen DINOv2 token (Section 3.3.3). Neither method enforces an explicit identity or reconstruction loss between the generated object patch and the reference, and AnydoorMed's high-frequency detail branch is computed from the destination context, not from the reference anomaly (Section 3.3.4). This architecture can transfer coarse semantics but has no dedicated channel for reference-specific fine structure such as microcalcifications. The paper's own failure cases confirm the risk: Fig. 4.2 shows microcalcifications not closely replicated, and Fig. 4.3 shows an out-of-domain horse collapsing to a car. The quantitative metrics used (FID, LPIPS, CLIP-I) are also insensitive to tiny clinical features, so they cannot establish the abstract's claim of 'structural integrity' and 'impressive detail preservation' for medical counterfactuals. The central claim therefore needs narrowing unless reference-fidelity is measured directly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes two reference-guided diffusion inpainting methods. MObI extends Paint-by-Example with 3D-bounding-box conditioning and cross-modal camera-lidar attention for object insertion in autonomous driving scenes. AnydoorMed adapts AnyDoor to mammography for anomaly insertion using a DINOv2 reference token and a destination-derived high-frequency detail map. Both methods are evaluated with FID, LPIPS, CLIP-I (and DINOv2 similarity for AnydoorMed), plus a BEVFusion detection check for MObI. The paper reports improvements over Paint-by-Example and AnyDoor and claims that foundation inpainting models can be readily adapted to camera-lidar and medical modalities.","tokens_in":44148,"tokens_out":6284,"duration_ms":70092,"significance":"If the central claim is correct, the contribution is a practical, parameter-efficient adaptation recipe for reference-guided inpainting in non-natural-image modalities, with direct applications to synthetic data generation for perception testing. The work has concrete strengths: MObI is already peer-reviewed at a CVPR workshop, the authors include a downstream detection evaluation, ablate their main design choices, and commit to releasing code and weights; AnydoorMed reports multiple seeds. However, the significance as presented is limited by weak baselines, lack of direct reference-fidelity measurement, and the use of test-set model selection; the claimed 'first-of-its-kind' and 'impressive detail preservation' go beyond what the evidence supports.","major_comments":[{"comment":"The abstract claims 'impressive detail preservation' and 'structural integrity' for AnydoorMed, but no evaluation in the paper directly measures fidelity of the generated anomaly to the reference. In §3.3.3 the reference enters only as a frozen DINOv2 token, and in §3.3.4 the high-frequency map is computed from the destination mammogram I (Ihf = (I⊗Kh + I⊗Kv)⊙I⊙Merode), so the network has no dedicated channel carrying reference-specific fine structure such as microcalcifications; the diffusion objective in §3.3 is the standard noise-prediction loss and contains no identity or reconstruction term. The metrics in Table 3.1 (FID, LPIPS, CLIP-I, DINOv2 similarity) are global or coarse and, as the paper's own Fig. 4.2 shows, a failure case where 'the inserted anomaly does not closely replicate the microcalcifications from the reference image' is not penalized by these numbers. To support the central claim, the authors should add a direct reference-fidelity metric (e.g., local structural similarity or a calcification-detection score inside the edit box) and compare reinsertion output against the original anomaly at the pixel or feature level; otherwise the 'detail preservation' claim should be removed from the abstract.","section":"§3.3.3–3.3.4, Fig. 4.2"},{"comment":"For the Insertion task in §3.4.1, the anomaly is inserted into a healthy scan and there is no ground-truth edited image, yet Table 3.1 reports LPIPS values (0.08) for this setting. The paper does not state what the LPIPS comparison is computed against; if it is compared to the unedited healthy scan, the score rewards outputs that alter the image as little as possible, which would not measure successful insertion. FID on the 426-sample validation set also has high variance, and no details are given on how many patches or crops are used. The evaluation protocol for insertion must be specified unambiguously, or these numbers should be reported only as a calibration/internal reference.","section":"§3.4.1, Table 3.1"},{"comment":"Both methods select the final model on the evaluation set. For MObI, §2.3.6 states the final model is selected 'based on the best FID achieved on a test set of 200 pre-selected images'; for AnydoorMed, §3.3.7 states selection is 'based on the best FID achieved on the test set with 426 samples'. This is a form of test-set model selection that biases the reported metrics upward and invalidates them as unbiased estimates of generation quality. Additionally, all MObI numbers are single-run with no error bars, so it is unclear whether the reported improvements over PbE are within run-to-run variability. The authors should move model selection to a validation split, report the test-set results of the chosen model only, and provide multiple seeds or bootstrapped confidence intervals for at least the main FID/LPIPS comparisons.","section":"§2.3.6, §3.3.7"},{"comment":"The comparison set is too narrow to support the generalization claims made in the abstract and §4.1. In §2.4.2 MObI is compared only to Paint-by-Example and copy-paste; GenMM, which is discussed in §2.2 as the closest multimodal object-inpainting method, is never evaluated quantitatively. Similarly, §3.4.1 compares AnydoorMed only to AnyDoor and copy-paste, and the related-work discussion in §3.2 lists recent medical counterfactual/inpainting methods (e.g., MedEdit [86], RadEdit [88], PRISM [78]) without any comparison. Without at least one stronger or domain-specific baseline, the claim of 'state-of-the-art results according to realism metrics' is an overstatement; the comparisons should be extended or the claim should be explicitly restricted to the tested baselines.","section":"§2.4.2, §3.4.1"}],"minor_comments":[{"comment":"The phrase 'first-of-its-kind framework for Multimodal Object Inpainting' is inaccurate because GenMM [48] is described in §2.2 as a prior multimodal object-inpainting method; the novelty claim should be narrowed to 'first end-to-end joint camera-lidar method'.","section":"Abstract, §2.2"},{"comment":"The text reports Insertion FID 4.89 and LPIPS 0.08 while the table lists 4.78 ± 0.14 and 0.08 ± 0.01; these should be harmonized.","section":"§3.4.2 vs Table 3.1"},{"comment":"The sentence 'adapting the newly added input and output adapters of the range autoencoder' appears to be a copy-paste error from the MObI chapter; AnydoorMed adapts a medical autoencoder, not a range autoencoder.","section":"§3.3.7"},{"comment":"The caption contains 'Left:' twice; the second occurrence should read 'Right:' for the score-distribution plot.","section":"Fig. 2.10 caption"},{"comment":"The thesis front matter (Declaration of originality, Copyright statement, Appendices B/C on ethics and planning) is not appropriate for a journal submission and should be removed or adapted to the journal's format.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written as a BSc dissertation, with thesis front matter and planning appendices; a journal submission would need substantial reformatting. The 'first-of-its-kind' claim also seems to conflict with GenMM, which the authors themselves cite, and this may raise novelty concerns in review. The core technical directions are plausible and the MObI part has already passed peer review at a CVPR workshop, so I do not see grounds for outright rejection; the missing reference-fidelity evaluation and the weak comparison set are, however, load-bearing for the stated claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a BSc dissertation, not a typical arXiv paper, and that shows. The genuinely new contribution is AnydoorMed, a transfer of AnyDoor to mammography anomaly inpainting; MObI is already published at CVPRW 2025, so in this document it's a re-publication with extra context. If you work on synthetic data for driving or medical imaging, the engineering here is believable and the writing is unusually honest about failure modes.\n\nWhat it does well: the MObI chapter is a solid write-up of a peer-reviewed method with ablations, a range-view lidar adaptation, and a BEVFusion detection check. AnydoorMed demonstrates that AnyDoor's machinery can be fine-tuned for mammography with a few thousand images, and the reported FID/LPIPS gains over AnyDoor and copy-paste are plausible. The discussion section explicitly lists the failure cases: microcalcifications not accurately replicated, out-of-domain references collapsing to in-domain classes (horse to car), anatomical implausibility when the box goes outside breast tissue, and copy-paste artefacts.\n\nThe soft spots are mostly where the reader and stress-test point, but with a nuance. The abstract and intro overclaim \"impressive detail preservation\" and \"controllable realism\" when the paper's own figures show the reference-fidelity premise holds only for a limited distribution. Neither method enforces an identity or reconstruction loss between generated patch and reference; both rely on a frozen CLIP/DINOv2 token plus a box. The metrics used (FID, LPIPS, CLIP-I) don't directly measure clinical fidelity or reference-specific fine structure, and there's no clinician evaluation or downstream medical task. So the central claim needs narrowing unless reference fidelity is measured directly. That said, the stress-test's concern is not a hidden flaw: the paper acknowledges it in Section 4.2. The more concrete empirical gaps are: no code released (only GitHub promises), no lidar baseline for MObI, no error bars for MObI, and evaluation limited to filtered placements.\n\nThe circularity concern in the reader take is a non-issue: this is straightforward empirical training on held-out images, no equation-level circularity.\n\nWho is this for? A reader who wants a worked example of adapting reference-guided inpainting to camera-lidar and medical modalities, and who wants an honest account of the limits. It's not a reorganisation of either field. I'd give it a serious referee if it came to a workshop, but I would push for narrowed claims, released artifacts, and a radiologist or downstream-task evaluation for AnydoorMed before relying on it.\n\nRecommendation: engage with it as a useful engineering report; do not cite it as a primary source for MObI (cite the CVPRW paper), and treat AnydoorMed as promising but unvalidated for clinical use.","headline":"A candid BSc dissertation that republishes MObI and adds AnydoorMed, a promising but unvalidated AnyDoor-to-mammography adaptation; honest limitations, overclaimed abstract.","tokens_in":44766,"tokens_out":2863,"would_cite":false,"duration_ms":31284,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This dissertation claims that reference-guided diffusion inpainting, built for natural images, can be adapted with lightweight fine-tuning to generate realistic and controllable counterfactual data in camera-lidar driving scenes and in…","keywords":["reference-guided inpainting","latent diffusion models","multimodal counterfactual generation","camera-lidar object insertion","3D bounding box conditioning","mammography anomaly synthesis","zero-initialised gated cross-attention"],"falsifier":"A direct test is to take references from well outside the training distribution—an animal or vehicle class never seen in nuScenes, or a synthetic anomaly with a distinctive microstructure—insert them at valid locations, and measure identity preservation between the reference and the inpainted region with CLIP-I or DINOv2 similarity. The paper's own failure figures predict that similarity will collapse for such references, so running this measurement across many out-of-distribution queries would settle whether the reference-guided adaptation reproduces the reference or merely re-renders a training-domain prior; a complementary check is a reader study counting how often microcalcification clusters survive AnydoorMed insertion.","tokens_in":43656,"feed_emoji":"🚗","tokens_out":11542,"duration_ms":123172,"temperature":0.7,"pith_summary":"This dissertation tries to show that diffusion inpainting models built for natural images can be re-purposed, with the base weights kept frozen and only small trainable adapters added, to generate realistic synthetic data in very different modalities: joint camera and lidar driving scenes, and mammography scans. It presents two systems built on that recipe. MObI inserts an object from a single RGB reference image into a driving scene at a position and orientation set by a 3D bounding box, producing the camera patch and the lidar range-view patch in a single denoising pass. AnydoorMed transplants an anomaly such as a mass, calcification, or architectural distortion from one mammogram into another, guided by the reference patch and a high-frequency detail map. If the claim is right, safety-critical perception systems could be stress-tested with controllable counterfactual scenes without collecting rare real-world data.","feed_headline":"One reference image drives camera-lidar and mammogram edits","feed_subtitle":"Two diffusion-based methods add controllable objects to driving scenes and anomalies to scans for safer AI stress-testing.","key_machinery":"The load-bearing mechanism is the zero-initialised gated cross-attention adapter: new conditioning tokens attend to the frozen diffusion model's features, and the attention output is multiplied by a gate that starts at zero, so fine-tuning begins from the exact pretrained behaviour and gradually steers it toward the new task without destroying the original prior. Around this sit the modality-specific pieces. MObI projects a 3D bounding box into each sensor view, encodes it through Fourier embeddings and an MLP into a shared conditioning token, and adapts the Stable Diffusion VAE to lidar range views through average-pooled downsampling, exponential-CDF intensity normalisation, object-aware depth normalisation, and fine-tuned residual-block input and output layers; camera and lidar features then attend to each other through cross-modal attention of the form $\\text{Attn} = \\mathrm{softmax}(QK^\\top/\\sqrt{d_{\\text{head}}})V$ with zero-initialised gating. AnydoorMed encodes the reference anomaly with DINOv2 and collages a Sobel-derived high-frequency map into the context, letting the denoising process preserve fine detail such as microcalcifications while the gated cross-attention binds the reference tokens to the target location.","core_discovery":"The central claim, stated on the paper's own terms, is that pretrained reference-guided inpainting foundations—Paint-by-Example for MObI and AnyDoor for AnydoorMed—can be adapted to new perceptual modalities with a shared, data-efficient recipe: keep the base diffusion model and reference encoders frozen, interleave zero-initialised gated cross-attention adapters that inject the new conditioning, and fine-tune only those adapters together with a lightly modified VAE that maps the new modality into the pretrained latent space. MObI is presented as the first framework for multimodal object inpainting that generates camera and lidar jointly from a single reference image, with a Fourier-embedded 3D bounding box token supplying spatial control and cross-modal attention keeping the two sensors coherent. AnydoorMed applies the same recipe to mammography, synthesising anomalies from a DINOv2-encoded reference patch with fine structures such as microcalcifications preserved, and the paper reports it outperforms its baselines on the realism metrics used. The reported reinsertion experiments show that an off-the-shelf multimodal detector still finds objects inserted by MObI with only a small drop in scene-level accuracy, which the paper takes as evidence that the synthetic insertions are usable for downstream perception testing.","pith_inferences":["The ablations suggest the binding constraint for a new modality is the VAE that lifts its data into the latent space, not the diffusion prior; a testable corollary is that improving the per-modality autoencoder would raise realism more than scaling the diffusion model.","The open-world failure mode (a horse becoming a brown car) points to the frozen reference encoder plus the strong class prior in the pretrained weights; unfreezing the reference encoder or adding an explicit reconstruction loss on the inpainted region are natural fixes the dissertation leaves untried.","The evaluation measures detector performance on generated data but not whether training on synthetic-augmented data improves downstream accuracy; measuring that training benefit for rare classes is the next logical experiment.","For the medical half, the dissertation's own ethics statement cautions against clinical use without rigorous validation, so the near-term practical value of AnydoorMed is in research stress-testing of diagnostic models rather than in diagnosis itself."],"forward_implications":["An off-the-shelf BEVFusion detector keeps detecting objects reinserted by MObI, with scene-level mAP nearly unchanged (0.89 to 0.88 for cars, 0.87 to 0.86 for pedestrians), so geometrically conditioned insertions are usable for downstream perception evaluation.","Conditioning on a 3D bounding box rather than an edit mask lets the same object be inserted at different positions, orientations, and scales in the same scene, enabling systematic novel-view and counterfactual testing.","The recipe transfers across two very different modalities, which the paper takes as evidence that other perceptual inputs could be added the same way, provided a suitable VAE adaptation exists.","A blank reference performs object deletion, and AnydoorMed can insert anomalies into healthy scans, so the same framework can both add and remove content for data augmentation in class-imbalanced settings."],"supporting_citations":[{"why":"Paint-by-Example: the reference-guided inpainting base that MObI extends, supplying the pretrained diffusion weights, CLIP reference encoding, and classifier-free guidance recipe.","marker":"[13]"},{"why":"AnyDoor: the reference-guided inpainting base that AnydoorMed extends, supplying the zoom-in cropping, DINOv2 reference encoding, high-frequency map, and detail encoder.","marker":"[16]"},{"why":"Latent diffusion / Stable Diffusion: provides the latent space and the pretrained VAE that both systems adapt to their new modalities.","marker":"[9]"},{"why":"The variational autoencoder formalism and the specific pretrained image VAE whose input and output layers both systems fine-tune.","marker":"[4]"},{"why":"ControlNet: provides the zero-initialised gating mechanism reused by the cross-attention adapters in both systems.","marker":"[12]"},{"why":"Flamingo: the interleaved gated cross-attention adaptation strategy that both systems use to inject new conditioning into a frozen base model.","marker":"[68]"},{"why":"nuScenes: the multimodal driving dataset on which MObI is trained and evaluated, including its lidar beam geometry used for range-view projection.","marker":"[63]"},{"why":"VinDr-Mammo: the mammography dataset with anomaly bounding-box annotations on which AnydoorMed is trained and evaluated.","marker":"[90]"},{"why":"BEVFusion: the off-the-shelf multimodal detector used to test whether objects reinserted by MObI remain detectable.","marker":"[50]"}],"fun_headline_variants":["Diffusion inpainting hops from driving scenes to mammograms","One reference image edits lidar, camera, and X-rays","Multimodal counterfactuals: from roads to mammograms","Adapting reference-guided inpainting to new sensory domains","Inpaint objects in 3D and anomalies in mammograms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fragile premise is that a single frozen reference encoding (CLIP for MObI, DINOv2 for AnydoorMed) plus a bounding box carries enough information about identity, orientation, scale, and fine structure for the model to reproduce the object at the target location; the paper's own figures showing a horse rendered as a brown car and microcalcifications lost in the output indicate this premise holds only when the reference sits inside the training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion inpainting hops from driving scenes to mammograms","One reference image edits lidar, camera, and X-rays","Multimodal counterfactuals: from roads to mammograms","Adapting reference-guided inpainting to new sensory domains","Inpaint objects in 3D and anomalies in mammograms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001372,"raw_usage":{"total_tokens":5629,"prompt_tokens":1083,"completion_tokens":4546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":699,"completion_tokens_details":{"reasoning_tokens":4460}},"tokens_in":699,"tokens_out":4546,"duration_ms":39377,"temperature":1.0,"reasoning_tokens":4460,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:04:54.871498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take references from well outside the training distribution—an animal or vehicle class never seen in nuScenes, or a synthetic anomaly with a distinctive microstructure—insert them at valid locations, and measure identity preservation between the reference and the inpainted region with CLIP-I or DINOv2 similarity. The paper's own failure figures predict that similarity will collapse for such references, so running this measurement across many out-of-distribution queries would settle whether the reference-guided adaptation reproduces the reference or merely re-renders a training-domain prior; a complementary check is a reader study counting how often microcalcification clusters survive AnydoorMed insertion.","supporting_citations":[{"cited_title":"Flamingo: A visual language model for few-shot learn- ing,","cited_arxiv_id":null,"evidence_quote":"Flamingo: the interleaved gated cross-attention adaptation strategy that both systems use to inject new conditioning into a frozen base model."},{"cited_title":"Nuscenes: A multimodal dataset for autonomous driv- ing,","cited_arxiv_id":null,"evidence_quote":"nuScenes: the multimodal driving dataset on which MObI is trained and evaluated, including its lidar beam geometry used for range-view projection."},{"cited_title":"Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography,","cited_arxiv_id":null,"evidence_quote":"VinDr-Mammo: the mammography dataset with anomaly bounding-box annotations on which AnydoorMed is trained and evaluated."},{"cited_title":"Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,","cited_arxiv_id":null,"evidence_quote":"BEVFusion: the off-the-shelf multimodal detector used to test whether objects reinserted by MObI remain detectable."}],"review_version":1}