Pith. sign in

REVIEW 4 major objections 5 minor 96 references

Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This dissertation claims that reference-guided diffusion inpainting, built for natural images, can be adapted with lightweight fine-tuning to generate realistic and controllable counterfactual data in camera-lidar driving scenes and in…

desk verdict A candid BSc dissertation that republishes MObI and adds AnydoorMed, a promising but unvalidated AnyDoor-to-mammography adaptation; honest limitations, overclaimed abstract. read the letter →

arxiv 2507.23058 v1 pith:NOEUJI4J submitted 2025-07-30 cs.CV cs.AI

Alexandru Buburuzan This is my paper
classification cs.CVcs.AI
keywords reference-guidedinpaintinglatentdiffusionmodelsmultimodalcounterfactualgenerationcamera-lidarobjectinsertion3Dboundingboxconditioningmammographyanomalysynthesiszero-initialisedgatedcross-attention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This dissertation tries to show that diffusion inpainting models built for natural images can be re-purposed, with the base weights kept frozen and only small trainable adapters added, to generate realistic synthetic data in very different modalities: joint camera and lidar driving scenes, and mammography scans. It presents two systems built on that recipe. MObI inserts an object from a single RGB reference image into a driving scene at a position and orientation set by a 3D bounding box, producing the camera patch and the lidar range-view patch in a single denoising pass. AnydoorMed transplants an anomaly such as a mass, calcification, or architectural distortion from one mammogram into another, guided by the reference patch and a high-frequency detail map. If the claim is right, safety-critical perception systems could be stress-tested with controllable counterfactual scenes without collecting rare real-world data.

What carries the argument

The load-bearing mechanism is the zero-initialised gated cross-attention adapter: new conditioning tokens attend to the frozen diffusion model's features, and the attention output is multiplied by a gate that starts at zero, so fine-tuning begins from the exact pretrained behaviour and gradually steers it toward the new task without destroying the original prior. Around this sit the modality-specific pieces. MObI projects a 3D bounding box into each sensor view, encodes it through Fourier embeddings and an MLP into a shared conditioning token, and adapts the Stable Diffusion VAE to lidar range views through average-pooled downsampling, exponential-CDF intensity normalisation, object-aware depth normalisation, and fine-tuned residual-block input and output layers; camera and lidar features then attend to each other through cross-modal attention of the form $\text{Attn} = \mathrm{softmax}(QK^\top/\sqrt{d_{\text{head}}})V$ with zero-initialised gating. AnydoorMed encodes the reference anomaly with DINOv2 and collages a Sobel-derived high-frequency map into the context, letting the denoising process preserve fine detail such as microcalcifications while the gated cross-attention binds the reference tokens to the target location.

What would settle it

A direct test is to take references from well outside the training distribution—an animal or vehicle class never seen in nuScenes, or a synthetic anomaly with a distinctive microstructure—insert them at valid locations, and measure identity preservation between the reference and the inpainted region with CLIP-I or DINOv2 similarity. The paper's own failure figures predict that similarity will collapse for such references, so running this measurement across many out-of-distribution queries would settle whether the reference-guided adaptation reproduces the reference or merely re-renders a training-domain prior; a complementary check is a reader study counting how often microcalcification clusters survive AnydoorMed insertion.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that pretrained reference-guided inpainting foundations—Paint-by-Example for MObI and AnyDoor for AnydoorMed—can be adapted to new perceptual modalities with a shared, data-efficient recipe: keep the base diffusion model and reference encoders frozen, interleave zero-initialised gated cross-attention adapters that inject the new conditioning, and fine-tune only those adapters together with a lightly modified VAE that maps the new modality into the pretrained latent space. MObI is presented as the first framework for multimodal object inpainting that generates camera and lidar jointly from a single reference image, with a Fourier-embedded 3D bounding box token supplying spatial control and cross-modal attention keeping the two sensors coherent. AnydoorMed applies the same recipe to mammography, synthesising anomalies from a DINOv2-encoded reference patch with fine structures such as microcalcifications preserved, and the paper reports it outperforms its baselines on the realism metrics used. The reported reinsertion experiments show that an off-the-shelf multimodal detector still finds objects inserted by MObI with only a small drop in scene-level accuracy, which the paper takes as evidence that the synthetic insertions are usable for downstream perception testing.

Load-bearing premise

The fragile premise is that a single frozen reference encoding (CLIP for MObI, DINOv2 for AnydoorMed) plus a bounding box carries enough information about identity, orientation, scale, and fine structure for the model to reproduce the object at the target location; the paper's own figures showing a horse rendered as a brown car and microcalcifications lost in the output indicate this premise holds only when the reference sits inside the training distribution.

Editorial extensions

If this is right

  • An off-the-shelf BEVFusion detector keeps detecting objects reinserted by MObI, with scene-level mAP nearly unchanged (0.89 to 0.88 for cars, 0.87 to 0.86 for pedestrians), so geometrically conditioned insertions are usable for downstream perception evaluation.
  • Conditioning on a 3D bounding box rather than an edit mask lets the same object be inserted at different positions, orientations, and scales in the same scene, enabling systematic novel-view and counterfactual testing.
  • The recipe transfers across two very different modalities, which the paper takes as evidence that other perceptual inputs could be added the same way, provided a suitable VAE adaptation exists.
  • A blank reference performs object deletion, and AnydoorMed can insert anomalies into healthy scans, so the same framework can both add and remove content for data augmentation in class-imbalanced settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ablations suggest the binding constraint for a new modality is the VAE that lifts its data into the latent space, not the diffusion prior; a testable corollary is that improving the per-modality autoencoder would raise realism more than scaling the diffusion model.
  • The open-world failure mode (a horse becoming a brown car) points to the frozen reference encoder plus the strong class prior in the pretrained weights; unfreezing the reference encoder or adding an explicit reconstruction loss on the inpainted region are natural fixes the dissertation leaves untried.
  • The evaluation measures detector performance on generated data but not whether training on synthetic-augmented data improves downstream accuracy; measuring that training benefit for rare classes is the next logical experiment.
  • For the medical half, the dissertation's own ethics statement cautions against clinical use without rigorous validation, so the near-term practical value of AnydoorMed is in research stress-testing of diagnostic models rather than in diagnosis itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes two reference-guided diffusion inpainting methods. MObI extends Paint-by-Example with 3D-bounding-box conditioning and cross-modal camera-lidar attention for object insertion in autonomous driving scenes. AnydoorMed adapts AnyDoor to mammography for anomaly insertion using a DINOv2 reference token and a destination-derived high-frequency detail map. Both methods are evaluated with FID, LPIPS, CLIP-I (and DINOv2 similarity for AnydoorMed), plus a BEVFusion detection check for MObI. The paper reports improvements over Paint-by-Example and AnyDoor and claims that foundation inpainting models can be readily adapted to camera-lidar and medical modalities.

Significance. If the central claim is correct, the contribution is a practical, parameter-efficient adaptation recipe for reference-guided inpainting in non-natural-image modalities, with direct applications to synthetic data generation for perception testing. The work has concrete strengths: MObI is already peer-reviewed at a CVPR workshop, the authors include a downstream detection evaluation, ablate their main design choices, and commit to releasing code and weights; AnydoorMed reports multiple seeds. However, the significance as presented is limited by weak baselines, lack of direct reference-fidelity measurement, and the use of test-set model selection; the claimed 'first-of-its-kind' and 'impressive detail preservation' go beyond what the evidence supports.

major comments (4)
  1. [§3.3.3–3.3.4, Fig. 4.2] The abstract claims 'impressive detail preservation' and 'structural integrity' for AnydoorMed, but no evaluation in the paper directly measures fidelity of the generated anomaly to the reference. In §3.3.3 the reference enters only as a frozen DINOv2 token, and in §3.3.4 the high-frequency map is computed from the destination mammogram I (Ihf = (I⊗Kh + I⊗Kv)⊙I⊙Merode), so the network has no dedicated channel carrying reference-specific fine structure such as microcalcifications; the diffusion objective in §3.3 is the standard noise-prediction loss and contains no identity or reconstruction term. The metrics in Table 3.1 (FID, LPIPS, CLIP-I, DINOv2 similarity) are global or coarse and, as the paper's own Fig. 4.2 shows, a failure case where 'the inserted anomaly does not closely replicate the microcalcifications from the reference image' is not penalized by these numbers. To support the central claim, the authors should add a direct reference-fidelity metric (e.g., local structural similarity or a calcification-detection score inside the edit box) and compare reinsertion output against the original anomaly at the pixel or feature level; otherwise the 'detail preservation' claim should be removed from the abstract.
  2. [§3.4.1, Table 3.1] For the Insertion task in §3.4.1, the anomaly is inserted into a healthy scan and there is no ground-truth edited image, yet Table 3.1 reports LPIPS values (0.08) for this setting. The paper does not state what the LPIPS comparison is computed against; if it is compared to the unedited healthy scan, the score rewards outputs that alter the image as little as possible, which would not measure successful insertion. FID on the 426-sample validation set also has high variance, and no details are given on how many patches or crops are used. The evaluation protocol for insertion must be specified unambiguously, or these numbers should be reported only as a calibration/internal reference.
  3. [§2.3.6, §3.3.7] Both methods select the final model on the evaluation set. For MObI, §2.3.6 states the final model is selected 'based on the best FID achieved on a test set of 200 pre-selected images'; for AnydoorMed, §3.3.7 states selection is 'based on the best FID achieved on the test set with 426 samples'. This is a form of test-set model selection that biases the reported metrics upward and invalidates them as unbiased estimates of generation quality. Additionally, all MObI numbers are single-run with no error bars, so it is unclear whether the reported improvements over PbE are within run-to-run variability. The authors should move model selection to a validation split, report the test-set results of the chosen model only, and provide multiple seeds or bootstrapped confidence intervals for at least the main FID/LPIPS comparisons.
  4. [§2.4.2, §3.4.1] The comparison set is too narrow to support the generalization claims made in the abstract and §4.1. In §2.4.2 MObI is compared only to Paint-by-Example and copy-paste; GenMM, which is discussed in §2.2 as the closest multimodal object-inpainting method, is never evaluated quantitatively. Similarly, §3.4.1 compares AnydoorMed only to AnyDoor and copy-paste, and the related-work discussion in §3.2 lists recent medical counterfactual/inpainting methods (e.g., MedEdit [86], RadEdit [88], PRISM [78]) without any comparison. Without at least one stronger or domain-specific baseline, the claim of 'state-of-the-art results according to realism metrics' is an overstatement; the comparisons should be extended or the claim should be explicitly restricted to the tested baselines.
minor comments (5)
  1. [Abstract, §2.2] The phrase 'first-of-its-kind framework for Multimodal Object Inpainting' is inaccurate because GenMM [48] is described in §2.2 as a prior multimodal object-inpainting method; the novelty claim should be narrowed to 'first end-to-end joint camera-lidar method'.
  2. [§3.4.2 vs Table 3.1] The text reports Insertion FID 4.89 and LPIPS 0.08 while the table lists 4.78 ± 0.14 and 0.08 ± 0.01; these should be harmonized.
  3. [§3.3.7] The sentence 'adapting the newly added input and output adapters of the range autoencoder' appears to be a copy-paste error from the MObI chapter; AnydoorMed adapts a medical autoencoder, not a range autoencoder.
  4. [Fig. 2.10 caption] The caption contains 'Left:' twice; the second occurrence should read 'Right:' for the score-distribution plot.
  5. [General] The thesis front matter (Declaration of originality, Copyright statement, Appendices B/C on ethics and planning) is not appropriate for a journal submission and should be removed or adapted to the journal's format.

Circularity Check

2 steps flagged · score 6.0 of 10

FID is partially circular because the final models are selected on the test-set FID that is later reported as the headline realism result; the core diffusion-training derivations and most external metrics remain independent.

  1. fitted input called prediction [Section 2.3.6 (Trainings details) and Table 2.2]
    "The final model is selected based on the best Fréchet Inception Distance (FID) [71] achieved on a test set of 200 pre-selected images, where objects are reinserted into the scenes using the previously-described filters."

    The MObI checkpoint is chosen by minimizing FID on a 200-image reinsertion test set, and then the same reinsertion protocol supplies the FID numbers reported in Table 2.2 as evidence of realism. The reported FID is therefore the model-selection criterion itself, not an independent held-out prediction, and the comparison against PbE and copy&paste is biased because those baselines were not selected by FID on that set. The circularity is limited to the FID claim: LPIPS, CLIP-I, D-LPIPS, I-LPIPS, and the BEVFusion detector evaluation were not used for selection.

  2. fitted input called prediction [Section 3.3.7 (Training details) and Table 3.1]
    "The final model is selected based on the best Fréchet Inception Distance (FID) [71] achieved on a test set comprising 426 samples from the validation set, where anomalies are reinserted into the scan."

    AnydoorMed's checkpoint is selected by minimizing FID on 426 reinsertion samples, and Table 3.1 reports FID for exactly that reinsertion task (1.83) plus replacement and insertion FIDs from the same selected model. The reinsertion FID is the selection objective renamed as an evaluation result, so it is not an independent measure of realism. The replacement and insertion FIDs are not the same task but still come from a model chosen by the reported reinsertion FID, so the headline 'outperforms AnyDoor and copy&paste' claim rests in part on a self-selected metric. LPIPS, CLIP-I, and DINOv2 scores were not selection criteria and provide partly independent evidence.

full rationale

The central derivations are not circular: both methods train a latent diffusion model with a noise-prediction objective conditioned on masked context, reference tokens, and bounding-box or detail encodings, and the evaluations use held-out scenes and external detectors or perceptual metrics. The self-citation to the author's own published MObI paper [14] is not load-bearing; it only identifies the prior publication from which the chapter is adapted. The main circularity is statistical rather than algebraic: FID is used as the model-selection criterion on the evaluation set and then reported as the primary realism metric, which makes the FID-based superiority claims partially circular by construction. Because other metrics and downstream evaluations are independent, the score is 6 rather than higher.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest on fine-tuning existing foundation models; the main costs are a few hand-chosen constants, frozen pretrained components, and the assumption that reference encodings carry the identity information needed for transfer. No new physical or mathematical entities are introduced.

free parameters (6)
  • Intensity normalisation exponent lambda = 4
    Used in the exponential CDF map i' = 2 e^(-lambda i/255) - 1 for lidar intensity; chosen experimentally in Section 2.3.2.
  • Depth normalisation scaling alpha = 0.75
    Controls the interval allocated to object depth values in the range-view normalisation (Figure 2.3, Section 2.3.2); set by hand and not ablated.
  • Reference sampling Beta distribution parameters = Beta(4, 1)
    Bias towards temporally distant reference patches during MObI training; chosen ad hoc in Section 2.3.6 and Figure 2.4.
  • Classifier-free guidance scale = 5
    Used for reference and bounding-box conditioning, inherited from Paint-by-Example in Section 2.3.4.
  • Empty-box augmentation fraction = 30%
    Share of training samples drawn from the empty-box database in Section 2.3.6.
  • Object selection thresholds = >=64 lidar points, >=100x100 px, IoU<=50%, visibility>=70%
    Filters that define the training and evaluation subset for MObI reinsertion and replacement in Sections 2.3.6 and 2.4.1.
assumptions (4)
  • domain assumption A frozen CLIP or DINOv2 reference encoder captures enough semantic identity of the object or anomaly to transfer it to a new scene.
    MObI Section 2.3.3 freezes CLIP tokens from Paint-by-Example; AnydoorMed Section 3.3.3 freezes DINOv2. If the reference encoding loses fine detail, the insertion will not preserve identity; open-world and microcalcification failures in Figures 4.1 and 4.2 support this risk.
  • domain assumption The projected 3D bounding box in camera and range-view coordinates is a sufficient spatial alignment signal between the two modalities.
    MObI's cross-modal attention in Section 2.3.4 is trained without explicit depth or geometric consistency losses; it relies on the boxes to create correspondence between camera and lidar features.
  • domain assumption The pretrained Stable Diffusion VAE latent space remains a valid representation after channel replacement and adapter fine-tuning.
    Both methods adapt the image VAE to lidar or mammography in Sections 2.3.2 and 3.3.2 but keep the frozen middle layers and the diffusion prior; this is plausible but is only verified by reconstruction metrics, not by an independent theory.
  • standard math The standard DDPM, DDIM, and latent diffusion formulations used in the background chapter are accepted as correct.
    The theory chapter is a recap of textbook material and does not introduce new mathematical claims, so it is treated as a standard background assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation." pith.science (2026). https://pith.science/paper/NOEUJI4J

@misc{pith2026250723058,
  author       = {Pith},
  title        = {Pith review of: Reference-Guided Diffusion Inpainting For Multimodal Counterfactual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOEUJI4J}},
  note         = {Machine review of arXiv:2507.23058}
}
read the original abstract

Safety-critical applications, such as autonomous driving and medical image analysis, require extensive multimodal data for rigorous testing. Synthetic data methods are gaining prominence due to the cost and complexity of gathering real-world data, but they demand a high degree of realism and controllability to be useful. This work introduces two novel methods for synthetic data generation in autonomous driving and medical image analysis, namely MObI and AnydoorMed, respectively. MObI is a first-of-its-kind framework for Multimodal Object Inpainting that leverages a diffusion model to produce realistic and controllable object inpaintings across perceptual modalities, demonstrated simultaneously for camera and lidar. Given a single reference RGB image, MObI enables seamless object insertion into existing multimodal scenes at a specified 3D location, guided by a bounding box, while maintaining semantic consistency and multimodal coherence. Unlike traditional inpainting methods that rely solely on edit masks, this approach uses 3D bounding box conditioning to ensure accurate spatial positioning and realistic scaling. AnydoorMed extends this paradigm to the medical imaging domain, focusing on reference-guided inpainting for mammography scans. It leverages a diffusion-based model to inpaint anomalies with impressive detail preservation, maintaining the reference anomaly's structural integrity while semantically blending it with the surrounding tissue. Together, these methods demonstrate that foundation models for reference-guided inpainting in natural images can be readily adapted to diverse perceptual modalities, paving the way for the next generation of systems capable of constructing highly realistic, controllable and multimodal counterfactual scenarios.

Figures

Figures reproduced from arXiv: 2507.23058 by the authors.

Figure 1.1
Figure 1.1. MObI uniquely enables realistic, 3D-conditioned object insertion across camera and lidar modal [PITH_FULL_IMAGE:figures/full_fig_p013_1_1.png] view at source ↗
Figure 1.1
Figure 1.1. (a) MObI enables the generation of multiple novel views from a single reference image while maintaining semantic consistency and multimodal coherence across camera and lidar modalities. The inserted object respects the geometric constraints imposed by an oriented 3D bounding box, with inpainting performed in a modality-agnostic latent space. (b) AnydoorMed inpaints an anomaly at a specific location within mammograph… view at source ↗
Figure 1.2
Figure 1.2. Directed graphical model of a VAE [4] comprising the observable discrete random variable x and the latent continuous random variable z. Solid lines represent the generative process pθ(z)pθ(x | z), while dashed lines represent the variational approximation qϕ(z | x) of the intractable true posterior pθ(z | x). Here, θ and ϕ denote function parameters. DKL(q ∥ p) denotes the Kullback–Leibler (KL) divergence, which qua… view at source ↗
Figures from the paper (25 more)
Figure 1.3
Figure 1.3. Figure 1.3: Directed graphical model of DDPM [6]. Dashed lines denote the forward diffusion process q(xt | xt−1). Solid lines denote the learnt denoising process pθ(xt−1 | xt). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_1_3.png]
Figure 1.4
Figure 1.4. Figure 1.4: Intuition for why the true denoising process q(xt−1 | xt) is approximately Gaussian, since the product of q(xt | xt−1) and q(xt−1 | xt) will be bell-shaped. This figure and the idea behind the explanation were adapted from [7]. The true posterior can be approximated …
Figure 1.5
Figure 1.5. Figure 1.5: Comparison between the denoising trajectories of DDPM and DDIM. DDPM follows a noisy path with small incremental steps, eventually reaching x0. In contrast, DDIM takes larger strides, which can lead to divergence from the true data distribution. This figure is only m…
Figure 1.6
Figure 1.6. Figure 1.6: Architecture and training pipeline of Paint-by-Example [13]. Inpainting Inpainting is the task of reconstructing missing or occluded regions of an image in a seman￾tically coherent and visually plausible manner. Within the diffusion framework, this is achieved by pre…
Figure 2.1
Figure 2.1. Figure 2.1: The proposed method can inpaint objects with a high degree of realism and controllability. Left: object inpainting methods based on edit masks alone such as Paint-by-Example [13] (PbE) achieve high realism but can lead to surprising results because there are often mu…
Figure 2.2
Figure 2.2. Figure 2.2: MObI architecture and training procedure. 2.3 Method This work extends Paint-by-Example [13] (PbE), a reference-based image inpainting method, to include bounding box conditioning and to jointly generate camera and lidar perception inputs. A diffusion model [6], [9],…
Figure 2.3
Figure 2.3. Figure 2.3: Normalisation strategy of the lidar depth, which influences the interval size allocated to the depth values of the object bounding box. within [−1, 1], see [PITH_FULL_IMAGE:figures/full_fig_p039_2_3.png]
Figure 2.4
Figure 2.4. Figure 2.4: The probability density function of the Beta distribution with parameters α = 4 and β = 1, used to sample reference patches of an object based on the normalised timestamp difference ∆t between tracked instances. Patches from further time points are sampled with highe…
Figure 2.6
Figure 2.6. Figure 2.6: Examples of object inpainting using MObI in the following settings: replacement (rows 1–4), insertion (row 5), and deletion (row 6, using a black reference). The proposed method can inpaint objects corresponding to a 3D bounding box with a high degree of realism whil…
Figure 2.7
Figure 2.7. Figure 2.7: Examples showcasing the controllability of the proposed method. From left to right: reference image xref extracted from a seperate source scene, original destination scene (original RGB image x (C), lidar range depth x (R) 0 and intensity x (R) 1 ), and edited scenes…
Figure 2.8
Figure 2.8. Figure 2.8: Spatial compositing of camera-lidar object inpainting in a scene with complex lighting. Note that some background points are not overridden due to lidar reflections on the hood of the inserted car (bottom). For FID and LPIPS, the evaluation is carried out on extended…
Figure 2.9
Figure 2.9. Figure 2.9: Additional examples showcasing the controllability of the proposed method. From left to right: reference image xref extracted from a seperate source scene, original destination scene (original RGB image x (C), Lidar range depth x (R) 0 and intensity x (R) 1 ), and ed…
Figure 2.10
Figure 2.10. Figure 2.10: Detection performance of an off-the-shelf BEVFusion [50] object detector on objects reinserted using the proposed method. Left: mAP is computed at the scene-level, and TP errors (translation, scale, and orientation) are computed on the reinserted objects only. Left:…
Figure 2.11
Figure 2.11. Figure 2.11: Comparison of detection results between the original scene and the same scene with the object shown in red replaced. BEVFusion [50] achieves good detection performance on the object reinserted using the proposed method, while leaving the boxes of the other objects u…
Figure 3.1
Figure 3.1. Figure 3.1: AnydoorMed architecture and training pipeline. The anomaly’s High Frequency map (HF map) was coloured purple for visualisation purposes. Medical image generation and inpainting Recent advances in medical image generation have seen diffusion models employed to synthes…
Figure 3.2
Figure 3.2. Figure 3.2: Samples from VinDR-Mammo dataset [90] with bounding box annotations. Anomaly processing In addition to image processing, anomalies associated with each scan were ex￾tracted from the corresponding annotations. The anomaly classes considered in this study include: Ar￾c…
Figure 3.3
Figure 3.3. Figure 3.3: Distribution of BI-RADS malignancy scores, showcasing class imbalance. 0 200 400 600 800 1000 Count Architectural Distortion Asymmetry Focal Asymmetry Global Asymmetry Mass Nipple Retraction Skin Retraction Skin Thickening Suspicious Calcification Suspicious Lymph No…
Figure 3.4
Figure 3.4. Figure 3.4: Distribution of anomalies based on their class, showcasing class imbalance. images are then normalised to a range of [−1, 1]. A mask is generated to in-fill the bounding box, and for the context image, the anomaly is erased by applying the mask with zero values. 3.3.…
Figure 3.5
Figure 3.5. Figure 3.5: Anomaly insertion results. AnydoorMed inserts the reference anomaly (second column), guided by the context and high-frequency map context (first column), into the healthy mammography scan (fourth column), producing the composited result (third column). The inpainted …
Figure 3.6
Figure 3.6. Figure 3.6: Anomaly reinsertion results. AnydoorMed reinserts the anomaly (second column), guided by the context and high-frequency map context (first column), into the mammography scan (fourth column), producing the composited result (third column). This is done by removing the…
Figure 3
Figure 3. Figure 3: ) [PITH_FULL_IMAGE:figures/full_fig_p067_3.png]
Figure 3.7
Figure 3.7. Figure 3.7: Anomaly replacement results. AnydoorMed replaces the anomaly from the original scan (fourth column), with the reference anomaly (second column), guided by the context and high-frequency map context (first column), producing the composited result (third column). This …
Figure 4.1
Figure 4.1. Figure 4.1: Object replacement results using hard references (different weather conditions or time of day, occlusions, etc.). MObI can successfully insert these hard references in the target bounding box. However, the quality in these examples is unsatisfactory. From top to bott…
Figure 4.2
Figure 4.2. Figure 4.2: Anomaly insertion results. AnydoorMed inserts the reference anomalies (second column), guided by the context and high-frequency collage (first column), into the healthy mammography scan (fourth column), producing the composited result (third column). However, these e…
Figure 4.3
Figure 4.3. Figure 4.3: Object insertion and replacement with out-of-domain and open-world references for MObI trained only on the pedestrian and car classes of nuScenes. (a) In the first two examples (top left), MObI inserts the correct object successfully but loses fine appearance details…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

96 extracted references · 52 canonical work pages

  1. [86]

    Mededit: Counterfactual diffusion-based image editing on brain mri,

    M. B. Alaya, D. M. Lang, B. Wiestler, J. A. Schnabel, and C. I. Bercea, “Mededit: Counterfactual diffusion-based image editing on brain mri,” in International W orkshop on Simulation and Synthe- sis in Medical Imaging, Springer, 2024, pp. 167–176 (cited on p. 57)

    work page 2024

  2. [88]

    Radedit: Stress-testing biomedical vision mod- els via diffusion image editing,

    F. Pérez-García, S. Bond-T aylor, P. P. Sanchez, et al., “Radedit: Stress-testing biomedical vision mod- els via diffusion image editing,” in European Conference on Computer Vision, Springer, 2024, pp. 358– 376 (cited on p. 57)

    work page 2024

  3. [78]

    Prism: High-resolution & precise counterfactual medical image generation using language-guided stable diffusion,

    A. Kumar, A. Kriz, M. Havaei, and T. Arbel, “Prism: High-resolution & precise counterfactual medical image generation using language-guided stable diffusion,” 2025 (cited on pp. 55, 57)

    work page 2025

  4. [1]

    Cognitive neuroscience of human counterfactual reasoning,

    N. V an Hoeck, P. D. W atson, and A. K. Barbey, “Cognitive neuroscience of human counterfactual reasoning,” Frontiers in human neuroscience, vol. 9, p. 420, 2015 (cited on pp. 12, 13)

    2015

  5. [2]

    Mental models and counterfactual thoughts about what might have been,

    R. M. Byrne, “Mental models and counterfactual thoughts about what might have been,” Trends in cognitive sciences, vol. 6, no. 10, pp. 426–431, 2002 (cited on p. 13)

    2002

  6. [3]

    Useful counterfactuals,

    T. Costello and J. McCarthy, “Useful counterfactuals,” 1999 (cited on p. 13)

    1999

  7. [4]

    Auto-encoding variational bayes,

    D. P. Kingma and M. W elling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013 (cited on pp. 15, 16, 28, 36, 38, 39, 44, 46, 60, 63)

    arXiv 2013

  8. [5]

    V ariational inference: A review for statisticians,

    D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “V ariational inference: A review for statisticians,” Journal of the American statistical Association, vol. 112, no. 518, pp. 859–877, 2017 (cited on p. 16)

    2017

Show all 96 references
  1. [6]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural infor- mation processing systems, vol. 33, pp. 6840–6851, 2020 (cited on pp. 18, 22, 24, 28, 35, 57)

    2020

  2. [7]

    Diffusion models,

    C. M. Bishop and H. Bishop, “Diffusion models,” in Deep Learning: Foundations and Concepts. Cham: Springer International Publishing, 2024, pp. 581–607, isbn: 978-3-031-45468-4. doi: 10. 1007/978-3-031-45468-4_20 . [Online]. Available:https://doi.org/10.1007/978-3- 031-45468-4_...

    2024 doi

  3. [8]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020 (cited on pp. 27, 42, 63). 77

    2010 arXiv

  4. [9]

    High-resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695 (cited on pp. 28, 35, 36, 38, 39, 44, 46, ...

    2022

  5. [10]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, S...

    2015

  6. [11]

    Attention is all you need,

    A. V aswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” Advances in neural informa- tion processing systems, vol. 30, 2017 (cited on p. 28)

    2017

  7. [12]

    Adding conditional control to text-to-image diffusion mod- els,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion mod- els,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3836– 3847 (cited on pp. 29, 34, 40, 62)

    2023

  8. [13]

    Paint by example: Exemplar-based image editing with diffusion models,

    B. Y ang, S. Gu, B. Zhang, et al., “Paint by example: Exemplar-based image editing with diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 381–18 391 (cited on pp. 29, 30, 32, 33, 35, 36, 39–41, 43, 44, 50, 51, 56)

    2023

  9. [14]

    Mobi: Multimodal object inpainting using diffusion models,

    A. Buburuzan, A. Sharma, J. Redford, P. K. Dokania, and R. Mueller, “Mobi: Multimodal object inpainting using diffusion models,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 1974–1984 (cited on p. 31)

    2025

  10. [15]

    Challenges in autonomous vehicle testing and validation,

    P. Koopman and M. W agner, “Challenges in autonomous vehicle testing and validation,” SAE In- ternational Journal of Transportation Safety, vol. 4, no. 1, pp. 15–24, 2016 (cited on p. 32)

    2016

  11. [16]

    Anydoor: Zero-shot object-level image customization,

    X. Chen, L. Huang, Y. Liu, Y. Shen, D. Zhao, and H. Zhao, “Anydoor: Zero-shot object-level image customization,” arXiv preprint arXiv:2307.09481, 2023 (cited on pp. 32, 36, 40, 55–57, 59, 61–63, 65)

    2023 arXiv

  12. [17]

    N. Ruiz, Y. Li, N. W adhwa, et al., Magic insert: Style-aware drag-and-drop, 2024. arXiv: 2407 . 02489 [cs.CV]. [Online]. Available:https://arxiv.org/abs/2407.02489 (cited on pp. 32, 55, 56)

    2024 arXiv

  13. [18]

    Kulal, T

    S. Kulal, T. Brooks, A. Aiken, et al., Putting people in their place: Affordance-aware human insertion into scenes, 2023. arXiv: 2304.14406 [cs.CV]. [Online]. Available:https://arxiv.org/abs/ 2304.14406 (cited on pp. 32, 56)

    2023 arXiv

  14. [19]

    Cadsim: Robust and scalable in-the-wild 3d reconstruc- tion for controllable sensor simulation,

    J. W ang, S. Manivasagam, Y. Chen, et al., “Cadsim: Robust and scalable in-the-wild 3d reconstruc- tion for controllable sensor simulation,” arXiv preprint arXiv:2311.01447, 2023 (cited on p. 32). 78

    2023 arXiv

  15. [20]

    Just add $100 more: Augmenting nerf-based pseudo-lidar point cloud for resolving class-imbalance problem,

    M. Chang, S. Lee, J. Kim, and N. Kim, “Just add $100 more: Augmenting nerf-based pseudo-lidar point cloud for resolving class-imbalance problem,” arXiv preprint arXiv:2403.11573, 2024 (cited on p. 32)

    2024 arXiv

  16. [21]

    Scene-conditional 3d object stylization and compo- sition,

    J. Zhou, T. Jakab, P. T orr, and C. Rupprecht, “Scene-conditional 3d object stylization and compo- sition,” arXiv preprint arXiv:2312.12419, 2023 (cited on p. 32)

    2023 arXiv

  17. [22]

    Editable scene simulation for autonomous driving via collaborative llm-agents,

    Y. W ei, Z. W ang, Y. Lu,et al., “Editable scene simulation for autonomous driving via collaborative llm-agents,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2024, pp. 15 077–15 087 (cited on p. 32)

    2024

  18. [23]

    Geosim: Realistic video simulation via geometry-aware com- position for self-driving,

    Y. Chen, F. Rong, S. Duggal, et al., “Geosim: Realistic video simulation via geometry-aware com- position for self-driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7230–7240 (cited on p. 32)

    2021

  19. [24]

    Drive-1-to-3: Enriching diffusion priors for novel view synthesis of real vehicles,

    C. Lin, B. Zhuang, S. Sun, Z. Jiang, J. Cai, and M. Chandraker, “Drive-1-to-3: Enriching diffusion priors for novel view synthesis of real vehicles,”arXiv preprint arXiv:2412.14494, 2024 (cited on p. 32)

    2024 arXiv

  20. [25]

    Multitest: Physical-aware object insertion for testing multi-sensor fusion perception systems,

    X. Gao, Z. W ang, Y. Feng, L. Ma, Z. Chen, and B. Xu, “Multitest: Physical-aware object insertion for testing multi-sensor fusion perception systems,” in Proceedings of the IEEE/ACM 46th Inter- national Conference on Software Engineering, ser. ICSE ’24, ACM, Apr. 2024, pp. 1–1...

    2024

  21. [26]

    Lift3d: Synthesize 3d training data by lifting 2d gan to 3d generative radiance field,

    L. Li, Q. Lian, L. W ang, N. Ma, and Y.-C. Chen, “Lift3d: Synthesize 3d training data by lifting 2d gan to 3d generative radiance field,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 332–341 (cited on p. 32)

    2023

  22. [27]

    W ayve, PRISM-1, https://wayve.ai/thinking/prism-1/ , Last accessed: 14.11.2024, 2024 (cited on p. 32)

    2024

  23. [28]

    Neurad: Neural rendering for autonomous driving,

    A. T onderski, C. Lindström, G. Hess, W . Ljungbergh, L. Svensson, and C. Petersson, “Neurad: Neural rendering for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2024, pp. 14 895–14 904 (cited on pp. 32, 33)

    2024

  24. [29]

    Unisim: A neural closed-loop sensor simulator,

    Z. Y ang, Y. Chen, J. W ang, et al., “Unisim: A neural closed-loop sensor simulator,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1389–1399 (cited on p. 32). 79

    2023

  25. [30]

    Diffusion models are geometry critics: Single image 3d editing using pre-trained diffusion priors,

    R. W ang, J. Xiang, J. Y ang, and X. T ong, “Diffusion models are geometry critics: Single image 3d editing using pre-trained diffusion priors,” in European Conference on Computer Vision, Springer, 2025, pp. 441–458 (cited on p. 32)

    2025

  26. [31]

    Neural assets: 3d-aware multi-object scene synthesis with image diffusion models,

    Z. Wu, Y. Rubanova, R. Kabra, et al., “Neural assets: 3d-aware multi-object scene synthesis with image diffusion models,” arXiv preprint arXiv:2406.09292, 2024 (cited on p. 32)

    2024 arXiv

  27. [32]

    Image sculpting: Precise object editing with 3d geometry control,

    J. Y enphraphai, X. Pan, S. Liu, D. Panozzo, and S. Xie, “Image sculpting: Precise object editing with 3d geometry control,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4241–4251 (cited on p. 32)

    2024

  28. [33]

    Diffusion han- dles enabling 3d edits for diffusion models by lifting activations to 3d,

    K. Pandey, P. Guerrero, M. Gadelha, Y. Hold-Geoffroy, K. Singh, and N. J. Mitra, “Diffusion han- dles enabling 3d edits for diffusion models by lifting activations to 3d,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7695–7704...

    2024

  29. [34]

    Object 3dit: Language- guided 3d-aware image editing,

    O. Michel, A. Bhattad, E. V anderBilt, R. Krishna, A. Kembhavi, and T. Gupta, “Object 3dit: Language- guided 3d-aware image editing,” Advances in Neural Information Processing Systems, vol. 36, 2024 (cited on p. 32)

    2024

  30. [35]

    Customnet: Zero-shot object customiza- tion with variable-viewpoints in text-to-image diffusion models,

    Z. Yuan, M. Cao, X. W ang, Z. Qi, C. Yuan, and Y. Shan, “Customnet: Zero-shot object customiza- tion with variable-viewpoints in text-to-image diffusion models,” arXiv preprint arXiv:2310.19784, 2023 (cited on p. 32)

    2023 arXiv

  31. [36]

    Magicdrive: Street view generation with diverse 3d geometry con- trol,

    R. Gao, K. Chen, E. Xie, et al., “Magicdrive: Street view generation with diverse 3d geometry con- trol,” arXiv preprint arXiv:2310.02601, 2023 (cited on pp. 32, 34, 40, 73)

    2023 arXiv

  32. [37]

    Drivingdiffusion: Layout-guided multi-view driving scene video gen- eration with latent diffusion model,

    X. Li, Y. Zhang, and X. Y e, “Drivingdiffusion: Layout-guided multi-view driving scene video gen- eration with latent diffusion model,” arXiv preprint arXiv:2310.07771, 2023 (cited on pp. 32, 34, 73)

    2023 arXiv

  33. [38]

    Panacea: Panoramic and controllable video generation for autonomous driving,

    Y. W en, Y. Zhao, Y. Liu, et al., “Panacea: Panoramic and controllable video generation for autonomous driving,” arXiv preprint arXiv:2311.16813, 2023 (cited on pp. 32, 73)

    2023 arXiv

  34. [39]

    T ext2street: Controllable text-to-image generation for street views,

    J. Su, S. Gu, Y. Duan, X. Chen, and J. Luo, “T ext2street: Controllable text-to-image generation for street views,” arXiv preprint arXiv:2402.04504, 2024 (cited on pp. 32, 34)

    2024 arXiv

  35. [40]

    Huang, Y

    B. Huang, Y. W en, Y. Zhao, et al., Subjectdrive: Scaling generative data in autonomous driving via subject control, 2024. arXiv: 2403.19438 [cs.CV] . [Online]. Available:https://arxiv.org/ abs/2403.19438 (cited on pp. 32, 34)

    2024 arXiv

  36. [41]

    W . Wu, X. Guo, W . T ang,et al., Drivescape: Towards high-resolution controllable multi-view driving video generation, 2024. arXiv: 2409 . 05463 [cs.CV]. [Online]. Available:https : / / arxiv . org/abs/2409.05463 (cited on pp. 32, 34, 73). 80

    2024 arXiv

  37. [42]

    H. Ran, V . Guizilini, and Y. W ang, Towards realistic scene generation with lidar diffusion models,

  38. [43]

    Zyrianov, X

    V . Zyrianov, X. Zhu, and S. W ang, Learning to generate realistic lidar point clouds, 2022. arXiv: 2209.03954 [cs.CV]. [Online]. Available:https://arxiv.org/abs/2209.03954 (cited on pp. 32, 34, 50)

    2022 arXiv

  39. [44]

    Q. Hu, Z. Zhang, and W . Hu, Rangeldm: Fast realistic lidar point cloud generation, 2024. arXiv: 2403.10094 [cs.CV]. [Online]. Available:https://arxiv.org/abs/2403.10094 (cited on pp. 32, 34)

    2024 arXiv

  40. [45]

    Xiong, W .-C

    Y. Xiong, W .-C. Ma, J. W ang, and R. Urtasun, Ultralidar: Learning compact representations for li- dar completion and generation, 2023. arXiv: 2311.01448 [cs.CV] . [Online]. Available:https: //arxiv.org/abs/2311.01448 (cited on pp. 32, 34)

    2023 arXiv

  41. [46]

    H. Bian, L. Kong, H. Xie, L. Pan, Y. Qiao, and Z. Liu, Dynamiccity: Large-scale lidar generation from dynamic scenes, 2024. arXiv: 2410.18084 [cs.CV]. [Online]. Available:https://arxiv. org/abs/2410.18084 (cited on pp. 32, 34)

    2024

  42. [47]

    X-drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios,

    Y. Xie, C. Xu, C. Peng, et al., “X-drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios,” arXiv preprint arXiv:2411.01123, 2024 (cited on pp. 32, 40)

    2024 arXiv

  43. [48]

    Genmm: Geomet- rically and temporally consistent multimodal data generation for video and lidar,

    B. Singh, V . Kulharia, L. Y ang, A. Ravichandran, A. T yagi, and A. Shrivastava, “Genmm: Geomet- rically and temporally consistent multimodal data generation for video and lidar,” arXiv preprint arXiv:2406.10722, 2024 (cited on pp. 32, 34)

    2024 arXiv

  44. [49]

    Liang, H

    T. Liang, H. Xie, K. Yu, et al., Bevfusion: A simple and robust lidar-camera fusion framework, 2022. arXiv: 2205.13790 [cs.CV] . [Online]. Available:https://arxiv.org/abs/2205.13790 (cited on p. 33)

    2022 arXiv

  45. [50]

    Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,

    Z. Liu, H. T ang, A. Amini, et al., “Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation,” in 2023 IEEE international conference on robotics and automation (ICRA), IEEE, 2023, pp. 2774–2781 (cited on pp. 33, 51, 52)

    2023

  46. [51]

    J. Gunn, Z. Lenyk, A. Sharma, et al., Lift-attend-splat: Bird’s-eye-view camera-lidar fusion using transformers, 2024. arXiv: 2312.14919 [cs.CV] . [Online]. Available:https://arxiv.org/ abs/2312.14919 (cited on p. 33)

    2024 arXiv

  47. [52]

    Synthesizing training data for object de- tection in indoor scenes,

    G. Georgakis, A. Mousavian, A. C. Berg, and J. Kosecka, “Synthesizing training data for object de- tection in indoor scenes,” arXiv preprint arXiv:1702.07836, 2017 (cited on pp. 33, 50, 55). 81

    2017 arXiv

  48. [53]

    Cut, paste and learn: Surprisingly easy synthesis for instance detection,

    D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1301– 1310 (cited on pp. 33, 50)

    2017

  49. [54]

    Simple copy-paste is a strong data augmentation method for instance segmentation,

    G. Ghiasi, Y. Cui, A. Srinivas, et al., “Simple copy-paste is a strong data augmentation method for instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2918–2928 (cited on pp. 33, 55)

    2021

  50. [55]

    Pointaugmenting: Cross-modal augmentation for 3d ob- ject detection,

    C. W ang, C. Ma, M. Zhu, and X. Y ang, “Pointaugmenting: Cross-modal augmentation for 3d ob- ject detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2021, pp. 11 794–11 803 (cited on pp. 33, 50, 55)

    2021

  51. [56]

    Second: Sparsely embedded convolutional detection,

    Y. Y an, Y. Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018 (cited on p. 33)

    2018

  52. [57]

    Cutmix: Regularization strategy to train strong classifiers with localizable features,

    S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Y oo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6023–6032 (cited on p. 33)

    2019

  53. [58]

    Exploring data augmentation for multi-modality 3d object detection,

    W . Zhang, Z. W ang, and C. C. Loy, “Exploring data augmentation for multi-modality 3d object detection,” arXiv preprint arXiv:2012.12741, 2020 (cited on pp. 33, 50)

    2012 arXiv

  54. [59]

    Exploring geometric consistency for monocular 3d object detection,

    Q. Lian, B. Y e, R. Xu, W . Y ao, and T. Zhang, “Exploring geometric consistency for monocular 3d object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2022, pp. 1685–1694 (cited on p. 33)

    2022

  55. [60]

    Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout,

    K. Y ang, E. Ma, J. Peng, Q. Guo, D. Lin, and K. Yu, “Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout,”arXiv preprint arXiv:2308.01661, 2023 (cited on p. 34)

    2023 arXiv

  56. [61]

    Synthetic lidar point cloud generation using deep gener- ative models for improved driving scene object recognition,

    Z. Xiang, Z. Huang, and K. Khoshelham, “Synthetic lidar point cloud generation using deep gener- ative models for improved driving scene object recognition,”Image and Vision Computing, vol. 150, p. 105 207, 2024, issn: 0262-8856. doi: https : / / doi . org / 10 . 1016 / j . im...

    2024

  57. [62]

    Deep unsupervised learning us- ing nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. W eiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning us- ing nonequilibrium thermodynamics,” in International conference on machine learning, PMLR, 2015, pp. 2256–2265 (cited on pp. 35, 57). 82

    2015

  58. [63]

    Nuscenes: A multimodal dataset for autonomous driv- ing,

    H. Caesar, V . Bankiti, A. H. Lang, et al., “Nuscenes: A multimodal dataset for autonomous driv- ing,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621–11 631 (cited on pp. 37, 43, 53, 72, 87)

    2020

  59. [64]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778 (cited on pp. 39, 60)

    2016

  60. [65]

    T aming transformers for high-resolution image synthe- sis,

    P. Esser, R. Rombach, and B. Ommer, “T aming transformers for high-resolution image synthe- sis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 873–12 883 (cited on pp. 39, 61)

    2021

  61. [66]

    Learning transferable visual models from natural lan- guage supervision,

    A. Radford, J. W . Kim, C. Hallacy, et al., “Learning transferable visual models from natural lan- guage supervision,” in International conference on machine learning, PMLR, 2021, pp. 8748–8763 (cited on pp. 39, 46, 56, 61)

    2021

  62. [67]

    Dinov2: Learning robust visual features without su- pervision,

    M. Oquab, T. Darcet, T. Moutakanni, et al., “Dinov2: Learning robust visual features without su- pervision,” arXiv preprint arXiv:2304.07193, 2023 (cited on pp. 40, 56, 61)

    2023 arXiv

  63. [68]

    Flamingo: A visual language model for few-shot learn- ing,

    J.-B. Alayrac, J. Donahue, P. Luc, et al., “Flamingo: A visual language model for few-shot learn- ing,” Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022 (cited on pp. 40, 62)

    2022

  64. [69]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022 (cited on p. 41)

    2022 arXiv

  65. [70]

    Placing objects in context via in- painting for out-of-distribution segmentation,

    P. de Jorge, R. V olpi, P. K. Dokania, P. H. T orr, and G. Rogez, “Placing objects in context via in- painting for out-of-distribution segmentation,” arXiv preprint arXiv:2402.16392, 2024 (cited on p. 42)

    2024 arXiv

  66. [71]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”Advances in neural information pro- cessing systems, vol. 30, 2017 (cited on pp. 44, 46, 64)

    2017

  67. [72]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. W ang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595 (cited on pp. 46, 50)

    2018

  68. [73]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,

    N. Ruiz, Y. Li, V . Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2023, pp. 22 500–22 510 (cited...

    2023

  69. [74]

    Going deeper with convolutions,

    C. Szegedy, W . Liu, Y. Jia, et al., “Going deeper with convolutions,” in Proceedings of the IEEE con- ference on computer vision and pattern recognition, 2015, pp. 1–9 (cited on p. 47)

    2015

  70. [75]

    Lidar data synthesis with denoising diffusion probabilistic mod- els,

    K. Nakashima and R. Kurazume, “Lidar data synthesis with denoising diffusion probabilistic mod- els,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 14 724–14 731 (cited on p. 50)

    2024

  71. [76]

    Swin transformer: Hierarchical vision transformer using shifted win- dows,

    Z. Liu, Y. Lin, Y. Cao, et al., “Swin transformer: Hierarchical vision transformer using shifted win- dows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012– 10 022 (cited on p. 52)

    2021

  72. [77]

    Controllable and efficient multi-class pathology nuclei data augmenta- tion using text-conditioned diffusion models,

    H.-J. Oh and W .-K. Jeong, “Controllable and efficient multi-class pathology nuclei data augmenta- tion using text-conditioned diffusion models,” in International Conference on Medical Image Com- puting and Computer-Assisted Intervention, Springer, 2024, pp. 36–46 (cited on pp...

    2024

  73. [79]

    Denoising diffusion models for 3d healthy brain tissue in- painting,

    A. Durrer, J. W olleb, F. Bieder, et al., “Denoising diffusion models for 3d healthy brain tissue in- painting,” in MICCAI W orkshop on Deep Generative Models, Springer, 2024, pp. 87–97 (cited on pp. 55, 57)

    2024

  74. [80]

    St-gan: Spatial transformer generative adversarial networks for image compositing,

    C.-H. Lin, E. Yumer, O. W ang, E. Shechtman, and S. Lucey, “St-gan: Spatial transformer generative adversarial networks for image compositing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9455–9464 (cited on p. 56)

    2018

  75. [81]

    I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, et al., Generative adversarial networks, 2014. arXiv: 1406.2661 [stat.ML] (cited on p. 56)

    2014 arXiv

  76. [82]

    Objectstitch: Object compositing with diffusion model,

    Y. Song, Z. Zhang, Z. Lin, et al., “Objectstitch: Object compositing with diffusion model,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 310– 18 319 (cited on p. 56)

    2023

  77. [83]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023 (cited on p. 56)

    2023 arXiv

  78. [84]

    Winter, M

    D. Winter, M. Cohen, S. Fruchter, Y. Pritch, A. Rav-Acha, and Y. Hoshen, Objectdrop: Bootstrap- ping counterfactuals for photorealistic object removal and insertion, 2024. arXiv: 2403.18818 [cs.CV]. [Online]. Available:https://arxiv.org/abs/2403.18818 (cited on p. 56)

    2024 arXiv

  79. [85]

    Anatomically-controllable medical image generation with segmentation-guided diffusion models,

    N. Konz, Y. Chen, H. Dong, and M. A. Mazurowski, “Anatomically-controllable medical image generation with segmentation-guided diffusion models,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2024, pp. 88–98 (cited on p. 57). 84

    2024

  80. [87]

    Diffusion models with implicit guid- ance for medical anomaly detection,

    C. I. Bercea, B. Wiestler, D. Rueckert, and J. A. Schnabel, “Diffusion models with implicit guid- ance for medical anomaly detection,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2024, pp. 211–220 (cited on p. 57)

    2024

  81. [89]

    Diffusion models for counterfac- tual generation and anomaly detection in brain images,

    A. Fontanella, G. Mair, J. W ardlaw, E. T rucco, and A. Storkey, “Diffusion models for counterfac- tual generation and anomaly detection in brain images,” IEEE Transactions on Medical Imaging, 2024 (cited on p. 57)

    2024

  82. [90]

    Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography,

    H. T. Nguyen, H. Q. Nguyen, H. H. Pham, et al., “Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography,”Scientific Data, vol. 10, no. 1, p. 277, 2023 (cited on pp. 58, 59, 63, 87)

    2023

  83. [91]

    D’Orsi, E

    C. D’Orsi, E. Sickles, E. Mendelson, E. Morris, et al., ACR BI-RADS Atlas, Breast Imaging Report- ing and Data System. Reston, Virginia: American College of Radiology, 2013 (cited on p. 59)

    2013

  84. [92]

    An image is worth 16x16 words: T ransformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An image is worth 16x16 words: T ransformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020 (cited on p. 61)

    2010 arXiv

  85. [93]

    Adam: A method for stochastic optimization,

    D. Kingma, “Adam: A method for stochastic optimization,” in Int Conf Learn Represent, 2014 (cited on p. 63)

    2014

  86. [94]

    W ovogen: W orld volume-aware diffusion for controllable multi-camera driving scene generation,

    J. Lu, Z. Huang, Z. Y ang, J. Zhang, and L. Zhang, “W ovogen: W orld volume-aware diffusion for controllable multi-camera driving scene generation,” in European Conference on Computer Vision, Springer, 2025, pp. 329–345 (cited on p. 73)

    2025

  87. [95]

    Simple open-vocabulary object detection,

    M. Minderer, A. Gritsenko, A. Stone, et al., “Simple open-vocabulary object detection,” in Euro- pean Conference on Computer Vision, Springer, 2022, pp. 728–755 (cited on p. 73). 85 Appendices A Reproducibility statement T o promote transparency and facilitate further research...

    2022

  88. [2024]

    [Online]

    arXiv: 2404.00815 [cs.CV] . [Online]. Available:https://arxiv.org/abs/2404. 00815 (cited on pp. 32, 34, 50)

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.