Pith. sign in

REVIEW 4 major objections 5 minor 59 references

ZoomLDM: Latent Diffusion Model for multi-scale image generation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single latent diffusion model, conditioned by scale through a shared embedding space, generates pathology patches at eight magnifications and, sampled jointly, yields coherent 4096×4096 images, 4× super-resolution, and…

desk verdict Solid multi-scale diffusion method with credible mid-scale gains, but the 4k coherence and state-of-the-art claims rest on unverified approximations and favorable evaluation choices. read the letter →

arxiv 2411.16969 v2 pith:ZGI3HH32 submitted 2024-11-25 cs.CV

Srikar Yellapragada , Alexandros Graikos , Kostas Triaridis , Prateek Prasanna , Rajarsi R. Gupta , Joel Saltz , Dimitris Samaras This is my paper · ORCID
classification cs.CV
keywords latentdiffusionmodelsmulti-scaleimagegenerationhistopathologywholeslideimagesself-supervisedconditioningcross-magnificationspacejointsamplingzero-shotsuper-resolutionmultipleinstancelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZoomLDM claims that one latent diffusion model, trained on 256×256 patches cut from whole-slide images at eight magnifications, can generate realistic histopathology images at any of those scales, and that sampling several scales jointly produces globally coherent 4096×4096 images in about eight minutes. The mechanism is a cross-magnification latent space: a trainable 'Summarizer' transformer compresses the SSL embedding array of a tissue region, together with a magnification token, into a fixed set of conditioning tokens, so a single weight-shared denoiser serves all eight scales and the data-scarce low magnifications inherit strength from the data-rich ones (FID 13.4 versus 106.1 at 0.15625× with real conditions). A second, small diffusion model learns to sample that conditioning space, allowing novel images to be drawn with no reference slide. If the claims hold, multi-scale generation, zero-shot 4× super-resolution, and slide-level classification features all come from one checkpoint instead of per-scale models.

What carries the argument

The load-bearing object is the cross-magnification latent space produced by the Summarizer, a 12-layer transformer that consumes the array of UNI SSL embeddings describing a region (up to 8×8 tokens, padded or pooled to a fixed size) together with a learned magnification embedding, and emits 65×512 conditioning tokens that the U-Net denoiser attends to. Projecting every scale into this shared space lets one weight-shared denoiser train across magnifications, and lets a Conditioning Diffusion Model (a Diffusion Transformer trained afterward) draw novel conditions from the space. The second mechanism is joint multi-scale sampling, an adaptation of a fast constrained-sampling algorithm for diffusion models in which the current estimate of the coarser image $\mathrm{Dec}(\hat{z}_0^L)$ plays the role of the measurement $y$ for the finer patches, with the constraint error propagated through the linear downsampling operator $A$; the expensive backpropagation through the decoder is replaced by a finite-difference step through the VQ-f4 encoder, justified by the assumption that the VQ-f4 autoencoder (the 4×-compressing VAE of the base latent diffusion model) has an approximately orthogonal decoder Jacobian (Supplementary Section 8.2, Eqs. 10–12).

What would settle it

Compute the empirical singular-value distribution of the VQ-f4 decoder Jacobian, $J = \partial \mathrm{Dec}(\hat{z}_0)/\partial \hat{z}_0$, at real 20× pathology latents (via finite-difference products $J^\top J$ or a few Hutchinson-style estimates); if the singular values deviate substantially from 1, the identity $\left(\partial \mathrm{Dec}/\partial \hat{z}_0\right)^\top e \approx \left(\partial \hat{z}_0/\partial \mathrm{Dec}\right) e$ behind Eq. 12 fails. A behavioral check that is feasible today: run 1024×1024 joint sampling with the paper's Algorithm 2 and with the exact backpropagation-based algorithm of [16], and compare CLIP and crop FID — statistical equivalence validates the approximation, while a gap shows the claimed coherence is not attributable to the stated mechanism.

Watch

Extended reading notes

Core claim

The central claim is that scale can be a conditioning variable in a latent diffusion model: one weight-shared denoiser generates patches at eight histopathology magnifications, 20× down to 0.15625×, and does so at the lowest FID reported at every scale, with the biggest margins where training data is thinnest. Because all scales share one learned conditioning space, the authors then sample scales jointly, using the current estimate of a coarse 256×256 context image at 1.25× as a constraint that guides the sixteen-by-sixteen 20× patches through the linear downsampling relation $x_{s+1}=A x_s$; the result is 4096×4096 images that keep global tissue structure and local detail, which they state is the first practical 4K histopathology synthesis: about 8 minutes per image, where prior methods took 4–12 hours and produced either blur or contextually wrong stitching. The same joint sampler, fed a single low-resolution image through an inversion step that optimizes conditioning embeddings (adapting textual inversion), performs 4× super-resolution with no super-resolution training. Finally, the paper argues the denoiser's intermediate features are themselves strong representations: fusing 20× and 5× ZoomLDM features raises BRCA-subtyping AUC to 94.91 and HRD AUC to 88.03, above UNI, Phikon, and CTransPath, and 20× features alone already beat UNI's 20× features.

Load-bearing premise

The 4096×4096 joint-sampling algorithm rests on the assumption that the VQ-f4 autoencoder's decoder Jacobian is approximately orthogonal, so the guidance direction can be computed by a cheap finite-difference pass through the encoder instead of backpropagation through the decoder; the paper states this justification in Supplementary Section 8.2 but gives no quantitative check, and if the Jacobian is far from orthogonal the guidance steps are not provably following the constraint gradient.

Editorial extensions

If this is right

  • A single ZoomLDM checkpoint replaces eight per-magnification generative models for histopathology, and the shared-weights training transfers strength to scarce scales: FID at 0.15625× drops from 106.1 to 13.4, and even CDM-sampled conditions (26.1) beat the single-scale baseline.
  • Joint multi-scale sampling makes 4096×4096 histopathology synthesis tractable at roughly 8 minutes per image on the reported hardware, where patch-stitching took about 4 hours and infinite-dimensional diffusion about 12 hours; the paper reports its 4K CLIP/crop FID trails the latter, while arguing that local detail and global structure are both present for the first time.
  • Zero-shot 4× super-resolution: given a 256×256 image at 5×, ZoomLDM infers conditioning by inversion and generates 1024×1024 output that beats diffusion super-resolution baselines (CompVis, ControlNet, ResShift, StableSR) on LPIPS and pathology-similarity metrics, without any SR-specific training.
  • Multi-scale denoiser features improve slide-level classification: 20×+5× fusion reaches 94.91 AUC for BRCA subtyping and 88.03 for HRD prediction, above the SSL encoders used to condition the model, implying that generative multi-scale training refines the representations of the conditioning encoder.
  • The recipe transfers to a second gigapixel domain, satellite imagery: four-resolution NAIP generation lowers crop FID for 1024×1024 synthesis from 43.8/48.7 to 13.3, supporting the paper's claim that the mechanism is domain-general.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Testable extension beyond the paper: the Conditioning Diffusion Model can in principle be trained on paired (summarizer-latent, biomarker) data, so that sampled conditions carry a specified mutation or grade; the paper's closing remarks gesture at this, and a concrete check is whether CDM-sampled images of BRCA-mutant versus wild-type slides differ in the expected histological features.
  • The orthogonality assumption in Supplementary Section 8.2 is the part most worth probing: if the VQ-f4 decoder Jacobian is far from orthogonal, the encoder finite-difference 'gradient' is a heuristic rather than the constraint gradient, and the observed 4K coherence may come mostly from the model's prior; a controlled comparison of Algorithm 2 against the exact backprop-based algorithm at 1024×102
  • Because the method couples scales by a linear downsampling constraint, joint three-scale (or deeper) sampling should compound coherence; the paper demonstrates two-scale pyramids, and a natural stress test is whether a 2.5×→5×→20× pyramid at 8192×8192 stays coherent or accumulates drift, especially given the patch-border stitching the paper itself notes at 4K.
  • The result that single-scale 20× features beat the very SSL encoder they were conditioned on suggests generation acts as representation refinement; a testable consequence is that fine-tuning ZoomLDM on a new tissue type should improve downstream MIL accuracy more than fine-tuning the conditioning SSL encoder alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ZoomLDM, a latent diffusion model trained on histopathology patches at eight magnifications, conditioned on a learned cross-magnification latent space built from SSL embeddings via a trainable Summarizer. A Conditioning Diffusion Model (CDM) is used to sample conditions without real images. The authors introduce joint multi-scale sampling to generate 4096x4096 images and perform zero-shot 4x super-resolution, and they extract multi-scale denoiser features for multiple instance learning. The paper claims state-of-the-art patch FID at all scales, globally coherent 4k synthesis under a tractable budget, and improved MIL performance.

Significance. If the central claims hold, the work is significant for large-image generative modeling in histopathology and satellite domains: a single weight-shared model with scale conditioning, a learned cross-scale latent space, and a tractable joint sampling scheme would be a practical step beyond patch-based or cascaded approaches. The paper has concrete strengths: code release, satellite-image generalization, a CDM for data-scarce novel sampling, ablations on the SSL encoder and summarizer, and an 8-minute 4k synthesis time versus hours for baselines. However, the load-bearing approximation in the joint sampling algorithm is unvalidated, and several stated claims are stronger than the quantitative evidence (notably Table 2 at 4096x4096 and the reconstruction-like behavior noted in Supp. 9.3).

major comments (4)
  1. [Supplementary Section 8.2, Eqs. (9)-(12)] The finite-difference approximation e ≈ [Enc(Dec(z0)+ζ e_img) − Enc(Dec(z0))]/ζ is exact only if the decoder Jacobian is orthogonal and Enc ≈ Dec^{-1}. The paper asserts this is enforced by VQ or KL training, but provides no quantitative check. Since Algorithm 2 is the mechanism behind the 4096x4096 coherence and 4x super-resolution claims, please report (a) the cosine similarity between the exact backpropagated e and the approximate e at representative timesteps and scales, (b) the decrease of the constraint C = ||A Dec(z0) − Dec(z0^L)||² over the sampling trajectory, and (c) for a tractable 1024x1024 case, a direct comparison against the backprop-based version of [16]. If the approximation fails, the guidance steps need not follow a valid descent direction, and the observed coherence may be a byproduct of the prior rather than of self-guidance.
  2. [Table 1 vs. Supplementary Section 9.3] Supp. 9.3 states that for magnifications below 5x the model 'can almost perfectly replicate the source image' because the SSL embeddings contain enough information to reconstruct the patch. This raises the concern that the large low-scale FID gains in Table 1 (e.g., 0.15625x: 13.42 vs. 106.14) partly reflect near-reconstruction of the conditioning real image rather than novel sample quality. The CDM row already addresses fully novel conditions, but for the real-condition samples the paper should report a novelty/diversity metric (e.g., distance to the nearest training neighbor) and, ideally, an FID computed over generated images whose conditions are sampled from the CDM only. The abstract's 'state-of-the-art generation quality across all scales' is stronger than the current evidence.
  3. [Table 2, 4096x4096 row] ZoomLDM's 4096x4096 CLIP FID (6.75) and Crop FID (18.90) are worse than both Graikos et al. (2.75 and 11.30) and ∞-Brush (2.63 and 14.76). The paper argues for qualitative superiority and practical inference time (8 min vs. hours), which is a legitimate strength, but the phrase 'we are the first to generate 4k pathology images with both detail and global coherency' is not supported by the reported metrics. Please either add a quantitative evaluation that directly supports the coherence/detail claim (e.g., layout or structure metrics, or a blinded human study) or explicitly restrict the state-of-the-art quality claim to 1024x1024.
  4. [Section 4.5 and Table 4] The MIL features are extracted from ZoomLDM's U-Net while conditioning on UNI embeddings (Section 4.5; see also Supp. Table 7). Comparing these features against UNI's own features is not a fair head-to-head because UNI embeddings are an input to the proposed feature extractor; the improved AUC may partly reflect the conditioning encoder's information being re-read out rather than a genuinely new multi-scale representation. A fair baseline would fuse UNI embeddings with a simple learned projector inside the same ABMIL pipeline, or use a different conditioning encoder for ZoomLDM (e.g., HIPT) and compare against that encoder's raw features.
minor comments (5)
  1. [Section 4.1.1] The sentence 'Our LDM configuration consists of a VQ-f4 autoencoder and a U-Net model pre-trained on ImageNet' is ambiguous; please state explicitly whether the U-Net is initialized from an ImageNet latent diffusion checkpoint (which one?) and what is trained from scratch.
  2. [Supplementary Section 8.1 and Figure 5] There is a typo in the supplementary heading ('Summerizer' should be 'Summarizer'), and the figure caption uses 'magt' where 'magnification embedding' is meant.
  3. [Throughout] The term 'Conditioning Diffusion Model' is used in Section 3.1, but 'Condition Diffusion Model' appears in the Conclusion; please unify the terminology.
  4. [Table 1 caption] The caption refers to 'SoTA' models, but for the lower magnifications the baselines are standalone models trained by the authors with the same architecture; please clarify which entries are published SoTA and which are internal baselines.
  5. [Supplementary Section 9.4] The admitted stitching discontinuities along patch borders and unresolved thin structures should be acknowledged in the main text near the 'globally coherent' 4096x4096 claim, not only in the supplementary.

Circularity Check

2 steps flagged · score 5.0 of 10

Low-scale FID and MIL comparisons are confounded by conditioning on target-derived/UNI embeddings; the core multi-scale training and joint-sampling derivation is otherwise self-contained.

  1. fitted input called prediction [Section 4.2, Table 1, and Supplementary Section 9.3]
    "For FID comparisons above, images were generated by randomly sampling SSL embeddings for different magnifications from the dataset. ... Interestingly, for magnifications below5× we find that the model can almost perfectly replicate the source image since the SSL embeddings used as conditioning contain enough information to reconstruct the patch at that scale perfectly."

    The FID reference set is real patches from the same dataset, and the conditioning for each generated patch is the SSL embedding matrix extracted from the base-resolution region of that same real patch. The paper itself states that for magnifications below 5x the conditioning contains enough information to reconstruct the source patch almost perfectly, so the low-scale 'ZoomLDM' FID row measures how well the model inverts SSL embeddings back into a near-copy of the reference image rather than how well it synthesizes novel samples. The gap to the SoTA baselines at those scales is therefore partly an artifact of conditioning on target-derived descriptors; only the CDM row is an independent generative FID, and it is consistently worse (e.g., 26.09 vs 13.42 at 0.15625x).

  2. other [Section 4.5 (Multiple Instance Learning) and Table 4]
    "For each patch in the WSI, we extract features from ZoomLDM’s U-Net output block 3 at a fixed timestep t=100, conditioned on UNI embeddings. ... We compare ZoomLDM’s features to those from SoTA encoders—Phikon [14], CTransPath [47], and UNI [8]."

    The proposed MIL feature extractor receives UNI embeddings as conditioning input and then produces denoiser features as a function of those embeddings. Comparing these features against the UNI encoder directly is not a comparison of an independent learned representation to UNI; any sufficiently expressive post-processing of UNI features could beat the raw UNI features. The paper's conclusion that 'by learning to generate across scales, ZoomLDM learns to produce more informative features' is not supported by this protocol, since the baseline encoder is an input to the evaluated extractor. The multi-scale fusion result is similarly confounded because the 5x features are conditioned on the same UNI embeddings of the base-resolution region.

full rationale

The central technical derivation—multi-scale training with the Summarizer, the cross-magnification latent space, and joint multi-scale sampling—is self-contained and does not reduce to its inputs. The joint sampling algorithm is described with explicit equations in Supplementary Section 8.2 (Eqs. 9-12), including the finite-difference approximation, so the citation to the same-group fast constrained sampling paper [16] is not the only load-bearing evidence; the algorithm is restated in the text. The unvalidated orthogonality assumption for the VQ-f4 decoder Jacobian is a genuine correctness risk, but it is not circularity. The two genuine confounds are in the evaluation protocol: the headline per-scale FID at low magnifications uses SSL embeddings derived from the reference images, leading to near-reconstruction as the paper admits, and the MIL 'outperforms UNI' result is obtained by feeding UNI embeddings into ZoomLDM and comparing the resulting features with UNI. Because the CDM-based FID row provides a partially independent check and the core generative method is not forced by a fit or a self-citation chain, the overall circularity score is moderate.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims rest on domain assumptions about SSL embedding transfer across scales, the linear A relationship between scales, and the orthogonal-Jacobian approximation in the VAE. The free parameters are mostly sampling and feature-extraction hyperparameters chosen without sensitivity analysis. No physically invented entities are introduced; the cross-magnification latent space is a model component rather than an independently verifiable object.

free parameters (5)
  • joint sampling step sizes lambda, delta, zeta, K = lambda=0.5, delta=0.005, zeta=0.005, K=1
    Chosen for joint sampling in Supplementary 8.2 with no sensitivity analysis; large-image and super-resolution results may depend on these values.
  • classifier-free guidance scale = 2.0
    Used for LDM and CDM sampling; standard but chosen without an ablation reported in the main text.
  • MIL feature extraction timestep and block = t=100, U-Net output block 3
    Hand-picked in Section 4.5; no ablation over timesteps or blocks, so the MIL conclusions may depend on this choice.
  • embedding inversion annealing schedule = t=950 to t=50 over 200 steps
    Supplementary 8.3; hyperparameters for the super-resolution conditioning inversion, with no sensitivity study.
  • SSL embedding pooling cap = 8x8 tokens
    Section 4.1.2: average pooling of SSL embeddings larger than 8x8 for low magnifications; a compression choice that affects conditioning fidelity.
assumptions (4)
  • domain assumption SSL embeddings from UNI, trained on 20x patches, remain informative for all lower magnifications after Summarizer compression.
    Section 3.1: lower-scale patches are conditioned on embeddings of their contained 20x regions. If this transfer fails, low-scale FID gains cannot be attributed to cross-scale generalization.
  • domain assumption Images at adjacent scales are related by a fixed linear downsampling operator A, specifically bicubic interpolation.
    Section 3.2, Eq. 1: joint sampling forces x_{s+1}=A x_s. Real whole-slide image pyramids may use other downsampling, which would bias the guidance.
  • domain assumption The LDM VAE decoder Jacobian is approximately orthogonal, allowing the gradient of the constraint to be approximated via finite differences through the encoder.
    Supplementary Section 8.2, Eqs. 10-12: this assumption is stated without quantitative verification and underpins the efficient 4096x4096 sampling algorithm.
  • domain assumption Denoiser features at timestep 100 from U-Net block 3 are a useful representation for MIL when conditioned on UNI embeddings.
    Section 4.5: the choice is empirical with no theoretical justification or ablation, so the MIL results may be sensitive to this fixed timestep and layer.
invented entities (1)
  • Cross-magnification latent space
    purpose: Shared conditioning space produced by the Summarizer to compress SSL embeddings from all scales into a fixed set of tokens for the denoiser.
    It is a learned construct; the paper's FID, large-image, and MIL experiments provide indirect internal support, but there is no external falsifiable handle that distinguishes it from a generic conditioning network.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZoomLDM: Latent Diffusion Model for multi-scale image generation." pith.science (2026). https://pith.science/paper/ZGI3HH32

@misc{pith2026241116969,
  author       = {Pith},
  title        = {Pith review of: ZoomLDM: Latent Diffusion Model for multi-scale image generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGI3HH32}},
  note         = {Machine review of arXiv:2411.16969}
}
abstract

Diffusion models have revolutionized image generation, yet several challenges restrict their application to large-image domains, such as digital pathology and satellite imagery. Given that it is infeasible to directly train a model on 'whole' images from domains with potential gigapixel sizes, diffusion-based generative methods have focused on synthesizing small, fixed-size patches extracted from these images. However, generating small patches has limited applicability since patch-based models fail to capture the global structures and wider context of large images, which can be crucial for synthesizing (semantically) accurate samples. To overcome this limitation, we present ZoomLDM, a diffusion model tailored for generating images across multiple scales. Central to our approach is a novel magnification-aware conditioning mechanism that utilizes self-supervised learning (SSL) embeddings and allows the diffusion model to synthesize images at different 'zoom' levels, i.e., fixed-size patches extracted from large images at varying scales. ZoomLDM synthesizes coherent histopathology images that remain contextually accurate and detailed at different zoom levels, achieving state-of-the-art image generation quality across all scales and excelling in the data-scarce setting of generating thumbnails of entire large images. The multi-scale nature of ZoomLDM unlocks additional capabilities in large image generation, enabling computationally tractable and globally coherent image synthesis up to $4096 \times 4096$ pixels and $4\times$ super-resolution. Additionally, multi-scale features extracted from ZoomLDM are highly effective in multiple instance learning experiments.

Figures

Figures reproduced from arXiv: 2411.16969 by the authors.

Figure 1
Figure 1. ZoomLDM can generate synthetic image patches at mul [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach. Left: We extract [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Large Images (4096 × 4096) generated from ZoomLDM. Our large image generation framework is the first to generate 4k pathology images with local details and global consistency, all within reasonable inference time. We provide more 4k examples and comparisons in the supplementary. magnifications, we train standalone models specifically for patches from those magnifications, keeping the architecture consistent with Zoo… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: We showcase 4× super-resolution results (256 × 256 → 1024 × 1024). Samples generated by other methods [39, 52] exhibit artifacts, inconsistencies, and blurriness that are not present in our outputs. Specifically, in blue boxes, we can observe that CompVis[39] generates…
Figure 5
Figure 5. Figure 5: Overview of the Summarizer and Condition Diffusion Model. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Joint sampling process across two different magnifications for the TCGA-BRCA ZoomLDM model. We jointly generate a [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Joint sampling process across two different resolutions for the Satellite ZoomLDM model. We jointly generate a [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Figure illustrating our pipeline for the image inversion used in the super-resolution task. For a given image we first use the denois [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Examples of the image inversion algorithm. Given a real image at any magnification, we infer the SSL embeddings that generated [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: We present 0.15625× images generated from our model and their nearest neighbors in the training dataset. Although only trained on ∼ 2500 images, our 400M parameter model did not memorize the training samples and successfully synthesized novel images at that magnificat…
Figure 11
Figure 11. Figure 11: Synthetic patches (256 × 256 pixel) generated by ZoomLDM juxtaposed with the corresponding real images from TCGA￾BRCA. Across all magnifications, ZoomLDM preserves the semantic features of the reference patches. 9.5. Comparison to previous works In [PITH_FULL_IMAGE:f…
Figure 12
Figure 12. Figure 12: Images synthesized by ZoomLDM using conditions sampled from our Conditioning Diffusion model (CDM). [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Synthetic patches (256 × 256 pixel) generated by ZoomLDM juxtaposed with the corresponding real images from NAIP 20 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: We present 4096×4096 images generated from our histopathology model. Our results exhibit correct global structures in terms of the arrangement of cells and tissue while also maintaining high-resolution details. We point out two weaknesses: The local model fails to mai…
Figure 15
Figure 15. Figure 15: We present 4096 × 4096 images generated from our satellite model. The results demonstrate images with reasonable global structures that also maintain high-resolution features. A similar weakness to the pathology images is visible, with slight discontinuities among the…
Figure 16
Figure 16. Figure 16: We compare with two recent previous methods that also generated large histopathology images. In this example, we compare [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Comparison between ∞ − Brush [26] and our method. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 43 canonical work pages

  1. [16]

    Fast constrained sampling in pre-trained diffusion models

    Alexandros Graikos, Nebojsa Jojic, and Dimitris Samaras. Fast constrained sampling in pre-trained diffusion models. arXiv preprint arXiv:2410.18804, 2024. 4, 5, 14, 15

    work page arXiv 2024

  2. [1]

    Bach: Grand challenge on breast cancer histology images

    Guilherme Aresta, Teresa Ara ´ujo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan, Varghese Alex, Bahram Marami, Marcel Prastawa, Monica Chan, Michael Donovan, Gerardo Fernandez, Jack Zeineh, Matthias Kohl, Christoph Walz, Florian Ludwig, Stefan Braunewell, Maximilian Baust, Quoc Dang Vu, Minh Nguyen Nhat To, Eal Kim, Jin Tae Kwak, Sameh Galal, Ver...

    work page 2019

  3. [2]

    Diffinfinite: Large mask-image synthesis via parallel random patch diffusion in histopathology

    Marco Aversa, Gabriel Nobis, Miriam H ¨agele, Kai Stand- voss, Mihaela Chirica, Roderick Murray-Smith, Ahmed Alaa, Lukas Ruff, Daniela Ivanova, Wojciech Samek, et al. Diffinfinite: Large mask-image synthesis via parallel random patch diffusion in histopathology. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmark...

    work page 2023

  4. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 1

    2023

  5. [4]

    The can- cer genome atlas pan-cancer analysis project

    JN Cancer Genome Atlas Research Network et al. The can- cer genome atlas pan-cancer analysis project. Nat. Genet, 45 (10):1113–1120, 2013. 5, 7

    work page 2013

  6. [5]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021. 1

    work page 2021

  7. [6]

    Pixart- σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation, 2024

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of diffu- sion transformer for 4k text-to-image generation, 2024. 3

    work page 2024

  8. [7]

    Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

    Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144–16155, 2022. 2, 3, 8, 12, 13

    work page 2022

Show all 59 references
  1. [8]

    A general-purpose self-supervised model for computa- tional pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song, Muhammad Shaban, et al. A general-purpose self-supervised model for computa- tional pathology. arXiv preprint arXiv:2308.15474, 2023. 5, 8

    2023 arXiv

  2. [9]

    Towards a general-purpose foundation model for com- putational pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song, Muhammad Shaban, et al. Towards a general-purpose foundation model for com- putational pathology. Nature Medicine, 2024. 2, 13, 18

    2024

  3. [10]

    Prompt-tuning latent diffusion models for inverse problems

    Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In Forty-first International Conference on Machine Learning, 2024. 8, 15

    2024

  4. [11]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informa- tion processing systems, 34:8780–8794, 2021. 2

    2021

  5. [12]

    Tweedie’s formula and selection bias

    Bradley Efron. Tweedie’s formula and selection bias. Jour- nal of the American Statistical Association, 106(496):1602– 1614, 2011. 5

    2011

  6. [13]

    Generate your own scotland: Satellite image generation conditioned on maps

    Miguel Espinosa and Elliot J Crowley. Generate your own scotland: Satellite image generation conditioned on maps. arXiv preprint arXiv:2308.16648, 2023. 3

    2023 arXiv

  7. [14]

    Scaling self-supervised learning for histopathology with masked image modeling

    Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Alice Mac Kain, Charlie Saillard, and Jean-Baptiste Schiratti. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, pages 2023–07, 2023. 8

    2023

  8. [15]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh Interna- tional Conference on Learning Representations, 2023. 8, 15, 16

    2023

  9. [17]

    Learned representation-guided diffusion models for large-image generation

    Alexandros Graikos, Srikar Yellapragada, Minh-Quan Le, Saarthak Kapse, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Learned representation-guided diffusion models for large-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  10. [18]

    Diffusion- based generation of histopathological whole slide images at a gigapixel scale

    Robert Harb, Thomas Pock, and Heimo M ¨uller. Diffusion- based generation of histopathological whole slide images at a gigapixel scale. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pages 5131–5140, 2024. 2, 3

    2024

  11. [19]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5

    2017

  12. [20]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 1, 3, 5, 15

    2022 arXiv

  13. [21]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2, 15

    2020

  14. [22]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Inter- national conference on machine learning, pages 2127–2136. PMLR, 2018. 8

    2018

  15. [23]

    Ex- plainable ai for computational pathology identifies model 9 limitations and tissue biomarkers

    Jakub R Kaczmarzyk, Joel H Saltz, and Peter K Koo. Ex- plainable ai for computational pathology identifies model 9 limitations and tissue biomarkers. ArXiv, pages arXiv–2409,

  16. [24]

    Si-mil: Taming deep mil for self-interpretability in gigapixel histopathology

    Saarthak Kapse, Pushpak Pati, Srijan Das, Jingwei Zhang, Chao Chen, Maria Vakalopoulou, Joel Saltz, Dimitris Sama- ras, Rajarsi R Gupta, and Prateek Prasanna. Si-mil: Taming deep mil for self-interpretability in gigapixel histopathology. In Proceedings of the IEEE/CVF Conferen...

    2024

  17. [25]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and vi...

    2020

  18. [26]

    ∞-brush: Controllable large image synthesis with diffusion models in infinite dimensions, 2024

    Minh-Quan Le, Alexandros Graikos, Srikar Yellapragada, Rajarsi Gupta, Joel Saltz, and Dimitris Samaras. ∞-brush: Controllable large image synthesis with diffusion models in infinite dimensions, 2024. 1, 2, 3, 6, 7, 12, 19, 23

    2024

  19. [27]

    Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning

    Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 202...

    2021

  20. [28]

    A visual- language foundation model for computational pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual- language foundation model for computational pathology. Nature Medicine, 30:863–874, 2024. 8

    2024

  21. [29]

    A morphology focused diffusion probabilistic model for synthesis of histopathology images

    Puria Azadi Moghadam, Sanne Van Dalen, Karina C Mar- tin, Jochen Lennerz, Stephen Yip, Hossein Farahani, and Ali Bashashati. A morphology focused diffusion probabilistic model for synthesis of histopathology images. In Proceed- ings of the IEEE/CVF Winter Conference on Applica...

    2000

  22. [30]

    A multimodal comparison of latent denois- ing diffusion probabilistic models and generative adversarial networks for medical image synthesis

    Gustav M ¨uller-Franzes, Jan Moritz Niehues, Firas Khader, Soroosh Tayebi Arasteh, Christoph Haarburger, Christiane Kuhl, Tianci Wang, Tianyu Han, Teresa Nolte, Sven Nebelung, et al. A multimodal comparison of latent denois- ing diffusion probabilistic models and generative ad...

    2023

  23. [31]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR,

  24. [32]

    Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, p...

    2022

  25. [33]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 5, 12

    2023 arXiv

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  27. [35]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 3

    2023 arXiv

  28. [36]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth Interna- tional Conference on Learning Representations, 2024. 1

    2024

  29. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  30. [38]

    Large scale high-resolution land cover mapping with multi- resolution data

    Caleb Robinson, Le Hou, Kolya Malkin, Rachel Soobit- sky, Jacob Czawlytko, Bistra Dilkina, and Nebojsa Jojic. Large scale high-resolution land cover mapping with multi- resolution data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 127...

    2019

  31. [39]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3, 5, 7

    2022

  32. [40]

    Image super-resolution via iterative refinement

    Chitwan Saharia, Jonathan Ho, William Chan, Tim Sal- imans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4713– 4726, 2022. 3

    2022

  33. [41]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  34. [42]

    Rsdiff: Remote sens- ing image generation from text using diffusion model

    Ahmad Sebaq and Mohamed ElHelw. Rsdiff: Remote sens- ing image generation from text using diffusion model. arXiv preprint arXiv:2309.02455, 2023. 3

    2023 arXiv

  35. [43]

    pytorch-fid: FID Score for PyTorch

    Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid ,

  36. [44]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2020. 3, 5

    2020

  37. [45]

    National agriculture imagery program (NAIP),

    USGS. National agriculture imagery program (NAIP),

  38. [46]

    Exploiting diffusion prior for 10 real-world image super-resolution

    Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for 10 real-world image super-resolution. International Journal of Computer Vision, pages 1–21, 2024. 17, 18

    2024

  39. [47]

    Transpath: Transformer-based self-supervised learning for histopatho- logical image classification

    Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Junzhou Huang, Wei Yang, and Xiao Han. Transpath: Transformer-based self-supervised learning for histopatho- logical image classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th...

    2021

  40. [48]

    Vit-dae: Transformer-driven diffusion autoen- coder for histopathology image analysis

    Xuan Xu, Saarthak Kapse, Rajarsi Gupta, and Prateek Prasanna. Vit-dae: Transformer-driven diffusion autoen- coder for histopathology image analysis. arXiv preprint arXiv:2304.01053, 2023. 3

    2023 arXiv

  41. [49]

    Pathldm: Text conditioned latent diffusion model for histopathology

    Srikar Yellapragada, Alexandros Graikos, Prateek Prasanna, Tahsin Kurc, Joel Saltz, and Dimitris Samaras. Pathldm: Text conditioned latent diffusion model for histopathology. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision (WACV) , pages 5...

  42. [50]

    Effi- cient diffusion model for image restoration by residual shift- ing

    Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Effi- cient diffusion model for image restoration by residual shift- ing. arXiv preprint arXiv:2403.07319, 2024. 17, 18

    2024 arXiv

  43. [51]

    Resshift: Efficient diffusion model for image super- resolution by residual shifting

    Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super- resolution by residual shifting. Advances in Neural Infor- mation Processing Systems, 36, 2024. 17, 18

    2024

  44. [52]

    Adding conditional control to text-to-image diffusion models, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 7, 8

    2023

  45. [53]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 8 11 ZoomLDM: Latent Diffusion Model for multi-scale image generation Supplementary Material We organize the sup...

    2018

  46. [55]

    However, gigapixel images also concern the remote sensing domain, where satellite images regularly are in the range of 10000 × 10000 pixels

    ZoomLDM on satellite images In the main text, we focused on the digital histopathology domain and how our multi-scale diffusion model can prove useful in generation and downstream tasks. However, gigapixel images also concern the remote sensing domain, where satellite images r...

  47. [56]

    Table 7 shows that replacing UNI with HIPT degrades performance and further replacing the ViT summarizer network with a simple 4-layer CNN leads to a greater decline

    Ablation on SSL encoder and Summarizer We retrain ZoomLDM with (i) a weaker SSL encoder (HIPT [7]) and (ii) both a weaker SSL encoder and a simpler summa- rizer network (CNN vs ViT). Table 7 shows that replacing UNI with HIPT degrades performance and further replacing the ViT ...

  48. [57]

    Summarizer-CDM training details Summarizer: We train the Summarizer jointly with the LDM

    Experiment details 8.1. Summarizer-CDM training details Summarizer: We train the Summarizer jointly with the LDM. The Summarizer processes the SSL embeddings extracted alongside the image patches and projects them to a latent space that is shared across all scales (cross-magni...

  49. [58]

    (8) However, to calculate g we need e = ∂C ∂ ˆz0 which we can calculate by backpropagating through the decoder model

    Algorithm 1 requires us to compute the direction e of ˆz0 towards which the constraint C is minimized and uses it to update the current diffusion latent as g = ˆz0(zt + δe) − ˆz0(zt) δ (7) z′ t = zt + λg. (8) However, to calculate g we need e = ∂C ∂ ˆz0 which we can calculate ...

  50. [59]

    More super-resolution baselines In Tables 8 and 9 we provide additional baselines for the super-resolution task

    Additional results 9.1. More super-resolution baselines In Tables 8 and 9 we provide additional baselines for the super-resolution task. We use ResShift [50, 51] and StableSR [46] to super-resolve pathology images and compare them to the zero-shot performance of ZoomLDM. Using...

  51. [2023]

    https : / / www . usgs . gov / centers / eros / science / usgs - eros - archive - aerial - photography - national - agriculture - imagery-program-naip. 5, 12

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.