REVIEW 4 major objections 5 minor 31 references
Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imaging
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A multi-level VQGAN trained on paired multiplex immunofluorescence and H&E images generates virtual H&E stains that outperform a standard conditional GAN on downstream nuclei segmentation and tissue-label consistency, on two colorectal canc
desk verdict Useful first paired mIF-to-H&E comparison, but the 'superior architecture' claim is not supported by the paper's own CODEX results and the post-hoc evaluation choices are too convenient. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the multi-level VQGAN: a convolutional encoder maps an mIF patch into feature maps at two spatial resolutions, each quantized by its own learned codebook of 1024 entries, and a decoder reconstructs the H&E image from the discrete codes. The hierarchy is intended to let the low-resolution codebook carry coarse tissue architecture while the high-resolution codebook carries cellular detail. Training uses a composite loss $L_{\mathrm{total}} = \lambda_{\mathrm{recon}} L_{\mathrm{recon}} + \lambda_{\mathrm{perc}} L_{\mathrm{perc}} + \lambda_{\mathrm{style}} L_{\mathrm{style}} + \lambda_{\mathrm{stain}} L_{\mathrm{stain}} + \lambda_{\mathrm{adv}} L_{\mathrm{adv}}$, where the
What would settle it
Retrain the cGAN and VQGAN on the Orion data with random class-balanced sampling instead of CONCH-based curation, and score semantic preservation with an independent tissue classifier or pathologist panel. If the VQGAN's label-consistency and segmentation margins persist, the architecture claim holds; if they shrink or reverse, the reported advantage is an artifact of the shared CONCH evaluator. A complementary check is sweeping the number of quantization levels on the 58-channel CODEX data to see whether the optimal hierarchy depth depends on input dimensionality.
Extended reading notes
Core claim
The paper's central claim is that hierarchical vector quantization can compress a 19- or 58-channel mIF input into discrete codes that decode into an H&E image whose scientific content matches ground-truth H&E better than a standard conditional GAN with a continuous latent space. The decisive evidence is functional, not perceptual: using a watershed segmenter, the two-level VQGAN reaches a mean IoU of 0.6800 on Orion versus 0.6376 for the cGAN; using CONCH as a tissue labeler, generated images agree with ground-truth labels 69.9% of the time for the two-level VQGAN versus 62.9% for the cGAN. On the 58-channel CODEX dataset, the single-level VQGAN leads with 53.5% label consistency and 0.8192
Load-bearing premise
The load-bearing premise is that using the same CONCH model to balance the Orion training patches and to score label consistency does not bias the architecture comparison; if CONCH's preferences shaped which patches were learned, the label-consistency metric may reward VQGAN outputs for matching CONCH's biases rather than true tissue semantics.
Editorial extensions
If this is right
- If the claim is correct, mIF images can be rendered as H&E-like views quickly enough to give morphological context without waiting for physical histology, and existing H&E-based CAD pipelines become applicable to mIF data.
- Discrete latent codes are sufficiently expressive that a finite codebook does not hurt a high-dimensional (19-58 channel) image translation task.
- For class-balanced, 19-channel data, adding a second quantization level improves reconstruction and downstream utility over a single level.
- VQGAN-generated virtual stains support watershed nuclei segmentation well enough that a traditional segmenter transfers better than deep segmentation models.
- The learned discrete codes are proposed as a possible foundation for other mIF downstream tasks, such as cell classification or tumor-microenvironment clustering.
Reading between the lines
- A consequence the authors leave implicit: the CONCH model is used twice, to balance the Orion training set and to score label consistency, so a direct test with independently curated patches and an independent semantic labeler is needed to know whether the VQGAN's margin is about tissue fidelity or alignment with CONCH's preferences.
- On the 58-channel CODEX data, the single-level VQGAN beats the two-level model on both label consistency and segmentation, suggesting the optimal number of quantization levels is dataset-dependent; a systematic sweep of codebook levels against channel depth would test this.
- The failure of Cellpose and StarDist to transfer between ground-truth and generated H&E suggests current virtual stains preserve low-frequency layout but not all high-frequency textural cues deep segmenters rely on; improving those cues is a concrete next target.
- The winning label-consistency scores (roughly 50-70%) mean virtual H&E is not yet interchangeable with physical H&E for diagnosis; the paper's own caution about clinical readiness leaves open how much agreement a diagnostic workflow would require.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a paired, supervised approach to virtual H&E staining from multiplex immunofluorescence (mIF) images using a multi-level VQGAN, and compares it against a single-level VQGAN and a pix2pix-style cGAN on two public colorectal cancer datasets (Orion, CODEX). The authors report reconstruction metrics (L1, L2, SSIM, PSNR) and two downstream evaluations: tissue label consistency using a pretrained CONCH model, and nuclei segmentation using a watershed algorithm. The central claim is that the multi-level VQGAN is a 'robust and superior architecture' for generating scientifically useful virtual stains.
Significance. If the claimed superiority held, the paper would provide a useful benchmark for discrete-latent generative models in virtual staining, a topic with limited prior work for high-dimensional mIF-to-H&E translation. The study uses paired data with patient-level splits, which is a sound design for avoiding data leakage, and it contributes a systematic comparison of cGAN, VQGAN, and hierarchical VQGAN on two datasets. However, the evaluation has serious methodological flaws, including post-hoc selection of the segmentation metric, double use of the CONCH model for dataset curation and outcome measurement, and absence of any statistical uncertainty estimates. Because the central claim depends on these confounded and internally inconsistent results, the paper's significance is not currently established.
major comments (4)
- [Abstract; Section IV.B; Tables IV and V] The abstract and conclusion claim that the multi-level VQGAN is a 'robust and superior architecture.' This is contradicted by the paper's own CODEX results: the single-level VQGAN achieves higher label consistency (53.5% vs 48.4%, Table IV) and higher watershed IoU (0.8192 vs 0.8028, Table V) than the multi-level VQGAN. The paper does not address this inconsistency, so the central superiority claim is unsupported.
- [Section III.A and Section IV.B.1] The semantic-preservation metric uses CONCH to classify generated images, but CONCH was also used to curate the Orion training set via zero-shot classification and class balancing. This creates a circular dependency: the test set is enriched for patches CONCH classifies confidently, and the metric rewards outputs aligned with CONCH's specific decision boundary. The paper neither controls for this nor discusses its potential bias, making the Orion label-consistency advantage (69.9% vs 62.9%) unreliable as evidence of general semantic preservation.
- [Section IV.B.2] The authors state that Cellpose and StarDist 'showed poor agreement' and then switched to a traditional watershed algorithm 'to enable a comparison.' This is an explicit post-hoc change of the evaluation metric after observing the first-choice deep-learning segmenters did not yield usable results. No pre-registration, multiple-testing correction, or acknowledgment of this selection bias is provided. The motivating application is the H&E-based CAD ecosystem, which largely relies on deep-learning segmentation; watershed results do not demonstrate utility for that ecosystem.
- [Tables II-V] No error bars, confidence intervals, or significance tests are reported for any metric. Differences such as Orion L1 (0.1491 vs 0.1545) and Orion IoU (0.6800 vs 0.6376) may be within noise, especially given patch-level evaluation and an unreported number of patients. Without variance estimates, the claimed improvements are not supported. This problem affects the reconstruction metrics as well as the downstream metrics.
minor comments (5)
- [Introduction] The sentence 'Hematoxylin and Eosin (H&E) staining has served as the foundation... (H&E) [2]' has a duplicate '(H&E)' and an awkward break. Please revise.
- [Throughout] The terms '1-level' and 'single-level' are used interchangeably. Standardize terminology for consistency.
- [Section IV.A] The text says 'Our proposed 2-level VQGAN' while the abstract and conclusion use 'multi-level VQGAN.' Clarify whether the contribution is specifically the 2-level variant or the hierarchical approach generally.
- [Section IV.B.2] Figure 4 is referenced as showing box plots for Orion, but the figure is not present in the provided text. Ensure all figures are included and captions match the discussion.
- [General] No information is given about code availability, hyperparameter search ranges, or the number of patients in the splits. Add these details to improve reproducibility.
Circularity Check
Mild circular dependency: CONCH is used both to curate the Orion evaluation set and to score semantic preservation, entangling the label-consistency metric; reconstruction metrics remain independent.
-
other
[Section III.A (Datasets) and Section IV.B.1 (Tissue Classification); claim echoed in Abstract]
"We employed a vision-language model, CONCH [20], for zero-shot classification of H&E patches into nine distinct tissue categories. The dataset was then sampled to create a balanced distribution of these tissue types. ... we evaluated whether a pre-trained CONCH foundation model assigned the same tissue labels to generated images as it would to the corresponding ground truth H&E images."
The Orion dataset—including the distribution used for evaluation—is constructed from CONCH's zero-shot classifications (the dataset was sampled to balance tissue categories according to CONCH's labels). The semantic-preservation metric is then defined as CONCH's label agreement between generated and ground-truth H&E. The same model therefore defines the test population and measures the outcome, so Table IV's Orion label-consistency values partly reflect CONCH's self-consistency on CONCH-selected patches rather than an independent tissue-semantics ground truth. The Abstract's inference that VQGAN-generated images show 'superior performance and agreement with ground-truth analysis' is thus not fully external to the curation model; it is a measurement dependency, not a reduction by constructi
full rationale
The paper's central reconstruction comparison (Tables II–III) is a conventional, self-contained train/test evaluation with patient-level splits; no fitted parameter is renamed as a prediction. The watershed segmentation (Table V) is also independent of training, although the metric was adopted only after Cellpose and StarDist 'showed poor agreement ... across all generative models'—a post-hoc evaluation choice that raises a validity concern but is not itself circular. The one genuine circular dependency is the Orion label-consistency evaluation: CONCH is used in Section III.A to curate the dataset and again in Section IV.B.1 to score semantic preservation, so the semantic-preservation claim is partially self-referential (the same model shapes the population and defines the outcome). This is mild rather than forced, because the VQGAN's advantage is not guaranteed by construction. The only self-citation ([29], Thota et al.) is used to rationalize modest label-consistency rates and is not load-bearing. Separately, the CODEX rows of Tables IV and V show the 1-level VQGAN beating the 2-level model (53.5% vs 48.4% label consistency; 0.8192 vs 0.8028 IoU), which contradicts the Abstract's blanket 'multi-level VQGAN is superior' phrasing—an internal-consistency/correctness problem, not a circularity. Score 3 reflects one secondary metric entangled with its own curation tool while the core image-quality comparison remains independent.
Assumptions & free parameters
free parameters (5)
- codebook_size =
1024
- commitment_cost =
[0.2] and [0.2, 0.1]
- adversarial_loss_weight =
0.02
- loss_weights (lambda values) =
not reported
- early_stopping_patience =
15
assumptions (4)
- domain assumption Paired mIF and H&E patches are co-registered and correspond to the same tissue region.
- domain assumption CONCH zero-shot classification provides a valid ground truth for tissue categories.
- ad hoc to paper Traditional watershed segmentation is a meaningful downstream task for evaluating virtual stains.
- domain assumption The reconstruction and perceptual losses (L1, SSIM, PSNR, KimiaNet perceptual) are valid proxies for H&E quality.
Cite this review
Pith. "Pith review of Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imaging." pith.science (2026). https://pith.science/paper/GNRQFLA7
@misc{pith2026250804734,
author = {Pith},
title = {Pith review of: Cross-Domain Image Synthesis: Generating H&E from Multiplex Biomarker Imaging},
year = {2026},
howpublished = {\url{https://pith.science/paper/GNRQFLA7}},
note = {Machine review of arXiv:2508.04734}
}
read the original abstract
While multiplex immunofluorescence (mIF) imaging provides deep, spatially-resolved molecular data, integrating this information with the morphological standard of Hematoxylin & Eosin (H&E) can be very important for obtaining complementary information about the underlying tissue. Generating a virtual H&E stain from mIF data offers a powerful solution, providing immediate morphological context. Crucially, this approach enables the application of the vast ecosystem of H&E-based computer-aided diagnosis (CAD) tools to analyze rich molecular data, bridging the gap between molecular and morphological analysis. In this work, we investigate the use of a multi-level Vector-Quantized Generative Adversarial Network (VQGAN) to create high-fidelity virtual H&E stains from mIF images. We rigorously evaluated our VQGAN against a standard conditional GAN (cGAN) baseline on two publicly available colorectal cancer datasets, assessing performance on both image similarity and functional utility for downstream analysis. Our results show that while both architectures produce visually plausible images, the virtual stains generated by our VQGAN provide a more effective substrate for computer-aided diagnosis. Specifically, downstream nuclei segmentation and semantic preservation in tissue classification tasks performed on VQGAN-generated images demonstrate superior performance and agreement with ground-truth analysis compared to those from the cGAN. This work establishes that a multi-level VQGAN is a robust and superior architecture for generating scientifically useful virtual stains, offering a viable pathway to integrate the rich molecular data of mIF into established and powerful H&E-based analytical workflows.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
J.-R. Lin, Y .-A. Chen, D. Campton, J. Cooper, S. Coy, C. Yapp, J. B. Tefft, E. McCarty, K. L. Ligon, S. J. Rodig et al. , “High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers,” Nature cancer, vol. 4, no. 7, pp. 1036–1052, 2023
work page 2023
-
[2]
Histopathological image analysis: A review,
M. N. Gurcan, L. E. Boucheron, A. Can, A. Madabhushi, N. M. Rajpoot, and B. Yener, “Histopathological image analysis: A review,” IEEE reviews in biomedical engineering , vol. 2, pp. 147–171, 2009
work page 2009
-
[3]
The wonderful colors of the hematoxylin–eosin stain in diagnostic surgical pathology,
J. K. Chan, “The wonderful colors of the hematoxylin–eosin stain in diagnostic surgical pathology,” International journal of surgical pathology, vol. 22, no. 1, pp. 12–32, 2014
work page 2014
-
[4]
Image analysis and machine learning in digital pathology: Challenges and opportunities,
A. Madabhushi and G. Lee, “Image analysis and machine learning in digital pathology: Challenges and opportunities,” Medical image analysis, vol. 33, pp. 170–175, 2016
work page 2016
-
[5]
Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry,
C. Giesen, H. A. Wang, D. Schapiro, N. Zivanovic, A. Jacobs, B. Hat- tendorf, P. J. Sch ¨uffler, D. Grolimund, J. M. Buhmann, S. Brandt et al., “Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry,” Nature methods, vol. 11, no. 4, pp. 417–422, 2014
work page 2014
-
[6]
Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning,
Y . Rivenson, H. Wang, Z. Wei, K. de Haan, Y . Zhang, Y . Wu, H. G ¨unaydın, J. E. Zuckerman, T. Chong, A. E. Sisk et al. , “Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning,” Nature biomedical engineering , vol. 3, no. 6, pp. 466– 477, 2019
work page 2019
-
[7]
S. Bao, H. H. Lee, Q. Yang, L. W. Remedios, R. Deng, C. Cui, L. Y . Cai, K. Xu, X. Yu, S. Chiron et al., “Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis,” Proceedings of machine learning research, vol. 227, p. 1406, 2024
work page 2024
-
[8]
A. Saad, N. Ghatwary, S. M. Gasser, and M. S. ElMahallawy, “Automatic image generation and stage prediction of breast cancer immunobiologi- cal through a proposed ihc-gan model,” BMC Medical Imaging, vol. 25, no. 1, p. 6, 2025
work page 2025
Show all 31 references
-
[9]
Diversity in stable gans: A systematic review of mode collapse mitigation strategies,
M. Cobbinah, H. Nunoo-Mensah, P. Ebenezer Adjei, F. Adoma Acheam- pong, I. Acquah, E. Tutu Tchao, A. Selasi Agbemenu, J. John Kponyo, and E. Abaidoo, “Diversity in stable gans: A systematic review of mode collapse mitigation strategies,” Engineering Reports , vol. 7, no. 6, p....
2025
-
[10]
Neural discrete representation learning,
A. Van Den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” in Advances in neural information processing systems, 2017, pp. 6306–6315
2017
-
[11]
Taming transformers for high- resolution image synthesis,
P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high- resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 12 873–12 883
2021
-
[12]
Generating diverse high-fidelity images with vq-vae-2,
A. Razavi, A. Van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” in Advances in neural information processing systems, 2019, pp. 14 866–14 876
2019
-
[13]
In silico labeling: predicting fluorescent labels in unlabeled images,
E. M. Christiansen, S. J. Yang, D. M. Ando, A. Javaherian, G. Skibinski, S. Lipnick, E. Mount, A. O’neil, K. Shah, A. K. Lee et al. , “In silico labeling: predicting fluorescent labels in unlabeled images,” Cell, vol. 173, no. 3, pp. 792–803, 2018
2018
-
[14]
Patholog- ical semantics-preserving learning for h&e-to-ihc virtual staining,
F. Chen, R. Zhang, B. Zheng, Y . Sun, J. He, and W. Qin, “Patholog- ical semantics-preserving learning for h&e-to-ihc virtual staining,” in International Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2024, pp. 384–394
2024
-
[15]
Staindiffuser: multitask dual diffusion model for virtual staining,
T. Kataria, B. Knudsen, and S. Y . Elhabian, “Staindiffuser: multitask dual diffusion model for virtual staining,” arXiv preprint arXiv:2403.11340 , 2024
2024 arXiv
-
[16]
Image-to-image translation with conditional adversarial networks,
P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125– 1134
2017
-
[17]
A survey on training challenges in generative adversarial networks for biomedical image analysis,
M. M. Saad, R. O’Reilly, and M. H. Rehmani, “A survey on training challenges in generative adversarial networks for biomedical image analysis,” Artificial Intelligence Review , vol. 57, no. 2, p. 19, 2024
2024
-
[18]
Gans for medical image analysis,
S. Kazeminia, C. Baur, A. Kuijper, B. Van Ginneken, N. Navab, S. Al- barqouni, and A. Mukhopadhyay, “Gans for medical image analysis,” Artificial intelligence in medicine , vol. 109, p. 101938, 2020
2020
-
[19]
Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front,
C. M. Sch ¨urch, S. S. Bhate, G. L. Barlow, D. J. Phillips, L. Noti, I. Zlobec, P. Chu, S. Black, J. Demeter, D. R. McIlwain et al. , “Coordinated cellular neighborhoods orchestrate antitumoral immunity at the colorectal cancer invasive front,” Cell, vol. 182, no. 5, pp. 1341–...
2020
-
[20]
A visual-language foundation model for computational pathology,
M. Y . Lu, B. Chen, D. F. Williamson, R. J. Chen, I. Liang, T. Ding, G. Jaume, I. Odintsov, L. P. Le, G. Gerber et al. , “A visual-language foundation model for computational pathology,” Nature medicine , vol. 30, no. 3, pp. 863–874, 2024
2024
-
[21]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241
2015
-
[22]
Fine-tuning and training of densenet for histopathology image representation using tcga diagnostic slides,
A. Riasatian, M. Babaie, D. Maleki, S. Kalra, M. Valipour, S. Hemati, M. Zaveri, A. Safarpoor, S. Shafiei, M. Afshari et al., “Fine-tuning and training of densenet for histopathology image representation using tcga diagnostic slides,” Medical image analysis , vol. 70, p. 102032, 2021
2021
-
[23]
A neural algorithm of artistic style,
L. A. Gatys, A. S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” arXiv preprint arXiv:1508.06576 , 2015
2015 arXiv
-
[24]
A method for normalizing histology slides for quantitative analysis,
M. Macenko, M. Niethammer, J. S. Marron, D. Borland, J. T. Woosley, X. Guan, C. Schmitt, and N. E. Thomas, “A method for normalizing histology slides for quantitative analysis,” in 2009 IEEE international symposium on biomedical imaging: from nano to macro . IEEE, 2009, pp. 1107–1110
2009
-
[25]
Decoupled weight decay regularization,
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2019
2019 arXiv
-
[26]
The pascal visual object classes (voc) challenge,
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisser- man, “The pascal visual object classes (voc) challenge,” International journal of computer vision , vol. 88, no. 2, pp. 303–338, 2010
2010
-
[27]
Measures of the amount of ecologic association between species,
L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology, vol. 26, no. 3, pp. 297–302, 1945
1945
-
[28]
Watersheds in digital spaces: an efficient algorithm based on immersion simulations,
L. Vincent and P. Soille, “Watersheds in digital spaces: an efficient algorithm based on immersion simulations,” IEEE Transactions on Pattern Analysis & Machine Intelligence , vol. 13, no. 06, pp. 583–598, 1991
1991
-
[29]
Demonstration of an adversarial attack against a multimodal vision language model for pathology imaging,
P. Thota, J. P. Veerla, P. S. Guttikonda, M. S. Nasr, S. Nilizadeh, and J. M. Luber, “Demonstration of an adversarial attack against a multimodal vision language model for pathology imaging,” in 2024 IEEE International Symposium on Biomedical Imaging (ISBI) . IEEE, 2024, pp. 1–5
2024
-
[30]
Cellpose: a generalist algorithm for cellular segmentation,
C. Stringer, T. Wang, M. Michaelos, and M. Pachitariu, “Cellpose: a generalist algorithm for cellular segmentation,” Nature methods, vol. 18, no. 1, pp. 100–106, 2021
2021
-
[31]
Stardist image segmentation improves circulating tumor cell detection,
M. Stevens, A. Nanou, L. W. Terstappen, C. Driemel, N. H. Stoecklein, and F. A. Coumans, “Stardist image segmentation improves circulating tumor cell detection,” Cancers, vol. 14, no. 12, p. 2916, 2022
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.