pith. sign in

arxiv: 1907.03728 · v1 · pith:ADYZHNUTnew · submitted 2019-07-08 · 💻 cs.CV · eess.IV

Correlation via synthesis: end-to-end nodule image generation and radiogenomic map learning based on generative adversarial network

Ziyue Xu , Xiaosong Wang , Hoo-Chang Shin , Dong Yang , Holger Roth , Fausto Milletari , Ling Zhang , Daguang Xu This is my paper

Pith reviewed 2026-05-25 00:57 UTC · model grok-4.3

classification 💻 cs.CV eess.IV
keywords generative adversarial networkradiogenomicsimage synthesisnon-small cell lung cancerend-to-end learninggene expression profilesnodule images
0
0 comments X

The pith

Conditioning a GAN on gene expression profiles generates realistic nodule images while learning their radiogenomic map in one process.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes an end-to-end generative adversarial network that fuses gene expression data with image features at multiple scales to synthesize corresponding nodule images from background inputs and gene profiles. This replaces the standard three-step pipeline of separate gene clustering into metagenes, independent image feature extraction, and later statistical correlation. A sympathetic reader would care because the approach avoids arbitrary choices in each isolated step and directly ties gene data to image synthesis. Results on a non-small cell lung cancer dataset show the generated images appear realistic.

Core claim

A generative adversarial network conditioned on both background images and gene expression profiles synthesizes the corresponding nodule image by fusing image and gene features at different scales, learning the radiogenomic map simultaneously rather than through independent clustering, extraction, and correlation steps.

What carries the argument

Multi-scale fusion of gene expression profiles into the conditional GAN generator to control synthesis and embed the gene-image relationship.

If this is right

  • Realistic synthetic nodule images can be produced directly from gene expression inputs.
  • Gene-image relationships emerge from the joint training without requiring metagenes or separate statistical tests.
  • Multi-scale feature fusion maintains image quality while embedding the conditioning information.
  • The method offers a single model for both image generation and radiogenomic mapping on NSCLC data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same conditioning strategy could be tested on other imaging modalities or cancer types to see if the end-to-end property holds.
  • Generated images conditioned on specific gene profiles might serve as data augmentation for downstream classification tasks.
  • If the internal map proves stable, it could support prediction of gene expression levels from new patient images alone.

Load-bearing premise

Conditioning the GAN on gene profiles at multiple scales will produce realistic images and a meaningful radiogenomic map without separate clustering or post-hoc correlation steps.

What would settle it

If expert radiologists cannot distinguish the synthetic images from real ones at rates above chance, or if the learned associations fail to align with independent gene validation data on held-out cases.

Figures

Figures reproduced from arXiv: 1907.03728 by Daguang Xu, Dong Yang, Fausto Milletari, Holger Roth, Hoo-Chang Shin, Ling Zhang, Xiaosong Wang, Ziyue Xu.

Figure 1
Figure 1. Figure 1: Proposed multi-conditional GAN for radiogenomic map learning and nodule synthesis. (a) Generator utilizes both background image and gene code to synthesize image together with nodule segmentation. (b) Fusion block at each resolution layer helps to fuse the information from background with that from previous layer and gene code. (c) With image, segmentation, and gene code, discriminator distinguishes three … view at source ↗
Figure 2
Figure 2. Figure 2: Examples of proposed synthesis GAN: (a, e) background image, (b, f) synthesized nodule image, (c, g) background weight image, (d, h) segmentation mask. Fig.2 shows two examples (a-d) and (e-h) for the proposed GAN. (a) is the background image, (b) is the synthesis result, (c) is the background weight map, and (d) is the resulting mask. (e-h) is the same as (a-d) but for another case with ground-glass opaci… view at source ↗
Figure 3
Figure 3. Figure 3: Result of nodule synthesis, first row: training image, whose genomic information is used to synthesize each column; second row: background image; third row: synthetic image generated by baseline method [9]; last tow: synthetic image generated by the proposed method [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of gene coding illustrated by 2D t-SNE map [7] : raw gene (5172-D) and gene code produced by baseline method (128-D) does not show obvious separation, while gene code produced by the proposed method (128-D) showed feasibility for clustering. Three groups of samples are drawn from clusters formed according to distance, and their corresponding image are shown [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
read the original abstract

Radiogenomic map linking image features and gene expression profiles is useful for noninvasively identifying molecular properties of a particular type of disease. Conventionally, such map is produced in three separate steps: 1) gene-clustering to "metagenes", 2) image feature extraction, and 3) statistical correlation between metagenes and image features. Each step is independently performed and relies on arbitrary measurements. In this work, we investigate the potential of an end-to-end method fusing gene data with image features to generate synthetic image and learn radiogenomic map simultaneously. To achieve this goal, we develop a generative adversarial network (GAN) conditioned on both background images and gene expression profiles, synthesizing the corresponding image. Image and gene features are fused at different scales to ensure the realism and quality of the synthesized image. We tested our method on non-small cell lung cancer (NSCLC) dataset. Results demonstrate that the proposed method produces realistic synthetic images, and provides a promising way to find gene-image relationship in a holistic end-to-end manner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper proposes an end-to-end GAN architecture conditioned on both background images and multi-scale gene expression profiles to simultaneously synthesize realistic nodule images and learn a radiogenomic map, avoiding the conventional three-step pipeline of gene clustering, feature extraction, and post-hoc correlation. The method is evaluated on an NSCLC dataset, with the abstract asserting that it produces realistic synthetic images and offers a promising holistic approach to gene-image relationships.

Significance. If the central claim holds with proper validation, the work could demonstrate that multi-scale gene conditioning in a GAN can yield both high-quality image synthesis and a biologically meaningful radiogenomic map without separate clustering or statistical post-processing steps. This would represent a substantive advance over conventional pipelines by reducing arbitrary measurement choices, provided the map component is shown to be non-incidental via ablation or correlation metrics.

major comments (2)
  1. [Abstract] Abstract: the claim that the method 'produces realistic synthetic images' is unsupported because no quantitative metrics (e.g., FID, PSNR, SSIM), baselines, error bars, or validation protocol are reported, preventing any assessment of whether the generator actually succeeds or merely produces plausible outputs while ignoring gene inputs.
  2. [Abstract] Abstract, paragraph 2: the assertion of a 'promising way to find gene-image relationship in a holistic end-to-end manner' lacks any description of how the radiogenomic map is extracted from the trained model, any correlation scores, ablation of the gene-conditioning input, or comparison against the conventional three-step method; without these, the map could be a spurious byproduct of the GAN objective rather than an independent, meaningful output.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We agree that the abstract claims require stronger quantitative support and explicit validation of the radiogenomic map component. Below we address each major comment and outline the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that the method 'produces realistic synthetic images' is unsupported because no quantitative metrics (e.g., FID, PSNR, SSIM), baselines, error bars, or validation protocol are reported, preventing any assessment of whether the generator actually succeeds or merely produces plausible outputs while ignoring gene inputs.

    Authors: We acknowledge that the current abstract relies primarily on qualitative visual results presented in the manuscript. To address this, we will revise the abstract to reference quantitative metrics (including FID scores against real images, with error bars over multiple runs) and will add a dedicated evaluation subsection describing the validation protocol, baselines (e.g., unconditional GAN and standard cGAN variants), and comparison results. These additions will be incorporated in the revised manuscript. revision: yes

  2. Referee: [Abstract] Abstract, paragraph 2: the assertion of a 'promising way to find gene-image relationship in a holistic end-to-end manner' lacks any description of how the radiogenomic map is extracted from the trained model, any correlation scores, ablation of the gene-conditioning input, or comparison against the conventional three-step method; without these, the map could be a spurious byproduct of the GAN objective rather than an independent, meaningful output.

    Authors: The radiogenomic map arises directly from the multi-scale gene-conditioning mechanism, which modulates image synthesis at different resolutions. We agree that explicit validation is needed and will add: (1) an ablation study that removes or randomizes the gene input and quantifies the resulting drop in image quality and feature alignment; (2) post-training extraction of correlation scores between gene expression vectors and synthesized image features; and (3) a brief comparison to a conventional three-step pipeline on the same NSCLC data. These elements will be described in the methods/results and referenced in the revised abstract. revision: yes

Circularity Check

0 steps flagged

No circularity: standard conditional GAN with multi-scale fusion; radiogenomic map is an emergent property of conditioning, not a fitted input renamed as output.

full rationale

The paper describes a conditional GAN architecture that fuses gene expression profiles with background images at multiple scales during synthesis. The radiogenomic map arises directly from the learned conditioning mechanism rather than from any post-training extraction that reduces to the training objective by construction. No equations, fitted parameters, or self-citations are presented that would make the map equivalent to its inputs. The method is self-contained as a generative modeling approach; image realism is the primary reported outcome, with the map-learning aspect positioned as a byproduct of the end-to-end training without circular reduction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no identifiable free parameters, axioms, or invented entities.

pith-pipeline@v0.9.0 · 5744 in / 1175 out tokens · 25418 ms · 2026-05-25T00:57:57.427888+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · 3 internal anchors

  1. [1]

    Bakr, S., Gevaert, O., Echegaray, S., Ayers, K., Zhou, M., Shafiq, M., Zheng, H., Zhang, W., Leung, A., Kadoch, M., Shrager, J., Quon, A., Rubin, D., Plevritis, S., Napel, S.: Data for NSCLC Radiogenomics Collection (2017), the Cancer Imaging Archive

    work page 2017

  2. [2]

    Proceedings of the National Academy of Sciences 105(13), 5213–5218 (2008)

    Diehn, M., Nardini, C., Wang, D.S., McGovern, S., Jayaraman, M., Liang, Y., Aldape, K., Cha, S., Kuo, M.D.: Identification of noninvasive imaging surrogates for brain tumor gene-expression modules. Proceedings of the National Academy of Sciences 105(13), 5213–5218 (2008)

    work page 2008

  3. [3]

    Radiology 264(2), 387–396 (2012)

    Gevaert, O., Xu, J., Hoang, C.D., Leung, A.N., Xu, Y., Quon, A., Rubin, D.L., Napel, S., Plevritis, S.K.: NonSmall Cell Lung Cancer: Identifying Prognostic Imag- ing Biomarkers by Leveraging Public Gene Expression Microarray DataMethods and Preliminary Results. Radiology 264(2), 387–396 (2012)

    work page 2012

  4. [4]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018

    Jin, D., Xu, Z., Tang, Y., Harrison, A.P., Mollura, D.J.: CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. pp. 732–740. Springer International Publishing, Cham (2018)

    work page 2018

  5. [5]

    A Style-Based Generator Architecture for Generative Adversarial Networks

    Karras, T., Laine, S., Aila, T.: A Style-Based Generator Architecture for Genera- tive Adversarial Networks. CoRR abs/1812.04948 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  6. [6]

    Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition

    Liu, S., Gibson, E., Grbic, S., Xu, Z., Setio, A.A.A., Yang, J., Georgescu, B., Co- maniciu, D.: Decompose to manipulate: Manipulable Object Synthesis in 3D Med- ical Images with Structured Image Decomposition. CoRR abs/1812.01737 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  7. [7]

    Journal of Machine Learning Research 9, 2579–2605 (Nov 2008)

    van der Maaten, L., Hinton, G.: Visualizing High-Dimensional Data Using t-SNE. Journal of Machine Learning Research 9, 2579–2605 (Nov 2008)

    work page 2008

  8. [8]

    In: 2017 IEEE International Conference on Com- puter Vision (ICCV)

    Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least Squares Gen- erative Adversarial Networks. In: 2017 IEEE International Conference on Com- puter Vision (ICCV). pp. 2813–2821 (Oct 2017)

    work page 2017

  9. [9]

    In: The British MachineVision Conference (BMVC) (2018)

    Park, H., Yoo, Y., Kwak, N.: MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis. In: The British MachineVision Conference (BMVC) (2018)

    work page 2018

  10. [10]

    IEEE Transactions on Medical Imaging 35(5), 1285–1298 (May 2016)

    Shin, H., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R.M.: Deep Convolutional Neural Networks for Computer-Aided De- tection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE Transactions on Medical Imaging 35(5), 1285–1298 (May 2016)

    work page 2016

  11. [11]

    Class-Aware Adversarial Lung Nodule Synthesis in CT Images

    Yang, J., Liu, S., Grbic, S., Setio, A.A.A., Xu, Z., Gibson, E., Chabin, G., Georgescu, B., Laine, A.F., Comaniciu, D.: Class-Aware Adversarial Lung Nodule Synthesis in CT Images. CoRR abs/1812.11204 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  12. [12]

    In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)

    Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: Stack- GAN: Text to Photo-Realistic Image Synthesis With Stacked Generative Adversar- ial Networks. In: The IEEE International Conference on Computer Vision (ICCV) (Oct 2017)

    work page 2017

  13. [13]

    Radiology 286(1), 307–315 (2018)

    Zhou, M., Leung, A., Echegaray, S., Gentles, A., Shrager, J.B., Jensen, K.C., Berry, G.J., Plevritis, S.K., Rubin, D.L., Napel, S., Gevaert, O.: NonSmall Cell Lung Can- cer Radiogenomics Map Identifies Relationships between Molecular and Imaging Phenotypes with Prognostic Implications. Radiology 286(1), 307–315 (2018)

    work page 2018