pith. sign in

arxiv: 2507.05843 · v2 · pith:OQN2QEWYnew · submitted 2025-07-08 · 💻 cs.CV

USIGAN: Unbalanced Self-Information Feature Transport for Weakly Paired Image IHC Virtual Staining

Yue Peng , Bing Xiong , Fuqiang Chen , De Eybo , RanRan Zhang , Wanming Hu , Jing Cai , Wenjian Qin This is my paper

Pith reviewed 2026-05-22 00:00 UTC · model grok-4.3

classification 💻 cs.CV
keywords virtual stainingIHCH&Eweakly pairedunbalanced optimal transportgenerative modelpathological consistencyimage-to-image translation
0
0 comments X

The pith

Removing weakly paired terms from the joint marginal distribution lets virtual staining models preserve pathological semantics even when H&E and IHC slices come from adjacent but non-identical tissue.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces USIGAN, a generative approach for turning H&E images into virtual IHC images when only weakly paired adjacent slices are available. It extracts global morphological semantics without requiring exact positional matches and deliberately drops the weakly paired contributions inside the joint marginal distribution. Two supporting mechanisms, UOT-CTM and PC-SCM, then build correlation matrices at image and intra-group levels to keep content and staining patterns aligned with real adjacent slices. Experiments on two public datasets show gains on clinically relevant scores such as IoD and Pearson-R correlation.

Core claim

By framing virtual staining as unbalanced self-information feature transport and explicitly removing weakly paired terms from the joint marginal distribution, the method constructs correlation matrices that reflect pathological semantics more accurately than standard paired or unpaired baselines, yielding virtual IHC images with improved content consistency and semantic fidelity to adjacent real slices.

What carries the argument

Unbalanced self-information feature transport, which removes weakly paired terms in the joint marginal distribution and uses UOT-CTM and PC-SCM to form correlation matrices between H&E, generated IHC, and real IHC sets.

If this is right

  • Generated virtual IHC images maintain higher structural fidelity to the input H&E morphology.
  • Pathological staining patterns in the output align more closely with those observed in real adjacent IHC slices.
  • The approach reduces the need for perfectly registered training pairs while still delivering clinically usable consistency.
  • Metrics such as IoD and Pearson-R correlation improve relative to prior virtual-staining models on the same data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same removal of mismatched terms could be tested in other medical image-translation settings where exact spatial correspondence is unavailable, such as multi-modal radiology.
  • If the correlation matrices prove robust, the framework might support training on larger but noisier clinical archives without additional registration steps.
  • A direct comparison of the learned matrices against expert-marked semantic regions on held-out slides would provide an independent check on whether global semantics suffice.

Load-bearing premise

Global morphological semantics taken without positional correspondence, once processed by the UOT-CTM and PC-SCM mechanisms, will yield correlation matrices that correctly capture pathological semantics across weakly paired adjacent slices.

What would settle it

Running the method on a new set of weakly paired H&E-IHC slides and finding that the resulting correlation matrices fail to align with independent pathologist annotations of semantic features, or that IoD and Pearson-R scores do not exceed strong baselines, would falsify the central claim.

Figures

Figures reproduced from arXiv: 2507.05843 by Bing Xiong, De Eybo, Fuqiang Chen, Jing Cai, Ranran Zhang, Wanming Hu, Wenjian Qin, Yue Peng.

Figure 1
Figure 1. Figure 1: Spatial heterogeneity between adjacent slices poses a major challenge [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: We leverage weakly paired IHC as an intermediate bridge to ensure global consistency between H&E and IHC while mining image-level self [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The features extracted from H&E images and weakly paired IHC data [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Selected representative methods exhibit varying performances in virtual IHC staining results visualization on the MIST dataset. The quantitative [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Selected representative methods exhibit varying performances in virtual IHC staining results visualization on the IHC4BC dataset. The quantitative [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The visualization results of pathological semantic consistency and content preservation across different methods demonstrate that our approach exhibits [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualize the impact of UOT-CTM and PC-SCM on USIGAN on [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The infulenced different batch size on the Pearson Correlation [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
read the original abstract

Immunohistochemical (IHC) virtual staining is a task that generates virtual IHC images from H\&E images while maintaining pathological semantic consistency with adjacent slices. This task aims to achieve cross-domain mapping between morphological structures and staining patterns through generative models, providing an efficient and cost-effective solution for pathological analysis. However, under weakly paired conditions, spatial heterogeneity between adjacent slices presents significant challenges. This can lead to inaccurate one-to-many mappings and generate results that are inconsistent with the pathological semantics of adjacent slices. To address this issue, we propose a novel unbalanced self-information feature transport for IHC virtual staining, named USIGAN, which extracts global morphological semantics without relying on positional correspondence.By removing weakly paired terms in the joint marginal distribution, we effectively mitigate the impact of weak pairing on joint distributions, thereby significantly improving the content consistency and pathological semantic consistency of the generated results. Moreover, we design the Unbalanced Optimal Transport Consistency (UOT-CTM) mechanism and the Pathology Self-Correspondence (PC-SCM) mechanism to construct correlation matrices between H\&E and generated IHC in image-level and real IHC and generated IHC image sets in intra-group level.. Experiments conducted on two publicly available datasets demonstrate that our method achieves superior performance across multiple clinically significant metrics, such as IoD and Pearson-R correlation, demonstrating better clinical relevance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript presents USIGAN, a generative model for virtual IHC staining from H&E images under weakly paired conditions. It extracts global morphological semantics without positional correspondence, removes weakly paired terms from the joint marginal distribution to mitigate distribution shift, and introduces the UOT-CTM mechanism for unbalanced optimal transport consistency and the PC-SCM mechanism for pathology self-correspondence to construct image-level and intra-group correlation matrices. Experiments on two public datasets report superior performance on clinical metrics such as IoD and Pearson-R correlation.

Significance. If the central claims hold, the approach could advance virtual staining methods for pathology by handling common spatial heterogeneity in adjacent slices without requiring perfect pairings, potentially improving content and semantic consistency in generated images. The combination of unbalanced transport with self-information features offers a targeted response to weak pairing challenges.

major comments (3)
  1. Abstract and Method section: The central claim that 'removing weakly paired terms in the joint marginal distribution' mitigates impact and improves pathological semantic consistency lacks an explicit equation or derivation showing the modified joint distribution and the resulting transport plan; without this, it is unclear whether the removal preserves necessary information or simply relaxes the problem by construction.
  2. Method (UOT-CTM and PC-SCM): The mechanisms are described as constructing correlation matrices between H&E/generated IHC (image-level) and real/generated IHC (intra-group), but no details are given on how global morphological semantics (extracted without positional correspondence) are mapped to local pathological variations; this directly engages the concern that global features may align tissue type while mismatching cellular patterns relevant to IoD.
  3. Experiments: Superior IoD and Pearson-R results are reported, yet the manuscript does not include ablations isolating the contribution of term removal or the two mechanisms, nor statistical tests confirming that gains exceed those from standard unpaired baselines; this weakens attribution of improvements to the proposed components.
minor comments (2)
  1. Abstract: the sentence describing PC-SCM ends abruptly ('in intra-group level..'); complete the description for clarity.
  2. Notation: define acronyms UOT-CTM and PC-SCM at first use and ensure consistent use of 'self-information' versus 'self-correspondence' throughout.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thorough review and valuable feedback on our manuscript. We address each of the major comments in detail below, providing clarifications and indicating the revisions we plan to make to strengthen the paper.

read point-by-point responses

  1. Referee: Abstract and Method section: The central claim that 'removing weakly paired terms in the joint marginal distribution' mitigates impact and improves pathological semantic consistency lacks an explicit equation or derivation showing the modified joint distribution and the resulting transport plan; without this, it is unclear whether the removal preserves necessary information or simply relaxes the problem by construction.

    Authors: We appreciate this observation, as it points to a potential gap in the mathematical presentation. The manuscript describes the process of removing weakly paired terms from the joint marginal distribution to mitigate distribution shift caused by spatial heterogeneity. This is achieved by identifying and excluding terms with low correspondence scores computed from the self-information features. However, to make this clearer, we will add an explicit mathematical formulation of the original and modified joint distributions, along with the derivation of the resulting unbalanced transport plan, in the revised Method section. This will demonstrate that the removal selectively filters unreliable pairings while preserving the overall structure necessary for consistent mapping. revision: yes

  2. Referee: Method (UOT-CTM and PC-SCM): The mechanisms are described as constructing correlation matrices between H&E/generated IHC (image-level) and real/generated IHC (intra-group), but no details are given on how global morphological semantics (extracted without positional correspondence) are mapped to local pathological variations; this directly engages the concern that global features may align tissue type while mismatching cellular patterns relevant to IoD.

    Authors: Thank you for raising this important point regarding the connection between global and local features. In USIGAN, the global morphological semantics are captured through a feature extractor that operates on the entire image without assuming positional alignment. These global features inform the construction of the correlation matrices in UOT-CTM for image-level consistency between H&E and generated IHC, and in PC-SCM for intra-group correspondence between real and generated IHC. The unbalanced optimal transport then allows the model to adaptively align local pathological variations by permitting mass creation or destruction in the transport plan, which helps in handling mismatches at the cellular level while respecting the global semantics. We will revise the Method section to include a more detailed explanation of this mapping process, potentially with an illustrative figure showing the flow from global features to local correlation adjustments. revision: yes

  3. Referee: Experiments: Superior IoD and Pearson-R results are reported, yet the manuscript does not include ablations isolating the contribution of term removal or the two mechanisms, nor statistical tests confirming that gains exceed those from standard unpaired baselines; this weakens attribution of improvements to the proposed components.

    Authors: We agree that additional experiments would better attribute the performance gains to the specific components of our method. The current evaluation compares USIGAN against several state-of-the-art unpaired and weakly paired baselines on two public datasets, showing improvements in IoD and Pearson-R. To address this, we will conduct ablation studies by disabling the weakly paired term removal, UOT-CTM, and PC-SCM one at a time, and report the resulting metrics. Additionally, we will include statistical tests, such as paired t-tests, to confirm the significance of the improvements over the baselines. These results will be added to the Experiments section in the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on proposed mechanisms rather than self-referential reduction.

full rationale

The paper introduces USIGAN with novel components UOT-CTM and PC-SCM to address weakly paired IHC staining by extracting global morphological semantics and removing weakly paired terms from the joint marginal distribution. No equations or self-citations are shown that reduce the central consistency claims to fitted inputs or prior author results by construction. The approach is presented as a new transport-based solution with independent content for improving semantic consistency, making the derivation self-contained against external benchmarks like IoD and Pearson-R on public datasets.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 3 invented entities

Only the abstract is available, so the ledger is necessarily incomplete; typical GAN-style training likely involves several free parameters whose values are not reported here.

free parameters (1)
  • transport regularization parameter
    Likely present in the unbalanced optimal transport formulation but not quantified in the abstract.
axioms (1)
  • domain assumption Global morphological semantics can be extracted independently of positional correspondence in adjacent tissue slices.
    Invoked when the method removes weakly paired terms from the joint marginal distribution.
invented entities (3)
  • Unbalanced Self-Information Feature Transport no independent evidence
    purpose: Core mapping mechanism that avoids reliance on spatial alignment.
    Introduced as the central novel component of USIGAN.
  • UOT-CTM mechanism no independent evidence
    purpose: Constructs image-level correlation matrix between H&E and generated IHC.
    Newly designed consistency module.
  • PC-SCM mechanism no independent evidence
    purpose: Constructs intra-group correlation between real and generated IHC.
    Newly designed consistency module.

pith-pipeline@v0.9.0 · 5790 in / 1439 out tokens · 43822 ms · 2026-05-22T00:00:19.606251+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 1 internal anchor

  1. [1]

    Pathological semantics-preserving learning for H&E-to-IHC virtual staining

    Fuqiang Chen et al. “Pathological semantics-preserving learning for H&E-to-IHC virtual staining”. In: In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention . Springer. 2024, pp. 384–394

    work page 2024

  2. [2]

    Scaling Algorithms for Unbalanced Transport Problems

    Lenaic Chizat et al. “Scaling algorithms for un- balanced transport problems”. In: arXiv preprint arXiv:1607.05816 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  3. [3]

    Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning

    B DoanNgan, DarrowMorgan Angus, LeeHan Sung, et al. “Label-free virtual HER2 immunohistochemical staining of breast tissue using deep learning”. In: BME frontiers (2022)

    work page 2022

  4. [4]

    Contrastive learning improves model robustness under label noise

    Aritra Ghosh and Andrew Lan. “Contrastive learning improves model robustness under label noise”. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2021, pp. 2703–2708

    work page 2021

  5. [5]

    Supervised Information Mining from Weakly Paired Images for Breast IHC Virtual Staining

    Xianchao Guan et al. “Supervised Information Mining from Weakly Paired Images for Breast IHC Virtual Staining”. In: IEEE Transactions on Medical Imaging (2025)

    work page 2025

  6. [6]

    Deep learning-based transformation of H&E stained tissues into special stains

    K de Haan et al. Deep learning-based transformation of H&E stained tissues into special stains. Nat Commun. 2021; 12: 4884

    work page 2021

  7. [7]

    Qs-attn: Query-selected attention for contrastive learning in i2i translation

    Xueqi Hu et al. “Qs-attn: Query-selected attention for contrastive learning in i2i translation”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 18291–18300

    work page 2022

  8. [8]

    Image-to-image translation with conditional adversarial networks

    Phillip Isola et al. “Image-to-image translation with conditional adversarial networks”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2017, pp. 1125–1134

    work page 2017

  9. [9]

    Ex- ploring Patch-Wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Tasks

    Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. “Ex- ploring Patch-Wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Tasks”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . June 2022, pp. 18260–18269

    work page 2022

  10. [10]

    Exploring patch-wise semantic relation for contrastive learning in image-to-image translation tasks

    Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. “Exploring patch-wise semantic relation for contrastive learning in image-to-image translation tasks”. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 18260–18269

    work page 2022

  11. [11]

    Unpaired Image-to-Image Transla- tion via Neural Schr ¨odinger Bridge

    Beomsu Kim et al. Unpaired Image-to-Image Transla- tion via Neural Schr ¨odinger Bridge. 2024. arXiv: 2305. 15086 [cs.CV]. URL: https : / / arxiv. org / abs / 2305 . 15086

    work page 2024

  12. [12]

    Adaptive supervised patchnce loss for learning h&e-to-ihc stain translation with inconsistent groundtruth image pairs

    Fangda Li et al. “Adaptive supervised patchnce loss for learning h&e-to-ihc stain translation with inconsistent groundtruth image pairs”. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer. 2023, pp. 632–641. JOURNAL OF IEEE TRANS, VOL. 18, NO. 9, JULY 2025 13

    work page 2023

  13. [13]

    High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers

    Jia-Ren Lin et al. “High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers”. In: Nature cancer 4.7 (2023), pp. 1036–1052

    work page 2023

  14. [14]

    Xiaoqi Lin. “Characteristic morphology and immuno- histochemical patterns of clear cell papillary renal cell tumours may be observed in renal cell carcinomas, a critical pitfall in renal biopsy cytopathology”. In: Cytopathology 35.4 (2024), pp. 481–487

    work page 2024

  15. [15]

    Bci: Breast cancer immunohisto- chemical image generation through pyramid pix2pix

    Shengjie Liu et al. “Bci: Breast cancer immunohisto- chemical image generation through pyramid pix2pix”. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition . 2022, pp. 1815– 1824

    work page 2022

  16. [16]

    Unpaired stain transfer using pathology-consistent constrained generative adversarial networks

    Shuting Liu et al. “Unpaired stain transfer using pathology-consistent constrained generative adversarial networks”. In: IEEE transactions on medical imaging 40.8 (2021), pp. 1977–1989

    work page 2021

  17. [17]

    Contrastive learning for unpaired image-to-image translation

    Taesung Park et al. “Contrastive learning for unpaired image-to-image translation”. In: Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16. Springer. 2020, pp. 319–345

    work page 2020

  18. [18]

    Advancing H&E-to-IHC Virtual Staining with Task-Specific Domain Knowledge for HER2 Scoring

    Qiong Peng et al. “Advancing H&E-to-IHC Virtual Staining with Task-Specific Domain Knowledge for HER2 Scoring”. In: International Conference on Medi- cal Image Computing and Computer-Assisted Interven- tion. Springer. 2024, pp. 3–13

    work page 2024

  19. [19]

    Virtual histological staining of un- labelled tissue-autofluorescence images via deep learn- ing

    Yair Rivenson et al. “Virtual histological staining of un- labelled tissue-autofluorescence images via deep learn- ing”. In: Nature biomedical engineering 3.6 (2019), pp. 466–477

    work page 2019

  20. [20]

    Quan- tification of histochemical staining by color deconvo- lution

    Arnout C Ruifrok, Dennis A Johnston, et al. “Quan- tification of histochemical staining by color deconvo- lution”. In: Analytical and quantitative cytology and histology 23.4 (2001), pp. 291–299

    work page 2001

  21. [21]

    Nonlinear imaging histopathology: a pipeline to correlate gold-standard hematoxylin and eosin stain- ing with modern nonlinear microscopy

    Tehrani. “Nonlinear imaging histopathology: a pipeline to correlate gold-standard hematoxylin and eosin stain- ing with modern nonlinear microscopy”. In: IEEE Jour- nal of Selected Topics in Quantum Electronics 29.4: Biophotonics (2023), pp. 1–8

    work page 2023

  22. [22]

    IHC Profiler: an open source plugin for the quantitative evaluation and automated scoring of immunohistochemistry images of human tissue samples

    Frency Varghese et al. “IHC Profiler: an open source plugin for the quantitative evaluation and automated scoring of immunohistochemistry images of human tissue samples”. In: PloS one 9.5 (2014), e96801

    work page 2014

  23. [23]

    High-resolution image synthe- sis and semantic manipulation with conditional gans

    Ting-Chun Wang et al. “High-resolution image synthe- sis and semantic manipulation with conditional gans”. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2018, pp. 8798–8807

    work page 2018

  24. [24]

    Instance-wise hard negative ex- ample generation for contrastive learning in unpaired image-to-image translation

    Weilun Wang et al. “Instance-wise hard negative ex- ample generation for contrastive learning in unpaired image-to-image translation”. In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 14020–14029

    work page 2021

  25. [25]

    Unpaired Multi-Domain Histopathol- ogy Virtual Staining using Dual Path Prompted In- version

    Bing Xiong et al. “Unpaired Multi-Domain Histopathol- ogy Virtual Staining using Dual Path Prompted In- version”. In: Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 39. 8. 2025, pp. 8780–8787

    work page 2025

  26. [26]

    Modulated contrast for versatile image synthesis

    Fangneng Zhan et al. “Modulated contrast for versatile image synthesis”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. 2022, pp. 18280–18290

    work page 2022

  27. [27]

    Unbalanced feature transport for exemplar-based image translation

    Fangneng Zhan et al. “Unbalanced feature transport for exemplar-based image translation”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 15028–15038

    work page 2021

  28. [28]

    MVFStain: multiple virtual func- tional stain histopathology images generation based on specific domain mapping

    Ranran Zhang et al. “MVFStain: multiple virtual func- tional stain histopathology images generation based on specific domain mapping”. In: Medical Image Analysis 80 (2022), p. 102520

    work page 2022

  29. [29]

    High-Resolution Medical Image Translation via Patch Alignment-Based Bidirectional Contrastive Learning

    Wei Zhang et al. “High-Resolution Medical Image Translation via Patch Alignment-Based Bidirectional Contrastive Learning”. In: International Conference on Medical Image Computing and Computer-Assisted In- tervention. Springer. 2024, pp. 178–188

    work page 2024

  30. [30]

    The Spatially-Correlative Loss for Various Image Transla- tion Tasks

    Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The Spatially-Correlative Loss for Various Image Transla- tion Tasks. 2021. arXiv: 2104.00854 [cs.CV]. URL: https://arxiv.org/abs/2104.00854

    work page arXiv 2021

  31. [31]

    Unpaired image-to-image transla- tion using cycle-consistent adversarial networks

    Jun-Yan Zhu et al. “Unpaired image-to-image transla- tion using cycle-consistent adversarial networks”. In: Proceedings of the IEEE international conference on computer vision. 2017, pp. 2223–2232

    work page 2017