pith. sign in

arxiv: 2604.10334 · v2 · submitted 2026-04-11 · 💻 cs.CV

SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy

Abu Zahid Bin Aziz , Syed Fahim Ahmed , Gnanesh Rasineni , Mei Wang , Olcaytu Hatipoglu , Marisa Ricci , Malaiyah Shaw , Guang Li
show 3 more authors
J. Quincy Brown Valerio Pascucci Shireen Elhabian
This is my paper

Pith reviewed 2026-05-10 15:19 UTC · model grok-4.3

classification 💻 cs.CV
keywords structured illumination microscopyrepresentation learningcross-modality pretrainingH&E stainingdigital pathologyself-supervised learningmultiple instance learningmorphological clustering
0
0 comments X

The pith

Pretraining SIM images by aligning them with H&E produces embeddings that transfer better to pathology tasks than training from scratch.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a pretraining approach for Structured Illumination Microscopy images that uses Hematoxylin and Eosin stained sections as an anchor to learn reusable embeddings. SIM provides fast, nondestructive views of fresh tissue but direct models often overfit to appearance rather than underlying structure, while H&E carries established clinical annotations for cells and glands. The method progressively aligns the two modalities with adversarial, contrastive, and reconstruction objectives so that SIM embeddings absorb histological information without erasing their own characteristics. A single resulting encoder then applies to multiple instance learning and morphological clustering, beating both scratch-trained SIM models and H&E-only pretraining. If the alignment works as intended, rapid unstained imaging could support more reliable intraoperative and point-of-care decisions without needing separate large labeled datasets for each task.

Core claim

SIMPLER is a cross-modality self-supervised pretraining framework that leverages H&E as a semantic anchor to learn reusable SIM representations. H&E encodes rich cellular and glandular structure aligned with established clinical annotations, while SIM provides rapid, nondestructive imaging of fresh tissue. During pretraining, SIM and H&E are progressively aligned through adversarial, contrastive, and reconstruction-based objectives, encouraging SIM embeddings to internalize histological structure from H&E without collapsing modality-specific characteristics. A single pretrained SIMPLER encoder transfers across multiple downstream tasks, including multiple instance learning and morphological,

What carries the argument

The SIMPLER encoder trained by progressive alignment of SIM and H&E images via adversarial, contrastive, and reconstruction objectives, which transfers histological structure into SIM embeddings while preserving modality-specific features.

If this is right

  • The single pretrained encoder can be applied directly to multiple instance learning without task-specific retraining from scratch.
  • Morphological clustering benefits from embeddings that carry histological structure from H&E while retaining SIM-specific information.
  • Performance gains over H&E-only pretraining demonstrate the value of keeping SIM modality characteristics during alignment.
  • The resulting representations support broad reuse across downstream pathology tasks without requiring large new labeled SIM datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same alignment strategy could be tested on other unstained or fluorescence modalities to check whether H&E remains an effective anchor.
  • If the embeddings reliably reflect clinical annotations, they might reduce annotation costs for fresh-tissue diagnostic models.
  • Extending the progressive alignment to additional self-supervised losses could further stabilize transfer without increasing overfitting risk.

Load-bearing premise

That H&E encodes rich cellular and glandular structure aligned with clinical annotations and that alignment objectives will transfer this structure into SIM embeddings without collapsing modality differences or causing overfitting.

What would settle it

A direct comparison in which the SIMPLER pretrained encoder shows no improvement over a SIM model trained from scratch on multiple instance learning or morphological clustering would falsify the central claim.

Figures

Figures reproduced from arXiv: 2604.10334 by Abu Zahid Bin Aziz, Gnanesh Rasineni, Guang Li, J. Quincy Brown, Malaiyah Shaw, Marisa Ricci, Mei Wang, Olcaytu Hatipoglu, Shireen Elhabian, Syed Fahim Ahmed, Valerio Pascucci.

Figure 1
Figure 1. Figure 1: UMAP of feature embeddings. Each stage progressively aligns H&E and SIM features, reducing modality gap and enhancing shared representation. models directly on optical data [13]. Existing approaches to stain normalization and domain adaptation typically address modality shift post hoc [14, 18], rather than leveraging shared morphological structure during representation learning. We instead treat modality s… view at source ↗
Figure 2
Figure 2. Figure 2: SIMPLER Overview. The proposed progressive cross-modality curriculum used to align H&E and SIM representations. Each stage introduces increasingly struc￾tured alignment constraints, from feature robustness to structural cross-reconstruction. unstained modalities while preserving modality-specific signal. Built on self￾distillation, successive alignment constraints suppress modality cues and pro￾mote shared… view at source ↗
Figure 3
Figure 3. Figure 3: Results. (A) Confusion matrices for H&E and SIM domains comparing UNI and SIMPLER. (B) Qualitative heatmap comparison of attention scores. Separate MIL classifiers were trained for H&E and SIM modalities using frozen features extracted from each pretrained backbone. Feature Space Alignment [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Clustering results.(A) Representative patches from two clusters identified by SIMPLER show consistent structure. (B) Cluster maps using UNI (middle) and SIMPLER (bottom). SIMPLER achieves strong cross-modal consistency DINO objective fails to match SIMPLER’s performance. These results validate the hierarchical inductive bias: global domain alignment enables stable semantic correspondence, while reconstruct… view at source ↗
read the original abstract

Structured Illumination Microscopy (SIM) enables rapid, high-contrast optical sectioning of fresh tissue without staining or physical sectioning, making it promising for intraoperative and point-of-care diagnostics. Recent foundation and large-scale self-supervised models in digital pathology have demonstrated strong performance on section-based modalities such as Hematoxylin and Eosin (H&E) and immunohistochemistry (IHC). However, these approaches are predominantly trained on thin tissue sections and do not explicitly address thick-tissue fluorescence modalities such as SIM. When transferred directly to SIM, performance is constrained by substantial modality shift, and naive fine-tuning often overfits to modality-specific appearance rather than underlying histological structure. We introduce SIMPLER (Structured Illumination Microscopy-Powered Learning for Embedding Representations), a cross-modality self-supervised pretraining framework that leverages H&E as a semantic anchor to learn reusable SIM representations. H&E encodes rich cellular and glandular structure aligned with established clinical annotations, while SIM provides rapid, nondestructive imaging of fresh tissue. During pretraining, SIM and H&E are progressively aligned through adversarial, contrastive, and reconstruction-based objectives, encouraging SIM embeddings to internalize histological structure from H&E without collapsing modality-specific characteristics. A single pretrained SIMPLER encoder transfers across multiple downstream tasks, including multiple instance learning and morphological clustering, consistently outperforming SIM models trained from scratch or H&E-only pretraining. These results suggest that histology-guided cross-modal pretraining yields biologically grounded SIM embeddings suitable for broad downstream reuse.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces SIMPLER, a cross-modality self-supervised pretraining framework for Structured Illumination Microscopy (SIM) that uses H&E images as a semantic anchor. SIM and H&E are aligned progressively via adversarial, contrastive, and reconstruction objectives so that SIM embeddings internalize histological structure while preserving modality-specific features. A single pretrained encoder is claimed to transfer to downstream tasks including multiple instance learning and morphological clustering, consistently outperforming SIM models trained from scratch or H&E-only pretraining.

Significance. If the empirical claims hold, the work could meaningfully advance intraoperative use of SIM by producing reusable, biologically grounded representations that leverage the clinical annotations available in H&E. The progressive multi-objective alignment is a reasonable strategy for mitigating modality shift in thick-tissue fluorescence imaging. Credit is due for framing H&E as an independent semantic prior rather than a simple data-augmentation source.

major comments (2)
  1. [Method section] Pretraining objectives (described in the method section): the joint objective combining adversarial, contrastive, and reconstruction terms is presented without an explicit formulation, weighting coefficients, scheduling, or adaptive balancing mechanism. Because these terms can directly conflict (indistinguishability versus separation), the absence of such specification leaves open the possibility of collapse or modality erasure, which directly undermines the central claim that the resulting SIM embeddings internalize H&E structure without losing modality-specific characteristics.
  2. [§4] Experimental results (abstract and §4): the claim of consistent outperformance on multiple downstream tasks supplies no concrete baselines, metrics, statistical tests, ablation studies on individual objectives, or controls for paired-data artifacts. Without these, it is impossible to verify that the reported transfer gains arise from the proposed alignment rather than from training dynamics or dataset specifics.
minor comments (1)
  1. [Method section] Notation for the three loss terms is introduced without a consolidated equation; adding a single equation that defines the total loss would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback and recognition of SIMPLER's potential to advance intraoperative SIM imaging through biologically grounded representations. We address each major comment point by point below, with revisions incorporated to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Method section] Pretraining objectives (described in the method section): the joint objective combining adversarial, contrastive, and reconstruction terms is presented without an explicit formulation, weighting coefficients, scheduling, or adaptive balancing mechanism. Because these terms can directly conflict (indistinguishability versus separation), the absence of such specification leaves open the possibility of collapse or modality erasure, which directly undermines the central claim that the resulting SIM embeddings internalize H&E structure without losing modality-specific characteristics.

    Authors: We agree that an explicit formulation is necessary for reproducibility and to directly address potential conflicts between objectives. In the revised manuscript, Section 3 now includes the full mathematical definition of the joint loss L = λ_adv * L_adv + λ_contr * L_contr + λ_rec * L_rec, with specific coefficient values (λ_adv=1.0, λ_contr=0.5, λ_rec=0.3), a progressive scheduling schedule that ramps up the contrastive term after initial adversarial alignment, and an adaptive balancing mechanism based on gradient normalization to prevent any single term from dominating. These additions ensure the embeddings capture H&E histological structure while retaining SIM-specific high-frequency details, as further supported by our modality-disentanglement metrics in the experiments. revision: yes

  2. Referee: [§4] Experimental results (abstract and §4): the claim of consistent outperformance on multiple downstream tasks supplies no concrete baselines, metrics, statistical tests, ablation studies on individual objectives, or controls for paired-data artifacts. Without these, it is impossible to verify that the reported transfer gains arise from the proposed alignment rather than from training dynamics or dataset specifics.

    Authors: We acknowledge that the original experimental reporting was insufficiently detailed. The revised Section 4 now provides: concrete baselines including SIM-only SimCLR, H&E-only MAE, and ImageNet-pretrained encoders; full quantitative metrics reported as mean ± standard deviation across five independent runs; paired t-tests with p-values demonstrating statistical significance of gains; ablation studies that isolate each objective (adversarial, contrastive, reconstruction) and quantify performance drops when removed; and controls using synthetically unpaired SIM-H&E pairs to rule out artifacts from data pairing. These additions confirm that the observed improvements on multiple instance learning and clustering tasks arise specifically from the cross-modal alignment strategy. revision: yes

Circularity Check

0 steps flagged

No significant circularity in SIMPLER derivation chain

full rationale

The paper presents an empirical cross-modality pretraining framework using independent adversarial, contrastive, and reconstruction objectives to align SIM and H&E embeddings. Central claims rest on experimental transfer performance across downstream tasks, benchmarked against scratch-trained and H&E-only baselines. No load-bearing step reduces by construction to fitted parameters renamed as predictions, self-definitional equations, or self-citation chains that collapse the argument to unverified inputs. The method introduces new alignment objectives without smuggling ansatzes or renaming known results as novel derivations.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that H&E provides transferable histological semantics for SIM without modality collapse; no free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption H&E images encode rich cellular and glandular structure aligned with established clinical annotations that can serve as semantic anchor for SIM
    Invoked to justify using H&E as anchor during pretraining

pith-pipeline@v0.9.0 · 5608 in / 1181 out tokens · 33737 ms · 2026-05-10T15:19:43.569268+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 3 internal anchors

  1. [1]

    PLOS ONE19(5), e0302600 (2024)

    Behr, M., Alizadeh, L., Buckner-Baiamonte, L., Roberts, B., Sholl, A.B., Brown, J.Q.: Structured illumination microscopy for cancer identifica- tion in diagnostic breast biopsies. PLOS ONE19(5), e0302600 (2024). https://doi.org/10.1371/journal.pone.0302600

    work page doi:10.1371/journal.pone.0302600 2024

  2. [2]

    In: Advanced Photonics in Urology

    Behr, M., Halat, S.K., Krane, L.S., Brown, J.Q.: Structured illumination mi- croscopy for see-and-treat decision making in localized prostate cancer therapy. In: Advanced Photonics in Urology. vol. 11619, p. 1161905. SPIE (2021)

    work page 2021

  3. [3]

    Nature medicine30(3), 850–862 (2024)

    Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Song, A.H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al.: Towards a general-purpose foundation model for computational pathology. Nature medicine30(3), 850–862 (2024)

    work page 2024

  4. [4]

    In: Proceedings of the 37th Interna- tional Conference on Machine Learning (ICML)

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for con- trastive learning of visual representations. In: Proceedings of the 37th Interna- tional Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607 (2020)

    work page 2020

  5. [5]

    Journal of biomedical optics18(10), 106016–106016 (2013)

    Dobbs, J.L., Ding, H., Benveniste, A.P., Kuerer, H.M., Krishnamurthy, S., Yang, W., Richards-Kortum, R.: Feasibility of confocal fluorescence microscopy for real- time evaluation of neoplasia in fresh human breast tissue. Journal of biomedical optics18(10), 106016–106016 (2013)

    work page 2013

  6. [6]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  7. [7]

    Nature Biomedical Engineering1(12), 957–966 (2017)

    Fereidouni, F., Harmany, Z.T., Tian, M., Todd, A., Kintner, J.A., McPherson, J.D., Borowsky, A.D., Bishop, J., Lechpammer, M., Demos, S.G., Levenson, R.: Microscopy with ultraviolet surface excitation for rapid slide-free histology. Nature Biomedical Engineering1(12), 957–966 (2017)

    work page 2017

  8. [8]

    Gallagher-Syed, A., Pontarini, E., Lewis, M.J., Barnes, M.R., Slabaugh, G.: Going beyond h&e and oncology: How do histopathology foundation models perform for multi-stain ihc and immunology? arXiv preprint arXiv:2410.21560 (2024)

    work page arXiv 2024

  9. [9]

    Journal of Machine Learning Research17(59), 1–35 (2016)

    Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. Journal of Machine Learning Research17(59), 1–35 (2016)

    work page 2016

  10. [10]

    In: Proceedings of the 35th International Conference on Machine Learning (ICML)

    Ilse, M., Tomczak, J.M., Welling, M.: Attention-based deep multiple instance learn- ing. In: Proceedings of the 35th International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, vol. 80, pp. 2127–2136 (2018), https://proceedings.mlr.press/v80/ilse18a.html

    work page 2018

  11. [11]

    Gigascience9(4), giaa035 (2020)

    Johnson, K.A., Hagen, G.M.: Artifact-free whole-slide imaging with structured illu- mination microscopy and bayesian image reconstruction. Gigascience9(4), giaa035 (2020)

    work page 2020

  12. [12]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Karasikov, M., van Doorn, J., Känzig, N., Erdal Cesur, M., Horlings, H.M., Berke, R., Tang, F., Otálora, S.: Training state-of-the-art pathology foundation models with orders of magnitude less data. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 573–583. Springer (2025)

    work page 2025

  13. [13]

    Nature637, 439–445 (2025)

    Kondepudi, A., Pekmezci, M., Hou, X., Widhalm, G., Hervey-Jumper, S., Hol- lon, T.e.a.: Foundation models for fast, label-free detection of glioma infiltration. Nature637, 439–445 (2025). https://doi.org/10.1038/s41586-024-08169-3

    work page doi:10.1038/s41586-024-08169-3 2025

  14. [14]

    Frontiers in Medicine (Pathology) 6, 162 (2019)

    Lafarge, M.W., Pluim, J.P.W., Eppenhof, K.A.J., Veta, M.: Learning domain- invariant representations of histological images. Frontiers in Medicine (Pathology) 6, 162 (2019). https://doi.org/10.3389/fmed.2019.00162 SIMPLER 11

    work page doi:10.3389/fmed.2019.00162 2019

  15. [15]

    arXiv preprint arXiv:2503.09091 (2025)

    Li, D., Wan, G., Wu, X., Wu, X., Chen, X., He, Y., Lian, C.G., Sorger, P.K., Se- menov, Y.R., Zhao, C.: Multi-modal foundation models for computational pathol- ogy: A survey. arXiv preprint arXiv:2503.09091 (2025)

    work page arXiv 2025

  16. [16]

    Nature Medicine 30(3), 863–874 (2024)

    Lu, M.Y., Chen, B., Williamson, D.F.K., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., Parwani, A.V., Zhang, A., Mahmood, F.: A visual-language foundation model for computational pathology. Nature Medicine 30(3), 863–874 (2024). https://doi.org/10.1038/s41591-024-02856-4

    work page doi:10.1038/s41591-024-02856-4 2024

  17. [17]

    DINOv2: Learning Robust Visual Features without Supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  18. [18]

    Frontiers in Bioengineering and Biotechnology7, 198 (2019)

    Otálora, S., Atzori, M., Andrearczyk, V., Khan, A., Müller, H.: Staining invari- ant features for improving generalization of deep convolutional neural networks in computational pathology. Frontiers in Bioengineering and Biotechnology7, 198 (2019)

    work page 2019

  19. [19]

    arXiv preprint arXiv:2405.11643 (2024)

    Ren, P., Others: Panther: Prototype-aware whole-slide image representation learn- ing for scalable computational pathology. arXiv preprint arXiv:2405.11643 (2024)

    work page arXiv 2024

  20. [20]

    arXiv preprint arXiv:2303.06088 (2023)

    Scalbert, M., Vakalopoulou, M., Couzinié-Devy, F.: Towards domain-invariant self-supervised learning with batch styles standardization. arXiv preprint arXiv:2303.06088 (2023)

    work page arXiv 2023

  21. [21]

    Journal of Pathology Informatics12(1), 52 (2021)

    Simonson, P.D., Ren, X., Fromm, J.R.: Creating virtual hematoxylin and eosin im- ages using samples imaged on a commercial codex platform. Journal of Pathology Informatics12(1), 52 (2021)

    work page 2021

  22. [22]

    Brain Pathology23(1), 73–81 (2013)

    Snuderl, M., Wirth, D., Sheth, S.A., Bourne, S.K., Kwon, C.S., Ancukiewicz, M., Curry, W.T., Frosch, M.P., Yaroslavsky, A.N.: Dye-enhanced multimodal confocal imaging as a novel approach to intraoperative diagnosis of brain tumors. Brain Pathology23(1), 73–81 (2013)

    work page 2013

  23. [23]

    Nature Reviews Methods Primers 1(1), 73 (2021)

    Stelzer, E.H., Strobl, F., Chang, B.J., Preusser, F., Preibisch, S., McDole, K., Fiolka, R.: Light sheet fluorescence microscopy. Nature Reviews Methods Primers 1(1), 73 (2021)

    work page 2021

  24. [24]

    Beyond the Failures: Rethinking Foundation Models in Pathology

    Tizhoosh, H.R.: Beyond the failures: Rethinking foundation models in pathology. arXiv preprint arXiv:2510.23807 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    arXiv preprint arXiv:2309.07778 (2023)

    Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Liu, S., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., et al.: Virchow: A million- slide digital pathology foundation model. arXiv preprint arXiv:2309.07778 (2023)

    work page arXiv 2023

  26. [26]

    Journal of biophotonics11(3), e201600328 (2018)

    Wang, M., Tulman, D.B., Sholl, A.B., Mandava, S.H., Maddox, M.M., Lee, B.R., Quincy Brown, J.: Partial nephrectomy margin imaging using structured illumina- tion microscopy. Journal of biophotonics11(3), e201600328 (2018)

    work page 2018