Deep histological synthesis from mass spectrometry imaging for multimodal registration

Alan M. Race; James M. Brown; Kimberley M. Bird; Xujiong Ye

REVIEW 4 major objections 5 minor 14 references

Deep histological synthesis from mass spectrometry imaging for multimodal registration

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a conditional GAN can turn mass spectrometry images of breast tissue into synthetic H&E histology similar enough to real slides to make unimodal registration a practical route.

desk verdict A plausible proof-of-concept for MSI-to-histology synthesis whose registration payoff is claimed but never tested. read the letter →

arxiv 2506.05441 v1 pith:P7ADORPW submitted 2025-06-05 eess.IV cs.CVcs.LG

Kimberley M. Bird , Xujiong Ye , Alan M. Race , James M. Brown This is my paper

classification eess.IVcs.CVcs.LG

keywords massspectrometryimaginghistologysynthesisimage-to-imagetranslationpix2pixconditionalGANmultimodalregistrationmutualinformationSSIM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a conditional generative adversarial network, pix2pix, can translate mass spectrometry imaging (MSI) data of breast tissue into synthetic H&E histology images that closely resemble the real slides. This matters because aligning histology with MSI is difficult: the two modalities are formed by different processes, have different dimensionalities, and come from adjacent tissue sections. If the translation is faithful, the registration problem becomes unimodal—synthetic histology can be aligned to real histology with standard methods—instead of requiring direct multimodal matching. The paper reports mutual information rising from 0.429 to 1.353 and SSIM from 0.530 to 0.949 compared with a U-Net baseline, with visually similar tissue shape and minimal artifacts.

What carries the argument

The central object is pix2pix, a conditional generative adversarial network for paired image-to-image translation, which learns a mapping from a three-channel MSI representation to an H&E-like image. The generator is trained with an L1 penalty of weight 200 plus the adversarial loss, at an input size of 512 by 512. A U-Net trained on 32-by-32 patches serves as the non-generative baseline. The paired inputs come from 111 DESI-MSI and H&E images that were rebinned, peak-picked, resized, padded with black or white borders, and aligned with affine control-point registration before training.

What would settle it

Run a standard unimodal registration algorithm between synthetic and real histology pairs and compare alignment accuracy against manually placed landmarks or segmentation overlap. If the synthetic images that score high on MI and SSIM produce poor registration accuracy, the central claim that synthesis enables unimodal registration fails.

Watch

Extended reading notes

Core claim

Using 111 paired DESI-MSI and H&E breast tissue sections, the paper reduces each MSI volume by rebinning spectra, selecting the top 50 peaks, and forming a three-channel image with PCA, t-SNE, or UMAP. A pix2pix model then maps that three-channel MSI representation to a 512-by-512 synthetic histology image. On the test set, pix2pix with white padding achieves MI 1.353 and SSIM 0.949, well above the U-Net baseline of MI 0.429 and SSIM 0.530. The paper's central claim is that these synthetic images are qualitatively and quantitatively similar to real histology, making them a promising basis for unimodal registration of MSI and histology.

Load-bearing premise

The paper assumes that high mutual information and SSIM between synthetic and real histology imply the synthetic images will support unimodal registration, but it never tests registration, landmarks, or downstream segmentation to verify that link.

Editorial extensions

If this is right

The approach gives a concrete route to unimodal registration: generate synthetic histology from MSI, then register that synthetic image to real histology.
Quantitative evaluation with MI and SSIM can rank synthesis models before a registration experiment is run.
The improvement over U-Net supports the choice of a conditional GAN over a plain segmentation-style network for this translation task.
The method reduces the intractable many-channel MSI registration problem to a three-channel image translation problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper never performs an actual registration, so a direct test would be to register synthetic to real histology and measure landmark or segmentation overlap; my expectation is that MI/SSIM alone may not guarantee geometric accuracy.
The model's failure to copy blue pen marks from the real slides suggests it learns biological content rather than arbitrary annotations, which would make synthetic images cleaner registration targets than the originals; this could be tested by adding artificial markings.
Selecting only the top 50 peaks and collapsing MSI to three channels discards most molecular information; testing whether registration benefits from additional peak-selected channels or learned embeddings is a natural extension.
If synthesis quality transfers across datasets, the same translation step could serve other coregistration tasks, such as aligning MSI with IHC or other stains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper applies off-the-shelf pix2pix to synthesize H&E-like histology from DESI-MSI data, aiming to turn a hard multimodal registration problem into a unimodal one. That specific modality pair is new, as far as I know—prior translation work in pathology moved between H&E and IHC or between molecular imaging modalities, not from mass spectrometry images to histology. The idea is sensible, the writing is clear, and the code and data are public. The qualitative examples show reasonable tissue shape, and the authors are honest that the results are preliminary.

The soft spots are real and load-bearing. The abstract and conclusion claim that this synthesis can “effectively enable” unimodal registration, but no registration experiment is performed. The only quantitative evidence is MI and SSIM between synthetic and real histology, computed on image pairs that were already affine-aligned during preprocessing. Those global similarity metrics do not tell you whether a registration algorithm could recover an unknown misalignment from the synthetic image. High SSIM can be partly driven by large matching background/padding regions, which likely inflates the reported 0.949. The comparison against the U-Net baseline is also less clean than it appears: the U-Net was trained on 32×32 patches with different software, loss, and hyperparameters, so the MI/SSIM gains are not from a controlled comparison, and there are no error bars or significance tests. The dataset is 111 pairs split 70/10/20, which is small for a generative model; the results may not generalize far.

The stress-test note is correct: the registration claim goes beyond the evidence. The synthesis results themselves may still be useful, but the paper’s central motivation remains untested. I can imagine a straightforward fix—introduce known misalignments, register the synthetic histology to the real one, and report landmark or Dice error against a standard multimodal baseline—and the paper would be much stronger with even a small experiment like that. Absent it, the conclusion should be softened to “may enable” rather than “effectively enable.”

The paper is coherent on its own terms, the literature is cited appropriately, and the gap is an evaluation gap rather than a reasoning failure. I’d send it to peer review for a workshop or short-paper venue, with the registration claim either removed or substantiated. For a top-tier conference, the missing registration experiment would be disqualifying in current form.

Referee Report

4 major / 5 minor

Summary. This paper proposes a pix2pix-based image-to-image translation pipeline that synthesises H&E-like histology images from desorption electrospray ionisation mass spectrometry imaging (DESI-MSI) data, with the stated goal of enabling subsequent unimodal registration between MSI and histology. The method is evaluated on 111 publicly available breast tissue pairs using mutual information (MI) and structural similarity index measure (SSIM) against a U-Net baseline, and the authors report improvements of +0.924 in MI and +0.419 in SSIM. The paper concludes that the synthetic histology images are qualitatively and quantitatively similar to real histology and are therefore a promising route for registration. The manuscript includes source code availability; however, no registration experiment is performed, and the evaluation is limited to similarity metrics computed on images that were already affine-aligned during preprocessing.

Significance. The problem addressed is relevant: robust MSI-histology registration is genuinely difficult, and a synthesis-based route to unimodal registration could simplify the matching problem. The paper appears to be the first to apply supervised image-to-image translation to MSI-to-histology synthesis, and the public code release is a practical strength. If the registration claim were directly validated, the contribution would be useful to the multimodal imaging community. As it stands, the evidence supports only a claim about appearance similarity, not a claim about registrability; the significance of the work therefore depends on a subsequent registration evaluation that is currently absent.

major comments (4)

[Abstract; Section 4] The central claim that synthesis can 'effectively enable unimodal registration' is not tested anywhere in the manuscript. No registration algorithm is run, no misaligned test pairs are generated, and no landmark error, target registration error, Dice overlap, or downstream segmentation metric is reported. MI and SSIM between synthetic and real histology on pre-aligned pairs measure appearance similarity, not whether an image registration method can recover unknown misalignments from the synthetic image. The authors should add a registration experiment, for example by applying a standard unimodal registration method to synthetic-real pairs with known synthetic misalignments and reporting alignment accuracy, or by using the synthetic histology in a downstream task such as segmentation and comparing against real histology. Without such an experiment, the conclusion in Section 4 is unsupported.
[Section 3; Table 1] The quantitative evaluation is not sufficient to support the qualitative claim of similarity. Table 1 reports MI and SSIM for a single value per condition, with no number of samples, no variance, and no statistical test; it is unclear whether the reported numbers are on the validation set, the test set, or an average over all images. In addition, the images were affine-aligned during preprocessing (Section 2), so the metrics are computed on pairs that are already geometrically registered. Because large background/padding regions are included, SSIM in particular can be inflated by trivial agreement in constant regions. The authors should report metrics on a defined held-out test set with confidence intervals and should compute similarity on tissue-only masks as a robustness check.
[Section 2] The input representation for the pix2pix model is underspecified. The text states that a three-channel MSI representation is used 'for registration' via PCA, t-SNE, or UMAP 'depending on its visual clarity', but it does not state which representation is actually fed into the synthesis network, how the top 50 peaks are combined into the input channels, or how 'visual clarity' was operationalised. Figure 1 shows a single m/z image, which suggests the model input may be a single ion image, but this is not stated. This ambiguity prevents reproducibility; the authors should specify the exact input channels, normalisation, and the rule for selecting among PCA, t-SNE, and UMAP, and should state whether the affine control-point registration is applied before or after the train/validation/test split.
[Section 2; Table 1] The comparison against the U-Net baseline is confounded by major differences in setup: the U-Net is trained on 32×32 patches with mean squared error loss, while pix2pix is trained on 512×512 images with an adversarial plus L1 loss; the learning rates and optimisers also differ. The phrase 'trained on histology' is ambiguous, and the U-Net is only evaluated with black padding. A fairer baseline comparison, or a clear statement that the baseline is intentionally weak, is needed before the reported MI/SSIM improvements can be interpreted as evidence for the proposed method's effectiveness.

minor comments (5)

[Section 2] Training details are incomplete: the number of epochs, the convergence criterion, the random seed, and the exact train/validation/test partition are not given, which limits reproducibility despite the code release.
[Figure 1] The figure caption and the main text refer to 'MSI (m/z: 885.55213)', but it is not explained how a single ion image relates to the model's input representation; labelling the panels with the actual network input would avoid confusion.
[Section 3; Table 1] The table reports U-Net only with black padding but pix2pix with both black and white padding; for a symmetric comparison, the U-Net should also be evaluated with white padding, or the reason for the asymmetry should be stated.
[Abstract; Section 1] The term 'unimodal registration' is used without a definition; since the paper's central motivation is registration, a one-sentence definition of what unimodal registration means in this context would improve clarity.
[Section 3] The statement that samples show 'slightly darker colouration in correct parts' is vague; a quantitative colour or stain-normalisation analysis, or a more precise description, would strengthen the qualitative claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the synthesis result is evaluated on held-out validation images with independent MI/SSIM metrics, and the registration claim, though under-supported, is an extrapolation rather than a reduction to the model's inputs.

full rationale

The paper's central quantitative claim is that a pix2pix model produces synthetic histology from MSI with higher MI and SSIM against real histology than a U-Net baseline. This is assessed on a held-out validation set (Section 2 splits data into 70% training, 10% validation, 20% testing) using metrics computed between synthetic and real images, so it is not a fitted parameter renamed as a prediction. The preprocessing steps (rebinning, peak picking, affine registration) are fixed and do not encode the evaluation metrics. The cited prior work by co-authors (Race et al., references 2, 4, and 12) is used only for standard dimensionality reduction and spectral processing tools, not as the justification for the synthesis claim or the registration conclusion. The paper's extrapolation that image similarity 'serve[s] as a promising solution for registration' is not demonstrated by a registration experiment, but this is an unsupported conclusion or validity gap, not a circular derivation: the MI/SSIM evaluation does not assume the registration conclusion. Accordingly, no load-bearing step reduces to its own inputs, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim that synthetic histology enables registration rests on the alignment quality of the training pairs, the validity of similarity metrics as registration proxies, and the retention of chemical-spatial information in the three-channel MSI reduction. These are unverified domain assumptions. The free parameters are training and preprocessing choices that are not justified by a tuning procedure.

free parameters (3)

Number of top MSI peaks = 50
All spectra were summed and the top 50 peaks were selected. This cutoff is chosen by hand without a stated criterion and affects the input to the generator.
MSI embedding method = PCA, t-SNE, or UMAP per sample
A three-channel representation was selected based on 'visual clarity', a subjective per-sample choice that affects which chemical information reaches the model.
pix2pix L1 penalty = 200
Chosen without a described tuning procedure; influences the trade-off between realism and fidelity in the generator.

assumptions (3)

domain assumption The 111 MSI-histology pairs are accurately aligned by affine control-point registration, so the generator learns a valid one-to-one mapping.
Section 2 states image pairs were rescaled and registered with affine control points; if this registration is imperfect, the synthetic histology inherits misalignment.
domain assumption MI and SSIM between synthetic and real histology are valid proxies for registration quality.
The paper argues that similarity to real histology enables unimodal registration, but no registration experiment verifies this link.
domain assumption A three-channel reduction of MSI (PCA, t-SNE, or UMAP) preserves sufficient chemical-spatial information to regenerate histological structure.
Section 2 uses a three-channel representation; if key information is lost in this reduction, the synthesis cannot be faithful.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Deep histological synthesis from mass spectrometry imaging for multimodal registration." pith.science (2026). https://pith.science/paper/P7ADORPW

@misc{pith2026250605441,
  author       = {Pith},
  title        = {Pith review of: Deep histological synthesis from mass spectrometry imaging for multimodal registration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7ADORPW}},
  note         = {Machine review of arXiv:2506.05441}
}

read the original abstract

Registration of histological and mass spectrometry imaging (MSI) allows for more precise identification of structural changes and chemical interactions in tissue. With histology and MSI having entirely different image formation processes and dimensionalities, registration of the two modalities remains an ongoing challenge. This work proposes a solution that synthesises histological images from MSI, using a pix2pix model, to effectively enable unimodal registration. Preliminary results show promising synthetic histology images with limited artifacts, achieving increases in mutual information (MI) and structural similarity index measures (SSIM) of +0.924 and +0.419, respectively, compared to a baseline U-Net model. Our source code is available on GitHub: https://github.com/kimberley/MIUA2025.

Figures

Figures reproduced from arXiv: 2506.05441 by the authors.

**Figure 1.** Two examples of synthesised histology tissue samples (C+F) compared to their corresponding MSI (A+D) and real histology (B+E). 4 Conclusion Our results show that image-to-image translation can synthesise histology images from MSI, which are qualitatively and quantitatively similar to real examples, serving as a promising solution for registration of the two modalities. The proposed pipeline utilises a pix2pix mode… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

Analytical Chemistry 90(1), 240–265 (2017)

Buchberger, A.R., DeLaney, K., Johnson, J., et al.: Mass Spectrometry Imaging: A Review of Emerging Advancements and Future Insights. Analytical Chemistry 90(1), 240–265 (2017)

work page 2017
[2]

Analytical Chemistry85(6), 3071–3078 (2013)

Race, A.M., Steven, R.T., Palmer, A.D., et al.: Memory Efficient Principal Compo- nent Analysis for the Dimensionality Reduction of Large Mass Spectrometry Imag- ing Data Sets. Analytical Chemistry85(6), 3071–3078 (2013)

work page 2013
[3]

Proceedings of the National Academy of Sciences113(43), 12244– 12249 (2016)

Abdelmoula, W.M., Balluff, B., Englert, S., et al.: Data-driven identification of prog- nostic tumor subpopulations using spatially mapped t-SNE of mass spectrometry imaging data. Proceedings of the National Academy of Sciences113(43), 12244– 12249 (2016)

work page 2016
[4]

Analytical Chemistry93(6), 3061–3071 (2021)

Race,A.M.,Sutton,D.,Hamm,G.,etal.:DeepLearning-BasedAnnotationTransfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal Data Integration. Analytical Chemistry93(6), 3061–3071 (2021)

work page 2021
[5]

In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops, pp

Bian, C., Phillips, B., Cootes, T., et al.: HEMIT: H&E to Multiplex- Immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops, pp. 184–197. Springer, Cham, Switzerland (2025)

work page 2025
[6]

Winter, D., Triltsch, N., Rosati, M., et al.: Utilizing Mask-Guided Cross-Image Attention for Zero-Shot In-Silico Histopathologic Image Generation with a Diffusion Model.In:2025IEEE22ndInternationalSymposiumonBiomedicalImaging(ISBI), pp. 1–5. IEEE, New Jersey, USA (2025)

work page 2025
[7]

In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp

Shi, L., Zhang, Y., Wong, I.H.M., et al.: MulHiST: Multiple Histological Staining for Thick Biological Samples via Unsupervised Image-to-Image Translation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp. 735–

work page 2023
[8]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp

Liu,S.,Zhu,C.,Xu,F.,etal.:BCI:BreastCancerImmunohistochemicalImageGen- eration Through Pyramid Pix2pix. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1815–1824. IEEE, New Jersey, USA (2022)

work page 2022

Show all 14 references

[9]

In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp

Isola, P., Zhu, J.-Y., Zhou, T., et al.: Image-To-Image Translation With Conditional Adversarial Networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1125–1134. IEEE, New Jersey, USA (2017)

2017
[10]

In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp

Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomed- ical Image Segmentation. In: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp. 234–241. Springer, Cham, Switzerland (2015)

2015
[11]

Cancer Research75(9), 1828–1837 (2015)

Guenther, S., Muirhead, L.J., Speller, A.V.M., et al.: Spatially Resolved Metabolic Phenotyping of Breast Cancer by Desorption Electrospray Ionization Mass Spec- trometry. Cancer Research75(9), 1828–1837 (2015)

2015
[12]

Analytical Chemistry88(19), 9451–9458 (2016)

Race, A.M., Palmer, A.D., Dexter, A., et al.: SpectralAnalysis: Software for the Masses. Analytical Chemistry88(19), 9451–9458 (2016)

2016
[13]

In: SIGGRAPH ’22: ACM SIGGRAPH 2022 Conference Proceedings, pp

Saharia,C.,Chan,W.,Chang,H.,etal.:Palette:Image-to-ImageDiffusionModels. In: SIGGRAPH ’22: ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10. Association for Computing Machinery, NY, USA (2022)

2022
[744]

Springer, Cham, Switzerland (2023)

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Analytical Chemistry 90(1), 240–265 (2017)

Buchberger, A.R., DeLaney, K., Johnson, J., et al.: Mass Spectrometry Imaging: A Review of Emerging Advancements and Future Insights. Analytical Chemistry 90(1), 240–265 (2017)

work page 2017

[2] [2]

Analytical Chemistry85(6), 3071–3078 (2013)

Race, A.M., Steven, R.T., Palmer, A.D., et al.: Memory Efficient Principal Compo- nent Analysis for the Dimensionality Reduction of Large Mass Spectrometry Imag- ing Data Sets. Analytical Chemistry85(6), 3071–3078 (2013)

work page 2013

[3] [3]

Proceedings of the National Academy of Sciences113(43), 12244– 12249 (2016)

Abdelmoula, W.M., Balluff, B., Englert, S., et al.: Data-driven identification of prog- nostic tumor subpopulations using spatially mapped t-SNE of mass spectrometry imaging data. Proceedings of the National Academy of Sciences113(43), 12244– 12249 (2016)

work page 2016

[4] [4]

Analytical Chemistry93(6), 3061–3071 (2021)

Race,A.M.,Sutton,D.,Hamm,G.,etal.:DeepLearning-BasedAnnotationTransfer between Molecular Imaging Modalities: An Automated Workflow for Multimodal Data Integration. Analytical Chemistry93(6), 3061–3071 (2021)

work page 2021

[5] [5]

In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops, pp

Bian, C., Phillips, B., Cootes, T., et al.: HEMIT: H&E to Multiplex- Immunohistochemistry Image Translation with Dual-Branch Pix2pix Generator. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024 Workshops, pp. 184–197. Springer, Cham, Switzerland (2025)

work page 2025

[6] [6]

Winter, D., Triltsch, N., Rosati, M., et al.: Utilizing Mask-Guided Cross-Image Attention for Zero-Shot In-Silico Histopathologic Image Generation with a Diffusion Model.In:2025IEEE22ndInternationalSymposiumonBiomedicalImaging(ISBI), pp. 1–5. IEEE, New Jersey, USA (2025)

work page 2025

[7] [7]

In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp

Shi, L., Zhang, Y., Wong, I.H.M., et al.: MulHiST: Multiple Histological Staining for Thick Biological Samples via Unsupervised Image-to-Image Translation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp. 735–

work page 2023

[8] [8]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp

Liu,S.,Zhu,C.,Xu,F.,etal.:BCI:BreastCancerImmunohistochemicalImageGen- eration Through Pyramid Pix2pix. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1815–1824. IEEE, New Jersey, USA (2022)

work page 2022