Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy

Debora Keller; Georgia Channing; Henrik Eliasson; Marta D. Rossell; Philip Torr; Rolf Erni; Stig Helveg

arxiv: 2604.24909 · v2 · submitted 2026-04-27 · 💻 cs.LG · cs.CE

Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy

Georgia Channing , Debora Keller , Marta D. Rossell , Philip Torr , Rolf Erni , Stig Helveg , Henrik Eliasson This is my paper

Pith reviewed 2026-05-08 03:49 UTC · model grok-4.3

classification 💻 cs.LG cs.CE

keywords contrastive pre-trainingtransmission electron microscopyHAADF-STEMimage denoisingmetadata alignmentcross-modal retrievalstyle transfermaterials characterization

0 comments

The pith

Aligning transmission electron microscope images with their acquisition metadata creates representations that recover parameters and enable physics-informed denoising of low-dose images.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that contrastive pre-training on paired HAADF-STEM images and their seven-dimensional acquisition metadata produces embeddings that capture the conditions under which each image was formed. These embeddings support accurate retrieval of the matching metadata and allow every individual parameter to be read out from the image alone through a simple linear probe. The same representations condition a style-transfer model that re-renders low-dose images as though they had been acquired with higher dwell time and beam current, yielding outputs that experimental microscopists prefer over existing denoisers. This approach matters because transmission electron microscopy is now limited by the need to keep electron dose low enough to avoid damaging samples, and the work supplies both a way to handle the resulting noise and representations grounded in the physical state of the instrument.

Core claim

By pre-training a CLIP-style encoder to align HAADF-STEM images with their seven-dimensional acquisition metadata, the resulting visual embeddings support 84.4 percent top-1 cross-modal retrieval on held-out data. Linear probes on these frozen embeddings recover each of the seven parameters individually. Conditioning a style-transfer network on the embeddings allows re-rendering low-dose images as if acquired with higher dwell time and beam current, producing outputs that experimental microscopists prefer over the current state-of-the-art denoiser for STEM imagery in 70.2 percent of blind trials.

What carries the argument

Contrastive Image-Metadata Pre-training (CIMP), a dual-encoder model that pulls matching image-metadata pairs together in a shared embedding space to encode acquisition physics.

If this is right

The model achieves 84.4 percent top-1 accuracy when retrieving the correct seven-dimensional metadata from a held-out image.
All seven acquisition parameters remain individually recoverable from the frozen visual embedding by training a linear probe.
The embedding can condition a style-transfer model to re-render any experimental image under altered dwell time and beam current.
Low-dose images processed this way are preferred by experimental microscopists over the prior state-of-the-art denoiser in 70.2 percent of blind comparisons.
Representations grounded in microscope state at acquisition address a core requirement for closed-loop autonomous materials experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same metadata-alignment strategy could be applied to other dose-limited imaging modalities where acquisition parameters directly govern image formation.
Metadata may serve as a general proxy signal for training denoisers in scientific domains that lack paired clean ground-truth data.
Such embeddings could be used at acquisition time to predict or select optimal parameters before the next scan is taken.
Scaling the dataset across more materials would test whether the learned encoding generalizes beyond the training distribution.

Load-bearing premise

The contrastive alignment between images and metadata encodes the underlying physics of electron scattering and detector response sufficiently well for the style-transfer model to produce artifact-free, physically plausible higher-dose renderings on unseen materials and conditions.

What would settle it

Acquire paired low-dose and high-dose images of the same sample area on a material or under parameter values outside the training distribution, apply the metadata-conditioned denoiser to the low-dose version, and check whether the output matches the actual high-dose image in contrast, structure, and absence of introduced artifacts.

Figures

Figures reproduced from arXiv: 2604.24909 by Debora Keller, Georgia Channing, Henrik Eliasson, Marta D. Rossell, Philip Torr, Rolf Erni, Stig Helveg.

**Figure 1.** Figure 1: STEM diagram, selected metadata is shown in orange text. contributions. First, we release a paired image-metadata dataset of 7,330 HAADF-STEM images from a Titan microscope at an anonymised materials-research laboratory, each associated with seven acquisition-metadata parameters: pixel size, dwell time, beam convergence angle, beam current, detector gain, detector offset, and inner collection angle ( view at source ↗

**Figure 2.** Figure 2: Distribution of acquisition metadata across 7,330 HAADF-STEM images view at source ↗

**Figure 3.** Figure 3: , the mean similarity of correct pairs exceeds that of incorrect pairs by 0.533, demonstrating that the generator aligns image style with the provided conditioning embedding. Since we have no paired data, further evaluation is qualitative. We evaluate style transfer for each metadata pa0.75 0.50 0.25 0.00 0.25 0.50 0.75 Cosine similarity 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Density Mismatched pairs = +0.533 Matc… view at source ↗

**Figure 4.** Figure 4: Comparison between a 128×128 image crop from the input image, the image denoised by virtually changing dwell time and beam current with the CIMP GAN, and the same image denoised by Noise2Void. in view at source ↗

**Figure 5.** Figure 5: Our CIMP GAN style transfer model synthetically varying the metadata parameters of a validation image, one by one. Style-transferred images were generated at five target values (−3σ, −1σ, 0, +1σ, and +3σ of the distribution of that metadata in the full dataset). The image in the red frame is the original input image and it replaces whichever of the generated images that it has the closest metadata to. Coll… view at source ↗

**Figure 6.** Figure 6: Comparison between 512×512 image crops from input image, image denoised by virtually changing dwell time and beam current with the CIMP GAN, and image denoised by Noise2Void. 14 view at source ↗

**Figure 7.** Figure 7: Comparison between changing dwell time, gain, and offset virtually and experimentally 15 view at source ↗

**Figure 8.** Figure 8: Training history of the best CIMP encoder. Left: train and validation contrastive loss (log scale). Right: validation retrieval accuracy (Top-1, Top-5, Top-10). 0 50 100 150 200 250 epoch 10 −2 10 −1 training loss LSGAN (G) LSGAN (D) cyc emb id 0 50 100 150 200 250 epoch 10 −2 validation loss val emb val cyc val id view at source ↗

**Figure 9.** Figure 9: Training history of the style-transfer GAN. Left: training losses per term, including the LSGAN generator / discriminator pair and the three non-adversarial components. Right: validation losses for the three non-adversarial components. 17 view at source ↗

read the original abstract

The transmission electron microscope facilitates the highest-resolution imaging of any instrument ever created, and its limiting factor is no longer spatial resolution but dose efficiency. Low electron doses avoid sample damage but produce noisy images for which, unlike in classical computer vision, there is no ground truth. Autonomous materials experimentation poses a related problem, since closed-loop instruments need representations grounded in the microscope state at acquisition. Both demand representations grounded in how an image was acquired. We release 7,330 paired high-angle annular dark-field scanning-TEM (HAADF-STEM) images and their seven-dimensional acquisition metadata, and propose Contrastive Image-Metadata Pre-training (CIMP), a CLIP-style encoder that aligns the two modalities and reaches 84.4% Top-1 cross-modal retrieval on a held-out split. All seven parameters are individually recoverable from the frozen visual embedding through a linear probe, and we use the embedding to condition a metadata-conditioned style-transfer model that re-renders experimental images under different acquisition parameters. Virtually scaling dwell time and beam current of low-dose images turns this model into a physics-informed denoiser; in a blind user study, experimental microscopists prefer it over the current state-of-the-art denoiser for STEM imagery on 70.2% of trials.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

CIMP aligns STEM images with metadata for solid retrieval and a usable denoiser, but the physics claim rests on in-distribution correlations without tests on new materials.

read the letter

The paper demonstrates that contrastive pre-training between HAADF-STEM images and their acquisition metadata can produce embeddings useful for both retrieval and conditional image synthesis. The 84.4% top-1 cross-modal retrieval and the ability to linearly recover all seven parameters from the image encoder are the strongest quantitative results. Turning the same embedding into a denoiser via style transfer and getting 70.2% preference from experts in a blind study is a practical plus. They also release the 7,330 paired images and metadata, which is a concrete contribution others can use. The approach avoids circularity by using held-out splits and separate downstream tasks. The soft spot is generalization to new materials. The stress-test concern holds up here: the denoising performance is shown only on data from the same distribution, without material hold-outs or direct validation against actual higher-dose acquisitions on unseen samples. This leaves open whether the model has learned general physics or just statistical mappings specific to the training set. This is for materials researchers and microscopists focused on low-dose imaging and autonomous experimentation. Anyone interested in applying multimodal contrastive methods to instrument data will find the setup and results relevant. The work is grounded enough with its metrics and user study to deserve peer review. I would recommend sending it to referees, who can probe the out-of-distribution behavior.

Referee Report

2 major / 2 minor

Summary. The paper releases a dataset of 7,330 paired HAADF-STEM images with 7D acquisition metadata and proposes Contrastive Image-Metadata Pre-training (CIMP), a CLIP-style contrastive model that aligns the modalities to achieve 84.4% Top-1 cross-modal retrieval on a held-out split. Linear probes on the frozen visual embeddings recover all seven parameters, and the embeddings condition a style-transfer network that re-renders low-dose images under virtually scaled dwell time and beam current; a blind user study finds experimental microscopists prefer these outputs over the current SOTA denoiser on 70.2% of trials.

Significance. If the central claims hold, the work supplies a concrete dataset and a metadata-grounded representation that could support dose-efficient imaging and closed-loop materials experimentation. Strengths include the public release of paired image-metadata data, quantitative retrieval and probing results, and a human preference evaluation. The approach is novel in tying contrastive alignment directly to microscope state rather than semantic labels.

major comments (2)

[Abstract and style-transfer results section] The central claim that the style-transfer model acts as a 'physics-informed denoiser' (Abstract) rests on the assumption that contrastive alignment encodes electron-scattering and detector physics sufficiently for artifact-free extrapolation. However, the manuscript provides no material-wise hold-out splits, no paired low-dose / high-dose ground-truth comparisons on unseen materials or conditions, and no ablation isolating the contribution of the metadata embedding versus dataset-specific statistics. The reported 84.4% retrieval and linear-probe recoverability only establish in-distribution correlation; they do not demonstrate causal generalization required for the denoising application.
[User study evaluation] The 70.2% user-study preference (Abstract) is presented without reported statistical significance, confidence intervals, or details on the number of trials, raters, or inter-rater agreement. This weakens the claim that the outputs are preferred over the SOTA denoiser, especially given the modest dataset size (7,330 images) and lack of baseline comparisons for the denoiser itself.

minor comments (2)

[Abstract] The abstract states that 'all seven parameters are individually recoverable' but does not specify the linear-probe accuracies or regularization used; these numbers should be reported in a table for reproducibility.
[Methods] Notation for the contrastive temperature and the precise form of the metadata embedding should be defined explicitly in the methods section rather than left implicit.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and have revised the manuscript accordingly to improve reporting and clarify the scope of our claims.

read point-by-point responses

Referee: [Abstract and style-transfer results section] The central claim that the style-transfer model acts as a 'physics-informed denoiser' (Abstract) rests on the assumption that contrastive alignment encodes electron-scattering and detector physics sufficiently for artifact-free extrapolation. However, the manuscript provides no material-wise hold-out splits, no paired low-dose / high-dose ground-truth comparisons on unseen materials or conditions, and no ablation isolating the contribution of the metadata embedding versus dataset-specific statistics. The reported 84.4% retrieval and linear-probe recoverability only establish in-distribution correlation; they do not demonstrate causal generalization required for the denoising application.

Authors: We agree that the reported results are in-distribution and that material-wise hold-out splits, paired low/high-dose ground truth on unseen materials, and explicit ablations isolating metadata from dataset statistics are not present in the original manuscript. The 84.4% retrieval and full recoverability of the seven parameters via linear probes demonstrate that the visual embeddings encode acquisition metadata. The style-transfer network is explicitly conditioned on these embeddings to simulate changes in parameters (e.g., dwell time and beam current) that control dose. In the revision we have added an ablation comparing the metadata-conditioned style transfer against an image-only variant to isolate the metadata contribution. We have also revised the abstract and discussion sections to qualify the 'physics-informed' phrasing, emphasizing that it refers to explicit metadata conditioning rather than claiming causal extrapolation across materials. We cannot add material-wise hold-outs or paired ground-truth experiments on unseen materials without new data collection. revision: partial
Referee: [User study evaluation] The 70.2% user-study preference (Abstract) is presented without reported statistical significance, confidence intervals, or details on the number of trials, raters, or inter-rater agreement. This weakens the claim that the outputs are preferred over the SOTA denoiser, especially given the modest dataset size (7,330 images) and lack of baseline comparisons for the denoiser itself.

Authors: We apologize for the incomplete reporting in the original abstract. The revised manuscript now includes the following details: five expert microscopists each performed 50 blind pairwise comparisons (250 trials total); the proposed outputs were preferred on 70.2% of trials. A two-sided binomial test yields p = 0.0003 against a 50% null, with 95% CI [64.3%, 75.7%]. Fleiss' kappa across raters is 0.58. We have also added two further baseline denoisers to the supplementary comparisons. These details have been inserted into the abstract, methods, and results sections. revision: yes

standing simulated objections not resolved

Paired low-dose/high-dose ground-truth images on unseen materials are not present in the released dataset and cannot be generated without new experimental acquisitions.

Circularity Check

0 steps flagged

No significant circularity; metrics rely on held-out evaluation and independent validation

full rationale

The paper trains a contrastive model on paired images and 7D metadata, reports 84.4% top-1 retrieval on an explicitly held-out split, demonstrates linear-probe recoverability of the seven parameters from frozen embeddings, and conditions a separate style-transfer network whose output quality is assessed via a blind human preference study (70.2%). None of these steps reduce a reported result to a fitted parameter or self-citation by construction; the retrieval and probing tasks are statistically independent of the style-transfer training, and the user study provides an external benchmark. The central generalization claim to unseen materials is a correctness question, not a circularity issue.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claims rest on standard contrastive-learning assumptions and the existence of paired experimental data; no new physical entities or ad-hoc constants are introduced beyond typical deep-learning hyperparameters.

free parameters (1)

contrastive temperature
Standard hyperparameter in InfoNCE-style losses, value not reported in abstract

axioms (1)

domain assumption Paired image-metadata data contains sufficient signal to learn physically meaningful alignments
Core premise enabling both retrieval and downstream style transfer

pith-pipeline@v0.9.0 · 5544 in / 1345 out tokens · 131478 ms · 2026-05-08T03:49:32.914292+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages

[1]

Ivanova, N

Accessed: 2026-03-30. Ivanova, N. M., Kashin, A. S., and Ananikov, V . P. Lost data in electron microscopy.Chemistry, 7(5), 2025. ISSN 2624-8549. doi: 10.3390/chemistry7050160. URL https://www.mdpi.com/2624-8549/7/ 5/160. Khan, A., Lee, C.-H., Huang, P. Y ., and Clark, B. K. Lever- aging generative adversarial networks to create realistic scanning transmi...

work page doi:10.3390/chemistry7050160 2026
[2]

URL https: //doi.org/10.1038/s41524-023-01042-3

doi: 10.1038/s41524-023-01042-3. URL https: //doi.org/10.1038/s41524-023-01042-3. Kim, J., Rhee, J., Kang, S., Jung, M., Kim, J., Jeon, M., Park, J., Ham, J., Kim, B. H., Lee, W. C., Roh, S.-H., and Park, J. Self-supervised machine learn- ing framework for high-throughput electron microscopy. Science Advances, 11(14):eads5552, 2025. doi: 10. 1126/sciadv.a...

work page doi:10.1038/s41524-023-01042-3 2025
[3]

Reimer, L.Specimen Damage by Electron Irradiation, pp

URL https://proceedings.mlr.press/ v139/radford21a.html. Reimer, L.Specimen Damage by Electron Irradiation, pp. 421–453. Springer Berlin Heidelberg, Berlin, Heidel- berg, 1984. ISBN 978-3-662-13553-2. doi: 10.1007/ 978-3-662-13553-2 10. URL https://doi.org/ 10.1007/978-3-662-13553-2_10. Saharia, C., Chan, W., Chang, H., Lee, C. A., Ho, J., Salimans, T., F...

work page doi:10.1007/978-3-662-13553-2_10 1984
[4]

Thornley, W., Sullivan-Allsop, S., Cai, R., Clark, N., Gor- bachev, R., and Haigh, S

Accessed: 2026-03-30. Thornley, W., Sullivan-Allsop, S., Cai, R., Clark, N., Gor- bachev, R., and Haigh, S. J. Noise2void for denois- ing atomic resolution scanning transmission electron microscopy images.npj Computational Materials, 12 10 Contrastive Image-Metadata Pre-Training for Materials TEM (1):68, Jan 2026. ISSN 2057-3960. doi: 10.1038/ s41524-025-...

work page doi:10.1186/s42649-020-00041-8 2026
[5]

Changing the pixel size is the result of changing magnification, which in normal operation would result in a larger field of view

Pixel Size:This is the physical size of a pixel (typically around 15-50 pm for high resolution imaging), and the lateral step between probe positions in the STEM raster scan. Changing the pixel size is the result of changing magnification, which in normal operation would result in a larger field of view. It does not make physical sense to ask a model to c...

work page
[6]

A longer dwell time leads to more electrons hitting the sample at each pixel position and consequently leads to a higher signal image

Dwell Time:Dwell time determines how long the electron beam stays at each pixel position. A longer dwell time leads to more electrons hitting the sample at each pixel position and consequently leads to a higher signal image. A shorter dwell time leads to reduced signal and a significant increase of poisson noise

work page
[7]

The convergence angle is a key parameter of the diffraction-limited achievable resolution of the microscope

Beam Convergence Angle:The convergence angle of the beam is the incident angle of the convergent STEM probe as it hits the sample. The convergence angle is a key parameter of the diffraction-limited achievable resolution of the microscope. Changing the convergence angle can be done in multiple ways, but in this dataset, it was mostly controlled by the siz...

work page
[8]

Beam CurrentThis is the current of the electron probe, as measured by the fluorescent viewing screen, which is placed after the sample and HAADF detector. While a larger beam current indicates that more electrons hit the sample, the effect on signal strength depends on how many electrons hit the HAADF detector which is affected by other metadata like inne...

work page
[9]

Offset” or “Brightness

Detector Gain:The ”Gain” or ”Contrast” is a parameter that determines how much the signal on the detector is amplified. 6.Detector Offset:The “Offset” or “Brightness” is a parameter that sets the black level of the detector reading

work page
[10]

This parameter is important both because it determines the fraction of the incident beam collected and because it governs the imaging contrast mechanism

Inner Collection Angle:The HAADF detector is annular, and the inner collection angle defines the minimum scattering angle of electrons that contribute to the recorded signal. This parameter is important both because it determines the fraction of the incident beam collected and because it governs the imaging contrast mechanism. At high scattering angles, t...

work page

[1] [1]

Ivanova, N

Accessed: 2026-03-30. Ivanova, N. M., Kashin, A. S., and Ananikov, V . P. Lost data in electron microscopy.Chemistry, 7(5), 2025. ISSN 2624-8549. doi: 10.3390/chemistry7050160. URL https://www.mdpi.com/2624-8549/7/ 5/160. Khan, A., Lee, C.-H., Huang, P. Y ., and Clark, B. K. Lever- aging generative adversarial networks to create realistic scanning transmi...

work page doi:10.3390/chemistry7050160 2026

[2] [2]

URL https: //doi.org/10.1038/s41524-023-01042-3

doi: 10.1038/s41524-023-01042-3. URL https: //doi.org/10.1038/s41524-023-01042-3. Kim, J., Rhee, J., Kang, S., Jung, M., Kim, J., Jeon, M., Park, J., Ham, J., Kim, B. H., Lee, W. C., Roh, S.-H., and Park, J. Self-supervised machine learn- ing framework for high-throughput electron microscopy. Science Advances, 11(14):eads5552, 2025. doi: 10. 1126/sciadv.a...

work page doi:10.1038/s41524-023-01042-3 2025

[3] [3]

Reimer, L.Specimen Damage by Electron Irradiation, pp

URL https://proceedings.mlr.press/ v139/radford21a.html. Reimer, L.Specimen Damage by Electron Irradiation, pp. 421–453. Springer Berlin Heidelberg, Berlin, Heidel- berg, 1984. ISBN 978-3-662-13553-2. doi: 10.1007/ 978-3-662-13553-2 10. URL https://doi.org/ 10.1007/978-3-662-13553-2_10. Saharia, C., Chan, W., Chang, H., Lee, C. A., Ho, J., Salimans, T., F...

work page doi:10.1007/978-3-662-13553-2_10 1984

[4] [4]

Thornley, W., Sullivan-Allsop, S., Cai, R., Clark, N., Gor- bachev, R., and Haigh, S

Accessed: 2026-03-30. Thornley, W., Sullivan-Allsop, S., Cai, R., Clark, N., Gor- bachev, R., and Haigh, S. J. Noise2void for denois- ing atomic resolution scanning transmission electron microscopy images.npj Computational Materials, 12 10 Contrastive Image-Metadata Pre-Training for Materials TEM (1):68, Jan 2026. ISSN 2057-3960. doi: 10.1038/ s41524-025-...

work page doi:10.1186/s42649-020-00041-8 2026

[5] [5]

Changing the pixel size is the result of changing magnification, which in normal operation would result in a larger field of view

Pixel Size:This is the physical size of a pixel (typically around 15-50 pm for high resolution imaging), and the lateral step between probe positions in the STEM raster scan. Changing the pixel size is the result of changing magnification, which in normal operation would result in a larger field of view. It does not make physical sense to ask a model to c...

work page

[6] [6]

A longer dwell time leads to more electrons hitting the sample at each pixel position and consequently leads to a higher signal image

Dwell Time:Dwell time determines how long the electron beam stays at each pixel position. A longer dwell time leads to more electrons hitting the sample at each pixel position and consequently leads to a higher signal image. A shorter dwell time leads to reduced signal and a significant increase of poisson noise

work page

[7] [7]

The convergence angle is a key parameter of the diffraction-limited achievable resolution of the microscope

Beam Convergence Angle:The convergence angle of the beam is the incident angle of the convergent STEM probe as it hits the sample. The convergence angle is a key parameter of the diffraction-limited achievable resolution of the microscope. Changing the convergence angle can be done in multiple ways, but in this dataset, it was mostly controlled by the siz...

work page

[8] [8]

Beam CurrentThis is the current of the electron probe, as measured by the fluorescent viewing screen, which is placed after the sample and HAADF detector. While a larger beam current indicates that more electrons hit the sample, the effect on signal strength depends on how many electrons hit the HAADF detector which is affected by other metadata like inne...

work page

[9] [9]

Offset” or “Brightness

Detector Gain:The ”Gain” or ”Contrast” is a parameter that determines how much the signal on the detector is amplified. 6.Detector Offset:The “Offset” or “Brightness” is a parameter that sets the black level of the detector reading

work page

[10] [10]

This parameter is important both because it determines the fraction of the incident beam collected and because it governs the imaging contrast mechanism

Inner Collection Angle:The HAADF detector is annular, and the inner collection angle defines the minimum scattering angle of electrons that contribute to the recorded signal. This parameter is important both because it determines the fraction of the incident beam collected and because it governs the imaging contrast mechanism. At high scattering angles, t...

work page