REVIEW 4 major objections 5 minor 46 references
Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Off-the-shelf diffusion features guide medical image registration when anatomy is missing.
desk verdict Fresh and convincing use of frozen diffusion features for registration, with a real ambiguity about shared noise and a missing error-bar analysis that should be fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the diffusion feature extractor g: the output of a chosen intermediate U-Net block of a pre-trained diffusion model, evaluated on a lightly noised image at a moderate timestep. For a pair of images, DGIR computes g on the warped moving image and the fixed image and applies LNCC to these feature maps as the similarity loss; the same noise vector (the paper's Eq. 3 uses ε ∼ N(0,I) without specifying whether it is shared) makes the features identical for perfectly aligned inputs. The paper's ablations show that mid-resolution blocks (8–11 and 29–32) and small-to-medium noise levels (t ∈ [10,150]) give the best guiding signal, with timestep 50 and block 29 used for the 2D results.
What would settle it
Run the DXA-to-X-ray experiment with independently sampled noise vectors for the warped and fixed images instead of one shared epsilon and compare Dice; if the advantage over LNCC disappears, the shared-noise assumption is what carries the result.
Extended reading notes
Core claim
The central claim is that intermediate features of an off-the-shelf diffusion model, trained exclusively on ImageNet RGB images, encode semantically meaningful correspondences in medical images, even across modalities and when whole anatomies are missing from one image. DGIR defines a feature extractor g(x) = h_n(√α_t x + √(1−α_t) ε, t), the n-th block output of the diffusion U-Net at timestep t, and trains a registration network with loss 1 − LNCC(g(A∘Φ), g(B)) plus a smoothness penalty on the deformation Φ. The paper reports that this loss outperforms intensity-based losses (LNCC, MSE, MIND, NGF), feature-based baselines (VGG, DINO, an autoencoder), and a segmentation-guided method on DXA-to-X-ray knee registration, and that it substantially improves brain-extracted to non-brain-extracted MRI registration while staying on par with standard methods when the anatomy is shared.
Load-bearing premise
The method assumes that the diffusion model's intermediate features, computed on both images with the same random noise, are stable and semantically comparable across the two scans despite the added randomness and the large domain shift from natural images to medical images.
Editorial extensions
If this is right
- Diffusion-feature similarity removes the need for segmentation labels in missing-anatomy registration, achieving higher Dice than a segmentation-guided baseline on the DXA-to-X-ray task.
- Because the diffusion model is off-the-shelf and ImageNet-pretrained, the similarity signal transfers without any medical pretraining or adaptation.
- The same 2D feature extractor can guide 3D volumetric registration by randomly sampling slices during training, covering the whole volume over time.
- On the brain MRI task, the method matches LNCC-trained networks and a foundation registration model when skull and neck are present in both scans, and clearly beats them when they are absent in one.
- The best-performing settings cluster at mid-resolution blocks and light noise, giving a practical recipe for choosing the feature layer and timestep.
Reading between the lines
- An implementation detail the paper leaves implicit is that the noise vector in Eq. (3) must be shared between the warped and fixed images for the features to be identical under perfect alignment; a practitioner should draw one epsilon per image pair.
- Because the sweet spot sits at mid-resolution features, the method likely trades semantic robustness against geometric precision; tuning the block and noise level per modality may extend it to other missing-anatomy settings such as pre/post-operative or tumor-resection registration.
- The same feature extractor could be used for unsupervised test-time registration, optimizing the deformation field directly, since the similarity is differentiable with respect to the warp.
- Newer diffusion backbones or multi-layer feature fusion may further close the gap to specialized registration models on more extreme appearance shifts, although the paper only tests the two presented tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Diffusion-Guided Image Registration (DGIR), a similarity loss for deformable registration computed on intermediate features of a frozen pre-trained 2D diffusion model. The loss is 1 minus LNCC applied to diffusion features g(A∘Φ) and g(B), where g is defined by adding Gaussian noise to the input and extracting a selected U-Net block output. The authors evaluate on 2D DXA-to-X-ray knee registration and 3D brain MRI registration in missing-anatomy scenarios, reporting higher Dice scores than intensity-based and feature-based baselines, including a segmentation-guided method. They also ablate the diffusion block index and noise timestep, and they release code.
Significance. If the claims hold, the paper makes a useful contribution by demonstrating that an off-the-shelf ImageNet-trained diffusion model can provide semantic correspondence signals for medical image registration without any task-specific training of the feature extractor. The method is cleanly integrated into a standard registration framework, the 2D ablations of block and timestep are informative, and the extension to 3D via randomly sampled slices is a practical way to reuse a 2D extractor. The public code release is a further strength. The main weakness is that the stochastic noise in the feature extractor is underspecified and no statistical uncertainty is reported, so the quantitative claims are currently suggestive rather than conclusive. The central idea is plausible and the reported gains are large, but the analysis needs to be tightened before the claim that perfectly aligned images produce identical features by construction can be accepted.
major comments (4)
- [§3.1, Eq. (3)] The noise vector ε in Eq. (3) is never specified as shared or independently drawn for the warped and fixed images. The statement in the text that for perfect image alignment these features would 'by construction be identical' is only true if the same ε realization is used for both inputs. If independent noise is drawn for each image, then even when A∘Φ = B the two feature maps differ, and the LNCC loss has a stochastic floor that can bias gradients; the magnitude depends on the timestep t and the chosen block n and is not quantified. Please specify the implementation exactly and, ideally, ablate shared versus independent noise or report the residual loss at identity alignment to show that the effect is negligible.
- [§4, Tables 1 and 2] All Dice scores and Jacobian percentages are reported as single point estimates, with no standard deviations, no number of seeds, and no significance tests. Because training involves stochastic optimization and the proposed loss itself depends on random noise, the run-to-run variance is unknown; for example, the reported 2D improvement over Seg-Guided-MMReg (0.9611 vs 0.9422 average Dice) could be within seed variance. Please add repeated runs or paired statistical tests, at least on the 2D test set, to support the claimed superiority.
- [§3.2, 3D experiments] The 3D protocol states that 'we randomly select N coronal, or sagittal, or axial slices for both fixed and warped 3D images' but does not specify N, the sampling distribution over axes, or whether the same slice indices are used for the fixed and warped volumes. These choices affect the loss and the effective training signal, and the 3D results in Table 2 cannot be reproduced without specifying them.
- [§5, Table 2] The claim that DGIR 'performs on par' with LNCC in the BE→BE scenario rests on a small difference (0.8079 vs 0.8156) with no variance information; moreover, the Jacobian determinant percentage for Ours is higher than for LNCC (0.268% vs 0.020%), so the trade-off between alignment accuracy and folding is not fully characterized. Please discuss or quantify this trade-off.
minor comments (5)
- [Acknowledgments and §5] There are typos: 'Acknoledgements' should be 'Acknowledgements', and 'foundationl' in Section 5 should be 'foundational'.
- [§5] The text says 'see Eq. (1)' when describing the training loss, but the relevant equation appears to be Eq. (4), not Eq. (1).
- [Table 2] The column headers interleave metric and scenario names (for example, 'Dice Score BE→NBE test %|J| BE→NBE test'), which makes the table hard to parse; grouped subheaders for metric and scenario would be clearer.
- [Figure 3] The figure shows two separate blocks labeled 'noise' but does not indicate whether the same noise tensor is passed to both feature extractors; please annotate this explicitly.
- [§5] The selection of block n=11 for 3D is described as 'comparable to the corresponding decoder block', but n=11 lies in the encoder range shown in Fig. 5a (blocks 8–11); please clarify the terminology.
Circularity Check
No significant circularity: the diffusion-feature loss is computed from a frozen pre-trained model and evaluated on held-out segmentation labels, and the authors' self-citations appear only as baselines or architecture references.
full rationale
The derivation chain is self-contained with respect to the evaluation labels. The proposed similarity loss in Eq. (4) is built from a frozen off-the-shelf diffusion model via Eq. (3), computing g(x) = h_n(sqrt(alpha_t)x + sqrt(1-alpha_t)epsilon, t) and then applying 1 - LNCC. No diffusion-model parameter is trained or fitted to the Dice labels, and no learned component is constructed from the evaluation target. The registration network is trained with the proposed loss, while Dice scores are computed on held-out segmentation masks that do not appear in the loss or in the training objective. The choice of block number and timestep is explicitly reported as validation-set model selection: 'Test dice scores for our method reported in Tab. 1 are based on the model with best parameters (timestep t = 50, block n = 29) on small validation set.' This is standard model selection, not a fitted input renamed as a prediction. The claim that off-the-shelf diffusion features identify semantic correspondences in medical images is corroborated by qualitative correspondence figures and by comparisons against other feature extractors (AE, VGG, DINO), so it does not reduce to the method's own construction. The authors' prior work appears only as baselines (Seg-Guided-MMReg [7], UniGradICON [40]) or as architectural references (ICON [13]); none of these citations carries the load-bearing argument for the central claim. One non-circular technical caveat is that Eq. (3) leaves unspecified whether the noise realization epsilon is shared between g(A composed phi) and g(B), so the statement that features 'by construction' are identical at perfect alignment is conditional on sharing the same noise; this affects interpretability and variance but is not a case of the prediction being equivalent to its inputs by definition or by fitted construction. Overall, no circular step meeting the required evidentiary standard was found.
Assumptions & free parameters
free parameters (3)
- diffusion block index n =
29 (2D), 11 (3D)
- noise timestep t =
50 (2D and 3D)
- regularization weight lambda =
1
assumptions (3)
- domain assumption Diffusion features trained on ImageNet transfer to medical images and capture anatomical correspondences
- domain assumption LNCC on diffusion features is a valid similarity measure for registration
- ad hoc to paper The random noise epsilon in feature extraction is applied consistently (or does not adversely affect the loss)
Cite this review
Pith. "Pith review of Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models." pith.science (2026). https://pith.science/paper/NU2UI4PL
@misc{pith2026250602419,
author = {Pith},
title = {Pith review of: Guiding Registration with Emergent Similarity from Pre-Trained Diffusion Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/NU2UI4PL}},
note = {Machine review of arXiv:2506.02419}
}
read the original abstract
Diffusion models, while trained for image generation, have emerged as powerful foundational feature extractors for downstream tasks. We find that off-the-shelf diffusion models, trained exclusively to generate natural RGB images, can identify semantically meaningful correspondences in medical images. Building on this observation, we propose to leverage diffusion model features as a similarity measure to guide deformable image registration networks. We show that common intensity-based similarity losses often fail in challenging scenarios, such as when certain anatomies are visible in one image but absent in another, leading to anatomically inaccurate alignments. In contrast, our method identifies true semantic correspondences, aligning meaningful structures while disregarding those not present across images. We demonstrate superior performance of our approach on two tasks: multimodal 2D registration (DXA to X-Ray) and monomodal 3D registration (brain-extracted to non-brain-extracted MRI). Code: https://github.com/uncbiag/dgir
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Neuroimage 54(3), 2033–2044 (2011)
Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C.: A repro- ducibleevaluationofantssimilaritymetricperformanceinbrainimageregistration. Neuroimage 54(3), 2033–2044 (2011)
work page 2011
-
[2]
IEEE Transactions on Medical Imaging38(8), 1788–1800 (2019)
Balakrishnan,G.,Zhao,A.,Sabuncu,M.R.,Guttag,J.,Dalca,A.V.:Voxelmorph:a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging38(8), 1788–1800 (2019)
2019
-
[3]
Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., Babenko, A.: Label- efficient semantic segmentation with diffusion models. ICLR (2021)
work page 2021
-
[4]
In: CVPR (2021)
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: CVPR (2021)
2021
-
[5]
Chen, X., Liu, Z., Xie, S., He, K.: Deconstructing denoising diffusion models for self-supervised learning. ICLR (2025)
work page 2025
-
[6]
In: Medical Imaging with Deep Learning
Czolbe, S., Krause, O., Feragen, A.: Semantic similarity metrics for learned image registration. In: Medical Imaging with Deep Learning. pp. 105–118. PMLR (2021)
work page 2021
-
[7]
Demir, B., Niethammer, M.: Multimodal image registration guided by few segmen- tations from one modality. In: MIDL (2024)
work page 2024
- [8]
Show all 46 references
-
[9]
NeurIPS 34, 8780–8794 (2021)
Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. NeurIPS 34, 8780–8794 (2021)
2021
-
[10]
Neuroimage62(2), 774–781 (2012)
Fischl, B.: FreeSurfer. Neuroimage62(2), 774–781 (2012)
2012
-
[11]
Physics in Medicine & Biology65(20), 20TR01 (2020)
Fu, Y.,Lei, Y., Wang, T., Curran,W.J., Liu, T., Yang, X.:Deep learning in medical image registration: a review. Physics in Medicine & Biology65(20), 20TR01 (2020)
2020
-
[12]
arXiv preprint arXiv:2407.00783 (2024)
Fuest, M., Ma, P., Gui, M., Fischer, J.S., Hu, V.T., Ommer, B.: Diffusion models and representation learning: A survey. arXiv preprint arXiv:2407.00783 (2024)
2024 arXiv
-
[13]
In: ICCV
Greer,H.,Kwitt,R.,Vialard,F.X.,Niethammer,M.:ICON:Learningregularmaps through inverse consistency. In: ICCV. pp. 3396–3405 (2021)
2021
-
[14]
In: MICCAI
Greer, H., Tian, L., Vialard, F.X., Kwitt, R., Bouix, S., San Jose Estepar, R., Rushmore, R., Niethammer, M.: Inverse consistency by construction for multistep deep registration. In: MICCAI. pp. 688–698. Springer (2023)
2023
-
[15]
In: MICCAI
Haber,E.,Modersitzki,J.:Intensitygradientbasedregistrationandfusionofmulti- modal images. In: MICCAI. pp. 726–733. Springer (2006) 10 N. Tursynbek et al
2006
-
[16]
Medical image analysis16(7), 1423–1435 (2012)
Heinrich, M.P., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F.V., Brady, M., Schnabel, J.A.: MIND: Modality independent neighbourhood descriptor for multi- modal deformable registration. Medical image analysis16(7), 1423–1435 (2012)
2012
-
[17]
NeurIPS33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS33, 6840–6851 (2020)
2020
-
[18]
MELBA1 (2022)
Hoopes, A., Hoffmann, M., Greve, D.N., Fischl, B., Guttag, J., Dalca, A.V.: Learn- ing the effect of registration hyperparameters with hypermorph. MELBA1 (2022)
2022
-
[19]
In: CVPR
Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Re- purposing diffusion-based image generators for monocular depth estimation. In: CVPR. pp. 9492–9502 (2024)
2024
-
[20]
ECCV (2022)
Kim, B., Han, I., Ye, J.C.: Diffusemorph: Unsupervised deformable image registra- tion along continuous trajectory using diffusion models. ECCV (2022)
2022
-
[21]
In: WBIR MICCAI (2024)
Kögl, F., Reithmeir, A., Sideri-Lampretsa, V., Machado, I., Braren, R., Rueckert, D., Schnabel, J.A., Zimmer, V.A.: General vision encoder features as guidance in medical image registration. In: WBIR MICCAI (2024)
2024
-
[22]
In: CVPR
Lee, H.Y., Tseng, H.Y., Yang, M.H.: Exploiting diffusion prior for generalizable dense prediction. In: CVPR. pp. 7861–7871 (2024)
2024
-
[23]
NeurIPS36 (2023)
Luo, G., Dunlap, L., Park, D.H., Holynski, A., Darrell, T.: Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NeurIPS36 (2023)
2023
-
[24]
Journal of cognitive neuroscience 19(9), 1498–1507 (2007)
Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C., Buckner, R.L.: Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience 19(9), 1498–1507 (2007)
2007
-
[25]
NeurIPS (2024)
Meng, B., Xu, Q., Wang, Z., Cao, X., Huang, Q.: Not all diffusion model activations have been evaluated as discriminative features. NeurIPS (2024)
2024
-
[26]
Springer (2024)
Mukhopadhyay, S., Gwilliam, M., Yamaguchi, Y., Agarwal, V., Padmanabhan, N., Swaminathan,A.,Zhou,T.,Ohya,J.,Shrivastava,A.:Dotext-freediffusionmodels learn discriminative visual representations? In: ECCV. Springer (2024)
2024
-
[27]
Protocol for the cohort study1, 2 (2006)
Nevitt, M., Felson, D., Lester, G.: The osteoarthritis initiative. Protocol for the cohort study1, 2 (2006)
2006
-
[28]
TMLR (2024)
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: DINOv2: Learning robust visual features without supervision. TMLR (2024)
2024
-
[29]
In: ACM SIGGRAPH 2023
Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: ACM SIGGRAPH 2023. pp. 1–11 (2023)
2023
-
[30]
In: MICCAI (2023)
Qin, Y., Li, X.: FSDiffReg: Feature-wise and score-wise diffusion-guided unsuper- vised deformable image registration for cardiac images. In: MICCAI (2023)
2023
-
[31]
In: CVPR
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022)
2022
-
[32]
In: MICCAI
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
2015
-
[33]
In: CVPR
Shi, Y., Xue, C., Liew, J.H., Pan, J., Yan, H., Zhang, W., Tan, V.Y., Bai, S.: DragDiffusion: Harnessing diffusion models for interactive point-based image edit- ing. In: CVPR. pp. 8839–8849 (2024)
2024
-
[34]
arXiv:1409.1556 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
2014 arXiv
-
[35]
MICCAI (2024)
Song, X., Xu, X., Yan, P.: General purpose image encoder DINOv2 for medical image registration. MICCAI (2024)
2024
-
[36]
arXiv preprint arXiv:2403.16776 (2024) Guiding Registration with Pre-trained Diffusion Models 11
Starck, S., Sideri-Lampretsa, V., Kainz, B., Menten, M., Mueller, T., Rueckert, D.: Diff-def: Diffusion-generated deformation fields for conditional atlases. arXiv preprint arXiv:2403.16776 (2024) Guiding Registration with Pre-trained Diffusion Models 11
2024 arXiv
-
[37]
PLoS medicine12(3), e1001779 (2015)
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al.: UK Biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine12(3), e1001...
2015
-
[38]
NeurIPS36, 1363–1389 (2023)
Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. NeurIPS36, 1363–1389 (2023)
2023
-
[39]
In: CVPR
Tian, J., Aggarwal, L., Colaco, A., Kira, Z., Gonzalez-Franco, M.: Diffuse at- tend and segment: Unsupervised zero-shot segmentation using stable diffusion. In: CVPR. pp. 3554–3563 (2024)
2024
-
[40]
In: MICCAI
Tian, L., Greer, H., Kwitt, R., Vialard, F.X., San José Estépar, R., Bouix, S., Rushmore, R., Niethammer, M.: unigradicon: A foundation model for medical im- age registration. In: MICCAI. pp. 749–760. Springer (2024)
2024
-
[41]
In: CVPR
Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. In: CVPR. pp. 1921–1930 (2023)
2023
-
[42]
In: MICCAI
Tursynbek, N., Niethammer, M.: Unsupervised discovery of 3d hierarchical struc- ture with generative diffusion features. In: MICCAI. pp. 320–330. Springer (2023)
2023
-
[43]
In: CVPR
Xiang, W., Yang, H., Huang, D., Wang, Y.: Denoising diffusion autoencoders are unified self-supervised learners. In: CVPR. pp. 15802–15812 (2023)
2023
-
[44]
NeuroImage158, 378–396 (2017)
Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage158, 378–396 (2017)
2017
-
[45]
In: CVPR
Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., Lu, J.: Unleashing text-to-image diffusion models for visual perception. In: CVPR. pp. 5729–5739 (2023)
2023
-
[46]
In: MICCAI (2024)
Zhuo, Y., Shen, Y.: Diffusereg: Denoising diffusion model for obtaining deformation fields in unsupervised deformable image registration. In: MICCAI (2024)
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.