REVIEW 4 major objections 6 minor 25 references
KLDivNet: An unsupervised neural network for multi-modality image registration
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A CNN maximising a Donsker-Varadhan KL bound gives unsupervised multi-modality registration that outperforms histogram mutual information and LNCC on brain and liver datasets.
desk verdict A plausible and novel application of MINE to multi-modality deformable registration, but the reported gains are thin and the estimator's shuffling trick has an unanalyzed bias. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the Donsker-Varadhan variational representation of KL divergence: for probability measures $\mu$ and $\lambda$, $D_{KL}(\mu\|\lambda)=\sup_\Phi \mathbb{E}_\mu[\Phi]-\log \mathbb{E}_\lambda[e^\Phi]$. Applied to image registration, $\mu$ is the joint distribution of the moved and fixed images and $\lambda$ is the product of their marginals, so the KL divergence is exactly their mutual information. KLDivNet is a Y-shaped CNN that parameterises $\Phi$ by $D_\theta(T(M),F)$, estimates the joint expectation as the mean over its output feature map, and estimates the marginal-product expectation by feeding the same network a voxel-shuffled fixed image $F_s$ and averaging $e^{D_\theta}$. Maximising this objective with respect to both $\theta$ and the registration transform is what the paper claims substitutes for computing histogram-based MI during training.
What would settle it
Take a synthetic image pair with a known ground-truth deformation and a known intensity model, then compare the KLDivNet estimate of the second term, $\operatorname{mean}_i e^{D_\theta(T(M),F_s)}$, with a Monte Carlo estimate of $\mathbb{E}_{P_{T(M)}P_F}[e^{D_\theta}]$ obtained from genuinely independent samples with the same intensity distribution. A large gap shows the shuffled-voxel estimator is biased. Separately, sweep translations of a synthetic T1/T2 pair and check whether the maximiser of $S$ coincides with the true alignment; if the peak of $S$ is systematically off-center, the claim that it acts as a cross-modality similarity fails.
Extended reading notes
Core claim
The paper's central claim is that the right-hand side of the Donsker-Varadhan representation, $S(T(M),F)=\sup_\theta \operatorname{mean}[D_\theta(T(M),F)]-\log\operatorname{mean}[e^{D_\theta(T(M),F_s)}]$, is a valid and effective similarity measure for deformable image registration across modalities. Here $D_\theta$ is a convolutional network, the first mean is over the moved image $T(M)$ paired with the fixed image $F$, and $F_s$ is the fixed image with voxels shuffled so that it is independent of $T(M)$ while sharing the marginal intensity distribution. The paper argues that maximising this lower bound behaves like maximising mutual information, and that embedding it in a U-Net-like registration network yields unsupervised training because no deformation ground truth is needed. The reported experiments show DivRegNet outperforming histogram-based MI, LNCC, and a conventional FFD method on Dice and average surface distance across intra- and inter-modality tasks on brain and liver data.
Load-bearing premise
The result stands on the assumption that averaging the critic over a voxel-shuffled fixed image equals averaging over true independent samples from the fixed image's intensity distribution, and that the CNN critic is flexible enough to make the lower bound tight; neither gap is measured.
Editorial extensions
If this is right
- DivRegNet can be trained end-to-end on unlabelled pairs of CT and MR scans, so deformation ground truth is not needed for cross-modality tasks.
- The KL-divergence similarity removes histogram-bin selection: unlike the Parzen-window MI baseline, there is no number-of-bins hyperparameter to tune.
- The same trained RegNet can handle both intra-modality and inter-modality registration, so a single architecture covers T2-T2, T1-T2 and PD-T2 brain tasks.
- Because KLDivNet is differentiable, backpropagation goes directly through the similarity loss without estimating a joint intensity histogram.
Reading between the lines
- Inference: the shuffled-voxel trick is likely biased for a convolutional critic, because shuffling destroys local spatial correlations inside the network's receptive fields; an honest product-of-marginals sample would require permuting patches or sampling an independent image with the same intensity histogram. This bias is not tested in the paper.
- Inference: if that bias is large, part of DivRegNet's edge over histogram MI could come from the U-Net's smoothness or other regularisation rather than from the KL objective itself; comparing DivRegNet against the same architecture with a corrected marginal estimator would isolate the effect.
- Inference: the same variational recipe could extend to other neural-dual distances, such as Wasserstein or Sinkhorn divergences, producing alternative unsupervised multi-modality registration losses with different invariance properties.
- Inference: alternating more critic updates per registration update, or pretraining KLDivNet on a large corpus of unlabelled modality pairs, would likely narrow the gap between the estimated lower bound and true mutual information.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KLDivNet, a CNN-based estimator of the Donsker-Varadhan lower bound on KL divergence, and embeds it as a similarity metric into DivRegNet, an unsupervised deformable registration network built on a VoxelMorph-style backbone. The method is evaluated on three datasets (AAL Brain, LiTS Liver, Hospital Liver) for both intra- and inter-modality registration, with comparisons to LNCC, histogram-based MI, and conventional FFD SEMI using Dice, ASD, and HD. The central claim is that maximizing the estimated Donsker-Varadhan bound provides an effective, differentiable surrogate for mutual information in multi-modality registration.
Significance. If the method works as claimed, it offers a principled and differentiable alternative to histogram-based MI, which is a genuine need for deep-learning-based multi-modality registration. The use of external anatomical metrics (Dice, ASD, HD) reduces circularity, and the Donsker-Varadhan formulation is a standard and attractive starting point. However, the theoretical derivation in Section 3.1 is garbled, the surrogate fidelity of Eq. (9) is not established, and the reported gains over a well-tuned MI baseline are small (Dice deltas of +0.002 to +0.006) with no statistical analysis. The idea is promising, but the current evidence does not support the claim of 'superior performance'.
major comments (4)
- [Section 3.1, Eq. (4)-(7)] The derivation of the Donsker-Varadhan bound is mathematically garbled. Eq. (4) introduces a function u(x) and a change of measure that is not justified; the manipulations do not follow from the preceding line. More importantly, Eq. (7) is labeled an 'upper bound form' and the text says 'This upper bound is tight,' but the Donsker-Varadhan representation is a lower bound on KL divergence. This is not a mere terminology slip: the method's foundation is the lower-bound inequality, and the mislabeling obscures the direction of the optimization. The derivation should be rewritten cleanly, and the terminology corrected.
- [Section 3.1, Eq. (9)] The marginal expectation E_{P_{T(M)}P_F}[e^Phi] is computed as mean_i e^{D_theta(T(M), F_s)_i}, where F_s is a voxel-shuffled fixed image. This is valid only if the critic D_theta has no spatial context beyond a single voxel. The manuscript describes 'patch pairs' and the architecture includes downsampling layers; if the receptive field exceeds one voxel, shuffled-image patches are not draws from the product of the marginal patch distributions, so Eq. (9) is a biased estimator. The paper neither states the effective receptive field nor compares Eq. (9) to a direct product-of-marginals estimate. This is load-bearing because the claim that S(T(M),F) estimates MI depends on this estimator.
- [Section 3.2] The inner supremum in Eq. (7) over Phi is not actually solved. In the one-stage training described in Section 3.2, the registration network and KLDivNet are optimized jointly by maximizing S, so the trained D_theta may be far from the optimal Phi*. Maximizing a lower bound with a suboptimal critic need not maximize true MI. The manuscript provides no analysis of the finite-capacity or optimization gap, and no experiment measures how close the trained critic is to the Donsker-Varadhan optimum. Without this, the method should be positioned as a heuristic, not as a faithful MI surrogate.
- [Tables 2, 3, and 4] The reported differences between DivRegNet and the best RegNet+MI baseline are very small: e.g., on AAL T1-T2, Dice 0.9546 vs 0.9524 and ASD 0.72 vs 0.75; on LiTS, Dice 0.9030 vs 0.9016; on Hospital Liver, Dice 0.8351 vs 0.8348. No error bars, standard deviations, repeated runs, or significance tests are given. The claim that DivRegNet delivers 'superior' or state-of-the-art performance is not statistically supported. Please provide uncertainty estimates or formal significance testing.
minor comments (6)
- [Abstract] The phrase 'most underlined processes' is unclear; it likely should be 'most fundamental processes' or 'most important processes'.
- [Section 3.1, Eq. (1)] The notation in Eq. (1), 'logµ(x)/λ(x)µ(dx)', is ambiguous; it should be written as log(µ(x)/λ(x)) µ(dx) to avoid confusion.
- [References] Reference [McAllester and Statos, 2018] should be spelled 'McAllester and Stratos'.
- [Figure 3] The caption does not fully identify which contour corresponds to which metric; labeling the subplots directly would improve readability.
- [Section 4.3] The abbreviation 'FFD SEMI' is used without definition; please expand it on first use.
- [Section 4.4] The PWDE-based MI loss is referenced but not described; a brief description or pseudocode would aid reproducibility.
Circularity Check
No significant circularity: the KL-divergence lower-bound loss is independently derived and registration quality is evaluated against external anatomical Dice/ASD/HD benchmarks.
full rationale
The derivation chain is self-contained. The KLDivNet similarity is constructed from the Donsker-Varadhan variational lower bound (Eq. 7), with the standard MINE-style estimator given in Eq. (9); the bound and its tightness condition are stated from the cited mathematics, not from the registration target. The registration network maximizes this declared loss, which is a stated design choice, not a hidden fit: no histogram bin count, deformation prior, or target Dice value is fitted and then reported as a prediction. The evaluation uses external anatomical labels and metrics (Dice, ASD, HD) on AAL Brain, LiTS, and Hospital Liver, so the reported registration accuracy is not implied by the loss definition. The voxel-shuffle approximation and the finite-capacity critic raise correctness and robustness concerns (the shuffled-image marginal estimate can be biased if the critic has spatial context, and one-stage joint training may not reach the optimal Phi*), but these are approximation-error issues, not circularity: the paper's equations do not reduce to their own inputs. The only self-citation (Zhuang et al. 2011) is used for the FFD-SEMI comparison baseline and as an FFD example; it is not invoked to justify the central premise. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (4)
- AAL phantom FFD control-point spacing =
20 mm
- AAL phantom FFD displacement standard deviation =
6 mm
- Histogram bins for RegNet+MI baseline =
64
- EMA (exponential moving average) =
not reported
assumptions (4)
- standard math Donsker-Varadhan variational representation of KL-divergence
- domain assumption Voxel-wise sampling assumption
- domain assumption Shuffle preserves marginal and independence
- domain assumption Bound tightness with finite critic
Cite this review
Pith. "Pith review of KLDivNet: An unsupervised neural network for multi-modality image registration." pith.science (2026). https://pith.science/paper/7BIVZKOM
@misc{pith2026190808767,
author = {Pith},
title = {Pith review of: KLDivNet: An unsupervised neural network for multi-modality image registration},
year = {2026},
howpublished = {\url{https://pith.science/paper/7BIVZKOM}},
note = {Machine review of arXiv:1908.08767}
}
read the original abstract
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a registration network to achieve the unsupervised deformable registration for multi-modality images. We employed three datasets, i.e., AAL Brain, LiTS Liver and Hospital Liver, with both the intra- and inter-modality image registration tasks for validation. Results showed that our similarity metric was effective, and the proposed registration network delivered superior performance compared to the state-of-the-art methods.
Figures
Reference graph
Works this paper leans on
-
[1]
An unsupervised learning model for deformable medical im- age registration
[Balakrishnan et al., 2018] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsupervised learning model for deformable medical im- age registration. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9252– 9260,
work page 2018
-
[5]
The Liver Tumor Segmentation Benchmark (LiTS)
[Bilic et al., 2019] Patrick Bilic, Patrick Ferdinand Christ, Eugene V orontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, J¨urgen Hesser, and Others. The Liver Tumor Segmentation Benchmark (LiTS). arXiv preprint arXiv:1901.04056,
arXiv 2019
-
[6]
Consistent linear- elastic transformations for image matching
[Christensen, 1999] Gary E Christensen. Consistent linear- elastic transformations for image matching. InBiennial In- ternational Conference on Information Processing in Med- ical Imaging, pages 224–237. Springer,
work page 1999
-
[8]
Asymptotic evaluation of cer- tain markov process expectations for large time—iii
[Donsker and Varadhan, 1976] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of cer- tain markov process expectations for large time—iii. Communications on pure and applied Mathematics , 29(4):389–461,
work page 1976
-
[11]
[Hill et al., 2001] Derek L G Hill, Philipp G Bathelor, Mark Holden, and David J Hawkes. Medical image registration. Physics in Medicine and Biology , 46:1–45,
work page 2001
-
[13]
Squeeze- and-excitation networks
[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 7132–7141,
work page 2018
-
[14]
Information theory and statistics
[Kullback, 1997] Solomon Kullback. Information theory and statistics. Courier Corporation,
work page 1997
-
[16]
Multi- modality image registration by maximization of mutual information
[Maes et al., 1997] Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens. Multi- modality image registration by maximization of mutual information. IEEE transactions on Medical Imaging , 16(2):187–198,
work page 1997
Show all 25 references
-
[17]
Formal limitations on the measurement of mutual information
[McAllester and Statos, 2018] David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251,
2018 arXiv
-
[18]
Automated anatomical abelling atlas
[Rolls et al., 2019] Edmund T Rolls, Chu-Chung Huang, Ching-Po Lin, Jianfeng Feng, and Marc Joliot. Automated anatomical abelling atlas
2019
-
[19]
Nonrigid registration using free-form de- formations: application to breast mr images
[Rueckert et al., 1999] Daniel Rueckert, Luke I Sonoda, Carmel Hayes, Derek LG Hill, Martin O Leach, and David J Hawkes. Nonrigid registration using free-form de- formations: application to breast mr images. IEEE trans- actions on medical imaging , 18(8):712–721,
1999
-
[20]
Airlab: Au- tograd image registration laboratory
[Sandk¨uhler et al., 2018] Robin Sandk ¨uhler, Christoph Jud, Simon Andermatt, and Philippe C Cattin. Airlab: Au- tograd image registration laboratory. arXiv preprint arXiv:1806.09907,
2018 arXiv
-
[21]
Networks for joint affine and non- parametric image registration
[Shen et al., 2019] Zhengyang Shen, Xu Han, Zhenlin Xu, and Marc Niethammer. Networks for joint affine and non- parametric image registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4224–4233,
2019
-
[22]
Studholme, D
[Studholme et al., 1999] C. Studholme, D. L. G. Hill, and D. J. Hawkes. An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognition, 32(1):71– 86, Jan
1999
-
[23]
Alignment by maximization of mutual in- formation
[Viola and Wells III, 1997] Paul Viola and William M Wells III. Alignment by maximization of mutual in- formation. International journal of computer vision , 24(2):137–154,
1997
-
[24]
Recursive cascaded networks for unsuper- vised medical image registration
[Zhao et al., 2019] Shengyu Zhao, Yue Dong, Eric I Chang, Yan Xu, et al. Recursive cascaded networks for unsuper- vised medical image registration. In Proceedings of the IEEE International Conference on Computer Vision, pages 10600–10610,
2019
-
[25]
A nonrigid registration framework using spatially encoded mutual in- formation and free-form deformations
[Zhuang et al., 2011] Xiahai Zhuang, Simon Arridge, David J Hawkes, and Sebastien Ourselin. A nonrigid registration framework using spatially encoded mutual in- formation and free-form deformations. IEEE Transactions on Medical Imaging, 30(10):1819–1828, 2011
2011
-
[1976]
Asymptotic evaluation of certain markov process expectations for large time
[Donsker and Varadhan, 1983] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36(2):183–212,
1983
-
[1983]
Deep learning in medical image registra- tion: A survey
[Haskins et al., 2019] Grant Haskins, Uwe Kruger, and Pingkun Yan. Deep learning in medical image registra- tion: A survey. arXiv preprint arXiv:1903.02026,
2019 arXiv
-
[1997]
Differentiable learning- to-normalize via switchable normalization
[Luo et al., 2019] Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning- to-normalize via switchable normalization. International Conference on Learning Representation (ICLR) ,
2019
-
[1999]
A deep learning framework for unsupervised affine and deformable image registration
[de V oset al., 2019] Bob D de V os, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Isgum. A deep learning framework for unsupervised affine and deformable image registration. Medical image analysis, 52:128–143,
2019
-
[2001]
Learning deep representa- tions by mutual information estimation and maximization
[Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670,
2018 arXiv
-
[2005]
Mutual information neu- ral estimation
[Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, De- von Hjelm, and Aaron Courville. Mutual information neu- ral estimation. In International Conference on Machine Learning, pages 530–539,
2018
-
[2018]
V oxel- morph: a learning framework for deformable medical im- age registration
[Balakrishnan et al., 2019] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. V oxel- morph: a learning framework for deformable medical im- age registration. IEEE transactions on medical imaging ,
2019
-
[2019]
Computing large defor- mation metric mappings via geodesic flows of diffeo- morphisms
[Beg et al., 2005] M Faisal Beg, Michael I Miller, Alain Trouv´e, and Laurent Younes. Computing large defor- mation metric mappings via geodesic flows of diffeo- morphisms. International journal of computer vision , 61(2):139–157,
2005
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.