Pith. sign in

REVIEW 4 major objections 6 minor 25 references

KLDivNet: An unsupervised neural network for multi-modality image registration

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A CNN maximising a Donsker-Varadhan KL bound gives unsupervised multi-modality registration that outperforms histogram mutual information and LNCC on brain and liver datasets.

desk verdict A plausible and novel application of MINE to multi-modality deformable registration, but the reported gains are thin and the estimator's shuffling trick has an unanalyzed bias. read the letter →

arxiv 1908.08767 v2 pith:7BIVZKOM submitted 2019-08-23 cs.CV

classification cs.CV
keywords multi-modalityimageregistrationdeformableKLdivergenceDonsker-Varadhanrepresentationmutualinformationestimationunsupervisedlearningconvolutionalneuralnetworkmedicalanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cross-modality image registration usually relies on mutual information estimated from intensity histograms, which needs bin tuning and complicated gradients in deep learning. This paper proposes to replace it with a CNN that maximises the Donsker-Varadhan lower bound on the KL divergence between the joint distribution of a moved/fixed image pair and the product of their marginals. The resulting similarity module, KLDivNet, is trained unsupervisedly and then attached to a U-Net-like deformable registration network, forming DivRegNet. On AAL Brain, LiTS Liver, and Hospital Liver datasets the authors report that DivRegNet achieves the best Dice and average surface distance among the compared methods, in both same-modality and cross-modality tasks. The point of the work is that a differentiable, histogram-free estimate of mutual information can serve as a practical training loss for multi-modality deformable registration.

What carries the argument

The load-bearing identity is the Donsker-Varadhan variational representation of KL divergence: for probability measures $\mu$ and $\lambda$, $D_{KL}(\mu\|\lambda)=\sup_\Phi \mathbb{E}_\mu[\Phi]-\log \mathbb{E}_\lambda[e^\Phi]$. Applied to image registration, $\mu$ is the joint distribution of the moved and fixed images and $\lambda$ is the product of their marginals, so the KL divergence is exactly their mutual information. KLDivNet is a Y-shaped CNN that parameterises $\Phi$ by $D_\theta(T(M),F)$, estimates the joint expectation as the mean over its output feature map, and estimates the marginal-product expectation by feeding the same network a voxel-shuffled fixed image $F_s$ and averaging $e^{D_\theta}$. Maximising this objective with respect to both $\theta$ and the registration transform is what the paper claims substitutes for computing histogram-based MI during training.

What would settle it

Take a synthetic image pair with a known ground-truth deformation and a known intensity model, then compare the KLDivNet estimate of the second term, $\operatorname{mean}_i e^{D_\theta(T(M),F_s)}$, with a Monte Carlo estimate of $\mathbb{E}_{P_{T(M)}P_F}[e^{D_\theta}]$ obtained from genuinely independent samples with the same intensity distribution. A large gap shows the shuffled-voxel estimator is biased. Separately, sweep translations of a synthetic T1/T2 pair and check whether the maximiser of $S$ coincides with the true alignment; if the peak of $S$ is systematically off-center, the claim that it acts as a cross-modality similarity fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the right-hand side of the Donsker-Varadhan representation, $S(T(M),F)=\sup_\theta \operatorname{mean}[D_\theta(T(M),F)]-\log\operatorname{mean}[e^{D_\theta(T(M),F_s)}]$, is a valid and effective similarity measure for deformable image registration across modalities. Here $D_\theta$ is a convolutional network, the first mean is over the moved image $T(M)$ paired with the fixed image $F$, and $F_s$ is the fixed image with voxels shuffled so that it is independent of $T(M)$ while sharing the marginal intensity distribution. The paper argues that maximising this lower bound behaves like maximising mutual information, and that embedding it in a U-Net-like registration network yields unsupervised training because no deformation ground truth is needed. The reported experiments show DivRegNet outperforming histogram-based MI, LNCC, and a conventional FFD method on Dice and average surface distance across intra- and inter-modality tasks on brain and liver data.

Load-bearing premise

The result stands on the assumption that averaging the critic over a voxel-shuffled fixed image equals averaging over true independent samples from the fixed image's intensity distribution, and that the CNN critic is flexible enough to make the lower bound tight; neither gap is measured.

Editorial extensions

If this is right

  • DivRegNet can be trained end-to-end on unlabelled pairs of CT and MR scans, so deformation ground truth is not needed for cross-modality tasks.
  • The KL-divergence similarity removes histogram-bin selection: unlike the Parzen-window MI baseline, there is no number-of-bins hyperparameter to tune.
  • The same trained RegNet can handle both intra-modality and inter-modality registration, so a single architecture covers T2-T2, T1-T2 and PD-T2 brain tasks.
  • Because KLDivNet is differentiable, backpropagation goes directly through the similarity loss without estimating a joint intensity histogram.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the shuffled-voxel trick is likely biased for a convolutional critic, because shuffling destroys local spatial correlations inside the network's receptive fields; an honest product-of-marginals sample would require permuting patches or sampling an independent image with the same intensity histogram. This bias is not tested in the paper.
  • Inference: if that bias is large, part of DivRegNet's edge over histogram MI could come from the U-Net's smoothness or other regularisation rather than from the KL objective itself; comparing DivRegNet against the same architecture with a corrected marginal estimator would isolate the effect.
  • Inference: the same variational recipe could extend to other neural-dual distances, such as Wasserstein or Sinkhorn divergences, producing alternative unsupervised multi-modality registration losses with different invariance properties.
  • Inference: alternating more critic updates per registration update, or pretraining KLDivNet on a large corpus of unlabelled modality pairs, would likely narrow the gap between the estimated lower bound and true mutual information.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes KLDivNet, a CNN-based estimator of the Donsker-Varadhan lower bound on KL divergence, and embeds it as a similarity metric into DivRegNet, an unsupervised deformable registration network built on a VoxelMorph-style backbone. The method is evaluated on three datasets (AAL Brain, LiTS Liver, Hospital Liver) for both intra- and inter-modality registration, with comparisons to LNCC, histogram-based MI, and conventional FFD SEMI using Dice, ASD, and HD. The central claim is that maximizing the estimated Donsker-Varadhan bound provides an effective, differentiable surrogate for mutual information in multi-modality registration.

Significance. If the method works as claimed, it offers a principled and differentiable alternative to histogram-based MI, which is a genuine need for deep-learning-based multi-modality registration. The use of external anatomical metrics (Dice, ASD, HD) reduces circularity, and the Donsker-Varadhan formulation is a standard and attractive starting point. However, the theoretical derivation in Section 3.1 is garbled, the surrogate fidelity of Eq. (9) is not established, and the reported gains over a well-tuned MI baseline are small (Dice deltas of +0.002 to +0.006) with no statistical analysis. The idea is promising, but the current evidence does not support the claim of 'superior performance'.

major comments (4)
  1. [Section 3.1, Eq. (4)-(7)] The derivation of the Donsker-Varadhan bound is mathematically garbled. Eq. (4) introduces a function u(x) and a change of measure that is not justified; the manipulations do not follow from the preceding line. More importantly, Eq. (7) is labeled an 'upper bound form' and the text says 'This upper bound is tight,' but the Donsker-Varadhan representation is a lower bound on KL divergence. This is not a mere terminology slip: the method's foundation is the lower-bound inequality, and the mislabeling obscures the direction of the optimization. The derivation should be rewritten cleanly, and the terminology corrected.
  2. [Section 3.1, Eq. (9)] The marginal expectation E_{P_{T(M)}P_F}[e^Phi] is computed as mean_i e^{D_theta(T(M), F_s)_i}, where F_s is a voxel-shuffled fixed image. This is valid only if the critic D_theta has no spatial context beyond a single voxel. The manuscript describes 'patch pairs' and the architecture includes downsampling layers; if the receptive field exceeds one voxel, shuffled-image patches are not draws from the product of the marginal patch distributions, so Eq. (9) is a biased estimator. The paper neither states the effective receptive field nor compares Eq. (9) to a direct product-of-marginals estimate. This is load-bearing because the claim that S(T(M),F) estimates MI depends on this estimator.
  3. [Section 3.2] The inner supremum in Eq. (7) over Phi is not actually solved. In the one-stage training described in Section 3.2, the registration network and KLDivNet are optimized jointly by maximizing S, so the trained D_theta may be far from the optimal Phi*. Maximizing a lower bound with a suboptimal critic need not maximize true MI. The manuscript provides no analysis of the finite-capacity or optimization gap, and no experiment measures how close the trained critic is to the Donsker-Varadhan optimum. Without this, the method should be positioned as a heuristic, not as a faithful MI surrogate.
  4. [Tables 2, 3, and 4] The reported differences between DivRegNet and the best RegNet+MI baseline are very small: e.g., on AAL T1-T2, Dice 0.9546 vs 0.9524 and ASD 0.72 vs 0.75; on LiTS, Dice 0.9030 vs 0.9016; on Hospital Liver, Dice 0.8351 vs 0.8348. No error bars, standard deviations, repeated runs, or significance tests are given. The claim that DivRegNet delivers 'superior' or state-of-the-art performance is not statistically supported. Please provide uncertainty estimates or formal significance testing.
minor comments (6)
  1. [Abstract] The phrase 'most underlined processes' is unclear; it likely should be 'most fundamental processes' or 'most important processes'.
  2. [Section 3.1, Eq. (1)] The notation in Eq. (1), 'logµ(x)/λ(x)µ(dx)', is ambiguous; it should be written as log(µ(x)/λ(x)) µ(dx) to avoid confusion.
  3. [References] Reference [McAllester and Statos, 2018] should be spelled 'McAllester and Stratos'.
  4. [Figure 3] The caption does not fully identify which contour corresponds to which metric; labeling the subplots directly would improve readability.
  5. [Section 4.3] The abbreviation 'FFD SEMI' is used without definition; please expand it on first use.
  6. [Section 4.4] The PWDE-based MI loss is referenced but not described; a brief description or pseudocode would aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the KL-divergence lower-bound loss is independently derived and registration quality is evaluated against external anatomical Dice/ASD/HD benchmarks.

full rationale

The derivation chain is self-contained. The KLDivNet similarity is constructed from the Donsker-Varadhan variational lower bound (Eq. 7), with the standard MINE-style estimator given in Eq. (9); the bound and its tightness condition are stated from the cited mathematics, not from the registration target. The registration network maximizes this declared loss, which is a stated design choice, not a hidden fit: no histogram bin count, deformation prior, or target Dice value is fitted and then reported as a prediction. The evaluation uses external anatomical labels and metrics (Dice, ASD, HD) on AAL Brain, LiTS, and Hospital Liver, so the reported registration accuracy is not implied by the loss definition. The voxel-shuffle approximation and the finite-capacity critic raise correctness and robustness concerns (the shuffled-image marginal estimate can be biased if the critic has spatial context, and one-stage joint training may not reach the optimal Phi*), but these are approximation-error issues, not circularity: the paper's equations do not reduce to their own inputs. The only self-citation (Zhuang et al. 2011) is used for the FFD-SEMI comparison baseline and as an FFD example; it is not invoked to justify the central premise. Therefore no circular step is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entity or fitted constant. It relies on standard variational representation theory plus several unverified statistical assumptions about using a CNN to estimate expectations from shuffled images.

free parameters (4)
  • AAL phantom FFD control-point spacing = 20 mm
    Hand-chosen for synthetic T2 image generation; controls deformation smoothness in the main brain experiment.
  • AAL phantom FFD displacement standard deviation = 6 mm
    Hand-chosen to sample random deformations for 250 phantom T2 images; controls task difficulty.
  • Histogram bins for RegNet+MI baseline = 64
    Chosen after AAL experiments as a speed/accuracy compromise and used for liver datasets; affects the strength of the MI comparison.
  • EMA (exponential moving average) = not reported
    Used to stabilize validation/test performance; decay value is not given, so exact reproduction is impossible.
assumptions (4)
  • standard math Donsker-Varadhan variational representation of KL-divergence
    Used in Eq. (7) to express KL-divergence as a supremum over functions; standard result from Donsker and Varadhan.
  • domain assumption Voxel-wise sampling assumption
    Eq. (9) averages over feature-map loci to estimate expectations, treating each locus as a sample from the intensity distribution despite spatial dependence.
  • domain assumption Shuffle preserves marginal and independence
    Shuffled fixed image F_s is assumed to be a valid draw from the product marginal P_F to estimate the second expectation; convolutional processing of shuffled images may bias this.
  • domain assumption Bound tightness with finite critic
    Optimizing S requires Phi* to be reachable; joint one-stage training of RegNet and KLDivNet may not keep the critic near-optimal, so the loss may diverge from true MI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KLDivNet: An unsupervised neural network for multi-modality image registration." pith.science (2026). https://pith.science/paper/7BIVZKOM

@misc{pith2026190808767,
  author       = {Pith},
  title        = {Pith review of: KLDivNet: An unsupervised neural network for multi-modality image registration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BIVZKOM}},
  note         = {Machine review of arXiv:1908.08767}
}
read the original abstract

Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, the lack of voxel-wise ground truth challenges the training of CNNs for an accurate registration. In this work, we propose a cross-modality similarity metric, based on the KL-divergence of image variables, and implement an efficient estimation method using a CNN. This estimation network, referred to as KLDivNet, can be trained unsupervisedly. We then embed the KLDivNet into a registration network to achieve the unsupervised deformable registration for multi-modality images. We employed three datasets, i.e., AAL Brain, LiTS Liver and Hospital Liver, with both the intra- and inter-modality image registration tasks for validation. Results showed that our similarity metric was effective, and the proposed registration network delivered superior performance compared to the state-of-the-art methods.

Figures

Figures reproduced from arXiv: 1908.08767 by the authors.

Figure 1
Figure 1. An example of Deformable Registration. The fixed and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework and architecture of DivRegNet, which consists of a registration network (RegNet) to perform the registration and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Plots of values of the similarity metrics for intra- and inter-modality images with different misalignments. The test images are from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the registration results in AAL Brain T1- [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    An unsupervised learning model for deformable medical im- age registration

    [Balakrishnan et al., 2018] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsupervised learning model for deformable medical im- age registration. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9252– 9260,

  2. [5]

    The Liver Tumor Segmentation Benchmark (LiTS)

    [Bilic et al., 2019] Patrick Bilic, Patrick Ferdinand Christ, Eugene V orontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, J¨urgen Hesser, and Others. The Liver Tumor Segmentation Benchmark (LiTS). arXiv preprint arXiv:1901.04056,

  3. [6]

    Consistent linear- elastic transformations for image matching

    [Christensen, 1999] Gary E Christensen. Consistent linear- elastic transformations for image matching. InBiennial In- ternational Conference on Information Processing in Med- ical Imaging, pages 224–237. Springer,

  4. [8]

    Asymptotic evaluation of cer- tain markov process expectations for large time—iii

    [Donsker and Varadhan, 1976] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of cer- tain markov process expectations for large time—iii. Communications on pure and applied Mathematics , 29(4):389–461,

  5. [11]

    Medical image registration

    [Hill et al., 2001] Derek L G Hill, Philipp G Bathelor, Mark Holden, and David J Hawkes. Medical image registration. Physics in Medicine and Biology , 46:1–45,

  6. [13]

    Squeeze- and-excitation networks

    [Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 7132–7141,

  7. [14]

    Information theory and statistics

    [Kullback, 1997] Solomon Kullback. Information theory and statistics. Courier Corporation,

  8. [16]

    Multi- modality image registration by maximization of mutual information

    [Maes et al., 1997] Frederik Maes, Andre Collignon, Dirk Vandermeulen, Guy Marchal, and Paul Suetens. Multi- modality image registration by maximization of mutual information. IEEE transactions on Medical Imaging , 16(2):187–198,

Show all 25 references
  1. [17]

    Formal limitations on the measurement of mutual information

    [McAllester and Statos, 2018] David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251,

  2. [18]

    Automated anatomical abelling atlas

    [Rolls et al., 2019] Edmund T Rolls, Chu-Chung Huang, Ching-Po Lin, Jianfeng Feng, and Marc Joliot. Automated anatomical abelling atlas

  3. [19]

    Nonrigid registration using free-form de- formations: application to breast mr images

    [Rueckert et al., 1999] Daniel Rueckert, Luke I Sonoda, Carmel Hayes, Derek LG Hill, Martin O Leach, and David J Hawkes. Nonrigid registration using free-form de- formations: application to breast mr images. IEEE trans- actions on medical imaging , 18(8):712–721,

  4. [20]

    Airlab: Au- tograd image registration laboratory

    [Sandk¨uhler et al., 2018] Robin Sandk ¨uhler, Christoph Jud, Simon Andermatt, and Philippe C Cattin. Airlab: Au- tograd image registration laboratory. arXiv preprint arXiv:1806.09907,

  5. [21]

    Networks for joint affine and non- parametric image registration

    [Shen et al., 2019] Zhengyang Shen, Xu Han, Zhenlin Xu, and Marc Niethammer. Networks for joint affine and non- parametric image registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4224–4233,

  6. [22]

    Studholme, D

    [Studholme et al., 1999] C. Studholme, D. L. G. Hill, and D. J. Hawkes. An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognition, 32(1):71– 86, Jan

  7. [23]

    Alignment by maximization of mutual in- formation

    [Viola and Wells III, 1997] Paul Viola and William M Wells III. Alignment by maximization of mutual in- formation. International journal of computer vision , 24(2):137–154,

  8. [24]

    Recursive cascaded networks for unsuper- vised medical image registration

    [Zhao et al., 2019] Shengyu Zhao, Yue Dong, Eric I Chang, Yan Xu, et al. Recursive cascaded networks for unsuper- vised medical image registration. In Proceedings of the IEEE International Conference on Computer Vision, pages 10600–10610,

  9. [25]

    A nonrigid registration framework using spatially encoded mutual in- formation and free-form deformations

    [Zhuang et al., 2011] Xiahai Zhuang, Simon Arridge, David J Hawkes, and Sebastien Ourselin. A nonrigid registration framework using spatially encoded mutual in- formation and free-form deformations. IEEE Transactions on Medical Imaging, 30(10):1819–1828, 2011

  10. [1976]

    Asymptotic evaluation of certain markov process expectations for large time

    [Donsker and Varadhan, 1983] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36(2):183–212,

  11. [1983]

    Deep learning in medical image registra- tion: A survey

    [Haskins et al., 2019] Grant Haskins, Uwe Kruger, and Pingkun Yan. Deep learning in medical image registra- tion: A survey. arXiv preprint arXiv:1903.02026,

  12. [1997]

    Differentiable learning- to-normalize via switchable normalization

    [Luo et al., 2019] Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li. Differentiable learning- to-normalize via switchable normalization. International Conference on Learning Representation (ICLR) ,

  13. [1999]

    A deep learning framework for unsupervised affine and deformable image registration

    [de V oset al., 2019] Bob D de V os, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Isgum. A deep learning framework for unsupervised affine and deformable image registration. Medical image analysis, 52:128–143,

  14. [2001]

    Learning deep representa- tions by mutual information estimation and maximization

    [Hjelm et al., 2018] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representa- tions by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670,

  15. [2005]

    Mutual information neu- ral estimation

    [Belghazi et al., 2018] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, De- von Hjelm, and Aaron Courville. Mutual information neu- ral estimation. In International Conference on Machine Learning, pages 530–539,

  16. [2018]

    V oxel- morph: a learning framework for deformable medical im- age registration

    [Balakrishnan et al., 2019] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. V oxel- morph: a learning framework for deformable medical im- age registration. IEEE transactions on medical imaging ,

  17. [2019]

    Computing large defor- mation metric mappings via geodesic flows of diffeo- morphisms

    [Beg et al., 2005] M Faisal Beg, Michael I Miller, Alain Trouv´e, and Laurent Younes. Computing large defor- mation metric mappings via geodesic flows of diffeo- morphisms. International journal of computer vision , 61(2):139–157,

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.