Closing the Gap between Deep and Conventional Image Registration using Probabilistic Dense Displacement Networks

Mattias P. Heinrich

arxiv: 1907.10931 · v1 · pith:FSDDMQZLnew · submitted 2019-07-25 · 💻 cs.CV · cs.LG

Closing the Gap between Deep and Conventional Image Registration using Probabilistic Dense Displacement Networks

Mattias P. Heinrich This is my paper

Pith reviewed 2026-05-24 16:21 UTC · model grok-4.3

classification 💻 cs.CV cs.LG

keywords image registrationdeep learningmedical imagingabdominal CTdisplacement networksmin-convolutionsmean field inferenceweakly-supervised learning

0 comments

The pith

A neural network approximates probabilistic dense displacement optimisation with min-convolutions to reach conventional registration accuracy on large-deformation abdominal CT.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a deep learning method for nonlinear image registration that incorporates constraints from probabilistic dense displacement optimisation. It builds a network around approximate min-convolutions and mean field inference to perform differentiable displacement regularisation in a discrete weakly-supervised setting. This design keeps the number of trainable weights very low, limited mostly to feature extraction layers. The resulting algorithm trains and runs quickly while achieving higher accuracy than earlier deep learning approaches on inter-patient abdominal CT registration.

Core claim

The central claim is that probabilistic dense displacement optimisation ideas can be approximated inside a neural network through min-convolutions and mean field inference, yielding a learnable registration algorithm with few trainable weights that outperforms previous deep learning methods by 15 percent Dice overlap on challenging inter-patient abdominal CT registration.

What carries the argument

Approximate min-convolutions and mean field inference for differentiable displacement regularisation in a discrete weakly-supervised registration network.

If this is right

The network trains and infers very fast.
It achieves state-of-the-art accuracies for inter-patient registration of abdominal CT.
It outperforms previous deep learning approaches by 15 percent Dice overlap.
It contains very few trainable weights primarily for feature extraction.
It is easier to train with few labelled scans.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same min-convolution regularisation could be tested on other modalities that involve large anatomical differences.
Reducing trainable weights to feature extraction layers may allow the method to generalise from smaller training sets than typical deep registration networks.
The discrete formulation might enable direct incorporation of additional anatomical constraints without retraining the entire network.

Load-bearing premise

The approximation of probabilistic dense displacement optimisation via min-convolutions and mean field inference inside the network preserves differentiability while delivering superior performance on large-deformation tasks.

What would settle it

A direct comparison experiment on an inter-patient abdominal CT dataset where the network fails to exceed prior deep learning Dice scores or loses its speed advantage.

Figures

Figures reproduced from arXiv: 1907.10931 by Mattias P. Heinrich.

**Figure 1.** Figure 1: Concept of probabilistic dense displacement network: 1) deformable convolution layers extract features for both fixed and moving image. 2) the correlation layer evaluates for each 3D grid point a dense displacement space yielding a 6D dissimilarity map. 3) spatial filters that promote smoothness act on dimensions 4-6 (min-convolutions) and dim. 1-3 (mean-field inference) in alternation. 4) the probabilist… view at source ↗

**Figure 2.** Figure 2: Visual outcome of proposed pdd-net method to register two patients and transfer a segmentation (moderate example). Most organs have been very well aligned and also anatomies that are not labelled in training (stomach, vertebras) can be registered. folding voxels (negative Jacobians) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

read the original abstract

Nonlinear image registration continues to be a fundamentally important tool in medical image analysis. Diagnostic tasks, image-guided surgery and radiotherapy as well as motion analysis all rely heavily on accurate intra-patient alignment. Furthermore, inter-patient registration enables atlas-based segmentation or landmark localisation and shape analysis. When labelled scans are scarce and anatomical differences large, conventional registration has often remained superior to deep learning methods that have so far mainly dealt with relatively small or low-complexity deformations. We address this shortcoming by leveraging ideas from probabilistic dense displacement optimisation that has excelled in many registration tasks with large deformations. We propose to design a network with approximate min-convolutions and mean field inference for differentiable displacement regularisation within a discrete weakly-supervised registration setting. By employing these meaningful and theoretically proven constraints, our learnable registration algorithm contains very few trainable weights (primarily for feature extraction) and is easier to train with few labelled scans. It is very fast in training and inference and achieves state-of-the-art accuracies for the challenging inter-patient registration of abdominal CT outperforming previous deep learning approaches by 15% Dice overlap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper folds probabilistic dense displacement ideas into a lightweight network via min-convolutions and mean-field inference, which keeps parameters low and targets large-deformation registration, but the 15% Dice claim rests on unshown experiments.

read the letter

This paper tries to close the gap between deep and conventional registration by embedding probabilistic dense displacement optimization into a neural net. It uses approximate min-convolutions and mean field inference to make the regularizer differentiable inside a weakly-supervised discrete setup. That design choice is the main new element and lets the model train with almost only feature-extraction weights, which is a practical win when labeled scans are few. The approach also stays fast at both training and inference, and it focuses on the real weakness of prior deep methods on inter-patient abdominal CT with big anatomical shifts. Those strengths line up with the stated goal of leveraging theoretically grounded constraints rather than learning everything from scratch. The soft spots sit in the validation. The abstract states a 15% Dice improvement over previous deep learning work, yet supplies no dataset names, no error bars, no ablations that isolate the regularizer, and no checks on how well the approximations hold for large deformations. The stress-test concern about losing regularization quality or introducing artefacts is therefore still open until the full results appear. This paper is for people already working on medical image registration who want to mix discrete probabilistic methods with deep feature learning. A reader dealing with atlas segmentation or radiotherapy planning would find the architecture details useful if the gains survive closer scrutiny. It deserves peer review because the core idea is grounded and the problem matters, though the experiments will need tightening.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes Probabilistic Dense Displacement Networks (PDD-Net) that embed approximate min-convolutions and mean-field inference inside a neural network to realize differentiable displacement regularisation drawn from probabilistic dense displacement optimisation. The method is presented in a discrete, weakly-supervised registration framework with only feature-extraction weights trained; it is claimed to be fast in both training and inference and to deliver state-of-the-art accuracy on inter-patient abdominal CT registration, outperforming prior deep-learning approaches by 15% Dice overlap.

Significance. If the differentiable approximations faithfully retain the regularisation properties of the underlying probabilistic model, the work would constitute a meaningful bridge between conventional optimisation-based registration and deep learning: it supplies a low-parameter, theoretically constrained network that handles large deformations without requiring extensive labelled data. The reported speed and accuracy gains on a clinically relevant task would be noteworthy.

major comments (2)

[§3.2] §3.2 (Differentiable Regularisation): the central performance claim (15% Dice improvement over prior DL methods) is attributed to the use of min-convolutions and mean-field inference as a drop-in replacement for probabilistic dense displacement optimisation, yet no quantitative error analysis, convergence bounds, or ablation isolating the contribution of these approximations versus a conventional smoothness penalty is supplied; without such evidence the 15% delta could equally be explained by dataset-specific feature learning.
[§4.3] §4.3 (Experiments on abdominal CT): the inter-patient registration results are presented without reported standard deviations across multiple folds or statistical significance tests against the strongest baseline; given that the method is positioned as superior on large-deformation tasks, the absence of these controls makes it impossible to judge whether the reported margin is robust.

minor comments (2)

The abstract states a 15% Dice improvement but supplies no dataset identifiers, number of cases, or baseline references; these details should be added for immediate readability.
Notation for the mean-field update equations is introduced without an explicit statement of the number of iterations or convergence criterion used at inference time.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond to each major comment below and indicate the changes planned for the revised manuscript.

read point-by-point responses

Referee: [§3.2] §3.2 (Differentiable Regularisation): the central performance claim (15% Dice improvement over prior DL methods) is attributed to the use of min-convolutions and mean-field inference as a drop-in replacement for probabilistic dense displacement optimisation, yet no quantitative error analysis, convergence bounds, or ablation isolating the contribution of these approximations versus a conventional smoothness penalty is supplied; without such evidence the 15% delta could equally be explained by dataset-specific feature learning.

Authors: The PDD-Net architecture is deliberately constructed so that the only learned components are the feature-extraction weights; the displacement regularisation is realised by fixed, differentiable approximations to the probabilistic dense displacement model. This design choice already limits the scope for purely dataset-specific feature learning. Nevertheless, we agree that an explicit ablation against a conventional smoothness term would strengthen the attribution of the observed gains. We will add this ablation study to the revised manuscript together with a short discussion of the approximation fidelity drawn from the underlying mean-field and min-convolution literature. We do not provide convergence bounds because the contribution of the work lies in the practical, end-to-end differentiable realisation rather than new theoretical guarantees. revision: partial
Referee: [§4.3] §4.3 (Experiments on abdominal CT): the inter-patient registration results are presented without reported standard deviations across multiple folds or statistical significance tests against the strongest baseline; given that the method is positioned as superior on large-deformation tasks, the absence of these controls makes it impossible to judge whether the reported margin is robust.

Authors: We accept that the current single-run presentation leaves the robustness of the 15 % Dice margin open to question. In the revision we will rerun the inter-patient abdominal CT experiments over multiple folds, report standard deviations, and add paired statistical significance tests against the strongest baseline. revision: yes

Circularity Check

0 steps flagged

No circularity: derivation relies on external probabilistic framework and differentiable approximations without self-referential reduction

full rationale

The abstract and provided text present the method as importing theoretically proven constraints from prior probabilistic dense displacement optimisation and approximating them via min-convolutions and mean-field inference inside a neural network. No equations, fitted parameters, or self-citations are shown that reduce a claimed prediction or result back to the paper's own inputs by construction. The central performance claim (SOTA Dice overlap) is positioned as an empirical outcome of the network design rather than a definitional or fitted tautology. This matches the default expectation of a self-contained derivation with independent content from the imported regularisation ideas.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The approach rests on the transferability of probabilistic dense displacement optimisation into a neural network with minimal learned parameters; no new entities are introduced.

free parameters (1)

trainable feature extraction weights
Described as the primary trainable components; exact count and values not stated in abstract.

axioms (1)

domain assumption Approximate min-convolutions and mean field inference can be implemented differentiably to enforce displacement regularisation within a discrete weakly-supervised registration network.
This is the core technical premise that enables the low-parameter design.

pith-pipeline@v0.9.0 · 5714 in / 1332 out tokens · 27697 ms · 2026-05-24T16:21:03.110565+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

[1]

IEEE Trans medical imaging (2019)

Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans medical imaging (2019)

work page 2019
[2]

In: Proc

Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T.: Flownet: Learning optical ﬂow with convo- lutional networks. In: Proc. ICCV. pp. 2758–2766 (2015)

work page 2015
[3]

Int J Computer Vision 70(1), 41–54 (2006)

Felzenszwalb, P.F., Huttenlocher, D.P.: Eﬃcient belief propagation for early vision. Int J Computer Vision 70(1), 41–54 (2006)

work page 2006
[4]

Medical image analysis 54, 1–9 (2019)

Heinrich, M.P., Oktay, O., Bouteldja, N.: Obelisk-net: Fewer layers to solve 3D multi-organ segmentation with sparse deformable convolutions. Medical image analysis 54, 1–9 (2019)

work page 2019
[5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Heinrich, M.P., Jenkinson, M., Papie˙ z, B.W., Brady, M., Schnabel, J.A.: Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 187–194. Springer (2013)

work page 2013
[6]

Medical image analysis 49, 1–13 (2018)

Hu, Y., Modat, M., Gibson, E., Li, W., Ghavami, N., Bonmati, E., Wang, G., Ban- dula, S., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Medical image analysis 49, 1–13 (2018)

work page 2018
[7]

Medical image analysis 36, 61–78 (2017)

Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Eﬃcient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)

work page 2017
[8]

In: NeurIPS

Kr¨ ahenb¨ uhl, P., Koltun, V.: Eﬃcient inference in fully connected CRFs with gaus- sian edge potentials. In: NeurIPS. pp. 109–117 (2011)

work page 2011
[9]

In: MICCAI DLMIA, pp

Krebs, J., Mansi, T., Mailh´ e, B., Ayache, N., Delingette, H.: Unsupervised prob- abilistic deformation modeling for robust diﬀeomorphic registration. In: MICCAI DLMIA, pp. 101–109. Springer (2018)

work page 2018
[10]

Computer methods and programs in biomedicine 98(3), 278–284 (2010)

Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., Fox, N.C., Ourselin, S.: Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine 98(3), 278–284 (2010)

work page 2010
[11]

IEEE Trans Medical Imaging 30(10), 1852–1862 (2011)

Rousseau, F., Habas, P.A., Studholme, C.: A supervised patch-based approach for human brain labeling. IEEE Trans Medical Imaging 30(10), 1852–1862 (2011)

work page 2011
[12]

IEEE Trans Medical Imaging 36(8), 1746–1757 (2017) Probabilistic Dense Displacement Networks 9

R¨ uhaak, J., Polzin, T., Heldmann, S., Simpson, I.J., Handels, H., Modersitzki, J., Heinrich, M.P.: Estimation of large motion in lung ct by integrating regularized keypoint correspondences into dense deformable registration. IEEE Trans Medical Imaging 36(8), 1746–1757 (2017) Probabilistic Dense Displacement Networks 9

work page 2017
[13]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Sentker, T., Madesta, F., Werner, R.: Gdl-ﬁre 4D: Deep learning-based fast 4D CT image registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 765–773. Springer (2018)

work page 2018
[14]

IEEE Trans Medical Imaging 35(11), 2459–2475 (2016)

Jimenez-del Toro, O., M¨ uller, H., Krenn, M., et al.: Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: Visceral anatomy benchmarks. IEEE Trans Medical Imaging 35(11), 2459–2475 (2016)

work page 2016
[15]

Medical image analysis 52, 128–143 (2019)

de Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Iˇ sgum, I.: A deep learning framework for unsupervised aﬃne and deformable image regis- tration. Medical image analysis 52, 128–143 (2019)

work page 2019
[16]

IEEE Trans Biomed Eng 63(8), 1563–1572 (2016)

Xu, Z., Lee, C., Heinrich, M.P., Modat, M., Rueckert, D., Ourselin, S., Abramson, R.G., Landman, B.: Evaluation of 6 registration methods for the human abdomen on clinically acquired CT. IEEE Trans Biomed Eng 63(8), 1563–1572 (2016)

work page 2016
[17]

NeuroImage 158, 378–396 (2017)

Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage 158, 378–396 (2017)

work page 2017
[18]

In: Proc

Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H.: Conditional random ﬁelds as recurrent neural networks. In: Proc. ICCV. pp. 1529–1537 (2015)

work page 2015

[1] [1]

IEEE Trans medical imaging (2019)

Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans medical imaging (2019)

work page 2019

[2] [2]

In: Proc

Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Van Der Smagt, P., Cremers, D., Brox, T.: Flownet: Learning optical ﬂow with convo- lutional networks. In: Proc. ICCV. pp. 2758–2766 (2015)

work page 2015

[3] [3]

Int J Computer Vision 70(1), 41–54 (2006)

Felzenszwalb, P.F., Huttenlocher, D.P.: Eﬃcient belief propagation for early vision. Int J Computer Vision 70(1), 41–54 (2006)

work page 2006

[4] [4]

Medical image analysis 54, 1–9 (2019)

Heinrich, M.P., Oktay, O., Bouteldja, N.: Obelisk-net: Fewer layers to solve 3D multi-organ segmentation with sparse deformable convolutions. Medical image analysis 54, 1–9 (2019)

work page 2019

[5] [5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Heinrich, M.P., Jenkinson, M., Papie˙ z, B.W., Brady, M., Schnabel, J.A.: Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 187–194. Springer (2013)

work page 2013

[6] [6]

Medical image analysis 49, 1–13 (2018)

Hu, Y., Modat, M., Gibson, E., Li, W., Ghavami, N., Bonmati, E., Wang, G., Ban- dula, S., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Medical image analysis 49, 1–13 (2018)

work page 2018

[7] [7]

Medical image analysis 36, 61–78 (2017)

Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Eﬃcient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)

work page 2017

[8] [8]

In: NeurIPS

Kr¨ ahenb¨ uhl, P., Koltun, V.: Eﬃcient inference in fully connected CRFs with gaus- sian edge potentials. In: NeurIPS. pp. 109–117 (2011)

work page 2011

[9] [9]

In: MICCAI DLMIA, pp

Krebs, J., Mansi, T., Mailh´ e, B., Ayache, N., Delingette, H.: Unsupervised prob- abilistic deformation modeling for robust diﬀeomorphic registration. In: MICCAI DLMIA, pp. 101–109. Springer (2018)

work page 2018

[10] [10]

Computer methods and programs in biomedicine 98(3), 278–284 (2010)

Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., Fox, N.C., Ourselin, S.: Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine 98(3), 278–284 (2010)

work page 2010

[11] [11]

IEEE Trans Medical Imaging 30(10), 1852–1862 (2011)

Rousseau, F., Habas, P.A., Studholme, C.: A supervised patch-based approach for human brain labeling. IEEE Trans Medical Imaging 30(10), 1852–1862 (2011)

work page 2011

[12] [12]

IEEE Trans Medical Imaging 36(8), 1746–1757 (2017) Probabilistic Dense Displacement Networks 9

R¨ uhaak, J., Polzin, T., Heldmann, S., Simpson, I.J., Handels, H., Modersitzki, J., Heinrich, M.P.: Estimation of large motion in lung ct by integrating regularized keypoint correspondences into dense deformable registration. IEEE Trans Medical Imaging 36(8), 1746–1757 (2017) Probabilistic Dense Displacement Networks 9

work page 2017

[13] [13]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Sentker, T., Madesta, F., Werner, R.: Gdl-ﬁre 4D: Deep learning-based fast 4D CT image registration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 765–773. Springer (2018)

work page 2018

[14] [14]

IEEE Trans Medical Imaging 35(11), 2459–2475 (2016)

Jimenez-del Toro, O., M¨ uller, H., Krenn, M., et al.: Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: Visceral anatomy benchmarks. IEEE Trans Medical Imaging 35(11), 2459–2475 (2016)

work page 2016

[15] [15]

Medical image analysis 52, 128–143 (2019)

de Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Iˇ sgum, I.: A deep learning framework for unsupervised aﬃne and deformable image regis- tration. Medical image analysis 52, 128–143 (2019)

work page 2019

[16] [16]

IEEE Trans Biomed Eng 63(8), 1563–1572 (2016)

Xu, Z., Lee, C., Heinrich, M.P., Modat, M., Rueckert, D., Ourselin, S., Abramson, R.G., Landman, B.: Evaluation of 6 registration methods for the human abdomen on clinically acquired CT. IEEE Trans Biomed Eng 63(8), 1563–1572 (2016)

work page 2016

[17] [17]

NeuroImage 158, 378–396 (2017)

Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage 158, 378–396 (2017)

work page 2017

[18] [18]

In: Proc

Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H.: Conditional random ﬁelds as recurrent neural networks. In: Proc. ICCV. pp. 1529–1537 (2015)

work page 2015