Zero Shot Learning for Multi-Modal Real Time Image Registration

Avinash Kori; Ganapathi Krishnamurthi

REVIEW 4 major objections 5 minor 11 references

Zero Shot Learning for Multi-Modal Real Time Image Registration

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free pipeline aligns brain MRI in 0.2 seconds per image.

desk verdict A fast affine registration pipeline that improves similarity metrics in practice, but the evaluation doesn't establish that the estimated transforms are correct. read the letter →

arxiv 1908.06213 v1 pith:TLQTFJYP submitted 2019-08-17 cs.CV cs.LG

Avinash Kori , Ganapathi Krishnamurthi This is my paper · ORCID

classification cs.CVcs.LG

keywords imageregistrationzero-shotlearningaffinetransformationcenter-of-masskeypointspre-trainedfeatureextractormultimodalMRIuncertaintyestimationreal-timemedicalimaging

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a registration pipeline in which no module is trained on medical data: a convolutional network pretrained on natural images serves as a fixed feature extractor, center-of-mass coordinates of its thresholded early feature maps become keypoints, and a small multilayer-perceptron regressor turns the keypoint pairs into affine transformation parameters. The regressor is trained only on random point sets and random affine matrices, which the authors call zero-shot learning because the training and testing data do not overlap. On multimodal brain MR images, the method reports larger improvements in Dice, mutual information, SSIM, and MSE than standard feature-based and elastic baselines, while running in about 0.2 seconds per image. The stated value is real-time, training-free registration for image-guided surgery.

What carries the argument

The load-bearing object is the center-of-mass keypoint: the intensity-weighted mean position of a thresholded feature map, which turns a high-dimensional image into a small coordinate set. These coordinates are fed, together with the moving-image coordinates, into a multilayer-perceptron regressor whose only training data are uniformly random points in [0,1] mapped through random affine matrices with entries in [-1,1]. The regressor outputs six affine parameters covering scale, rotation, shear, and translation, and ten resampled keypoint sets are averaged to reduce noise. The pretrained feature extractor is used only for keypoint generation; its internal weights are never updated.

What would settle it

Measure the estimated affine matrix against known ground-truth transforms on brain MRI pairs with rotations near 15 degrees and translations of tens of pixels; if the error grows sharply as the deformation approaches the edge of the synthetic training range, or if the reported Dice and MI improvements disappear on same-modality volumes with small deformations, the zero-shot generalization claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a transformation estimator trained entirely on synthetic random coordinates can register real medical images, as long as the keypoints it receives are produced by a frozen feature extractor that captures edges and corners. Feature maps from the first layers of a pretrained deep network are thresholded at 95 percent of maximum activation, and each map is reduced to an intensity-weighted center of mass, so an image is represented by a small fixed number of two-dimensional keypoints. This dimensionality reduction makes the runtime essentially independent of image size. The paper further claims that averaging predictions over ten bootstrap samples of the keypoints yields stable affine parameters and that blackening random pixels gives a per-transformation uncertainty estimate, producing clinically useful guidance without ground truth.

Load-bearing premise

The regressor is trained on uniformly random points in [0,1] and random affine matrices with entries in [-1,1], and is assumed to generalize to the keypoint geometry produced by real brain MRI and to the actual deformations between fixed and moving scans; if the synthetic point distribution or transform range does not match real geometry, the estimated transformations are biased.

Editorial extensions

If this is right

Because only keypoint coordinates, not pixel grids, enter the regressor, registration time stays near 0.2 seconds per image regardless of input resolution.
Medical sites without paired training images can still register modalities by reusing the same frozen feature extractor and the synthetic-only regressor.
The bootstrap-by-blackening procedure gives an uncertainty map for every registration, which can flag regions where the estimated transform is unreliable during surgery.
The same pipeline should work for other affine transforms by retraining the small regressor on synthetic data with the corresponding transform range.
Within the tested ranges (translation up to 50 pixels, rotation up to 0.3 radians, shear up to 0.03 radians), the method reduces registration error on multimodal MRI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the synthetic training distribution were matched to the actual keypoint geometry of brain MRI rather than uniform [0,1], the same architecture would likely hold across wider transform ranges.
The zero-shot claim could be tested more sharply by holding out entire modalities and measuring transform-matrix error against known ground-truth transforms, rather than only image-similarity gains.
The same keypoint-plus-synthetic-regressor design may extend to deformable registration by replacing the affine regressor with a dense-flow estimator trained on synthetic displacement fields.
Because error metrics improve substantially on same-modality pairs as well as multimodal pairs, the gain may come less from cross-modal feature invariance than from the keypoint reduction itself; a study ablating network layers would separate these factors.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a simple, fast affine registration pipeline that combines thresholded VGG-19 feature-map centers of mass with an MLP regressor trained on synthetic correspondences. It runs at 0.2 seconds per image and reports better Dice, MI, SSIM, and MSE than SIFT and SimpleElastix on three brain MRI datasets. The idea is genuinely new as a combination, and it has practical appeal: no medical training data, fixed keypoint count, and a built-in uncertainty estimate.Given the reported results, the paper does more than most workshop submissions: it evaluates on BraTS, ALBERT, and hospital data, compares against two baseline methods, and includes an uncertainty map. The runtime improvement is real and useful. The synthetic training strategy is also clever, though it carries a load-bearing assumption that is never tested.

The soft spots are significant. The biggest one is the gap between the synthetic training distribution and the actual keypoint geometry. The MLP is trained on uniformly random points in [0,1] with affine matrices having entries in [-1,1], but it is deployed on centers of mass of 95%-thresholded VGG-19 feature maps from real MRI. Those CoMs are not affine-equivariant: thresholding and center-of-mass computation change in content-dependent ways under rotation, scale, and shear. So the i-th fixed-image CoM does not necessarily correspond to the i-th moving-image CoM under the true transformation. Averaging ten 64-point subsets reduces sampling noise, not this systematic mismatch. The paper never validates the estimated affine parameters against the known artificial transformations described in Section 2.8, so a biased estimator that still improves overlap metrics cannot be distinguished from a correct one. The stress-test note is right about this, and it is the central weakness.

The evaluation has other issues: no error bars or statistical tests, no ground-truth transformation error, and no deep-learning baseline. The Dice and MI formulas in Equations 3 and 5 are incorrectly written, which is careless and casts some doubt on the exact numbers. The 95% threshold is selected post hoc. And calling this “zero-shot” overstates things; the regressor is trained, just on synthetic data.

That said, the method is plausible. The reported improvements are consistent with a pipeline that, despite the correspondence issue, may still estimate a reasonable affine transform because the MLP has some tolerance for noise. I would not dismiss it. It deserves peer review because it is a new combination with practical speed and enough evidence to warrant scrutiny. A serious referee should ask for parameter error on the synthetic transforms, code, and error bars.

For a reading group, it’s a decent example of how fast unsupervised registration can be built from off-the-shelf components, but it is not a deep methods paper. I would not cite it in my own work yet.

Recommendation: send it to peer review, but expect heavy revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes an unsupervised, real-time affine image registration pipeline. It uses a fixed VGG-19 network pretrained on ImageNet as a feature extractor, thresholds each feature map at 95% of its maximum activation, computes centers of mass (CoMs) of the thresholded maps from the first two layers, and feeds 128 such CoMs from fixed and moving images into an MLP regressor that outputs six affine parameters. The MLP is trained on synthetic point sets sampled uniformly in [0,1] and random affine matrices with entries in [-1,1]. Ten random subsets of 64 keypoints are averaged to reduce noise. Uncertainty is estimated by randomly blackening pixels in the moving image and computing the variance of the estimated parameters over 10 repetitions. The method is evaluated on BraTS, ALBERT, and hospital brain MRI data, with reported improvements in Dice, MI, SSIM, and MSE over SIFT and SimpleElastix at 0.2 seconds per image.

Significance. The method is conceptually simple and the training strategy for the regressor is clever: by training exclusively on synthetic coordinate pairs, the network is image-agnostic and the inference is extremely fast, which is attractive for intra-operative use. Evaluating on multiple MRI datasets is a strength. However, the paper does not currently validate the central claim of robust affine registration: there are no error bars or statistical tests, the evaluation metrics as written are incorrect, and the regressor's domain shift from synthetic uniform points to VGG-19 CoM keypoints is untested. These issues are fixable with additional experiments, and the core idea may be worth publishing after revision.

major comments (4)

[Section 2.5 and Section 2.8] The regressor is trained on synthetic point sets X ~ U[0,1] and random affine matrices with entries in [-1,1], but it is applied to CoM keypoints extracted from thresholded VGG-19 feature maps. This transfer assumes (i) that the CoM coordinates follow the same distribution as the synthetic points and (ii) that the ordered CoMs of fixed and moving images correspond under the true affine transform. Assumption (ii) is especially fragile because thresholding at 95% of the maximum activation and taking the center of mass is not affine-equivariant: after rotation, scale, or shear, the set of supra-threshold locations changes, so the CoM positions are biased in a content-dependent way. Averaging over ten sampled subsets reduces random sampling noise but cannot remove this systematic bias. The paper never evaluates the estimated affine parameters against the known artificial transformations introduced in Section 2.8, so a biased estimator that still improves overlap metrics cannot be distinguished from a correct one. This gap is load-bearing for the claim of robust affine registration.
[Section 2.6, Eqs. (3) and (5)] The Dice and MI formulas are mathematically incorrect. Eq. (3) writes Dice = 2|F∩M|/|F∪M|, which gives 2 for identical images instead of 1; the correct form uses |F|+|M| in the denominator. Eq. (5) is not mutual information: MI is the sum (or integral) of P_FM log(P_FM/(P_F P_M)) over joint outcomes, not E[P_FM] times the log ratio. Because the central comparisons in Table 1 and Fig. 4 are based on these metrics, the quantitative claims need to be re-derived with correct formulas.
[Table 1 and Fig. 4] The evaluation reports single aggregate numbers with no standard deviations, no sample sizes, no indication of the number of image pairs per dataset, and no statistical significance tests. The claim that the method 'proved to be robust' is not supported by the evidence: with one point per method per metric, the improvements could lie within inter-subject variability. The authors should report per-pair distributions, error bars, and at least a paired test (e.g., Wilcoxon signed-rank) for each metric and dataset.
[Section 2.5, Training Strategy and Parameter Estimation] The MLP architecture and training protocol are underspecified: the number of layers, hidden units, activation, optimizer, loss function, number of training pairs, epochs, and regularization are not reported. This makes the core transformation estimator non-reproducible. The iterative estimation mentioned in Section 2.8 ('with a pre-defined learning rate') is also not described. Please provide full details or code.

minor comments (5)

[Section 2.1 and reference [3]] The text refers to the 'ALBERT dataset' but reference [3] is the IXI dataset; the citation and dataset name do not match and should be corrected.
[Section 2.6, Eq. (4)] The text states that SSIM ∈ [0,1], but SSIM can be negative; the range statement is incorrect.
[Section 2.7] The uncertainty estimation description is inconsistent: the text says the variance of the transform parameters is the uncertainty, while Figures 3(e) and 3(j) display pixel-wise uncertainty maps. It is not explained how parameter variance is converted into per-pixel variance, nor is the number of blackened pixels specified.
[Abstract and Section 4] The term 'zero-shot learning' is misleading because the MLP regressor is trained specifically for the affine registration task, albeit on synthetic data; consider using 'unsupervised' or 'synthetic-data-trained' instead.
[Figure 4] The subfigures have no axis labels; the caption lists coordinate ranges but the axes themselves are unlabeled, making the scatter plots difficult to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the affine regressor is trained on independent synthetic correspondences and the reported metrics are evaluated on held-out image pairs.

full rationale

The derivation chain is self-contained with respect to the reported registration results. The affine regressor (Section 2.5, Training Strategy) is trained on synthetic point clouds X ~ U[0,1] and random affine matrices M with entries in [-1,1] using Y = MX; this training data is independent of the BraTS, ALBERT, and hospital MR images and of the Dice, MI, SSIM, and MSE metrics used in Section 3. The metrics are computed only after the estimated transform is applied to the moving image, and no equation in the paper shows a similarity metric entering the regressor's training objective or being used to fit the transform parameters. VGG-19 features come from a public ImageNet-pretrained model cited as [8,9], and the uncertainty estimate is a bootstrap-style procedure following the external reference [10]; neither is a load-bearing self-citation from the present authors. The 95% activation threshold is described as an experimentally determined hyperparameter, which is mild parameter selection rather than a prediction that reduces by construction to the reported outcome. The potential mismatch between the synthetic coordinate distribution and the VGG-CoM keypoint distribution is a correctness and robustness concern, not circularity, because the regressor is not fitted to the evaluation data or metrics. Therefore no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical entities or conserved quantities. Its load-bearing assumptions are all domain assumptions about feature transferability, keypoint reliability, and synthetic-to-real generalization, none of which are validated in the paper. The threshold and sampling parameters are hand-tuned and affect the reported outcomes.

free parameters (4)

feature map threshold = 0.95
Eq. 1 thresholds feature maps at 95% of maximum activation; described as experimentally determined, affecting which regions become CoM keypoints.
number of keypoints = 128
Section 2.5: 128 keypoints are used, equal to the number of feature maps in the first two layers of VGG-19; authors note this can be tuned as a hyperparameter.
keypoints sampled per set = 64
Section 2.5: ten sets of 64 keypoints are sampled with replacement from the 128 keypoints; this is a heuristic choice for bootstrap averaging.
affine matrix sampling range = [-1, 1]
Section 2.5: random transformation matrices M are generated from a uniform distribution between -1 and 1; this range may not match the deformations encountered at test time.

assumptions (5)

domain assumption ImageNet-pretrained VGG-19 features transfer to brain MRI for keypoint extraction
Section 2.3: VGG-19 trained on natural images is used without validation that its first-layer features preserve cross-modality correspondence in MRI.
domain assumption CoM of thresholded feature maps provides reliable and repeatable keypoints across fixed and moving images
Section 2.4: the entire registration pipeline depends on CoM keypoints, yet no analysis of keypoint repeatability, modality invariance, or failure cases is given.
domain assumption An MLP trained on random points and random affine matrices generalizes to real CoM keypoint distributions
Section 2.5: synthetic training data are drawn from uniform distributions, but the distribution of VGG-19 CoM keypoints from real MRI is unknown and unmodeled.
domain assumption Affine transformations are sufficient for the registration task
Only affine transforms are considered; the residual non-affine deformation is not measured, limiting the claim to affine cases.
standard math Standard backpropagation and MSE loss are suitable for training the regressor
Section 2.5: the regressor is trained with conventional supervised learning; this is standard practice and not a source of novel error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero Shot Learning for Multi-Modal Real Time Image Registration." pith.science (2026). https://pith.science/paper/TLQTFJYP

@misc{pith2026190806213,
  author       = {Pith},
  title        = {Pith review of: Zero Shot Learning for Multi-Modal Real Time Image Registration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLQTFJYP}},
  note         = {Machine review of arXiv:1908.06213}
}

read the original abstract

In this report we present an unsupervised image registration framework, using a pre-trained deep neural network as a feature extractor. We refer this to zero-shot learning, due to nonoverlap between training and testing dataset (none of the network modules in the processing pipeline were trained specifically for the task of medical image registration). Highlights of our technique are: (a) No requirement of a training dataset (b) Keypoints i.e.locations of important features are automatically estimated (c) The number of key points in this model is fixed and can possibly be tuned as a hyperparameter. (d) Uncertaintycalculation of the proposed, transformation estimates (e) Real-time registration of images. Our technique was evaluated on BraTS, ALBERT, and collaborative hospital Brain MRI data. Results suggest that the method proved to be robust for affine transformation models and the results are practically instantaneous, irrespective of the size of the input image

Figures

Figures reproduced from arXiv: 1908.06213 by the authors.

**Figure 2.** Above figure shows architecture used in transformation network. Network [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Intra and Inter modality MR registration. Time taken: 0.21 sec [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Scatter plots between similarity/ error score obtained before and after [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 8 canonical work pages

[1]

B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. I s gum, ``End-to-end unsupervised deformable image registration with a convolutional neural network,'' in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support , pp. 204--212, Springer, 2017

work page 2017
[2]

Masci, U

J. Masci, U. Meier, D. Cire s an, and J. Schmidhuber, ``Stacked convolutional auto-encoders for hierarchical feature extraction,'' in International Conference on Artificial Neural Networks , pp. 52--59, Springer, 2011

work page 2011
[3]

2018]..''

``Ixi - information extraction from images available at: http://www.brain-development.org/ [accessed 08 april. 2018]..''

work page 2018
[4]

B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al. , ``The multimodal brain tumor image segmentation benchmark (brats),'' IEEE transactions on medical imaging 34 (10), pp. 1993--2024, 2015

work page 1993
[5]

Bakas, H

S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, ``Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,'' Scientific data 4 , p. 170117, 2017

work page 2017
[6]

Bakas, H

S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, J. Freymann, K. Farahani, and C. Davatzikos, ``Segmentation labels and radiomic features for the pre-operative scans of the tcga-lgg collection,'' The Cancer Imaging Archive , 2017

work page 2017
[7]

Simonyan and A

K. Simonyan and A. Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' arXiv preprint arXiv:1409.1556 , 2014

arXiv 2014
[8]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, `` ImageNet: A Large-Scale Hierarchical Image Database ,'' in CVPR09 , 2009

work page 2009

Show all 11 references

[9]

Kybic, ``Bootstrap resampling for image registration uncertainty estimation without ground truth,'' IEEE Transactions on Image Processing , 2010

J. Kybic, ``Bootstrap resampling for image registration uncertainty estimation without ground truth,'' IEEE Transactions on Image Processing , 2010

2010
[10]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

B. D. de Vos, F. F. Berendsen, M. A. Viergever, M. Staring, and I. I s gum, ``End-to-end unsupervised deformable image registration with a convolutional neural network,'' in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support , pp. 204--212, Springer, 2017

work page 2017

[2] [2]

Masci, U

J. Masci, U. Meier, D. Cire s an, and J. Schmidhuber, ``Stacked convolutional auto-encoders for hierarchical feature extraction,'' in International Conference on Artificial Neural Networks , pp. 52--59, Springer, 2011

work page 2011

[3] [3]

2018]..''

``Ixi - information extraction from images available at: http://www.brain-development.org/ [accessed 08 april. 2018]..''

work page 2018

[4] [4]

B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest, et al. , ``The multimodal brain tumor image segmentation benchmark (brats),'' IEEE transactions on medical imaging 34 (10), pp. 1993--2024, 2015

work page 1993

[5] [5]

Bakas, H

S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, ``Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,'' Scientific data 4 , p. 170117, 2017

work page 2017

[6] [6]

Bakas, H

S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. Kirby, J. Freymann, K. Farahani, and C. Davatzikos, ``Segmentation labels and radiomic features for the pre-operative scans of the tcga-lgg collection,'' The Cancer Imaging Archive , 2017

work page 2017

[7] [7]

Simonyan and A

K. Simonyan and A. Zisserman, ``Very deep convolutional networks for large-scale image recognition,'' arXiv preprint arXiv:1409.1556 , 2014

arXiv 2014

[8] [8]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, `` ImageNet: A Large-Scale Hierarchical Image Database ,'' in CVPR09 , 2009

work page 2009