REVIEW 3 major objections 5 minor 39 references
Few Labeled Atlases are Necessary for Deep-Learning-Based Segmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Brain MRI segmentation with only three labeled atlases lands within four overlap-score points of full supervision, and the proposed semi-supervised variant improves boundary accuracy.
desk verdict A solid empirical study with a plausible central claim; the quantitative 'within four Dice points' needs error bars, but the core finding deserves serious review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a learning-based registration function $g_{\theta}(I,I^*)=\phi$ implemented as a convolutional encoder-decoder network, which warps one image onto another. It is trained with an unsupervised loss $L_{\text{img}} + \lambda L_{\text{smooth}}$ on pairs of one atlas and one unlabeled scan; the proposed semi-supervised extension occasionally feeds two atlases and adds $\gamma L_{\text{seg}}$, where $L_{\text{seg}}$ is the mean Dice overlap of the warped atlas label maps. A second mechanism is spatial augmentation: each atlas and its label map are deformed by a smooth random field before training, which creates new atlas-like examples. At test time, the trained network warps the $N$ labeled atlases plus $N_I$ augmented copies, propagates their one-hot label probabilities, and fuses them by voxel-wise maximum likelihood.
What would settle it
Train the same MAS-SS and SegNet-DA pipelines on a new anatomy (or a different MRI protocol) using only the small labeled set, with no auxiliary unlabeled images for registration training, and measure the Dice gap relative to a fully supervised model; if the gap is substantially larger than four points at three atlases, the claim that few labels suffice would be refuted.
Extended reading notes
Core claim
Using 7,329 unlabeled brain MRI scans to train an unsupervised registration network and 18 labeled atlases (with 10 held out for test), the paper shows that the number of labeled atlases needed for accurate segmentation is small. In multi-atlas segmentation, warping one-hot label probabilities from one to seven atlases and fusing them yields Dice scores that approach the fully supervised upper bound; adding random deformation augmentation to the atlas images helps most when only one or two atlases are used. In the supervised paradigm, training the same encoder-decoder network on the few labeled atlases with random smooth deformations also reaches near-upper-bound Dice once three or more atlases are present. The proposed semi-supervised method, MAS-SS, adds a segmentation-overlap (Dice) term to the registration loss during a fraction of training iterations, using atlas-to-atlas pairs, and this consistently improves Dice over the unsupervised MAS baseline while keeping mean surface distance below 0.4 mm and maximum surface distance below 7 mm. The paper concludes that deep-learning segmentation does not inherently require many labeled scans and that the topology-preserving registration framework is especially beneficial for small anatomical structures and boundary accuracy.
Load-bearing premise
The comparison rests on having thousands of unlabeled brain MRI scans to train the registration network; if such an unlabeled pool is not available for a new segmentation problem, the near-upper-bound performance with few labeled atlases is not established.
Editorial extensions
If this is right
- With only three labeled atlases, both multi-atlas segmentation and a data-augmented supervised network reach within four Dice points of a fully supervised model, so large annotation efforts are not automatically required for competitive brain MRI segmentation.
- Adding a semi-supervised Dice term to registration training improves Dice over the unsupervised baseline at every atlas count, and the improvement is largest for small structures such as the amygdala and hippocampus.
- Random smooth deformation augmentation alone makes supervised segmentation usable with very few labels, but it leaves larger surface distances than the registration-based approach.
- For one or two atlases, the semi-supervised multi-atlas method is the better choice; the supervised method catches up in Dice at roughly three atlases but not in boundary accuracy.
Reading between the lines
- An implicit consequence is that the scarce resource in few-label segmentation is unlabeled images rather than labels: the registration network is trained on thousands of unlabeled scans, so the recipe should transfer to any anatomy with a large unlabeled image pool.
- A testable extension is that distribution shift between the unlabeled corpus and the target scans will widen the reported Dice gap; the current experiments use brain MRIs from several public datasets, so the four-point gap is likely optimistic for a new scanner or protocol.
- Because the two paradigms differ more in surface distance than in Dice, boundary-sensitive applications such as surgical planning should prefer the semi-supervised multi-atlas method even when overlap scores look similar.
- The same semi-supervised registration loss could be tested in other few-label segmentation domains, and a plausible prediction is that the topology-preserving advantage will persist wherever anatomy has consistent structure across subjects.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies brain MRI segmentation when only N = 1..7 labeled atlases are available. It compares multi-atlas segmentation (MAS) with learning-based registration, including a proposed semi-supervised variant MAS-SS, against supervised CNNs (SegNet, SegNet-DA) and a fully supervised upper bound (SegNet-Full). The registration network is trained on 7,329 unlabeled T1 scans from eight public datasets, and the small labeled dataset is the Buckner40 set. The authors report that both paradigms achieve Dice scores within four points of the upper bound with only three atlases, that MAS-SS consistently improves over MAS/MAS-DA in Dice, and that MAS-SS yields significantly lower mean and maximum surface distances than SegNet-DA. The paper concludes that deep-learning segmentation does not necessarily require large labeled datasets.
Significance. The empirical finding is significant and potentially practice-relevant: if robust, it challenges the conventional assumption that CNN segmentation needs large labeled corpora and provides a systematic comparison of multi-atlas and supervised paradigms in the low-label regime. The paper has clear strengths: it sweeps N from 1 to 7, uses public datasets, reports two complementary metrics (Dice and surface distance), and proposes a simple semi-supervised registration variant that is a natural extension of voxel-based registration learning. The main risk is that the key quantitative claims currently lack explicit uncertainty quantification, so the strength of the conclusion exceeds what the reported numbers can support at this stage.
major comments (3)
- [Section 3.1.3 and Figure 3] The number n of random atlas-set permutations is never reported, and Figure 3 shows only mean curves without error bars, confidence intervals, or significance tests. With a test set of only 10 subjects, the claims that the best methods are 'within four Dice points' of the upper bound and that MAS-SS is 'consistently' and 'significantly better' in surface distance may reflect only the particular atlas sets and test subjects chosen. Please report n explicitly, add per-N variability (e.g., shaded standard deviations or box plots over atlas-set permutations), and perform paired statistical tests (e.g., Wilcoxon signed-rank) for the claimed improvements.
- [Sections 3.1.2 and 3.2] The favorable MAS results, including MAS-SS, depend on training the registration network with 7,329 unlabeled T1 scans from eight public datasets. Thus the title and abstract claim that 'few labeled atlases are necessary' is only valid when a large unlabeled image corpus is also available; without such unlabeled data, the proposed MAS-SS method cannot be trained as described. Please state this dependence prominently in the abstract and conclusion, and clarify whether SegNet-DA is the only method in the comparison that does not require unlabeled images.
- [Section 2.4 and Section 3.1.3] The number NI of augmented atlases used at evaluation time is introduced in Section 2.4 but never specified in the experimental section. This value directly determines how many warped label maps are fused and therefore affects the reported MAS Dice and surface-distance numbers. Please report NI and the remaining training hyperparameters (number of iterations, learning rate, batch size, and the details of the 'small single scenario' used to select the 10% supervised-iteration frequency) so the experiments are reproducible.
minor comments (5)
- [Section 3.1.3] There is a typo: 'Bucker40' should be 'Buckner40'.
- [Section 3.2] The word 'acheive' should be 'achieve'.
- [Abstract] The phrase 'These method' should be 'These methods'.
- [Figure 5 and Section 3.1.3] The symbol n is used in Figure 5 for the number of atlases, while Section 3.1.3 uses N for the number of atlases and n for the number of random permutations; please use consistent notation.
- [Section 3.1.2] Please clarify how many of the 7,829 scans in the eight public datasets have FreeSurfer segmentations used to train SegNet-Full, given that the same section states segmentations are 'only used for evaluation' and also that labels from these datasets are 'not used during training the registration network'.
Circularity Check
No significant circularity; the paper is an empirical comparison with an external held-out test set and no fitted prediction that reduces to its inputs.
full rationale
The paper is an empirical study comparing multi-atlas segmentation and supervised segmentation in the few-labeled-atlas regime. The proposed MAS-SS training loss (Eq. 2) combines an NCC image-similarity term, a smoothness term, and a Dice term measuring overlap between two atlas label maps after warping; the evaluation, in contrast, measures Dice and surface distance of the final fused segmentation against manual annotations on a held-out Buckner40 test set (Section 3.1.2 and Section 3.1.5). The training objective is therefore not the test metric by construction, and no model parameter is fitted to the test set. The headline result, that with three atlases the best MAS and SegNet methods are within four Dice points of the fully supervised upper bound, is a measured outcome of the experiments, not a consequence of a self-referential definition. Author self-citations such as VoxelMorph [7] are used to borrow network architecture and an unsupervised registration loss, but the central comparison also includes SegNet variants and external manual segmentations, so the load-bearing empirical claim does not reduce to a self-citation. The absence of error bars and the unspecified number of random atlas-set permutations are statistical reporting concerns, not circularity. No step in the paper's derivation chain equates a prediction with its input by construction.
Assumptions & free parameters
free parameters (4)
- lambda (smoothness weight) =
1.5
- gamma (Dice supervision weight) =
1.0
- supervised iteration frequency =
10%
- random deformation augmentation parameters =
not specified
assumptions (4)
- domain assumption Unlabeled brain MRI scans from eight public datasets are representative of the Buckner40 test distribution for training the registration network.
- domain assumption Random smooth deformations of the few labeled atlases generate realistic training examples.
- domain assumption Dice overlap between atlas segmentation maps is a valid supervisory signal for image registration, equivalent to anatomical correspondence.
- domain assumption Architecture and hyperparameters from VoxelMorph transfer to the few-atlas setting.
Cite this review
Pith. "Pith review of Few Labeled Atlases are Necessary for Deep-Learning-Based Segmentation." pith.science (2026). https://pith.science/paper/3BZ65UZQ
@misc{pith2026190804466,
author = {Pith},
title = {Pith review of: Few Labeled Atlases are Necessary for Deep-Learning-Based Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3BZ65UZQ}},
note = {Machine review of arXiv:1908.04466}
}
read the original abstract
We tackle biomedical image segmentation in the scenario of only a few labeled brain MR images. This is an important and challenging task in medical applications, where manual annotations are time-consuming. Current multi-atlas based segmentation methods use image registration to warp segments from labeled images onto a new scan. In a different paradigm, supervised learning-based segmentation strategies have gained popularity. These method consistently use relatively large sets of labeled training data, and their behavior in the regime of a few labeled biomedical images has not been thoroughly evaluated. In this work, we provide two important results for segmentation in the scenario where few labeled images are available. First, we propose a straightforward implementation of efficient semi-supervised learning-based registration method, which we showcase in a multi-atlas segmentation framework. Second, through an extensive empirical study, we evaluate the performance of a supervised segmentation approach, where the training images are augmented via random deformations. Surprisingly, we find that in both paradigms, accurate segmentation is generally possible even in the context of few labeled images.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
The ADHD Consortium. The adhd-200 consortium: A model to advance the translational potential of neuroimaging in clinical neuroscience. Frontiers in Systems Neuroscience, 6, 09 2012
work page 2012
-
[2]
Deep learning for brain mri segmentation: State of the art and future directions
Zeynettin Akkus, Alfiia Galimzianova, Assaf Hoogi, and Rubin. Deep learning for brain mri segmentation: State of the art and future directions. Journal of Digital Imaging, pages 1–11, 6 2017
work page 2017
-
[3]
X. Artaechevarria, A. Munoz-Barrutia, and C. Ortiz-de-Solorzano. Combination strategies in multi-atlas image segmentation: Application to brain mr data. IEEE Transactions on Medical Imaging, 28(8):1266–1277, Aug 2009
work page 2009
-
[4]
A fast diffeomorphic image registration algorithm
John Ashburner. A fast diffeomorphic image registration algorithm. Neuroimage, 38(1):95–113, 2007
work page 2007
-
[5]
Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffeo- morphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26–41, 2008
work page 2008
-
[6]
Multiresolution elastic matching
Ruzena Bajcsy and Stane Kovaˇciˇc. Multiresolution elastic matching. Computer vision, graphics, and image processing, 46(1):1–21, 1989
work page 1989
-
[7]
Guha Balakrishnan, Amy Zhao, Mert Sabuncu, John Guttag, and Adrian V . Dalca. V oxelmorph: A learning framework for deformable medical image registration. IEEE TMI, 2019. 7
work page 2019
-
[8]
Computing large deforma- tion metric mappings via geodesic flows of diffeomorphisms
M Faisal Beg, Michael I Miller, Alain Trouvé, and Laurent Younes. Computing large deforma- tion metric mappings via geodesic flows of diffeomorphisms. International journal of computer vision, 61(2):139–157, 2005
work page 2005
Show all 39 references
-
[9]
Semi- supervised and task-driven data augmentation
Krishna Chaitanya, Neerav Karani, Christian Baumgartner, and Ender Konukoglu. Semi- supervised and task-driven data augmentation. arXiv preprint arXiv:1902.05396, 2019
1902 arXiv
-
[10]
Harvard aging brain study: dataset and accessibility
Alexander Dagley, Molly LaPoint, Willem Huijbers, Trey Hedden, Donald G McLaren, Jas- meer P Chatwal, Kathryn V Papp, Rebecca E Amariglio, Deborah Blacker, Dorene M Rentz, et al. Harvard aging brain study: dataset and accessibility. NeuroImage, 2017
2017
-
[11]
Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces
Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu. Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. Medical Image Analysis, 2019
2019
-
[12]
Patch-based discrete registration of clinical brain images
Adrian V Dalca, Andreea Bobu, Natalia S Rost, and Polina Golland. Patch-based discrete registration of clinical brain images. International Workshop on Patch-based Techniques in Medical Imaging, (60–67), 2016
2016
-
[13]
Anatomical priors in convolutional networks for unsupervised biomedical segmentation
Adrian V Dalca, John Guttag, and Mert R Sabuncu. Anatomical priors in convolutional networks for unsupervised biomedical segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9290–9299, 2018
2018
-
[14]
A deep learning framework for unsupervised affine and deformable image registration
Bob de V os, Floris F Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Išgum. A deep learning framework for unsupervised affine and deformable image registration. Medical image analysis, 52:128–143, 2019
2019
-
[15]
The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism
Adriana Di Martino, Chao-Gan Yan, Qingyang Li, Erin Denio, Francisco X Castellanos, Kaat Alaerts, Jeffrey S Anderson, Michal Assaf, Susan Y Bookheimer, Mirella Dapretto, et al. The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain arch...
2014
-
[16]
L. R. Dice. Measures of the amount of ecologic association between species. Ecology, 1945
1945
-
[17]
B. Fischl. Freesurfer. Neuroimage, 62(2), 774-781., 2012
2012
-
[18]
The mcic collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia
Randy L Gollub, Jody M Shoemaker, Margaret D King, White, et al. The mcic collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia. Neuroinformatics, 11(3):367–388, 2013
2013
-
[19]
Automatic brain tumor detection and segmentation using u-net based fully convolutional networks
Dong Hao, Yang Guang, Liu Fangde, Mo Yuanhan, and Guo Yike. Automatic brain tumor detection and segmentation using u-net based fully convolutional networks. annual conference on medical image understanding and analysis, pages 506–517, 2017
2017
-
[20]
Brain genomics superstruct project initial data release with structural, functional, and behavioral measures
Avram J Holmes, Marisa O Hollinshead, Timothy M O’Keefe, et al. Brain genomics superstruct project initial data release with structural, functional, and behavioral measures. Scientific data, 2015
2015
-
[21]
Weakly-supervised convolutional neural networks for multimodal image registration
Yipeng Hu, Marc Modat, Eli Gibson, Wenqi Li, Nooshin Ghavami, Ester Bonmati, Guotai Wang, Steven Bandula, Caroline M Moore, Mark Emberton, et al. Weakly-supervised convolutional neural networks for multimodal image registration. Medical image analysis, 2018
2018
-
[22]
Differential data augmenta- tion techniques for medical imaging classification tasks
Zeshan Hussain, Francisco Gimenez, Darvin Yi, and Daniel Rubin. Differential data augmenta- tion techniques for medical imaging classification tasks. Annual Symposium proceedings. AMIA Symposium, 2017:979–984, 04 2018
2017
-
[23]
Multi-atlas segmentation of biomedical images: a survey
Juan Eugenio Iglesias and Mert R Sabuncu. Multi-atlas segmentation of biomedical images: a survey. Medical image analysis, 24(1):205–219, 2015
2015
-
[24]
Deep multi-class segmentation without ground-truth labels
Thomas Joyce, Agisilaos Chartsias, and Sotirios A Tsaftaris. Deep multi-class segmentation without ground-truth labels. International Conference on Medical Imaging with Deep Learning, 2018
2018
-
[25]
K Kamnitsas, C Ledig, V . F. Newcombe, et al. Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis, (36):61–78, 2017. 8
2017
-
[26]
Cnn-based segmentation of medical imaging data
Baris Kayalibay, Grady Jensen, and Patrick van der Smagt. Cnn-based segmentation of medical imaging data. arXiv preprint arXiv:1701.03056, 2017
2017 arXiv
-
[27]
Evaluation of 14 nonlinear deformation algorithms applied to human brain mri registration
Arno Klein, Jesper Andersson, A Ardekani, et al. Evaluation of 14 nonlinear deformation algorithms applied to human brain mri registration. NeuroImage, 46:786–802, 02 2009
2009
-
[28]
L. M. Koch, M. Rajchl, W. Bai, C. F. Baumgartner, T. Tong, J. Passerat-Palmbach, P. Aljabar, and D. Rueckert. Multi-atlas segmentation using partially annotated data: Methods and annotation strategies. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7):1683–...
2018
-
[29]
Learn- ing a probabilistic model for diffeomorphic registration
Julian Krebs, Hervé e Delingette, Boris Mailhé, Nicholas Ayache, and Tommaso Mansi. Learn- ing a probabilistic model for diffeomorphic registration. IEEE transactions on medical imaging, 2019
2019
-
[30]
Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults
Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csernansky, John C Morris, and Randy L Buckner. Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience, 19(9):1...
2007
-
[31]
The parkinson progression marker initiative (ppmi)
Kenneth Marek, Danna Jennings, Shirley Lasch, et al. The parkinson progression marker initiative (ppmi). Progress in neurobiology, 2011
2011
-
[32]
V-net: Fully convolutional neural networks for volumetric medical image segmentation
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. Fourth International Conference on 3D Vision, pages 565–571, 2016
2016
-
[33]
Ways toward an early diagnosis in alzheimer’s disease: the alzheimer’s disease neuroimaging initiative (adni)
Susanne G Mueller, Michael W Weiner, Thal, et al. Ways toward an early diagnosis in alzheimer’s disease: the alzheimer’s disease neuroimaging initiative (adni). Alzheimer’s & Dementia, 1(1):55–66, 2005
2005
-
[34]
Pereira, A
S. Pereira, A. Pinto, V . Alves, and C. A. Silva. Brain tumor segmentation using convolutional neural networks in mri images. IEEE Transactions on Medical Imaging, 35(5):1240–1251, May 2016
2016
-
[35]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. MICAAI, pages 234–241, 2015
2015
-
[36]
A generative model for image segmenta- tion based on label fusion
Mert R Sabuncu, BT Thomas Yeo, Van Leemput, et al. A generative model for image segmenta- tion based on label fusion. IEEE TMI, 29(10):1714–1729, 2010
2010
-
[37]
Multi-atlas segmentation with joint label fusion
Hongzhi Wang, Jung W Suh, Sandhitsu R Das, John B Pluta, Caryne Craige, and Paul A Yushkevich. Multi-atlas segmentation with joint label fusion. IEEE Trans Pattern Anal Mach Intell, 35(3):611–623, Mar 2013
2013
-
[38]
Quicksilver: Fast predictive image registration–a deep learning approach
Xiao Yang, Roland Kwitt, Martin Styner, and Marc Niethammer. Quicksilver: Fast predictive image registration–a deep learning approach. NeuroImage, 158:378–396, 2017
2017
-
[39]
Data augmentation using learned transforms for one-shot medical image segmentation
Amy Zhao, Guha Balakrishnan, Frédo Durand, John V Guttag, and Adrian V Dalca. Data augmentation using learned transforms for one-shot medical image segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 9
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.