REVIEW 3 major objections 5 minor 14 references
Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read CNN training variability is not merely a reproducibility problem: it can be harvested as data augmentation for downstream neuroimaging tasks like brain-age regression.
desk verdict The variability measurement is worth a look, but the augmentation claim is contradicted by the paper's own Figure 3. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the ensemble of 'numerical model families' created by perturbing training: Monte Carlo Arithmetic perturbs every floating-point operation with a random rounding error at a chosen virtual precision, while random seeds alter the pseudorandom initial state (weight sampling, dropout) with data order fixed to keep uncertainty epistemic. These perturbations turn one training run into a family of distinct but comparable segmentations. The paper then feeds the resulting per-subject ROI volumes from multiple FastSurfer runs into random-forest and SVM regressors, so the ensemble acts as a data augmentation operator: repetitions add diversity to the feature set without adding s
What would settle it
Train FastSurfer under MCA to full convergence (70 epochs) across all three planes and recompute min-Dice variability against FreeSurfer: if cortical variability no longer exceeds FreeSurfer, the paper's headline comparison is a checkpoint artifact. Separately, run brain-age regression with 20+ seed repetitions per subject: if test MAE stops falling or rises after 10 repetitions, the augmentation benefit saturates at exactly the regime the paper tested.
Extended reading notes
Core claim
Stated on the paper's own terms: training FastSurfer with Monte Carlo Arithmetic (randomly perturbed floating-point operations) or with different random seeds produces whole-brain segmentations that differ measurably from each other, and these differences are not noise that degrades the model. In subcortical regions FastSurfer's variability is comparable to FreeSurfer's, while in all cortical ROIs tested it is statistically higher. MCA-induced and seed-induced variability are statistically distinct (p = 4.45e-313), yet both track the IEEE baseline's loss curves and achieve comparable performance. The constructive claim is that this multiplicity of valid outputs constitutes a numerical ensemb
Load-bearing premise
The comparison relies on an MCA FastSurfer checkpoint at epoch 10 from the coronal model only—the paper's own footnote says its Dice score differs by 4%—so if a fully trained MCA model is less variable, the cortical-excess result may be an artifact.
Editorial extensions
If this is right
- If repeated FastSurfer runs under different seeds are valid segmentations, then single-run segmentation studies understate uncertainty; reporting ensemble spread (min Dice across runs) becomes a practical reliability metric.
- Brain-age regression improves with augmentation from repeated runs (test MAE 7.40 years at 10 repetitions vs. higher with fewer), so other ROI-based downstream models could adopt the same strategy without new data.
- The finding that FastSurfer is more variable than FreeSurfer in all cortical ROIs would redirect reproducibility checks toward cortical parcellation, not just subcortical volumes.
- Because MCA and seed perturbations are statistically distinct, combining both in one ensemble may yield more diversity than either alone.
- The loss variability concentrated at cosineWarmRestarts restarts implies that model checkpoints near learning-rate restarts are especially diverse, so choosing checkpoints may control ensemble diversity.
Reading between the lines
- Editorial extension: the augmentation benefit probably transfers beyond brain age—any ROI-feature classifier (e.g., Alzheimer's vs control, or cortical-thickness group comparisons) could treat a few FastSurfer seeds as a cheap ensemble; the paper only demonstrates the one task.
- Editorial extension: if MCA variability were measured at full convergence rather than epoch 10, the cortical 'more variable than FreeSurfer' result could shrink; a completed MCA training would settle whether the comparison holds.
- Editorial extension: the observed saturation ('a few repetitions capture most of the benefit') suggests the practical recipe is 3-10 runs, and beyond that the gains may be outweighed by compute; the paper doesn't test the saturation point directly.
- Editorial extension: learning-rate restart points could be exploited deliberately—harvesting checkpoints at restarts might generate diverse segmentations at lower cost than full retrainings.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies training-time numerical variability in FastSurfer, a CNN whole-brain segmentation pipeline, using Monte Carlo Arithmetic (MCA) and random-seed perturbations, and compares this variability with FreeSurfer. It claims that FastSurfer training variability is comparable to or larger than FreeSurfer's in cortical regions, that perturbed models perform on par with the baseline, and that the variability can be exploited as data augmentation to improve downstream brain-age regression. The paper includes reproducibility code and a small MNIST sanity check.
Significance. The topic is timely: if the claims held, training-time instability in deep learning would be recast as a usable resource rather than only a reproducibility hazard. The authors provide concrete external validation (FreeSurfer comparison, brain-age regression) and share code, which are strengths. However, the central 'useful' claim is internally contradicted by the paper's own SVM result, and the variability comparison rests on a partially trained coronal model. The manuscript's data do not currently support the title's promise.
major comments (3)
- [§3, Figure 3] The text states that 'increasing repetitions consistently reduces test MAE,' but the reported SVM Pearson R=0.803 (p=5.154e-3) is a statistically significant positive correlation: MAE increases with the number of repetitions. The sentence about extreme MAE values in the first and last repetitions does not explain a monotone positive trend across the full range. This direct contradiction invalidates the 'consistently' claim and, with it, the data-augmentation conclusion. The Random Forest correlation (R=-0.830) cannot rescue the claim without per-repetition error bars and a complete description of the protocol.
- [§3, footnote 1 and Figure 1] The MCA FastSurfer results are available only for the coronal model and are reported at epoch 10; the paper admits the Dice score at that point differs by 4%. The comparison with FreeSurfer is therefore between a partially trained FastSurfer model and a complete FreeSurfer pipeline. If the epoch-10 checkpoint is not representative of the fully trained model, the conclusion that FastSurfer is more variable in cortical regions may be an artifact of incomplete training. The authors must justify this proxy or provide fully trained checkpoint comparisons.
- [§3, brain-age regression analysis] The augmentation protocol is underspecified. It is not stated how the 10 FastSurfer repetitions per subject are partitioned into training and test sets, whether the same subject appears in both training and test, or whether the reported MAE (e.g., 7.40 years) is averaged over repeated test segmentations. Without this information, the correlation in Figure 3 and the claimed improvement cannot be interpreted. A no-augmentation baseline is also missing, so 'improves predictive performance' is not established.
minor comments (5)
- [Abstract / §3] The abstract and conclusion claim FastSurfer variability 'exceeds' FreeSurfer in cortical regions, but Figure 1 only shows the left hemisphere and the MCA results are at epoch 10; the claims should be qualified accordingly.
- [Figure 1] The y-axis label 'Sorensen Dice Scores' should be 'Sørensen–Dice Scores'.
- [Figure 2 caption] The caption says 'Training loss variability across random seeds,' but the figure also shows MCA and Default IEEE curves; the caption should be updated.
- [References] The reference '[fas]' (PyTorch Implementation of FastSurferCNN) lacks full bibliographic information; please include authors and year.
- [Title page] The author name 'Y ohan Chatelain' contains a spacing typo.
Circularity Check
No circular derivation found; the paper's claims are empirical comparisons against external benchmarks and independent downstream tasks.
full rationale
The paper's central claims are (i) FastSurfer training exhibits numerical variability comparable to or exceeding FreeSurfer, (ii) ensembles from perturbed training runs perform comparably to baseline, and (iii) random-seed ensembles can be used as data augmentation for brain age regression. None of these claims reduce to the paper's inputs by construction. The variability measurements are empirical: FastSurfer and FreeSurfer are compared via Sørensen-Dice scores on a CoRR test subset, an external dataset. The brain-age augmentation result is evaluated by test MAE against the number of repetitions for Random Forest and SVM regressors; no parameter is fitted to the downstream target and then reported as a prediction. The paper cites the authors' own prior tools and results (Fuzzy PyTorch, prior inference-variability work, MCA-based augmentation), but these citations are used as tools, context, or motivation rather than as load-bearing justification for the empirical findings. The Monte Carlo Arithmetic perturbation formula is explicitly stated, and the comparison to FreeSurfer is an external benchmark. One concern noted in Figure 3 is an internal inconsistency: the text claims increasing repetitions consistently reduces test MAE, but the SVM result shows a positive correlation (R=0.803, p=5.154e-3). This is a correctness or interpretation issue, not circularity. There is no self-definitional step, no fitted input renamed as a prediction, and no self-citation chain forcing the claimed conclusions.
Assumptions & free parameters
free parameters (3)
- MCA virtual precision t =
not reported
- Epoch-10 reporting point =
10
- Number of ensemble repetitions =
10
assumptions (4)
- domain assumption FastSurfer retraining reproduces the official model behavior
- ad hoc to paper Random seed and MCA variability are structurally comparable sources of training uncertainty
- ad hoc to paper The epoch-10 coronal MCA model is a fair proxy for fully trained FastSurfer variability
- domain assumption FreeSurfer segmentations are an appropriate reference standard for evaluating FastSurfer variability
Cite this review
Pith. "Pith review of Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation." pith.science (2026). https://pith.science/paper/3FY3ANFI
@misc{pith2026250905238,
author = {Pith},
title = {Pith review of: Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3FY3ANFI}},
note = {Machine review of arXiv:2509.05238}
}
read the original abstract
Deep learning (DL) has transformed neuroimaging by delivering state-of-the-art performance with reduced computation times. Yet, the numerical uncertainty inherent to DL training remains largely underexplored despite its potential to significantly impact the reliability of model outcomes. We show that training the FastSurfer segmentation model introduces substantial numerical uncertainty that exceeds its non-DL counterpart (FreeSurfer 7.3.2) in cortical regions, potentially impacting downstream clinical results. We also characterize this training-time uncertainty using random seed perturbations and demonstrate that seed-induced variability is structurally comparable to numerical variability. We then show that seed variability can be leveraged as a data augmentation technique through ensembling to improve downstream brain age regression performance. These findings position numerical uncertainty during DL training as a substantive factor in neuroimaging reliability, with measurable consequences for downstream tasks, and demonstrate that it can simultaneously be harnessed as a data augmentation technique.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/Deep-MI/FastSurfer
PyTorch Implementation of FastSurferCNN. https://github.com/Deep-MI/FastSurfer. Accessed: 2024-09-28. Yohan Chatelain, Nigel Yong Sao Young, Gregory Kiar, and Tristan Glatard. Pytracer: Automatically profiling numerical instabilities in python.IEEE Transactions on Computers, 72(6):1792–1803,
work page 2024
-
[9]
Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic
Ali Salari, Yohan Chatelain, Gregory Kiar, and Tristan Glatard. Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic. InUncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis: 3rd International Workshop, UNSURE 2021, and 6th International ...
work page 2021
-
[10]
doi: 10.1016/j.neuroimage.2013.05.041. Xi-Nian Zuo, Jeffrey S Anderson, Pierre Bellec, Rasmus M Birn, Bharat B Biswal, Janusch Blautzik, John Breitner, Randy L Buckner, Vince D Calhoun, F Xavier Castellanos, et al. An Open Science Resource for Establishing Reliability and Reproducibility in Functional Connectomics.Scientific Data, 1(1):1–13,
-
[12]
compiler. A.2.2 FastSurfer FastSurfer [Henschel et al., 2022] is a CNN model that performs whole-brain segmentation, cortical surface reconstruction, fast spherical mapping, and cortical thickness analysis from anatomical MRIs. The FastSurfer CNN is composed of three 2D fully convolutional neural networks—each associated with a different 2D slice orientat...
work page 2022
-
[13]
We obtained access to all training and validation datasets used by the authors (HCP Van Essen et al
and Github repository fas. We obtained access to all training and validation datasets used by the authors (HCP Van Essen et al. [2013], ABIDE-I, ABIDE-II, ADNI Mueller et al. [2005], IXI, 7 LA5C, MIRIAD Malone et al. [2013], OASIS1 Marcus et al. [2007], OASIS2 Marcus et al. [2009]), with the exception of the Rhineland Study dataset, due to data access res...
work page 2013
-
[2005]
Douglass Stott Parker.Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic
doi: 10.1016/j.jalz.2005.06.003. Douglass Stott Parker.Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic. University of California (Los Angeles). Computer Science Department,
-
[2007]
doi: 10.1162/jocn.2007.19.9.1498. Daniel S. Marcus, Alexander F. Fotenos, John G. Csernansky, John C. Morris, and Randy L. Buckner. Open access series of imaging studies (oasis): Longitudinal mri data in nondemented and demented older adults.Journal of Cognitive Neuroscience, 22(12):2677–2684,
-
[2009]
doi: 10.1162/jocn. 2009.21407. Susanne G. Mueller, Michael W. Weiner, Leon J. Thal, Ronald C. Petersen, Clifford R. Jack, William Jagust, John Q. Trojanowski, Arthur W. Toga, and Laurel Beckett. Ways toward an early diagnosis in alzheimer’s disease: the alzheimer’s disease neuroimaging initiative (adni).Alzheimer’s & Dementia, 1(1):55–66,
arXiv 2009
Show all 14 references
-
[2013]
Daniel S
doi: 10.1016/j.neuroimage.2012.12.044. Daniel S. Marcus, Tracy H. Wang, Jill Parker, John G. Csernansky, John C. Morris, and Randy L. Buckner. Open access series of imaging studies (oasis): Cross-sectional mri data in young, middle aged, nondemented, and demented older adults....
2012 doi
-
[2014]
fuzzy libmath
6 A Experimental Reproducibility The code to reproduce this work is located at https://github.com/InesGP/cnn_training_ variability. A.1 Computational Infrastructure The experiments were conducted on several clusters. Analysis was conducted on a server equipped with 8 compute n...
2002
-
[2016]
doi: 10.1109/ARITH.2016.31
IEEE Computer Society. doi: 10.1109/ARITH.2016.31. URLhttps://doi.ieeecomputersociety.org/10.1109/ARITH.2016.31. Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Haselgrove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness,...
2016 doi
-
[2022]
Data augmentation through monte carlo arithmetic leads to more generalizable classification in connectomics.arXiv preprint arXiv:2109.09649,
Gregory Kiar, Yohan Chatelain, Ali Salari, Alan C Evans, and Tristan Glatard. Data augmentation through monte carlo arithmetic leads to more generalizable classification in connectomics.arXiv preprint arXiv:2109.09649,
-
[2023]
The CNN consists of four competitive dense blocks (CDB) in the encoder and decoder part, separated by a bottleneck layer
Figure 4: Illustration of FastSurfer’s architecture. The CNN consists of four competitive dense blocks (CDB) in the encoder and decoder part, separated by a bottleneck layer. Figure reproduced from Henschel et al. [2020]. A.3 Data Use Acknowledgment Data used in the preparatio...
2020
-
[2025]
Krzysztof J Gorgolewski and Russell A Poldrack
Accessed: 2025-09-03. Krzysztof J Gorgolewski and Russell A Poldrack. A practical guide for improving transparency and reproducibility in neuroimaging research.PLoS biology, 14(7):e1002506,
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.