Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read CNN training variability is not merely a reproducibility problem: it can be harvested as data augmentation for downstream neuroimaging tasks like brain-age regression.

desk verdict The variability measurement is worth a look, but the augmentation claim is contradicted by the paper's own Figure 3. read the letter →

arxiv 2509.05238 v2 pith:3FY3ANFI submitted 2025-09-05 math.NA cs.AIcs.NA

classification math.NAcs.AIcs.NA
keywords numericaluncertaintyCNNtrainingvariabilityFastSurferMonteCarloArithmeticdataaugmentationbrainageregressionensemblingneuroimagingreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that training a convolutional segmentation network (FastSurfer) carries genuine numerical uncertainty, and that this uncertainty is a usable resource rather than only a nuisance. It finds that FastSurfer's training-time variability is comparable to or larger than the traditional FreeSurfer pipeline in cortical regions, and that models trained under different random seeds or floating-point perturbations perform about as well as a stable baseline while producing distinct segmentations. The lead application is brain-age regression: pooling multiple FastSurfer repetitions per subject as augmented training data lowers test mean absolute error as the number of repetitions grows, reaching 7.40 years with ten repetitions, without collecting new scans. A careful reader would care because, if true, any research group that can retrain a segmentation model a few times gets a free, principled augmentation strategy and a better estimate of segmentation uncertainty.

What carries the argument

The central mechanism is the ensemble of 'numerical model families' created by perturbing training: Monte Carlo Arithmetic perturbs every floating-point operation with a random rounding error at a chosen virtual precision, while random seeds alter the pseudorandom initial state (weight sampling, dropout) with data order fixed to keep uncertainty epistemic. These perturbations turn one training run into a family of distinct but comparable segmentations. The paper then feeds the resulting per-subject ROI volumes from multiple FastSurfer runs into random-forest and SVM regressors, so the ensemble acts as a data augmentation operator: repetitions add diversity to the feature set without adding s

What would settle it

Train FastSurfer under MCA to full convergence (70 epochs) across all three planes and recompute min-Dice variability against FreeSurfer: if cortical variability no longer exceeds FreeSurfer, the paper's headline comparison is a checkpoint artifact. Separately, run brain-age regression with 20+ seed repetitions per subject: if test MAE stops falling or rises after 10 repetitions, the augmentation benefit saturates at exactly the regime the paper tested.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms: training FastSurfer with Monte Carlo Arithmetic (randomly perturbed floating-point operations) or with different random seeds produces whole-brain segmentations that differ measurably from each other, and these differences are not noise that degrades the model. In subcortical regions FastSurfer's variability is comparable to FreeSurfer's, while in all cortical ROIs tested it is statistically higher. MCA-induced and seed-induced variability are statistically distinct (p = 4.45e-313), yet both track the IEEE baseline's loss curves and achieve comparable performance. The constructive claim is that this multiplicity of valid outputs constitutes a numerical ensemb

Load-bearing premise

The comparison relies on an MCA FastSurfer checkpoint at epoch 10 from the coronal model only—the paper's own footnote says its Dice score differs by 4%—so if a fully trained MCA model is less variable, the cortical-excess result may be an artifact.

Editorial extensions

If this is right

  • If repeated FastSurfer runs under different seeds are valid segmentations, then single-run segmentation studies understate uncertainty; reporting ensemble spread (min Dice across runs) becomes a practical reliability metric.
  • Brain-age regression improves with augmentation from repeated runs (test MAE 7.40 years at 10 repetitions vs. higher with fewer), so other ROI-based downstream models could adopt the same strategy without new data.
  • The finding that FastSurfer is more variable than FreeSurfer in all cortical ROIs would redirect reproducibility checks toward cortical parcellation, not just subcortical volumes.
  • Because MCA and seed perturbations are statistically distinct, combining both in one ensemble may yield more diversity than either alone.
  • The loss variability concentrated at cosineWarmRestarts restarts implies that model checkpoints near learning-rate restarts are especially diverse, so choosing checkpoints may control ensemble diversity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the augmentation benefit probably transfers beyond brain age—any ROI-feature classifier (e.g., Alzheimer's vs control, or cortical-thickness group comparisons) could treat a few FastSurfer seeds as a cheap ensemble; the paper only demonstrates the one task.
  • Editorial extension: if MCA variability were measured at full convergence rather than epoch 10, the cortical 'more variable than FreeSurfer' result could shrink; a completed MCA training would settle whether the comparison holds.
  • Editorial extension: the observed saturation ('a few repetitions capture most of the benefit') suggests the practical recipe is 3-10 runs, and beyond that the gains may be outweighed by compute; the paper doesn't test the saturation point directly.
  • Editorial extension: learning-rate restart points could be exploited deliberately—harvesting checkpoints at restarts might generate diverse segmentations at lower cost than full retrainings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies training-time numerical variability in FastSurfer, a CNN whole-brain segmentation pipeline, using Monte Carlo Arithmetic (MCA) and random-seed perturbations, and compares this variability with FreeSurfer. It claims that FastSurfer training variability is comparable to or larger than FreeSurfer's in cortical regions, that perturbed models perform on par with the baseline, and that the variability can be exploited as data augmentation to improve downstream brain-age regression. The paper includes reproducibility code and a small MNIST sanity check.

Significance. The topic is timely: if the claims held, training-time instability in deep learning would be recast as a usable resource rather than only a reproducibility hazard. The authors provide concrete external validation (FreeSurfer comparison, brain-age regression) and share code, which are strengths. However, the central 'useful' claim is internally contradicted by the paper's own SVM result, and the variability comparison rests on a partially trained coronal model. The manuscript's data do not currently support the title's promise.

major comments (3)
  1. [§3, Figure 3] The text states that 'increasing repetitions consistently reduces test MAE,' but the reported SVM Pearson R=0.803 (p=5.154e-3) is a statistically significant positive correlation: MAE increases with the number of repetitions. The sentence about extreme MAE values in the first and last repetitions does not explain a monotone positive trend across the full range. This direct contradiction invalidates the 'consistently' claim and, with it, the data-augmentation conclusion. The Random Forest correlation (R=-0.830) cannot rescue the claim without per-repetition error bars and a complete description of the protocol.
  2. [§3, footnote 1 and Figure 1] The MCA FastSurfer results are available only for the coronal model and are reported at epoch 10; the paper admits the Dice score at that point differs by 4%. The comparison with FreeSurfer is therefore between a partially trained FastSurfer model and a complete FreeSurfer pipeline. If the epoch-10 checkpoint is not representative of the fully trained model, the conclusion that FastSurfer is more variable in cortical regions may be an artifact of incomplete training. The authors must justify this proxy or provide fully trained checkpoint comparisons.
  3. [§3, brain-age regression analysis] The augmentation protocol is underspecified. It is not stated how the 10 FastSurfer repetitions per subject are partitioned into training and test sets, whether the same subject appears in both training and test, or whether the reported MAE (e.g., 7.40 years) is averaged over repeated test segmentations. Without this information, the correlation in Figure 3 and the claimed improvement cannot be interpreted. A no-augmentation baseline is also missing, so 'improves predictive performance' is not established.
minor comments (5)
  1. [Abstract / §3] The abstract and conclusion claim FastSurfer variability 'exceeds' FreeSurfer in cortical regions, but Figure 1 only shows the left hemisphere and the MCA results are at epoch 10; the claims should be qualified accordingly.
  2. [Figure 1] The y-axis label 'Sorensen Dice Scores' should be 'Sørensen–Dice Scores'.
  3. [Figure 2 caption] The caption says 'Training loss variability across random seeds,' but the figure also shows MCA and Default IEEE curves; the caption should be updated.
  4. [References] The reference '[fas]' (PyTorch Implementation of FastSurferCNN) lacks full bibliographic information; please include authors and year.
  5. [Title page] The author name 'Y ohan Chatelain' contains a spacing typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the paper's claims are empirical comparisons against external benchmarks and independent downstream tasks.

full rationale

The paper's central claims are (i) FastSurfer training exhibits numerical variability comparable to or exceeding FreeSurfer, (ii) ensembles from perturbed training runs perform comparably to baseline, and (iii) random-seed ensembles can be used as data augmentation for brain age regression. None of these claims reduce to the paper's inputs by construction. The variability measurements are empirical: FastSurfer and FreeSurfer are compared via Sørensen-Dice scores on a CoRR test subset, an external dataset. The brain-age augmentation result is evaluated by test MAE against the number of repetitions for Random Forest and SVM regressors; no parameter is fitted to the downstream target and then reported as a prediction. The paper cites the authors' own prior tools and results (Fuzzy PyTorch, prior inference-variability work, MCA-based augmentation), but these citations are used as tools, context, or motivation rather than as load-bearing justification for the empirical findings. The Monte Carlo Arithmetic perturbation formula is explicitly stated, and the comparison to FreeSurfer is an external benchmark. One concern noted in Figure 3 is an internal inconsistency: the text claims increasing repetitions consistently reduces test MAE, but the SVM result shows a positive correlation (R=0.803, p=5.154e-3). This is a correctness or interpretation issue, not circularity. There is no self-definitional step, no fitted input renamed as a prediction, and no self-citation chain forcing the claimed conclusions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on several unstated or weakly supported premises: that the epoch-10 coronal MCA model proxies full training, that random-seed variation stands in for numerical variation despite a t-test showing the two are statistically distinct, and that the ambiguous pretrained/retrained FastSurfer setup matches the official model. No new physical or mathematical entities are introduced.

free parameters (3)
  • MCA virtual precision t = not reported
    Controls perturbation magnitude in inexact(x)=x+2^(ex-t) xi; no value is given, yet it sets the noise level for all MCA experiments.
  • Epoch-10 reporting point = 10
    Ad hoc choice because MCA training was ongoing; the Dice score at this epoch differs by 4% from the final value (footnote 1).
  • Number of ensemble repetitions = 10
    The fixed number of retrainings used for the brain-age augmentation experiment; no saturation or cost-benefit analysis is given.
assumptions (4)
  • domain assumption FastSurfer retraining reproduces the official model behavior
    The paper is ambiguous about whether the pretrained v2.4.0 checkpoint or a locally retrained model is used (Appendix A.2.2).
  • ad hoc to paper Random seed and MCA variability are structurally comparable sources of training uncertainty
    The paper asserts comparability based on similar magnitudes but reports a two-sided t-test (p=4.45e-313) showing the two distributions are distinct.
  • ad hoc to paper The epoch-10 coronal MCA model is a fair proxy for fully trained FastSurfer variability
    Footnote 1 admits the Dice score at this point differs by 4%; this premise is load-bearing for the FastSurfer/FreeSurfer comparison.
  • domain assumption FreeSurfer segmentations are an appropriate reference standard for evaluating FastSurfer variability
    FastSurfer is trained on FreeSurfer output, so variability relative to the teacher may not generalize to other reference standards.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation." pith.science (2026). https://pith.science/paper/3FY3ANFI

@misc{pith2026250905238,
  author       = {Pith},
  title        = {Pith review of: Uncertain but Useful: Leveraging CNN Training Variability into Data Augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FY3ANFI}},
  note         = {Machine review of arXiv:2509.05238}
}
read the original abstract

Deep learning (DL) has transformed neuroimaging by delivering state-of-the-art performance with reduced computation times. Yet, the numerical uncertainty inherent to DL training remains largely underexplored despite its potential to significantly impact the reliability of model outcomes. We show that training the FastSurfer segmentation model introduces substantial numerical uncertainty that exceeds its non-DL counterpart (FreeSurfer 7.3.2) in cortical regions, potentially impacting downstream clinical results. We also characterize this training-time uncertainty using random seed perturbations and demonstrate that seed-induced variability is structurally comparable to numerical variability. We then show that seed variability can be leveraged as a data augmentation technique through ensembling to improve downstream brain age regression performance. These findings position numerical uncertainty during DL training as a substantive factor in neuroimaging reliability, with measurable consequences for downstream tasks, and demonstrate that it can simultaneously be harnessed as a data augmentation technique.

Figures

Figures reproduced from arXiv: 2509.05238 by the authors.

Figure 1
Figure 1. Variability comparison of minimum Dice scores across FastSurfer and FreeSurfer regions of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Training loss variability across random seeds for FastSurfer coronal model. Axial, sagittal, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Linear regression illustrating the effect of data augmentation on performance mean absolute [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Illustration of FastSurfer’s architecture. The CNN consists of four competitive dense [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of sources of variability across validation loss for MNIST model [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison of sources of variability across test accuracy and loss for MNIST model [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

  1. [1]

    https://github.com/Deep-MI/FastSurfer

    PyTorch Implementation of FastSurferCNN. https://github.com/Deep-MI/FastSurfer. Accessed: 2024-09-28. Yohan Chatelain, Nigel Yong Sao Young, Gregory Kiar, and Tristan Glatard. Pytracer: Automatically profiling numerical instabilities in python.IEEE Transactions on Computers, 72(6):1792–1803,

  2. [9]

    Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic

    Ali Salari, Yohan Chatelain, Gregory Kiar, and Tristan Glatard. Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic. InUncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis: 3rd International Workshop, UNSURE 2021, and 6th International ...

  3. [10]

    Xi-Nian Zuo, Jeffrey S Anderson, Pierre Bellec, Rasmus M Birn, Bharat B Biswal, Janusch Blautzik, John Breitner, Randy L Buckner, Vince D Calhoun, F Xavier Castellanos, et al

    doi: 10.1016/j.neuroimage.2013.05.041. Xi-Nian Zuo, Jeffrey S Anderson, Pierre Bellec, Rasmus M Birn, Bharat B Biswal, Janusch Blautzik, John Breitner, Randy L Buckner, Vince D Calhoun, F Xavier Castellanos, et al. An Open Science Resource for Establishing Reliability and Reproducibility in Functional Connectomics.Scientific Data, 1(1):1–13,

  4. [12]

    compiler. A.2.2 FastSurfer FastSurfer [Henschel et al., 2022] is a CNN model that performs whole-brain segmentation, cortical surface reconstruction, fast spherical mapping, and cortical thickness analysis from anatomical MRIs. The FastSurfer CNN is composed of three 2D fully convolutional neural networks—each associated with a different 2D slice orientat...

  5. [13]

    We obtained access to all training and validation datasets used by the authors (HCP Van Essen et al

    and Github repository fas. We obtained access to all training and validation datasets used by the authors (HCP Van Essen et al. [2013], ABIDE-I, ABIDE-II, ADNI Mueller et al. [2005], IXI, 7 LA5C, MIRIAD Malone et al. [2013], OASIS1 Marcus et al. [2007], OASIS2 Marcus et al. [2009]), with the exception of the Rhineland Study dataset, due to data access res...

  6. [2005]

    Douglass Stott Parker.Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic

    doi: 10.1016/j.jalz.2005.06.003. Douglass Stott Parker.Monte Carlo Arithmetic: Exploiting Randomness in Floating-Point Arithmetic. University of California (Los Angeles). Computer Science Department,

  7. [2007]

    Daniel S

    doi: 10.1162/jocn.2007.19.9.1498. Daniel S. Marcus, Alexander F. Fotenos, John G. Csernansky, John C. Morris, and Randy L. Buckner. Open access series of imaging studies (oasis): Longitudinal mri data in nondemented and demented older adults.Journal of Cognitive Neuroscience, 22(12):2677–2684,

  8. [2009]

    2009.21407

    doi: 10.1162/jocn. 2009.21407. Susanne G. Mueller, Michael W. Weiner, Leon J. Thal, Ronald C. Petersen, Clifford R. Jack, William Jagust, John Q. Trojanowski, Arthur W. Toga, and Laurel Beckett. Ways toward an early diagnosis in alzheimer’s disease: the alzheimer’s disease neuroimaging initiative (adni).Alzheimer’s & Dementia, 1(1):55–66,

Show all 14 references
  1. [2013]

    Daniel S

    doi: 10.1016/j.neuroimage.2012.12.044. Daniel S. Marcus, Tracy H. Wang, Jill Parker, John G. Csernansky, John C. Morris, and Randy L. Buckner. Open access series of imaging studies (oasis): Cross-sectional mri data in young, middle aged, nondemented, and demented older adults....

  2. [2014]

    fuzzy libmath

    6 A Experimental Reproducibility The code to reproduce this work is located at https://github.com/InesGP/cnn_training_ variability. A.1 Computational Infrastructure The experiments were conducted on several clusters. Analysis was conducted on a server equipped with 8 compute n...

  3. [2016]

    doi: 10.1109/ARITH.2016.31

    IEEE Computer Society. doi: 10.1109/ARITH.2016.31. URLhttps://doi.ieeecomputersociety.org/10.1109/ARITH.2016.31. Bruce Fischl, David H Salat, Evelina Busa, Marilyn Albert, Megan Dieterich, Christian Haselgrove, Andre Van Der Kouwe, Ron Killiany, David Kennedy, Shuna Klaveness,...

  4. [2022]

    Data augmentation through monte carlo arithmetic leads to more generalizable classification in connectomics.arXiv preprint arXiv:2109.09649,

    Gregory Kiar, Yohan Chatelain, Ali Salari, Alan C Evans, and Tristan Glatard. Data augmentation through monte carlo arithmetic leads to more generalizable classification in connectomics.arXiv preprint arXiv:2109.09649,

  5. [2023]

    The CNN consists of four competitive dense blocks (CDB) in the encoder and decoder part, separated by a bottleneck layer

    Figure 4: Illustration of FastSurfer’s architecture. The CNN consists of four competitive dense blocks (CDB) in the encoder and decoder part, separated by a bottleneck layer. Figure reproduced from Henschel et al. [2020]. A.3 Data Use Acknowledgment Data used in the preparatio...

  6. [2025]

    Krzysztof J Gorgolewski and Russell A Poldrack

    Accessed: 2025-09-03. Krzysztof J Gorgolewski and Russell A Poldrack. A practical guide for improving transparency and reproducibility in neuroimaging research.PLoS biology, 14(7):e1002506,

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.