{"id":"3897462c-e20f-4be8-9063-d3719a0642ef","arxiv_id":"2509.05238","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Training-time variability in FastSurfer is comparable in scale to FreeSurfer's, but the paper's evidence that this variability improves brain-age regression is internally contradictory.","lead":"This paper measures how much brain-segmentation outputs vary when the same CNN is trained with different random seeds or floating-point noise, and tests whether that variation can be used as extra training data for brain-age prediction. It is a candidate read for anyone who wants to know whether training-time nondeterminism in deep learning is a problem or a resource.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 3's SVM result contradicts the central augmentation claim: MAE increases with repetitions (R=0.803), while the text claims 'consistently reduces test MAE.'","rationale":"The paper explores a novel question and includes useful details (reproducibility link, MNIST sanity check, explicit methodology), but its central augmentation claim is undercut by an internal contradiction. The reader's verdict is REJECT, and the reader's rationale already notes the SVM positive correlation, but their formal 'weakest_assumption' focused on the epoch-10 MCA proxy. I find the SVM contradiction more load-bearing because it directly invalidates the central claim that ensembling repetitions improves downstream performance. The epoch-10 issue affects the variability comparison but could plausibly be fixed by reporting final checkpoints; the SVM contradiction cannot be fixed without changing the reported data or the conclusion. Thus I agree with the REJECT verdict but for a slightly different primary reason.","tokens_in":10024,"tokens_out":4678,"duration_ms":43653,"concrete_test":"Extract the underlying per-repetition MAE values from the code/data repository (https://github.com/InesGP/cnn_training_variability) for both Random Forest and SVM. Compute the mean and standard error of test MAE for each repetition count, and run both linear and Spearman rank correlations separately for each model. If the SVM correlation is positive and significant, or if the Random Forest negative correlation disappears when excluding the repetition-1 outlier, the claim that ensembling 'consistently reduces' MAE is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that seed-induced variability can be leveraged as data augmentation through ensembling to improve downstream brain age regression. This claim is directly contradicted by the paper's own data: Figure 3 reports Pearson's R=0.803 (p=5.154e-3) for the SVM regressor, indicating a statistically significant positive correlation between the number of repetitions and test MAE. The text states 'increasing repetitions consistently reduces test MAE,' which is false for the SVM. This is not a minor caveat; it undermines the entire 'useful' half of the paper. The paper attributes 'extreme MAE values in the first and last repetitions' to small sample sizes, but a positive correlation across the full range is not explained by extremes. The Random Forest result (R=-0.830) may be driven by a few points and lacks per-repetition error bars, so it cannot rescue the claim. This internal inconsistency invalidates the data augmentation conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies training-time numerical variability in FastSurfer, a CNN whole-brain segmentation pipeline, using Monte Carlo Arithmetic (MCA) and random-seed perturbations, and compares this variability with FreeSurfer. It claims that FastSurfer training variability is comparable to or larger than FreeSurfer's in cortical regions, that perturbed models perform on par with the baseline, and that the variability can be exploited as data augmentation to improve downstream brain-age regression. The paper includes reproducibility code and a small MNIST sanity check.","tokens_in":10343,"tokens_out":4873,"duration_ms":51457,"significance":"The topic is timely: if the claims held, training-time instability in deep learning would be recast as a usable resource rather than only a reproducibility hazard. The authors provide concrete external validation (FreeSurfer comparison, brain-age regression) and share code, which are strengths. However, the central 'useful' claim is internally contradicted by the paper's own SVM result, and the variability comparison rests on a partially trained coronal model. The manuscript's data do not currently support the title's promise.","major_comments":[{"comment":"The text states that 'increasing repetitions consistently reduces test MAE,' but the reported SVM Pearson R=0.803 (p=5.154e-3) is a statistically significant positive correlation: MAE increases with the number of repetitions. The sentence about extreme MAE values in the first and last repetitions does not explain a monotone positive trend across the full range. This direct contradiction invalidates the 'consistently' claim and, with it, the data-augmentation conclusion. The Random Forest correlation (R=-0.830) cannot rescue the claim without per-repetition error bars and a complete description of the protocol.","section":"§3, Figure 3"},{"comment":"The MCA FastSurfer results are available only for the coronal model and are reported at epoch 10; the paper admits the Dice score at that point differs by 4%. The comparison with FreeSurfer is therefore between a partially trained FastSurfer model and a complete FreeSurfer pipeline. If the epoch-10 checkpoint is not representative of the fully trained model, the conclusion that FastSurfer is more variable in cortical regions may be an artifact of incomplete training. The authors must justify this proxy or provide fully trained checkpoint comparisons.","section":"§3, footnote 1 and Figure 1"},{"comment":"The augmentation protocol is underspecified. It is not stated how the 10 FastSurfer repetitions per subject are partitioned into training and test sets, whether the same subject appears in both training and test, or whether the reported MAE (e.g., 7.40 years) is averaged over repeated test segmentations. Without this information, the correlation in Figure 3 and the claimed improvement cannot be interpreted. A no-augmentation baseline is also missing, so 'improves predictive performance' is not established.","section":"§3, brain-age regression analysis"}],"minor_comments":[{"comment":"The abstract and conclusion claim FastSurfer variability 'exceeds' FreeSurfer in cortical regions, but Figure 1 only shows the left hemisphere and the MCA results are at epoch 10; the claims should be qualified accordingly.","section":"Abstract / §3"},{"comment":"The y-axis label 'Sorensen Dice Scores' should be 'Sørensen–Dice Scores'.","section":"Figure 1"},{"comment":"The caption says 'Training loss variability across random seeds,' but the figure also shows MCA and Default IEEE curves; the caption should be updated.","section":"Figure 2 caption"},{"comment":"The reference '[fas]' (PyTorch Implementation of FastSurferCNN) lacks full bibliographic information; please include authors and year.","section":"References"},{"comment":"The author name 'Y ohan Chatelain' contains a spacing typo.","section":"Title page"}],"recommendation":"reject","confidential_remarks":"The internal contradiction in Figure 3 (SVM R=0.803, p=5.154e-3, positive trend) directly undermines the paper's central augmentation claim, and the epoch-10 proxy for MCA training is a serious validity concern for the variability comparison. These are load-bearing issues that would require new experiments or a substantial reframing, not just local edits. For a workshop paper, the internal inconsistency is disqualifying in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou should know two things about arXiv:2509.05238. First, it reports a real, reproducible attempt to measure training-time numerical variability in FastSurfer, and that part is worth reading. Second, its central 'useful' claim—that seed variability can be leveraged as data augmentation—is directly contradicted by the paper's own Figure 3. That contradiction is not a minor caveat; it breaks the abstract's main promise.\n\nWhat's new and good: the authors train FastSurfer under MCA and random seeds, with controlled data ordering, and compare the variability of segmentations against FreeSurfer under MCA. They find that in cortical regions FastSurfer appears more variable, and they show that the different training runs produce structurally distinct but valid outputs. The methods section is transparent: they link to code, specify the MCA formulation (Fuzzy PyTorch), and list training details. That is solid, reproducible empirical work.\n\nThe soft spots. First, Figure 3 shows a linear regression of test MAE on the number of repetitions. For the SVM, the correlation is positive (R=0.803, p=5.15e-3), meaning more repetitions worsen the MAE. The text says 'increasing repetitions consistently reduces test MAE,' which is simply false for the SVM. The Random Forest shows a negative correlation (R=-0.830), but that doesn't rescue a claim that is supposed to hold 'across models.' The authors dismiss the SVM as arising from 'extreme MAE values in the first and last repetitions,' but a positive correlation across the full range is not explained by extremes. This is a load-bearing flaw. Second, the FastSurfer/FreeSurfer variability comparison relies on MCA results from a single coronal model reported at epoch 10, and the footnote admits the Dice at that point differs by 4%. That makes the conclusion that FastSurfer is more variable in cortical regions conditional on a partially trained checkpoint. It could be true, but the evidence as presented is not strong enough.\n\nThe paper is honest about its limitations, and the code and data availability are a plus. But as written, the central claim doesn't hold up. A serious referee would likely ask for a re-analysis or a revised claim.\n\nMy recommendation: this deserves a serious referee in the sense that the topic is timely and the empirical work is genuinely reproducible, but I would not accept it in its current form. The authors need to either fix the augmentation claim (e.g., report per-repetition error bars, explain the SVM result, or restrict the claim to the Random Forest) or reframe the paper as a variability measurement study.","headline":"The variability measurement is worth a look, but the augmentation claim is contradicted by the paper's own Figure 3.","tokens_in":10730,"tokens_out":4611,"would_cite":false,"duration_ms":38243,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CNN training variability is not merely a reproducibility problem: it can be harvested as data augmentation for downstream neuroimaging tasks like brain-age regression.","keywords":["numerical uncertainty","CNN training variability","FastSurfer","Monte Carlo Arithmetic","data augmentation","brain age regression","ensembling","neuroimaging reproducibility"],"falsifier":"Train FastSurfer under MCA to full convergence (70 epochs) across all three planes and recompute min-Dice variability against FreeSurfer: if cortical variability no longer exceeds FreeSurfer, the paper's headline comparison is a checkpoint artifact. Separately, run brain-age regression with 20+ seed repetitions per subject: if test MAE stops falling or rises after 10 repetitions, the augmentation benefit saturates at exactly the regime the paper tested.","tokens_in":9997,"feed_emoji":"🧠","tokens_out":7461,"duration_ms":73841,"temperature":0.7,"pith_summary":"The paper tries to show that training a convolutional segmentation network (FastSurfer) carries genuine numerical uncertainty, and that this uncertainty is a usable resource rather than only a nuisance. It finds that FastSurfer's training-time variability is comparable to or larger than the traditional FreeSurfer pipeline in cortical regions, and that models trained under different random seeds or floating-point perturbations perform about as well as a stable baseline while producing distinct segmentations. The lead application is brain-age regression: pooling multiple FastSurfer repetitions per subject as augmented training data lowers test mean absolute error as the number of repetitions grows, reaching 7.40 years with ten repetitions, without collecting new scans. A careful reader would care because, if true, any research group that can retrain a segmentation model a few times gets a free, principled augmentation strategy and a better estimate of segmentation uncertainty.","feed_headline":"Retraining noise improves brain-age predictions","feed_subtitle":"FastSurfer's seed-to-seed variability acts as free data augmentation for downstream brain-age regression.","key_machinery":"The central mechanism is the ensemble of 'numerical model families' created by perturbing training: Monte Carlo Arithmetic perturbs every floating-point operation with a random rounding error at a chosen virtual precision, while random seeds alter the pseudorandom initial state (weight sampling, dropout) with data order fixed to keep uncertainty epistemic. These perturbations turn one training run into a family of distinct but comparable segmentations. The paper then feeds the resulting per-subject ROI volumes from multiple FastSurfer runs into random-forest and SVM regressors, so the ensemble acts as a data augmentation operator: repetitions add diversity to the feature set without adding s","core_discovery":"Stated on the paper's own terms: training FastSurfer with Monte Carlo Arithmetic (randomly perturbed floating-point operations) or with different random seeds produces whole-brain segmentations that differ measurably from each other, and these differences are not noise that degrades the model. In subcortical regions FastSurfer's variability is comparable to FreeSurfer's, while in all cortical ROIs tested it is statistically higher. MCA-induced and seed-induced variability are statistically distinct (p = 4.45e-313), yet both track the IEEE baseline's loss curves and achieve comparable performance. The constructive claim is that this multiplicity of valid outputs constitutes a numerical ensemb","pith_inferences":["Editorial extension: the augmentation benefit probably transfers beyond brain age—any ROI-feature classifier (e.g., Alzheimer's vs control, or cortical-thickness group comparisons) could treat a few FastSurfer seeds as a cheap ensemble; the paper only demonstrates the one task.","Editorial extension: if MCA variability were measured at full convergence rather than epoch 10, the cortical 'more variable than FreeSurfer' result could shrink; a completed MCA training would settle whether the comparison holds.","Editorial extension: the observed saturation ('a few repetitions capture most of the benefit') suggests the practical recipe is 3-10 runs, and beyond that the gains may be outweighed by compute; the paper doesn't test the saturation point directly.","Editorial extension: learning-rate restart points could be exploited deliberately—harvesting checkpoints at restarts might generate diverse segmentations at lower cost than full retrainings."],"forward_implications":["If repeated FastSurfer runs under different seeds are valid segmentations, then single-run segmentation studies understate uncertainty; reporting ensemble spread (min Dice across runs) becomes a practical reliability metric.","Brain-age regression improves with augmentation from repeated runs (test MAE 7.40 years at 10 repetitions vs. higher with fewer), so other ROI-based downstream models could adopt the same strategy without new data.","The finding that FastSurfer is more variable than FreeSurfer in all cortical ROIs would redirect reproducibility checks toward cortical parcellation, not just subcortical volumes.","Because MCA and seed perturbations are statistically distinct, combining both in one ensemble may yield more diversity than either alone.","The loss variability concentrated at cosineWarmRestarts restarts implies that model checkpoints near learning-rate restarts are especially diverse, so choosing checkpoints may control ensemble diversity."],"supporting_citations":[{"why":"Defines the FastSurferViNN pipeline whose CNN training is the object of study; also the source of the training recipe and models.","marker":"Henschel et al. [2022]"},{"why":"Prior work showing FastSurfer inference variability on CoRR; supplies the comparison baseline and the data subset for this study.","marker":"Gonzalez Pepe et al. [2023]"},{"why":"CoRR dataset provides the holdout test-retest subjects used to compare FastSurfer and FreeSurfer variability.","marker":"Zuo et al. [2014]"},{"why":"Defines Monte Carlo Arithmetic, the perturbation model used to inject numerical noise into FastSurfer training.","marker":"Parker [1997]"},{"why":"Fuzzy PyTorch tool implements MCA perturbations at runtime for the FastSurfer training runs.","marker":"Gonzalez-Pepe et al. [2025]"},{"why":"Provides the MCA-instrumented fuzzy libmath used to measure FreeSurfer's numerical variability for comparison.","marker":"Salari et al. [2021]"},{"why":"Classical FreeSurfer whole-brain segmentation pipeline that serves as the non-DL comparator.","marker":"Fischl et al. [2002]"},{"why":"Prior demonstration that MCA-based augmentation improves connectomics classification; the template the paper extends to segmentation ensembling.","marker":"Kiar et al. [2021]"},{"why":"Supports the claim that a few repetitions capture most of the augmentation benefit, justifying the 10-run design.","marker":"Lei et al. [2019]"}],"fun_headline_variants":["Training noise improves brain-age regression","CNN training variability as data augmentation for brain age","Harness seed randomness for better brain-age predictions","Numerical uncertainty enhances brain-age regression","Turn training wobble into free data augmentation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The comparison relies on an MCA FastSurfer checkpoint at epoch 10 from the coronal model only—the paper's own footnote says its Dice score differs by 4%—so if a fully trained MCA model is less variable, the cortical-excess result may be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Training noise improves brain-age regression","CNN training variability as data augmentation for brain age","Harness seed randomness for better brain-age predictions","Numerical uncertainty enhances brain-age regression","Turn training wobble into free data augmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2354,"prompt_tokens":670,"completion_tokens":1684,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":1629}},"tokens_in":414,"tokens_out":1684,"duration_ms":13042,"temperature":1.0,"reasoning_tokens":1629,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:29:22.775599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train FastSurfer under MCA to full convergence (70 epochs) across all three planes and recompute min-Dice variability against FreeSurfer: if cortical variability no longer exceeds FreeSurfer, the paper's headline comparison is a checkpoint artifact. Separately, run brain-age regression with 20+ seed repetitions per subject: if test MAE stops falling or rises after 10 repetitions, the augmentation benefit saturates at exactly the regime the paper tested.","supporting_citations":[{"cited_title":"Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic","cited_arxiv_id":null,"evidence_quote":"Provides the MCA-instrumented fuzzy libmath used to measure FreeSurfer's numerical variability for comparison."},{"cited_title":"Accurate simulation of operating system updates in neuroimaging using monte-carlo arithmetic","cited_arxiv_id":null,"evidence_quote":"Prior demonstration that MCA-based augmentation improves connectomics classification; the template the paper extends to segmentation ensembling."}],"review_version":1}