{"id":"e31ab223-4ab1-4e1e-ae4d-b0af285f72e2","arxiv_id":"2411.19564","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A voxel-spacing agnostic nnU-Net, trained with sparse annotations, iterative label cleaning, and 12,740 pseudo-labelled images, reaches DSC 85.6% for perivascular space segmentation and is extended to midbrain, hippocampus, FLAIR, and T2w inputs.","lead":"This paper trains a nnU-Net to automatically outline fluid-filled perivascular spaces in brain MRI scans, reporting Dice scores of 85.6% on 30 healthy subjects after iterative label cleaning. The result matters as an open-source tool for studying brain clearance in aging and neurodegeneration, though the headline performance depends on model-assisted label cleanup and a small validation set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 85.6% DSC is measured against labels revised with model assistance; the model's DSC against pre-cleaning labels is 66.2%, so independent validation is required.","rationale":"This is load-bearing because all central quantitative claims — the 85.6% DSC, the robustness across 3T/7T, and the cluster-count CCC — flow from the cleaned labels. The cleaning protocol in Section 2.4.4 has the model in the loop, and the 5FCV in Section 2.9 uses those cleaned labels as ground truth. Section 4.1.5 contains the critical admission and the 66.2% result, which the authors interpret as improved reproducibility rather than improved detection. However, without an independent, model-blinded gold standard, one cannot separate 'better labels' from 'labels that conform to the model.' This is not a claim of misconduct; it is an evaluation-design gap. The reader's weakest assumption is exactly this issue, and the conditional verdict already flags it, so no verdict change is needed, but the condition should explicitly require independent-rater validation before the headline accuracy claim is accepted as accuracy rather than reproducibility.","tokens_in":27204,"tokens_out":3490,"duration_ms":30949,"concrete_test":"Have an independent rater, blinded to all model outputs, manually segment PVS in a held-out set of 10 raw T1w scans spanning the three acquisition protocols, using the same sparse-slice protocol as Section 2.4.1. Run the final pseudo-label-trained T1w nnU-Net on those raw scans and compute DSC, sensitivity, PPV, and cluster-count CCC against these independent labels. If the independent-rater DSC is close to 66% rather than 85.6%, the headline performance claim is an artifact of model-assisted label cleaning and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — a robust framework with 5FCV DSC 85.6±1.4% — rests on ground truth labels that were revised in a model-in-the-loop process (Section 2.4.4: two rounds of model training and manual quality checking of PVS segmentations). When a human corrects labels while viewing model predictions, the 'gold standard' can drift toward the model's own output; the subsequent 5FCV then measures agreement with these model-influenced labels, not with an independent truth. The paper's own Section 4.1.5 comparison against the original raw-T1w segmentations yields mean DSC 66.2±7.0% and states that the improvement 'likely arises by removing poor-quality or noisy labels rather than increasing the model's sensitivity to PVS.' That 19-point gap shows the headline metric partly reflects label realignment rather than improved detection. Without an independently segmented test set, DSC 85.6% is an upper bound on reproducibility within the cleaning loop, not evidence of accurate PVS quantification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comprehensive pipeline for automated segmentation of perivascular spaces (PVS) in brain MRI using the nnU-Net ResEnc architecture. The authors train and evaluate T1w models for white-matter and basal-ganglia PVS using a sparse annotation strategy on 30 healthy participants from three datasets, optimize target voxel spacing and preprocessing (NLMF, AHE), perform iterative label cleaning with model assistance, and add semi-supervised learning with 12,740 pseudo-labels from 18 public datasets. They report a 5-fold cross-validation DSC of 85.6±1.4% for the final T1w model and develop pilot models for midbrain and hippocampal PVS, a T1w+FLAIR model for concurrent WMH segmentation, and a T2w model. The authors claim the resulting framework is robust and holistic for automated PVS quantification, with models made publicly available.","tokens_in":27438,"tokens_out":2337,"duration_ms":20833,"significance":"If the reported performance is accurate, the paper would provide a practically useful, open-source tool for PVS segmentation on heterogeneous T1w MRI, a task of growing interest in cerebral small vessel disease and neurodegenerative research. The paper has notable strengths: it is unusually candid in reporting the comparison against original un-cleaned labels (Section 4.1.5), it makes models publicly available, it evaluates on multi-scanner, multi-field-strength data, and it documents the voxel-spacing-agnostic finding, which is a practically relevant methodological contribution. However, the main numerical claim of 85.6% DSC is measured against labels that were revised with model assistance, and the paper's own analysis shows a 19-point drop (to 66.2%) against the original manual labels. This does not invalidate the engineering contributions, but it means the headline metric cannot be read as an independent accuracy estimate. The pilot models for other regions and sequences are evaluated on very small test sets, further limiting the strength of the 'comprehensive framework' claim.","major_comments":[{"comment":"The central claim of robust segmentation at DSC=85.6±1.4% rests on ground-truth labels that were iteratively cleaned with the assistance of the model's own predictions (Section 2.4.4). The paper's own Section 4.1.5 reports that the same final model achieves mean DSC=66.2±7.0% against the original manual labels, and attributes the improvement to 'removing poor-quality or noisy labels rather than increasing the model's sensitivity to PVS.' This is a load-bearing issue: the headline metric partly measures agreement with model-influenced labels, not agreement with an independent gold standard. I recommend that the abstract and conclusion either report both numbers with equal emphasis, or explicitly frame the 85.6% figure as a reproducibility metric within the cleaning loop, and provide an external validation set with independent manual segmentations to support the 'robust' claim.","section":"§2.4.4 and §4.1.5"},{"comment":"The T1w+FLAIR and T2w models are evaluated on only four and three held-out images respectively, with a train/test split rather than cross-validation. The paper acknowledges this in the Limitations section, but the abstract and conclusion still describe the overall framework as 'comprehensive' and 'robust' without qualification. Given the small test sizes, the reported DSCs (81.1±4.3% and 54.4±4.9%) are not sufficient to support strong claims about these models. Please temper the claims for these modalities or add additional validation data; at minimum, state the test-sample sizes in the abstract.","section":"§2.6, §2.7, and Table 2"},{"comment":"The comparisons among image handling and preprocessing strategies are based on mean DSC values and standard deviations, without any statistical testing. For example, the pseudo-label model (85.6±1.4%) is not statistically different from the iteration-2 label-cleaning model (85.7±1.2%), yet the paper claims that semi-supervised learning improved the model on the basis of concordance metrics alone. I recommend reporting per-fold paired comparisons, confidence intervals, or a repeated-measures test for the main T1w model comparisons, so that readers can assess whether observed differences are within fold-to-fold variability.","section":"§3.1 and §2.9.2"}],"minor_comments":[{"comment":"There is an inconsistency in the target spacing values: Section 2.4.2 lists 0.75, 0.80, 0.87, and 1.00 mm, while Section 4.1.2 lists 0.7, 0.8, 0.87, and 1.0 mm. Please correct the discussion text to match the methods.","section":"§2.4.2 and §4.1.2"},{"comment":"Please clarify whether the pseudo-labels (n=12,740) were kept entirely out of the validation folds in the 5FCV procedure. The text says all pseudo-labels were 'included alongside the training data in each training fold,' which is the expected design, but it should be stated explicitly to avoid ambiguity about possible data leakage.","section":"§2.9"},{"comment":"The multi-class dice loss in Eq. (2) uses u and v without defining the superscript k or the summation over i and k clearly in the text; a short sentence defining u_i^k and v_i^k as probability and one-hot label for class k at voxel i would improve readability.","section":"§2.8, Eq. (2)"},{"comment":"The caption for Figure 8 (top row) refers to 'middle-left, middle, and middle-right' panels, which is confusing; please use explicit panel labels (a), (b), (c), etc. throughout the figure.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its main weakness in Section 4.1.5, which is commendable, but the abstract and conclusion still lead with the 85.6% DSC without the critical caveat. The fit to a neuroimaging methods journal is appropriate. I would like to see the authors either obtain an external validation set with independent manual segmentations or substantially reframe the claims so that the headline number is not presented as an independent accuracy estimate. The pilot models for T1w+FLAIR and T2w are interesting but too thinly validated for the current wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the headline 85.6% DSC is not measured against an independent gold standard. The ground truth labels were cleaned in two rounds where the model's own predictions were shown to the human rater, so the 5FCV partly measures agreement with model-influenced labels. Against the original manual labels, the final model scores 66.2% DSC, and the paper itself says the gain likely comes from removing noisy labels rather than raising sensitivity. The stress-test note is on target, and the reader's conditional verdict is fair.\n\nThat said, there is real value here. The voxel-spacing agnostic nnU-Net clearly beats resampling to a common resolution, which is an informative empirical result for the PVS community. The sparse annotation strategy is new in this setting and looks genuinely efficient. The midbrain and hippocampal pilots, the T1w+FLAIR model, the pseudo-label training on 12,740 scans, and the open-source release of the trained models are useful. The paper is honest: Section 4.1.5 and the limitations section name the small validation sample and the label-cleaning effect explicitly, which is more than most engineering papers do.\n\nThe soft spots are real but not fatal. Validation is limited to 30 healthy subjects, all internal; there is no external test set. Model comparisons lack statistical testing, so the spacing and preprocessing ordering rests on point estimates. The T1w+FLAIR and T2w models are tested on four and three images, which cannot support robustness claims. The midbrain and hippocampal models have decent DSC but poor voxel-count concordance (CCC 0.19 and 0.11), so those pilots are not yet reliable quantification tools. The pseudo-label training uses the same model family to generate labels, and while held-out folds and human review help, it is not an independent signal.\n\nMy overall read: the central engineering findings probably survive scrutiny, but the paper overclaims in the abstract and conclusion. This is for imaging biomarker researchers who want a ready-to-use PVS tool; they should treat the quantitative claims cautiously. It deserves a serious referee, with major revision: add an independent test set whose labels were not touched by the model, run proper statistical comparisons, report the 66.2% figure in the abstract, and dial the 'robust and holistic' language down to match the evidence. Labels should also be released, not just the models.","headline":"Useful, honest engineering work, but the headline DSC is measured against model-affected labels and needs independent validation before the robustness claims can be trusted.","tokens_in":27957,"tokens_out":2338,"would_cite":true,"duration_ms":22043,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a voxel-spacing-agnostic nnU-Net can segment brain perivascular spaces on T1w MRI at DSC 85.6±1.4% by combining sparse annotation, iterative label cleaning, and pseudo-labelling, while conceding the gain comes mostly…","keywords":["perivascular spaces","Virchow-Robin spaces","nnU-Net","brain MRI segmentation","T1-weighted MRI","sparse annotation","pseudo-labelling","voxel spacing"],"falsifier":"Have independent experts annotate a fresh set of T1w scans without model assistance and compare the released model's predictions to those labels; if the Dice score lands near the 66.2% the paper reports against its un-revised ground truth rather than near 85.6%, the framework's generalisation claim on raw data would be contradicted. Alternatively, re-run the five-fold cross-validation with the first-round pre-cleaning labels as ground truth; the model's DSC should then be substantially below 80% if the improvement is label realignment.","tokens_in":27032,"feed_emoji":"🧠","tokens_out":10327,"duration_ms":78422,"temperature":0.7,"pith_summary":"The paper aims to establish an automated, publicly released pipeline for segmenting perivascular spaces (PVS), the fluid-filled sleeves around brain blood vessels whose enlargement is a candidate biomarker for small vessel disease, Alzheimer's disease, and Parkinson's disease, from T1-weighted MRI. Working with 30 healthy participants scanned on three different 3T and 7T protocols, the authors train a self-configuring deep-learning segmentation network (nnU-Net) using sparse manual labels on ten slices per scan. Their core finding is that a voxel-spacing agnostic configuration, which does not resample scans to a common resolution, clearly outperforms resampled variants (DSC 64.3% vs. 40.5–55%). After two rounds of iterative label cleaning and retraining with 12,740 pseudo-labelled scans, the model reaches a five-fold cross-validated Dice score of about 85.6%. A sympathetic reader would care because the work supplies a practical tool and a set of design choices, sparse annotation, no resampling, label cleaning, and pseudo-labelling, that other small-structure segmentation projects could reuse.","feed_headline":"Deep learning model maps brain perivascular spaces at 86% Dice score","feed_subtitle":"Skipping voxel resampling proved key: the model works across 3T and 7T scans and is released openly.","key_machinery":"The load-bearing machinery is the nnU-Net Residual Encoder, a 3D convolutional segmentation network with self-configuring preprocessing, configured in a voxel-spacing agnostic mode where voxel spacing metadata is ignored and no resampling to a common grid is performed, so small PVS structures are not blurred by interpolation. Around this sits a sparse annotation strategy in which only ten axial slices per scan are manually labelled and the remaining slices are masked with an ignore label trained under a partial loss, an iterative label cleaning loop in which non-local-means denoising plus adaptive histogram equalisation make PVS boundaries clearer for manual revision, and a pseudo-labelling stage where the enhanced model labels 12,740 unannotated T1w images that are added as training data. The combination of these mechanisms, not any single one, is what the paper credits for the final performance.","core_discovery":"On its own terms, the paper claims that its final T1-weighted nnU-Net pipeline is a practical, openly released framework for automated PVS quantification in brain MRI. Specifically, a voxel-spacing agnostic nnU-Net Residual Encoder trained from sparse ten-slice manual labels and refined by iterative label cleaning plus semi-supervised pseudo-labelling segments white matter and basal ganglia PVS across 3T and 7T T1w data with a mean five-fold cross-validated DSC of 85.6±1.4% (white matter 88.6±0.9%, basal ganglia 82.6±2.4%) and Lin's concordance correlation coefficient of 0.89 for predicted versus manual PVS cluster counts. The paper also reports pilot models for midbrain and hippocampal PVS (DSC 64.3±6.5% and 67.8±5.0%) and extended models using T1w+FLAIR and T2w inputs. Importantly, the paper's own comparison against the original pre-cleaning ground truth yields a mean DSC of 66.2%, and it explicitly states that the improvement to 85.6% likely arises from removing poor-quality or noisy labels rather than increasing the model's sensitivity to PVS.","pith_inferences":["Beyond the paper, the finding that label cleaning explains the DSC gain implies that users of the released model should expect raw-image accuracy closer to the 66% measured against the pre-cleaning labels, and should validate on their own unrevised ground truth.","Beyond the paper, the voxel-spacing agnostic result carries a general lesson for other small-structure segmentation tasks such as microbleeds or cortical lesions: resampling to a common grid may be silently reducing accuracy, a hypothesis testable by re-running nnU-Net on raw-resolution data.","Beyond the paper, the divergence between DSC and cluster-count concordance in the pseudo-label experiment suggests count-based metrics may be more clinically meaningful than voxel overlap for PVS quantification, and they could be adopted as primary endpoints in future evaluations.","Beyond the paper, the low voxel-count concordance in the midbrain and hippocampal models (CCC 0.11–0.29) points to applying the paper's own label-cleaning and enhancement steps to those pilot models as the immediate next test."],"forward_implications":["Other researchers can apply the released nnU-Net to heterogeneous T1w scans acquired at 3T or 7T without resampling to a common voxel grid, since the spacing-agnostic configuration was the top performer.","Sparse annotation with a partial loss can generate full-brain PVS training labels from ten manually labelled slices per scan, reducing the annotation burden that has limited PVS segmentation datasets.","Adding pseudo-labels from large open-access repositories does not raise the Dice score but improves agreement of predicted versus manual PVS cluster counts (CCC 0.89), so unlabelled data can be used to make quantification more reliable.","The T1w+FLAIR model separates white matter hyperintensities from PVS in one pass, letting studies of cerebral small vessel disease measure both markers without separate lesion-masking tools.","Pilot midbrain and hippocampal PVS models give voxel-wise quantification in regions previously measured mainly by coarse rating scales, enabling finer-grained study of PVS there."],"supporting_citations":[{"why":"Supplies the nnU-Net self-configuring segmentation framework that all models in the paper are built on.","marker":"Isensee et al., 2021"},{"why":"Defines the Residual Encoder (ResEnc) variant whose encoder is the architecture used throughout the paper.","marker":"Isensee et al., 2022"},{"why":"Provides the sparse annotation method with ignore labels and a partial loss that the paper adapts for ten-slice PVS labelling.","marker":"Gotkowski et al., 2024"},{"why":"Motivates the pseudo-labelling strategy the paper uses to train on 12,740 unannotated T1w images.","marker":"Ferreira et al., 2023"},{"why":"Systematic review of prior automated PVS quantification methods that gives the performance baseline the paper aims to surpass.","marker":"Waymont et al., 2024"},{"why":"Defines PVS anatomy, physiology, and disease relevance, grounding the clinical motivation for automated segmentation.","marker":"Wardlaw et al., 2020"},{"why":"Prior multi-modal T1w+FLAIR network for PVS segmentation that the paper's dual-channel model extends with regional labels and concurrent WMH segmentation.","marker":"Rashid et al., 2023"},{"why":"Supplies the non-local means denoising filter used in the enhanced preprocessing that aided manual label cleaning.","marker":"Coupé et al., 2012"},{"why":"Lesion Prediction Algorithm used to generate white matter hyperintensity masks for the T1w+FLAIR model.","marker":"Schmidt et al., 2019"}],"fun_headline_variants":["PVS segmentation hits 86% Dice with label-cleaning trick","nnU-Net maps perivascular spaces at 86% Dice across 3T and 7T","Skipping voxel resampling boosts PVS Dice from 64% to 86%","Label cleaning drives PVS segmentation to 86% Dice","Open source nnU-Net segments brain PVS at 86% Dice on 3T/7T"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the iteratively cleaned manual segmentations, revised using the model's own predictions, are an unbiased gold standard, yet the paper concedes the DSC gain comes mainly from deleting noisy labels, so the headline metric partly reflects agreement with those revised labels rather than raw detection accuracy.","fun_headline_variants_meta":{"raw":{"variants":["PVS segmentation hits 86% Dice with label-cleaning trick","nnU-Net maps perivascular spaces at 86% Dice across 3T and 7T","Skipping voxel resampling boosts PVS Dice from 64% to 86%","Label cleaning drives PVS segmentation to 86% Dice","Open source nnU-Net segments brain PVS at 86% Dice on 3T/7T"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000925,"raw_usage":{"total_tokens":4099,"prompt_tokens":1216,"completion_tokens":2883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":832,"completion_tokens_details":{"reasoning_tokens":2770}},"tokens_in":832,"tokens_out":2883,"duration_ms":16604,"temperature":1.0,"reasoning_tokens":2770,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:03:27.137823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent experts annotate a fresh set of T1w scans without model assistance and compare the released model's predictions to those labels; if the Dice score lands near the 66.2% the paper reports against its un-revised ground truth rather than near 85.6%, the framework's generalisation claim on raw data would be contradicted. Alternatively, re-run the five-fold cross-validation with the first-round pre-cleaning labels as ground truth; the model's DSC should then be substantially below 80% if the improvement is label realignment.","supporting_citations":[],"review_version":1}