{"id":"80115f39-53c3-49b4-9eaa-48da3e4609d6","arxiv_id":"2507.12092","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A standard nnU-Net with blob loss achieves the best cortical lesion segmentation in a multi-center benchmark, with out-of-domain F1 of 0.50.","lead":"This paper tests several deep learning models for finding cortical lesions in multiple sclerosis from multi-center MRI scans. The best model, a standard nnU-Net with a blob-aware loss, reaches a detection F1 score of 0.64 on in-domain data and 0.50 on out-of-domain data, and the trained models will be released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD generalization claim is confounded: site D differs in annotation protocol, cohort composition, and preprocessing, not only scanner vendor; F1=0.50 cannot be attributed to scanner shift alone.","rationale":"The reader's weakest assumption correctly identifies the central fragility: manual annotations from four sites are treated as comparable ground truth despite different raters, contrasts, and lesion criteria. My stress-test pass converged on the same point and sharpened it: the OOD claim, the headline numerical evidence for generalization, is confounded because site D differs in annotation protocol, cohort composition, and preprocessing in addition to scanner vendor. This is not an internal inconsistency or a challenge to the reported numbers; it is a validity limitation of the interpretation. The conditional verdict is appropriate: the benchmark and model release are valuable, but the central generalization claim should not be ACCEPT-level until the annotation confound is addressed or explicitly quantified. I therefore recommend no change to the reader's verdict, and the proposed re-annotation experiment is the concrete check that would settle whether the concern lands.","tokens_in":20131,"tokens_out":3962,"duration_ms":45932,"concrete_test":"Conduct a matched-label re-annotation study: take a stratified random subset of about 30 MS scans (10 each from sites A, B, and D), discard the original annotations, and have two independent expert raters, blinded to site and model output, re-annotate cortical lesions on a common contrast (e.g., MP2RAGE or MPRAGE) using one written protocol. Then recompute the best model's per-site F1, precision, and recall against these matched labels and also compute inter-rater F1. If the A-versus-D gap persists under matched labels and is comparable to inter-rater variability, the scanner-shift interpretation is supported; if the gap shrinks or falls within rater disagreement, the reported OOD F1=0.50 is largely an artifact of annotation and cohort differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a standard nnU-Net with Blob BCE + Dice Loss 'effectively segments cortical lesions across different imaging centers and MRI protocols' rests heavily on the out-of-domain result (Section 6, abstract). Section 3.1.2 designates site D as Test-out because of the different scanner manufacturer, but Section 3.1.1 shows that D also used a different annotation pipeline: three raters (a neuroscientist and two neurologists), detection on 0.7 mm DIR plus MP2RAGE, and manual delineation by a trained neuroscientist. Sites A-C used different rater teams, different source contrasts (e.g., site A only MP2RAGE; site B FLAIR/DIR/MP2RAGE; site C 7T MP2RAGE and T2*w GRE), and different consensus procedures. Site D additionally contains 40 MS-mimic patients, is registered to 3D EPI, and uses different skull-stripping (Section 3.1.2, Table 2). Thus the Test-out comparison changes vendor, sequence set, annotation criteria, rater expertise, cohort composition, and preprocessing simultaneously. The reported Table 5 gap (F1 0.74 for A vs 0.44 for D, with #FNL about 6.5 per scan in D vs about 3 in A/B) may reflect label-recall differences or annotation stringency rather than model generalization failure. The same confound undermines the in-domain cross-site comparison: the site performance hierarchy in Table 5 is not attributable to the model while the reference masks were built under different protocols. The paper's own limitation section acknowledges ambiguous lesion definitions but does not quantify their impact on the headline OOD number. Therefore the strongest empirical support for the cross-center claim is not yet isolated from label heterogeneity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a multi-center benchmark of deep learning cortical lesion (CL) segmentation in multiple sclerosis, built on 656 scans from four institutions with expert-consensus annotations. The authors systematically compare nnU-Net architecture variants, loss functions (standard BCE+Dice vs. Blob BCE+Dice), and input resampling strategies, report that the vanilla nnU-Net with Blob loss performs best, and characterize its per-site, per-modality, and out-of-domain behavior. They also analyze lesion-level errors and bottleneck features for interpretability and release the implementation and model weights. The central claim is that a standard nnU-Net with task-specific adjustments such as Blob BCE+Dice loss effectively segments cortical lesions across different imaging centers and MRI protocols, supported by reported in-domain F1 of 0.64 and out-of-domain F1 of 0.50.","tokens_in":20443,"tokens_out":9332,"duration_ms":102190,"significance":"The study is valuable as a large, multi-center benchmark for CL segmentation, a clinically relevant but technically challenging task. Its strengths include the size and heterogeneity of the dataset (656 scans, 3T/7T, multiple vendors and protocols), the use of standard errors and FDR-corrected statistical tests, the inclusion of negative results (upsampling degrades precision, U-Mamba fails to converge), and the public release of models and code. If the out-of-domain claim were clean, the paper would provide a practical baseline for clinical deployment. However, the central generalization claim rests on a confounded out-of-domain comparison and on model selection performed on the test set, so the significance is conditional on addressing these issues.","major_comments":[{"comment":"The out-of-domain generalization claim is confounded. Site D differs from sites A-C not only in scanner vendor but also in annotation protocol (three raters, detection on 0.7 mm DIR plus MP2RAGE, manual delineation by a trained neuroscientist, as described in Section 3.1.1), cohort composition (40 MS-mimic patients), preprocessing (registration to 3D EPI, skull-stripping with HD-BET to MPRAGE plus dilation), and the set of available modalities. Therefore, the reported out-of-domain F1 of 0.50 in Table 5 cannot be attributed to scanner shift alone. The abstract and conclusions should either reframe this as evaluation on a held-out site with multiple simultaneous differences, or the authors should isolate the scanner effect, for example by analyzing the MS-only subset of site D or by having an independent annotation protocol applied to a subset of site D data. As written, the central claim in Section 6 that the model 'effectively segments cortical lesions across different imaging centers and MRI protocols' is not fully supported.","section":"Section 3.1.2, Table 5, Section 6"},{"comment":"The selection of the 'best model' (Vanilla nnU-Net with Blob loss) is based on the same Test-in set that is later used to report final performance. The architecture and loss comparisons in Tables 3 and 4 are performed on Test-in, and the model with the highest metrics on that set is then analyzed in detail. No separate validation split or nested cross-validation is used. This procedure introduces an optimistic bias in the reported in-domain F1, DSC, and related metrics. The authors should disclose this selection-on-test issue explicitly and, if possible, re-evaluate the model selection with a proper validation partition or cross-validation. At minimum, the reported Test-in numbers should be labeled as model-selected estimates rather than unbiased performance estimates.","section":"Sections 4.1 and 4.2, Tables 3 and 4"},{"comment":"The cross-site performance comparison is interpreted primarily as a property of the model, but the reference masks from different sites were generated under different annotation protocols, rater expertise, and lesion criteria. For example, the much larger number of false-negative lesions per scan in site D (6.5, versus roughly 3 in sites A and B) and the very high FNL count in site C 7T (55.5) may reflect differences in annotation recall or lesion definition rather than model failure. The discussion in Section 5.2 should temper claims about site-specific model limitations and should explicitly acknowledge that without a common reference standard, the observed gaps are not directly attributable to the segmentation model. A quantitative analysis of inter-rater or inter-protocol variability would strengthen this point.","section":"Section 4.2.1 and Section 5.2, Table 5"},{"comment":"The benefit of Blob BCE+Dice loss is presented as a positive finding, but the only statistically significant improvement is out-of-domain precision (p<0.05), while in-domain improvements in nDSC, DSC, F1, and precision are not reported as significant. Given the multiple comparisons performed, the evidence for the loss-function advantage is weaker than the text and abstract imply. The authors should state clearly which comparisons reached significance after FDR correction and avoid implying a general improvement from Blob loss based on non-significant trends.","section":"Section 4.1.2 and Table 4"}],"minor_comments":[{"comment":"The text 'We employed the nnU-Net framework [24, 27]' cites reference [24], which is the HD-BET brain extraction paper, not the nnU-Net paper. The nnU-Net method paper is reference [22] (Isensee et al., 2021). Please correct the citation.","section":"Section 3.2, references"},{"comment":"The column header 'A B D C' appears inconsistent with the row totals and with the text stating that site D is used for out-of-domain testing. The table is difficult to parse, especially the rows marked with a checkmark for 7T and the MS-mimic rows. Please reformat the table so that the site columns, the 7T indicator, and the MS-mimic indicator are unambiguous.","section":"Table 2"},{"comment":"The definition of h in the nDSC formula is ambiguous: 'h is the ratio between the positive and the negative classes in the ground truth scan segmentation' does not specify which class is in the numerator. Please clarify whether h = N_negative / N_positive or the inverse, and define all symbols explicitly.","section":"Section 3.3.1, Eq. (1)"},{"comment":"In the MS-mimic rows, F1-score is listed but Recall is shown as '-'. Please explain why recall is undefined for these subjects and how F1 is computed without it, or add a footnote clarifying that the MS-mimic patients have no cortical lesion ground truth and that the reported value is precision under a different definition.","section":"Table 5"},{"comment":"The sentence 'the median FNL volume is 15mL' is implausible for cortical lesions, which are typically on the order of cubic millimeters. If the value is in mm^3, please correct the unit; if it is truly 15 mL, please explain how such large lesions are classified as cortical.","section":"Section 4.2.2"},{"comment":"There are several minor language issues, including 'widespread in adoption' in the Contributions section, 'U-Mamba architectures could not finally be included' in Section 4.1.1, and 'leuko-/intra-cortical' hyphenation inconsistencies. A careful proofreading pass is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and the data release is a useful contribution. The main concerns are the confounded out-of-domain comparison and the model selection on the test set; both are addressable with reframing and additional analysis, so I do not recommend rejection. The Table 2 formatting issue suggests the authors should double-check the accuracy of the data split description before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This one is worth reading before you invest in CL segmentation: it delivers a large multi-center dataset, public models, and an honest comparison of nnU-Net variants, but the headline out-of-domain number (F1=0.50) is less clean than it looks because the OOD site changes annotation protocol, cohort composition, and preprocessing, not just the scanner vendor.\n\nWhat is actually new: the OOD evaluation on 224 subjects is by far the largest for cortical lesions, and the systematic comparison of architectures, losses, and upsampling is useful. The paper also includes negatives without spin: upsampling pushes recall up but collapses precision, U-Mamba fails to converge, and larger ResEncUNet variants do not pay off. The blob BCE+Dice loss gives a modest, consistent precision gain, mostly out-of-domain. The error analysis is sensible: missed lesions are usually small leukocortical ones, and many false positives overlap white matter or juxtacortical lesions. Releasing trained models and code is a concrete contribution.\n\nSoft spots. The OOD claim is confounded. Site D differs in annotation team (three raters, DIR-based detection, different delineator), cohort (40 MS mimics), registration to EPI, and skull-stripping. So F1=0.50 cannot be attributed purely to scanner shift; label heterogeneity could easily account for part of the gap. The paper acknowledges ambiguity but does not quantify it. Second, despite the 'benchmark' framing, there are no comparisons to existing CL methods like CLAIMS or FLAWS-based pipelines, so the value is as an internal nnU-Net comparison, not a state-of-the-art benchmark. Model selection on Test-in is a minor issue; it can inflate the in-domain numbers. These are fixable rather than fatal.\n\nWho this is for: MS imaging researchers who want a solid T1-only baseline and realistic performance expectations. It deserves a serious referee. I would send it to review, with a request to address the OOD confound — perhaps by stratifying site-D results and discussing label heterogeneity — and to add at least one external comparison if feasible.","headline":"A genuinely useful multi-center benchmark and public model for cortical lesion segmentation, but the headline OOD number is confounded by annotation and cohort differences, so read the generalization claim with caution.","tokens_in":21096,"tokens_out":2363,"would_cite":true,"duration_ms":25919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard nnU-Net with a blob-aware loss detects MS cortical lesions on routine 3T MRI, with F1 of 0.64 in-domain and 0.50 on a held-out scanner vendor.","keywords":["multiple sclerosis","cortical lesions","lesion segmentation","nnU-Net","deep learning","MRI","out-of-distribution generalization","explainable AI"],"falsifier":"Have a single expert panel re-annotate a matched subset of scans from the in-domain sites and the out-of-domain site under one common protocol, then recompute per-site F1: if the out-of-domain F1 of about 0.50 rises toward the in-domain 0.64, the scanner-generalization claim is largely label heterogeneity, while if it stays near 0.50, true domain shift is confirmed. A complementary check is measuring inter-rater agreement between in-domain and out-of-domain annotators on identical scans, since model performance at or above that agreement means the network is near the annotation ceiling.","tokens_in":19926,"feed_emoji":"🧠","tokens_out":14590,"duration_ms":146417,"temperature":0.7,"pith_summary":"Multiple sclerosis cortical lesions are clinically valuable because they carry high diagnostic specificity and enter the McDonald criteria, yet their small size, faint MRI appearance, and heavy annotation burden keep them out of routine use. This paper tries to establish that a standard, self-configuring deep learning pipeline can close that gap on routine 3T T1-weighted MRI: using 656 scans from four institutions with expert-consensus annotations, it claims a plain nnU-Net trained with a blob-aware loss reaches a lesion-detection F1-score of 0.64 in-domain and 0.50 on a held-out center with a different scanner vendor. The paper further argues that architectural complexity does not pay off, that the blob loss is the one modification that reliably helps, and that the remaining failures concentrate in small leukocortical lesions and the inherent leukocortical-versus-juxtacortical ambiguity. If the claim is right, an off-the-shelf model with released weights and known error modes is a practical baseline for clinical and research cortical lesion assessment.","feed_headline":"F1 0.64: plain nnU-Net finds MS cortical lesions on routine MRI","feed_subtitle":"A 656-scan, four-center benchmark shows the vanilla model with blob loss holds up on an unseen scanner at F1 0.50.","key_machinery":"The carrying mechanism is the nnU-Net self-configuring segmentation framework, a 3D U-Net pipeline that automatically sets preprocessing, resolution, patch size, augmentation, ensembling, and post-processing from the data, combined with the Blob BCE + Dice loss, an instance-aware loss that reweights small connected components so that tiny lesions contribute as much as large ones. The benchmark design is the second mechanism: a stratified split of three in-domain sites into train and test sets, plus a deliberately out-of-domain fourth site with a different scanner vendor and a mix of MS and MS-mimic subjects, which gives the generalization numbers their meaning. The error typology, which classifies true-positive, false-positive, and false-negative lesions by lesion type and volume, and the bottleneck-feature analysis are the instruments that connect the model's behavior to clinically interpretable causes.","core_discovery":"On its own terms, the paper's central discovery is that a vanilla 3D U-Net configured by nnU-Net, trained on MP2RAGE and MPRAGE data with the Blob BCE + Dice loss, segments cortical lesions across sites and protocols well enough to act as a practical baseline: lesion-detection F1-score of 0.641 in-domain and 0.502 on an out-of-domain site that differs in scanner manufacturer, acquisition protocol, patient cohort, and annotation workflow, including MS-mimic patients. The comparative experiments show that larger residual-encoder variants are not significantly better, that U-Mamba variants failed to converge, and that upsampling all data to 0.5 mm isotropic raises recall but collapses precision, cutting F1 by roughly 45 percent, so the authors reject it. Error analysis attributes most missed lesions to the smallest leukocortical lesions, most false positives to juxtacortical white matter lesions that raters themselves struggle to classify, and the worst performance to 7T subpial lesions that lower-field T1-weighted imaging cannot reliably show. Bottleneck-feature projections reveal that the network encodes site- and modality-specific information alongside task-relevant features, and the authors conclude that a standard nnU-Net with task-specific loss adjustments effectively segments cortical lesions across centers and protocols, with the models released for public use.","pith_inferences":["If the out-of-domain site's scans were re-annotated by the in-domain sites' raters, the 0.50 out-of-domain F1 could shift measurably in either direction; part of what the paper attributes to scanner-shift generalization may be rater and lesion-definition shift, so the true cross-vendor gap is not yet pinned down.","Because many false positives fall inside white-matter lesion masks, the measured F1 likely understates clinical usefulness: some detections counted as wrong are real lesions that the ground truth did not label, which is exactly what a second-opinion reading tool would want to surface.","The sharp site and modality clustering in the bottleneck features suggests that lightweight domain adaptation, such as feature-statistics alignment or test-time adaptation, could recover a large share of the out-of-domain gap without any new annotated data.","The paper's own table reports no recall for the MS-mimic subjects in the out-of-domain set, so the headline 0.50 F1 blends detection on MS patients with false-positive control on mimics; a site planning deployment should read the MS-only and mimic-only numbers separately."],"forward_implications":["The released vanilla nnU-Net with Blob loss can be applied directly to new 3T MPRAGE/MP2RAGE data as an off-the-shelf cortical lesion detection baseline, with expected performance near the reported in-domain F1 of 0.64.","Clinical sites gain nothing from the largest architectures: since the vanilla model matches or beats the residual-encoder M, L, and XL variants, lighter models with lower memory and energy costs are the justified choice.","The Blob BCE + Dice loss gives a modest but consistent precision gain, most notable on out-of-domain data, making it the recommended default for small-lesion segmentation tasks.","Upsampling heterogeneous data to the finest available resolution is counterproductive for cortical lesion segmentation, since the false-positive explosion outweighs the recall gain.","Because missed lesions are mostly small leukocortical ones and false positives overlap juxtacortical white matter lesions, further progress depends on clearer annotation guidelines and possibly soft labels, not on bigger networks."],"supporting_citations":[{"why":"Supplies the self-configuring nnU-Net framework that the paper benchmarks and adapts.","marker":"[22]"},{"why":"Defines the Blob loss whose instance-size reweighting is the key task-specific modification behind the reported gains.","marker":"[30]"},{"why":"Documents the residual-encoder and state-space variants tested and the convergence-instability rationale for excluding U-Mamba.","marker":"[27]"},{"why":"Prior dedicated 7T cortical-lesion detection framework (CLAIMS) whose evaluation approach and results this benchmark builds on and compares with.","marker":"[20]"},{"why":"Prior joint white-matter and cortical lesion segmentation at 3T, the methodological baseline whose inflated CL detection motivates the CL-specific evaluation here.","marker":"[14]"},{"why":"Recent 3T CL detection with FLAWS contrasts, the comparative standard for specialized-contrast approaches this work contrasts against.","marker":"[21]"},{"why":"Supplies the standard evaluation pipeline and lesion-detection metrics (F1, precision, recall, connected-component analysis) used throughout.","marker":"[31]"},{"why":"Provides the HD-BET brain extraction used in the preprocessing pipeline that all models share.","marker":"[24]"}],"fun_headline_variants":["Vanilla nnU-Net scores F1 0.64 on MS cortical lesions","Four-center benchmark: nnU-Net handles MS cortical lesions","Cortical lesion segmentation: simple U-Net beats complexity","nnU-Net with blob loss nails MS lesion detection across sites","Out-of-domain F1 0.50: nnU-Net generalizes on MS MRI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark treats the manual lesion masks from the four sites as interchangeable ground truth even though each site used different raters, different source sequences, and different lesion-inclusion criteria, so if those label differences are large, the reported site-to-site gaps and the out-of-domain drop may reflect annotation mismatch rather than scanner or protocol shift.","fun_headline_variants_meta":{"raw":{"variants":["Vanilla nnU-Net scores F1 0.64 on MS cortical lesions","Four-center benchmark: nnU-Net handles MS cortical lesions","Cortical lesion segmentation: simple U-Net beats complexity","nnU-Net with blob loss nails MS lesion detection across sites","Out-of-domain F1 0.50: nnU-Net generalizes on MS MRI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1444,"prompt_tokens":1094,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":710,"tokens_out":350,"duration_ms":4116,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:54:00.700809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a single expert panel re-annotate a matched subset of scans from the in-domain sites and the out-of-domain site under one common protocol, then recompute per-site F1: if the out-of-domain F1 of about 0.50 rises toward the in-domain 0.64, the scanner-generalization claim is largely label heterogeneity, while if it stays near 0.50, true domain shift is confirmed. A complementary check is measuring inter-rater agreement between in-domain and out-of-domain annotators on identical scans, since model performance at or above that agreement means the network is near the annotation ceiling.","supporting_citations":[{"cited_title":"blob loss: instance imbalance aware loss functions for semantic segmentation","cited_arxiv_id":"2205.08209","evidence_quote":"Defines the Blob loss whose instance-size reweighting is the key task-specific modification behind the reported gains."},{"cited_title":"Isensee, T","cited_arxiv_id":null,"evidence_quote":"Documents the residual-encoder and state-space variants tested and the convergence-instability rationale for excluding U-Mamba."},{"cited_title":"La Rosa, E","cited_arxiv_id":null,"evidence_quote":"Prior dedicated 7T cortical-lesion detection framework (CLAIMS) whose evaluation approach and results this benchmark builds on and compares with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior joint white-matter and cortical lesion segmentation at 3T, the methodological baseline whose inflated CL detection motivates the CL-specific evaluation here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recent 3T CL detection with FLAWS contrasts, the comparative standard for specialized-contrast approaches this work contrasts against."}],"review_version":1}