{"id":"7e46c9f3-3699-4a26-afcb-a448452d989e","arxiv_id":"2505.22537","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ConfLUNet, an end-to-end instance segmentation network for multiple sclerosis lesions, improves separation and counting of confluent lesion units compared with connected components and automatic splitting.","lead":"This paper presents ConfLUNet, a deep learning model that finds and outlines individual multiple sclerosis lesions on brain MRI, including lesions that have merged into one confluent mass. It also introduces formal definitions and new metrics for confluent lesion units and reports that ConfLUNet outperforms existing methods on a private test set.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-rater ground truth for confluent lesions is acknowledged to be unreliable, so the reported performance gains may be specific to that rater's labels.","rationale":"The reader identified the single-rater ground truth as the weakest assumption, and the paper itself explicitly flags both the difficulty of annotating confluent lesions (Fig. A.10) and the potential bias of single-rater annotations (Limitations). This is load-bearing because the reference annotations serve simultaneously as training targets and evaluation labels; any systematic tendency to merge or arbitrarily split confluent lesions will be learned by ConfLUNet and then used to judge all methods, potentially favoring the method that best mimics that rater's particular bias. The paper's extensive evaluation and Holm-Bonferroni corrections do not mitigate this, because they only address statistical variability given the labels, not the validity of the labels themselves. I also considered a second concern: the baseline U-Net used to generate CC/ACLS predictions is trained with nnUNet's Poly learning-rate schedule while ConfLUNet uses a constant rate (Sec. 3.2 vs. Experiments 2/3), so part of the measured gain could be due to optimization choices rather than the instance heads. However, that concern affects the attribution of the improvement rather than the existence of the empirical difference, whereas unreliable ground truth undermines the entire evaluation. Therefore the reader's CONDITIONAL verdict remains appropriate: the claims are plausible and well-presented but not definitive until the reference labels are shown to be reproducible or the results are confirmed on an independent multi-rater dataset.","tokens_in":40340,"tokens_out":5066,"duration_ms":60859,"concrete_test":"Have a second trained rater, naive to the first annotations and using the same protocol and supporting contrasts, re-annotate the 13 test-set patients (or at minimum the 7 patients with CLUs). Compute inter-rater agreement on instance partitions (e.g., instance-wise F1 between annotators). Then recompute Tables 4-6 using the second rater's labels as ground truth, and also using the voxel-wise intersection or consensus of the two raters. If ConfLUNet's advantage over CC and ACLS in PQ/F1/F1CLU does not broadly replicate on the alternate ground truth, the central claim is not robust to annotation variability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation rests on manual instance annotations made by one neurobiologist and reviewed by one neurologist (Sec. 4.1.2). Appendix A (Fig. A.10) states that for large confluent lesions it is 'extremely difficult, if not impossible' to distinguish individual lesion units, and the Limitations section concedes that 'single-rater annotations may introduce bias.' If those labels are wrong in confluent regions, the reference masks used for both training and evaluation are wrong, so every reported metric (PQ, F1, F1CLU) and the ranking of ConfLUNet vs. CC/ACLS inherit that error. The formal definitions of CLUs (Eq. 2-4) are relative to these reference instances, and ConfLUNet's center/offset training targets are derived from them, so the model may be fitting the rater's idiosyncratic splitting rather than true lesion units. The exclusion of the two most confluent patients (Sec. 4.1.1) removes precisely the cases where the ground truth is most ambiguous, further narrowing the validity of the claim. Without evidence of inter-rater reproducibility, the observed superiority of ConfLUNet could vanish under a different (equally plausible) annotation of the same test images.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of instance segmentation of multiple sclerosis (MS) lesions, with a focus on confluent lesions that appear as merged clusters on MRI. The authors formalize definitions of confluent lesions and confluent lesion units (CLUs), propose CLU-aware evaluation metrics, systematically compare two existing post-processing approaches (connected components, CC, and automated confluent lesion splitting, ACLS), and introduce ConfLUNet, an end-to-end 3D instance segmentation network with semantic, center, and offset heads. On a held-out test set of 13 patients, ConfLUNet reportedly achieves significantly higher Panoptic Quality (42.0% vs 37.5%/36.8%) and lesion detection F1 (67.3% vs 61.6%/59.9%) than CC and ACLS, and the highest F1CLU (81.5%). The paper also provides open-source code and a detailed evaluation framework.","tokens_in":40560,"tokens_out":4983,"duration_ms":56535,"significance":"If the results hold, the paper makes a valuable contribution by formalizing the CLU evaluation problem, providing reproducible metrics, and demonstrating that an end-to-end model can achieve a better precision-recall balance than connectivity-based or center-splitting post-processing. The experimental design is careful in several respects: a held-out test set, 5-fold cross-validation, an architecture-matched baseline (3D U-Net with the same backbone), paired Wilcoxon tests, Holm-Bonferroni-corrected analyses, and explicit disclosure of limitations. The release of code and containers supports reproducibility. The main significance is in the evaluation framework and in establishing a concrete baseline for future instance segmentation methods in MS. However, the strength of the central claim is tempered by the heavy reliance on single-rater manual annotations for confluent regions, which the paper itself acknowledges are extremely difficult to label, and by the small test set with very few confluent cases.","major_comments":[{"comment":"The evaluation rests entirely on manual instance annotations made by a single neurobiologist and reviewed by a single neurologist. The paper itself states in Appendix A (Fig. A.10) that for large confluent lesions it is 'extremely difficult, if not impossible, to distinguish individual lesion units,' and the Limitations section concedes that 'single-rater annotations may introduce bias.' Because the reference instances define the CLU sets L_CLU and L_CLU+ (Eqs. 3–4) and the center/offset training targets (Section 3.1), all reported metrics (PQ, F1, F1CLU) and the ranking of ConfLUNet versus CC/ACLS inherit this potential bias. Without inter-rater reproducibility data or at least a sensitivity analysis on a subset of images re-annotated by a second rater, the central claim that ConfLUNet better separates true confluent lesion units is not fully established.","section":"Section 4.1.2, Appendix A, Limitations"},{"comment":"The paper highlights F1CLU as a key result, but the difference between ConfLUNet (81.5%) and CC (79.8%) is not reported as statistically significant, and the F1CLU+ improvement over CC is marginal (p = 0.046) and may not survive the Holm-Bonferroni correction shown in Appendix D. The CLU-specific comparisons are further limited to the 7 test patients with at least two CLUs (Section 5.3), giving very low statistical power. The claims about improved CLU detection should be softened accordingly, and the exact p-values (including non-significant ones) for the F1CLU comparisons should be reported.","section":"Section 5.3, Table 5"},{"comment":"The test set is small (n=13, with only 7 patients exhibiting at least two CLUs), and the two patients with the most extreme confluency were excluded by design because their confluent lesions were deemed indiscernible. Excluding the cases where the ground truth is most ambiguous removes exactly the cases most relevant to the method's intended use case, which narrows the external validity of the reported gains. The authors acknowledge this in the Limitations, but they do not quantify how the conclusions would change if those two patients were included (even with their uncertain labels). Reporting performance on those two patients as a sensitivity analysis would help assess robustness.","section":"Section 4.1.1, Section 5.3"},{"comment":"The abstract and Section 3 describe ConfLUNet as 'the first end-to-end instance segmentation framework for MS lesions,' but the authors' own prior work, reference [21] (ISBI 2024), already proposes ConfLUNet for confluent lesion identification. The present manuscript should explicitly state the architectural and methodological differences from [21] and adjust the novelty claim accordingly. As written, the 'first' claim is at odds with the cited prior work and could be misleading to readers.","section":"Abstract, Section 3, Reference [21]"},{"comment":"The construction of FPCLU (predicted lesions that match a reference lesion but are not the best match for that reference) implies that any method that never oversplits, such as CC, will automatically achieve PrecisionCLU = 100% regardless of how many reference CLUs it merges. Table 5 confirms that CC reaches 100% PrecisionCLU. As a result, F1CLU comparisons between CC and splitting methods are essentially recall-only comparisons, and the proposed metric does not directly penalize the merging failure mode that motivates the paper. The authors should justify this asymmetry or introduce an additional metric that penalizes merging explicitly.","section":"Section 4.2, FPCLU definition"}],"minor_comments":[{"comment":"There is a formatting error in the abstract: 'textF 1CLU' should be 'F1CLU'.","section":"Abstract"},{"comment":"The capitalization of the method name is inconsistent: the text uses both 'ConfLUNet' and 'ConfLUnet' (e.g., Figure D.12), and 'ConfLUnet' appears in some figures and tables. The authors should standardize to one spelling.","section":"Throughout"},{"comment":"The text cites 'SAMSEG [114]' for the SAMSEG tool, but the main reference list contains only [1]–[47]; [114] appears in the Appendix E reference list (Cerri et al.). This citation mismatch should be fixed so that the main text uses the correct reference number.","section":"Section 4.3, Reference list"},{"comment":"The ACLS description cites '[20, 120]' and 'our previous work [29]', but [120] is only in the Appendix E bibliography, not in the main reference list. The authors should ensure all citations in the main text point to the main reference list or move the relevant entries.","section":"Section 2.1, References"},{"comment":"The matching procedure uses a low IoU threshold (λ = 0.1) and mutual-best matching. The authors should briefly justify why this threshold is appropriate for 3D lesions and discuss its impact on reported PQ and F1 values, given that small spatial errors can cause a predicted lesion to fall below the threshold.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its experimental setup and provides a useful benchmark, but the load-bearing issue of single-rater ground truth in confluent regions needs to be addressed with reproducibility or sensitivity analyses, and the novelty claim relative to the authors' own ISBI 2024 paper should be clarified. The citation mismatch for SAMSEG and [120] also needs correction. I am recommending major revision rather than rejection because the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new things here are the formal definitions of CLU and CLU+, the CLU-aware detection metrics, and the first end-to-end instance segmentation model for MS lesions. The paper also gives CC and ACLS their first systematic side-by-side evaluation in a common instance-segmentation framework. That combination is useful and, as far as I can tell, not done before. They ship code and a Docker container, which is real and reproducible evidence.\n\nWhat the paper does well is the experimental discipline. There is a held-out test set, 5-fold cross-validation, an architecture-matched baseline, paired Wilcoxon tests, and Holm-Bonferroni corrections in the supplementary. The limitations are stated plainly: single-rater annotations, small test set, two extreme-confluency patients excluded, private data. They do not oversell; the abstract's uncorrected p-values are a minor issue because the corrected ones are in Appendix D.\n\nNow the soft spots. The stress-test worry about single-rater ground truth is legitimate and the authors themselves concede it. Large confluent lesions are, by their own admission, nearly impossible to split reliably from cross-sectional FLAIR. That means the reference instances in confluent regions are uncertain, and every PQ, F1, and CLU metric inherits that uncertainty. But this is a limitation that any instance-segmentation study in this domain would face without a multi-rater or longitudinal annotation effort, and the paper does not hide it. The comparative result—ConfLUNet beats CC and ACLS on the same labels—is still meaningful within this dataset, though it may not generalize to other raters or scanners. The test set (n=13, 7 patients with CLUs) is small, and some CLU-level differences are not significant. Those caveats matter but do not sink the paper.\n\nWho is this for? Anyone working on MS lesion analysis, especially on lesion counting or paramagnetic rim pipelines, will get value from the CLU definitions and the evaluation framework. The specific claim that end-to-end beats post-processing should be treated as promising but not definitive until validated on larger multi-centric data with more rater information.\n\nI would send this to peer review. It is an honest, well-executed study with a clear contribution, and the weaknesses are both disclosed and addressable. I would also cite it for the CLU formalism, which is something the field has been missing.","headline":"A solid, transparent instance-segmentation paper for MS lesions whose main risk is the acknowledged single-rater ground truth, but which deserves peer review.","tokens_in":41161,"tokens_out":1262,"would_cite":true,"duration_ms":18683,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ConfLUNet, the first end-to-end instance segmentation framework for MS lesions, significantly outperforms connected components and automated confluent lesion splitting on the held-out test set in panoptic quality, lesion detection F1, and…","keywords":["multiple sclerosis","instance segmentation","confluent lesions","MRI","FLAIR","panoptic quality","lesion detection","deep learning"],"falsifier":"Re-annotate the 13 test-set patients with several independent raters who each mark CLU instances inside confluent regions, then recompute ConfLUNet's CLU detection F1 against each rater and against a multi-rater consensus reference; if ConfLUNet's advantage over connected components shrinks or disappears under these alternative references, the reported gains are an artifact of a single annotator's judgment rather than a property of the method.","tokens_in":40114,"feed_emoji":"🧠","tokens_out":8597,"duration_ms":88596,"temperature":0.7,"pith_summary":"This paper argues that multiple sclerosis (MS) lesion analysis on MRI should be evaluated and built as instance segmentation, not just semantic segmentation, because a single connected 'confluent' lesion can contain several distinct disease units and standard post-processing cannot separate them. The authors formalize confluent lesions, define their constituent confluent lesion units (CLUs) and an extended CLU+ definition, and introduce CLU-aware precision, recall, and F1 metrics. On top of this, they present ConfLUNet, described as the first end-to-end instance segmentation framework for MS lesions, which jointly predicts a semantic lesion map, a lesion center heatmap, and voxel-to-center offset vectors from a single FLAIR image. On a held-out test set of 13 patients, ConfLUNet significantly surpasses connected components (CC) and automated confluent lesion splitting (ACLS) in Panoptic Quality (42.0% vs 37.5% and 36.8%) and detection F1 (67.3% vs 61.6% and 59.9%), and achieves the highest CLU detection F1 (81.5%). If correct, this shows that jointly optimizing detection and delineation can resolve lesion units that post-processing either merges or oversplits, and it gives the field a metric framework to measure that.","feed_headline":"Deep network splits fused MS lesions better than post-processing","feed_subtitle":"Trained on 50 patients, it raises panoptic quality to 42% and CLU detection F1 to 81.5% on 13 held-out scans","key_machinery":"The argument rests on two pieces of machinery. One is formal definition: a confluent lesion is a connected component that overlaps at least two reference lesion instances, and each of those instances is a confluent lesion unit (CLU); dilating the semantic mask before the same test defines extended CLU+, capturing near-touch lesions. These definitions convert a clinically messy phenomenon into countable objects that can be scored. The other is ConfLUNet's architecture, which adds two output heads to a standard self-configuring 3D encoder-decoder segmentation backbone: a heatmap of lesion centers (Gaussian-smoothed centers of mass) and a 3D offset vector field pointing from each voxel to its lesion center. At inference, voxels are displaced by their predicted offsets and assigned to the nearest detected center, so lesion boundaries reflect learned geometric evidence rather than pure spatial connectivity.","core_discovery":"On its own terms, the paper's central claim is that an end-to-end model that simultaneously learns what is lesion, where lesion centers are, and which voxel belongs to which center can separate confluent MS lesion units better than deriving instances from a semantic mask by connected components or by statistical splitting. The evidence is the held-out comparison: ConfLUNet reaches Panoptic Quality 42.0% against 37.5% for CC and 36.8% for ACLS (p = 0.017 and 0.005), lesion-wise F1 67.3% against 61.6% and 59.9% (p = 0.028 and 0.013), and CLU F1 81.5%, with statistically significant recall gains over CC (+12.5%) and precision gains over ACLS (+31.2%). The paper also establishes the failure modes of the existing methods: CC systematically undercounts CLUs while ACLS systematically oversplits lesions, and these behaviors hold across three different semantic segmentation tools, supporting the claim that the default evaluation practice is misaligned with clinical needs.","pith_inferences":["If the single-rater ground truth is noisy in exactly the confluent regions the method targets, the reported advantage might change in either direction; a multi-rater study would tell whether ConfLUNet is exploiting signal or label bias.","The CLU+ dilation-based definition effectively forgives boundary disagreements of one voxel, which suggests the practical near-term role of such a model may be to flag candidate confluent regions and their units for human confirmation.","The same center-and-offset formulation could be tested on longitudinal data, since the authors note that temporal appearance can disambiguate confluent lesions; that would be a natural and testable extension for CLU separation.","Reporting patient-wise averages hides the clinical worst case; the lesion-wise analysis in the paper shows CC's CLU recall falling to 41.5%, implying future evaluations should report both patient-wise and lesion-wise CLU metrics."],"forward_implications":["Connected components, the default post-processing across the MS lesion literature, systematically underestimates confluent lesion unit counts and should be reconsidered for instance-level evaluation.","ACLS overestimates lesion counts and oversplits, so its high recall comes at a precision cost that the new metrics expose.","ConfLUNet offers a balanced precision-recall operating point, which the authors argue is closer to clinical needs for lesion counting and lesion-level biomarkers.","The formal CLU and CLU+ definitions and metrics give future work a shared reference for measuring instance segmentation in MS, replacing the current reliance on CC-derived counts.","Because ConfLUNet needs only a single FLAIR volume, the approach is a practical candidate for clinical pipelines that do not routinely acquire research-grade multi-contrast scans."],"supporting_citations":[{"why":"Supplies the statistical confluent-splitting post-processing that the paper evaluates as the main alternative baseline.","marker":"[20]"},{"why":"Provides the center-plus-offset bottom-up instance segmentation scheme that ConfLUNet adapts to 3D MRI.","marker":"[24]"},{"why":"Provides the self-configuring encoder-decoder backbone and training protocol on which ConfLUNet is built.","marker":"[27]"},{"why":"Earlier comparison of connected components and ACLS for MS lesion counting that motivates the evaluation framework.","marker":"[29]"},{"why":"Gives the metric and matching recommendations (panoptic quality, IoU threshold, patient-wise statistics) used throughout.","marker":"[37]"},{"why":"Introduced the ACLS post-processing, evaluated here as the second baseline.","marker":"[120]"},{"why":"Prior version of ConfLUNet that this paper extends with formal CLU definitions and the full evaluation.","marker":"[21]"},{"why":"Clinical source for the lesion size threshold used to filter predicted instances.","marker":"[32]"}],"fun_headline_variants":["ConfLUNet beats connected components for MS lesion instances","End-to-end MS model outperforms post-processing on lesion splits","New network raises CLU F1 to 81.5% on confluent MS lesions","First end-to-end segmentation for MS lesions surpasses CC and ACLS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation stands on the single-rater manual annotations being a correct partition of confluent lesions into units, yet the paper itself acknowledges that in large confluent lesions distinguishing individual units is extremely difficult, if not impossible, for human raters.","fun_headline_variants_meta":{"raw":{"variants":["ConfLUNet beats connected components for MS lesion instances","End-to-end MS model outperforms post-processing on lesion splits","New network raises CLU F1 to 81.5% on confluent MS lesions","First end-to-end segmentation for MS lesions surpasses CC and ACLS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3445,"prompt_tokens":1164,"completion_tokens":2281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":780,"completion_tokens_details":{"reasoning_tokens":2202}},"tokens_in":780,"tokens_out":2281,"duration_ms":21647,"temperature":1.0,"reasoning_tokens":2202,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:05:14.201164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 13 test-set patients with several independent raters who each mark CLU instances inside confluent regions, then recompute ConfLUNet's CLU detection F1 against each rater and against a multi-rater consensus reference; if ConfLUNet's advantage over connected components shrinks or disappears under these alternative references, the reported gains are an artifact of a single annotator's judgment rather than a property of the method.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the statistical confluent-splitting post-processing that the paper evaluates as the main alternative baseline."},{"cited_title":"Grahl, V","cited_arxiv_id":null,"evidence_quote":"Clinical source for the lesion size threshold used to filter predicted instances."}],"review_version":1}