{"id":"fa7e2e43-9a5e-4212-b010-76b08e721652","arxiv_id":"2505.07573","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An nnU-Net model trained only on public CT data matches human-level kidney and abnormality segmentation and beats TotalSegmentator and BAMF on most tested datasets.","lead":"Researchers trained a public kidney and kidney-abnormality segmentation model on two open CT datasets and tested it on more than 1,500 scans from three centers, reporting higher Dice scores than two existing models. The paper is a validation study: if its numbers hold, radiologists get a freely available, externally checked tool for measuring kidney volumes and lesions from contrast-enhanced CT.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hilum protocol mismatch between KiTS and Radboudumc training labels is load-bearing: a hilum-inclusive prediction on hilum-excluded Radboudumc references caps kidney Dice near 0.95, so the reported human-level and SOTA comparisons may be partly artifacts of label inconsistency.","rationale":"The reader identified the same weakest assumption, and I agree. The concern is sharpened by a quantitative observation: the reported kidney Dice of 0.95 on the Radboudumc test set is close to what a hilum-inclusive predictor would score against a hilum-excluded reference when h/V is around 0.1. This makes the annotation-protocol mismatch a concrete correctness risk rather than a purely semantic convention, because it could inflate the apparent human-level agreement and the margin over BAMF. The paper's own Discussion acknowledges the decision not to re-annotate and frames the benefit as a hypothesis, but no supporting ablation is given. At the same time, the paper has real independent support: external evaluation on 1,510 Charite scans, public-data-only training, released code and weights, and a pre-registered-style evaluation protocol with multiple metrics. The concern is testable by re-annotation, and the overall verdict remains conditional rather than reject or accept: the central claims are plausible and well-evidenced in parts, but the label-protocol ambiguity should be resolved before the human-level and outperformance claims are taken at face value.","tokens_in":17839,"tokens_out":9010,"duration_ms":93149,"concrete_test":"Have an experienced radiologist re-annotate the renal hilum (non-fat hilar tissue) on 30 Radboudumc B30 and 20 TCGA-KIRC test cases, then recompute kidney and kidney+abnormality Dice/HD95 for the proposed model, BAMF, and TotalSegmentator under a single, hilum-excluded protocol. If the proposed model's Radboudumc kidney Dice drops materially (e.g., from 0.95 to below 0.93) or its advantage over BAMF shrinks, the merged-label target inconsistency is load-bearing and the reported human-level comparison is biased. If the metrics are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1.3 states that KiTS23 includes the renal hilum in the kidney mask while Radboudumc excludes it, and that the authors deliberately did not re-annotate either dataset. Because KiTS contributes 489 scans and Radboudumc 215, the merged kidney target is contradictory: the same hilar tissue is foreground in the majority of training data and background in the minority. The Radboudumc private test set follows the hilum-excluded protocol. For a model that, like the KiTS majority, predicts kidney as parenchyma plus hilum while the reference is parenchyma only, kidney Dice equals 2V/(2V+h), where V is the parenchyma volume and h is the non-fat hilar tissue volume. With h/V around 0.1, this yields a Dice of about 0.95, close to the reported kidney Dice on Radboudumc B30. The same mechanism could contribute to the 0.95 value on TCGA-KIRC if that reference uses KiTS-style hilum-inclusive labels. This is not merely a labeling convention: the paper motivates kidney volume as a biomarker, and a systematic hilum over-segmentation would bias volume estimates. The Discussion's claim that exposure to annotation variability improves robustness is a plausible hypothesis, but no ablation, re-annotation, or protocol-sensitivity analysis is provided to support it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents Renal-Net, a kidney and kidney-abnormality segmentation framework based on nnU-Net v2 (ResEnc-L, full-resolution, 5-fold ensemble), trained exclusively on two public datasets: KiTS23 and the Radboudumc kidney abnormality dataset. The pipeline uses TotalSegmentator-based ROI cropping and a rule-based post-processing step. The model is evaluated on a private Radboudumc test set (50 cases), the public TCGA-KIRC set (28 cases), and a large private Charité test set (1,510 scans). Performance is reported as Dice and HD95 for the kidney, kidney+abnormality, and abnormality regions, with comparisons against TotalSegmentator and the BAMF model, detection metrics, subgroup analyses, and qualitative case reviews. The authors claim that the model generalizes to external data, outperforms existing state-of-the-art models across all tested datasets, and reaches human-level performance on the Radboudumc test set, and they release code and weights.","tokens_in":18123,"tokens_out":7200,"duration_ms":65076,"significance":"If the validation withstands scrutiny, this is a practically valuable contribution: it shows that a model trained only on public data can reach high Dice values on large external cohorts, and it provides one of the largest external validations of a kidney/abnormality segmentation model (1,510 clinical CT scans). The explicit comparison with human inter-observer variability, the release of code and weights via GitHub, Zenodo, and grand-challenge.org, and the reporting of detection and subgroup results are all strengths. The main risk to the central claims is the inconsistency in the kidney-label protocol between the two training datasets, which directly affects the interpretation of kidney-region Dice on the Radboudumc test set and, consequently, the human-level and superiority claims.","major_comments":[{"comment":"The merged kidney label is not a well-defined training target because KiTS23 includes the renal hilum in the kidney mask (489 scans) while the Radboudumc training set excludes it (215 scans), and the authors state that they deliberately did not re-annotate either dataset. The Radboudumc private test set follows the hilum-excluded protocol. For a model that follows the KiTS majority convention, agreement with this reference is capped at Dice = 2V/(2V+h), where V is parenchyma volume and h is non-fat hilar tissue volume; with h/V around 0.1 this cap is about 0.95, which is exactly the range reported for the Radboudumc B30 (0.95 ± 0.04) and B20 (0.96 ± 0.01) kidney Dice. The reported kidney Dice therefore does not by itself establish anatomically correct hilum handling, and it is not a clean basis for the human-level comparison or for the superiority claim over TotalSegmentator and BAMF on the kidney region. Since the paper motivates kidney volume as a biomarker, a systematic hilum over-segmentation bias would have clinical consequences. Please quantify the hilum effect, for example by reporting metrics with the hilum region excluded, by re-annotating a subset under a single protocol, or by evaluating against a hilum-inclusive reference, and temper the Discussion's hypothesis that exposure to annotation variability improves robustness unless it is supported by an ablation or sensitivity analysis.","section":"Section 2.1.3, Figure 2, Tables A.4-A.6"},{"comment":"The statement that the proposed model 'significantly outperforms' the BAMF model on the Charité test set rests on Mann-Whitney tests over 1,510 scans, but the absolute differences in mean Dice are extremely small (kidney 0.84 vs 0.83; kidney+abnormality 0.89 vs 0.89; abnormality 0.83 vs 0.81). With this sample size, statistical significance can be reached for differences that are not practically meaningful. Please report effect sizes, paired difference distributions, or 95% confidence intervals, and revise the abstract's claim of outperforming state-of-the-art models 'across all tested datasets' to reflect the magnitude of the advantage on the Charité set.","section":"Section 3.2, Section 4.2.3, Table A.6"},{"comment":"The subgroup analyses are presented as evidence of robustness, but no statistical comparisons are made between subgroups, and the TCGA-KIRC analysis uses only 28 scans split into four age groups and two sex groups. The boxplots alone do not support the conclusion of 'consistent high performance' across subgroups. Please either add statistical testing or explicitly state that the subgroup results are descriptive only, and avoid the strong robustness language in the abstract until this is supported.","section":"Section 4.4, Figures 5 and 6"}],"minor_comments":[{"comment":"The BAMF row for the kidney region appears to contain a formatting error: '0.90 ± 0.02 0.90 ± 0.04' is ambiguous and the HD95 values for B20 and B30 are missing or misplaced. Please correct the table.","section":"Appendix A, Table A.4"},{"comment":"The abstract and title use 'Renal Mass Segmentation' while the body consistently uses 'Kidney Abnormality Segmentation'; please align the terminology throughout.","section":"Abstract and Section 2.3"},{"comment":"Footnote 1 states that all URLs will be made public after acceptance, while the abstract and text say the algorithm and code are publicly accessible at the given GitHub and Zenodo links. Please clarify the current availability status and provide the exact DOIs or version identifiers.","section":"Footnote 1, Abstract, Section 2.3"},{"comment":"The single-kidney evaluation rule for the Charité set, in which the predicted region overlapping the reference is selected and a zero score is assigned when there is no overlap, should be discussed as a potential source of favorable bias; consider a sensitivity analysis that evaluates both predicted kidney regions.","section":"Section 3.1.4"},{"comment":"The boxplots state that the whiskers show the full distribution while outliers are omitted; this is contradictory and should be clarified in the figure captions.","section":"Figures 3-6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing about this paper is the resource: a public nnU-Net trained on KiTS23 plus the Radboudumc public dataset, with weights and code released, and a validation footprint of 1,510 Charité scans plus TCGA-KIRC and a same-center test set. If you work on kidney or tumor segmentation, this is one of the few public models with that scale of external data. The authors should get credit for shipping the artifact and for reporting detection metrics and human inter-observer comparison, not just Dice.\n\nThe main numerical claims are credible in the sense that no parameters were fitted to the test sets and the external cohorts are real. But the paper overreaches in two places. First, “outperforms existing state-of-the-art models across all tested datasets” is not fully supported: on Charité the only comparator is BAMF, and the mean Dice advantage is tiny (0.84 vs 0.83 kidney; 0.83 vs 0.81 abnormality); significance comes from the large n. On TCGA there is no BAMF comparison because BAMF trained on that data, so the only comparator is TotalSegmentator, which is not a kidney-abnormality model. The claim is directionally likely but not demonstrated.\n\nSecond, the hilum protocol mismatch between the two training datasets is real and under-analyzed. KiTS includes the hilum in the kidney mask; Radboudumc excludes it. The authors deliberately did not re-annotate. The stress-test arithmetic checks out: if the model predicts the KiTS-style kidney (parenchyma plus hilum) on a Radboudumc-style reference, kidney Dice is capped near 2/(2+0.1) ≈ 0.95. The reported 0.95 kidney Dice on Radboudumc and TCGA sits suspiciously close to that ceiling. This does not definitively sink the paper—the model also trained on 215 Radboudumc scans with hilum-excluded labels, and the abnormality Dice is the more clinically relevant number and is not explained by this effect—but the authors’ claim that exposure to annotation variability improves robustness is a hypothesis with no supporting ablation. For kidney volume as a biomarker, this matters. They should either re-annotate a subset, measure hilum volume, or at least test sensitivity to excluding the hilum from predictions.\n\nMinor soft spots: subgroup analysis is just boxplots with no statistical tests; the same-center Radboudumc test set shares protocol with training; and the detection F1 is low (0.37–0.74), which the paper is honest about but which tempers the “robust” language.\n\nWho it’s for: groups wanting a ready-made kidney-abnormality segmentation model or a benchmark for external validation. It deserves a serious referee; the weaknesses are addressable with additional comparisons and a hilum-sensitivity analysis. I’d engage with it.","headline":"A genuinely useful public model with a large external validation, but the kidney-Dice claim is partially undercut by an unanalyzed hilum protocol mismatch.","tokens_in":18733,"tokens_out":3057,"would_cite":true,"duration_ms":28559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CT kidney-and-abnormality segmentation model trained only on public datasets can match human-level agreement at home and outperform existing public segmenters on external cohorts.","keywords":["kidney segmentation","kidney abnormality segmentation","nnU-Net","computed tomography","external validation","Dice similarity coefficient","subgroup analysis"],"falsifier":"Re-annotate a random subset of the external test scans under both annotation protocols—hilum included versus excluded, and cyst/tumour split versus merged—and recompute Dice and HD95 against each reference; if the scores diverge by more than the reported inter-observer spread, the claimed external generalizability is largely an artefact of protocol agreement rather than anatomical accuracy. A simpler first check is to run the released model on non-contrast CTs, where the paper already concedes failure, and measure the Dice drop.","tokens_in":17637,"feed_emoji":"🩻","tokens_out":10700,"duration_ms":90486,"temperature":0.7,"pith_summary":"The paper sets out to show that a CT kidney-and-abnormality segmentation model trained exclusively on publicly available data can match human-level agreement on its home-institution test set and still perform well on external cohorts from other institutions. If true, this matters because kidney volume and lesion size—currently often assessed by eye—could be measured automatically and reproducibly, supporting staging, treatment decisions, and active surveillance. The authors report that their model, built on an nnU-Net backbone, reaches Dice scores comparable to an independent human observer on the local private test set and significantly outperforms two established reference models on all tested datasets. Subgroup analyses across patient sex, age, CT contrast phase, and tumour histologic subtype show small performance differences. The model weights and code are released for reuse.","feed_headline":"Public data alone trains a kidney AI that beats prior models","feed_subtitle":"Trained without private data, it beats two established segmenters across three CT test sets.","key_machinery":"The load-bearing machinery is a 3D nnU-Net v2 model with Residual Encoder L presets, a self-configuring neural architecture for medical image segmentation, trained as a five-fold ensemble with the full-resolution configuration selected by cross-validation. Before inference and training, a pre-processing step uses the TotalSegmentator anatomical segmenter to locate the lower lung lobes and the urinary bladder and crops a region of interest around the kidneys, reducing compute and standardizing fields of view. A post-processing step removes predicted abnormalities not connected to the kidney unless they exceed 100 cm³ and keeps attached abnormalities with axial diameter above 3 mm, using kidney-volume and RECIST-inspired heuristics. The training target merges cyst and tumour labels from the public challenge into a single 'abnormality' class, while deliberately leaving the two datasets' differing hilum-inclusion protocols untouched.","core_discovery":"On its own terms, the paper's central claim is that a kidney-abnormality segmentation model built from the nnU-Net framework and trained only on the public kidney-tumour challenge dataset plus a Dutch public kidney-abnormality dataset generalizes to unseen CT scans. The reported numbers are Dice 0.95–0.96 for the kidney region and 0.66 for the abnormality region on the local 50-scan private test set, 0.95 and 0.86 on a 28-scan public clear-cell carcinoma cohort, and 0.84 and 0.83 on a 1510-scan German surgical cohort, with the model outperforming the TotalSegmentator and BAMF reference models on every comparison at corrected significance. The paper further claims that the model's performance lies within human inter-observer variability on the local test set and that subgroup analyses reveal no large biases by sex, age, contrast phase, or tumour subtype.","pith_inferences":["A consequence the authors leave implicit: the reported Dice values are measured against reference masks that follow each site's local protocol, so on the challenge dataset's hilum-inclusive protocol the kidney scores could shift; cross-dataset numbers are protocol-relative rather than anatomy-absolute.","Because the public clear-cell carcinoma cohort's reference masks were AI-generated and radiologist-corrected, part of the high external Dice may reflect agreement with an automated annotation style rather than pure anatomical truth; a fully manual re-annotation subset would separate these contributions.","Detection precision at a 0.5 IoU threshold on the home test set is markedly lower than on the large surgical cohort, suggesting small or subtle abnormalities in a less-selected population remain a weak point; a reader study could quantify the clinical cost of false negatives.","A direct testable extension is to retrain the identical pipeline on versions of the challenge data with hilum included or excluded and compare Dice under both reference protocols, quantifying the label-mismatch bias explicitly."],"forward_implications":["If the central claim holds, a segmentation model with human-comparable kidney Dice can be built without any proprietary training data, lowering the barrier for reproducible model development and external validation.","Automated kidney and abnormality volumes could replace subjective visual estimates in staging, nephrometry scoring, and longitudinal monitoring such as active surveillance.","The released weights and code let other centres test the model on their own CT protocols; the paper explicitly notes it fails on non-contrast scans, so retraining with non-contrast examples is the likely remedy.","Significant superiority over two public reference models suggests that organ-specific fine-tuning on merged public datasets is a practical route when general-purpose segmenters miss kidney abnormalities."],"supporting_citations":[{"why":"Supplies the nnU-Net self-configuring segmentation framework that the proposed model is built on.","marker":"[15]"},{"why":"Provides the public kidney-tumour challenge training dataset and the evaluation convention used for the merged kidney-plus-abnormality region.","marker":"[24]"},{"why":"Supply the Dutch public kidney-abnormality training set and the inter-observer Dice values used as the human performance baseline.","marker":"[29, 30]"},{"why":"Motivates the Residual Encoder L presets and the choice of nnU-Net over a competing out-of-the-box framework.","marker":"[28]"},{"why":"TotalSegmentator provides the anatomical landmarks used for ROI cropping and is the first comparison baseline.","marker":"[16]"},{"why":"Introduces the BAMF model and the AI-generated, radiologist-corrected annotations used for the public clear-cell carcinoma test cohort.","marker":"[25]"},{"why":"Defines the public clear-cell renal carcinoma imaging collection used as one external test set.","marker":"[31]"},{"why":"Supplies the recommendation to report the 95th-percentile Hausdorff distance alongside overlap metrics.","marker":"[34]"},{"why":"Sets the RECIST lesion-size thresholds that motivate the 3 mm post-processing cutoff for attached abnormalities.","marker":"[33]"}],"fun_headline_variants":["Public-data-only kidney AI out-segments prior models in CT","Kidney AI on open data beats existing CT segmenters","Open-data kidney AI outperforms two prior models across CT test sets","Renal-Net: open-data AI beats existing segmenters across subgroups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The merged training labels define one coherent segmentation target even though the two source datasets disagree about whether the renal hilum is part of the kidney and whether cysts should be a separate class, and the authors re-annotated neither.","fun_headline_variants_meta":{"raw":{"variants":["Public-data-only kidney AI out-segments prior models in CT","Kidney AI on open data beats existing CT segmenters","Open-data kidney AI outperforms two prior models across CT test sets","Renal-Net: open-data AI beats existing segmenters across subgroups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001418,"raw_usage":{"total_tokens":5739,"prompt_tokens":974,"completion_tokens":4765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":4692}},"tokens_in":590,"tokens_out":4765,"duration_ms":31612,"temperature":1.0,"reasoning_tokens":4692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:13:43.193826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of the external test scans under both annotation protocols—hilum included versus excluded, and cyst/tumour split versus merged—and recompute Dice and HD95 against each reference; if the scores diverge by more than the reported inter-observer spread, the claimed external generalizability is largely an artefact of protocol agreement rather than anatomical accuracy. A simpler first check is to run the released model on non-contrast CTs, where the paper already concedes failure, and measure the Dice drop.","supporting_citations":[{"cited_title":"Isensee, P","cited_arxiv_id":null,"evidence_quote":"Supplies the nnU-Net self-configuring segmentation framework that the proposed model is built on."},{"cited_title":"Wasserthal, H.-C","cited_arxiv_id":null,"evidence_quote":"TotalSegmentator provides the anatomical landmarks used for ROI cropping and is the first comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the public clear-cell renal carcinoma imaging collection used as one external test set."}],"review_version":1}