{"id":"7f277769-f20c-4005-b12f-8125bd9f571f","arxiv_id":"1908.02625","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A three-U-net logical ensemble with volumetric post-processing segments kidneys and tumors in CT, achieving an average F1 of 0.6758 on a KiTS19 test split.","lead":"Kidney and tumor segmentation from CT scans is tackled by combining three 2D U-net models with logical AND operations and hand-crafted 3D cleanup. On the KiTS19 challenge data the method reaches an average F1 of 0.6758 for kidney and tumor classes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-frame 80/10/10 split may leak same-patient slices into the test set, inflating the reported mean F1.","rationale":"The reader flagged evaluation-space and threshold-tuning issues, but the more fundamental concern is test-set contamination via per-frame splitting. The phrase 'held-out 10 percent test split' in the verdict's strongest claim is ambiguous; §2.3 does not state patient-level separation. Because CT frames from one patient are not independent, a random frame split violates the i.i.d. assumption and makes the reported F1 an optimistic estimate of case-level performance. This is checkable because the code is linked. If the code confirms patient-level split, the concern would be dissolved and the original conditional verdict would stand; if not, the central claim requires revision. I retain CONDITIONAL rather than REJECT because the actual split is not yet verified from the manuscript text alone.","tokens_in":5146,"tokens_out":4381,"duration_ms":46984,"concrete_test":"Open the GitHub repository's data-loading/split routine (e.g., the script that creates train/val/test sets) and determine whether it groups by patient ID/case folder or randomly samples frame indices. If the split is per-frame, retrain the three U-nets using a patient-disjoint split (for example, 168/21/21 of the 210 cases) and recompute Table 1. If the mean F1 (particularly tumor F1, currently 0.5019) drops below the reported values by more than the variance across re-splits, then the central claim is not supported for patient-level generalization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central number (mean F1 = 0.6758, from §3 Table 1) is computed on a 10% test split described only as 'an 80:10:10 split was performed to separate data into training, validation, and test sets' (§2.3). Because the unit of the split is not specified as patients, a random per-frame split is plausible: the 210 scans contain 45,424 frames; the models are trained on individual frames. Adjacent axial slices from the same patient are highly correlated, and the tumor-positive frames for a given case form a small contiguous block. If any slices of a patient appear in training, test slices from that same patient are no longer independent; the network can exploit patient-specific intensity/texture and slice-level context, inflating both kidney and tumor F1. The tumor class is particularly vulnerable because the test set is small (roughly 10% of 5,696 tumor frames) and patient-specific. The paper also evaluates against preprocessed ground truth (§2.5) and states that restoring original format reduces the score, but the split leakage is a more direct threat to the validity of the reported F1 as a statement about generalizing to new patients, which is the KiTS19 benchmark. The volumetric-validation thresholds (§2.4) are another tuning-on-data risk, but the undefined split granularity is the most load-bearing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes a pipeline for kidney and tumor segmentation in CT scans from the KiTS19 challenge, consisting of three binary 2D U-net classifiers (two kidney-and-tumor models and one tumor-only model), a logical ensemble rule, and heuristic volumetric validation in 3D. The authors preprocess the images with a series of computer-vision operations, train the U-nets on 256x256 frames, and post-process the stacked segmentations by rejecting kidney regions below a volume threshold or outside the central scan depth and by confirming tumor regions based on volume, morphology, consistency, and sphericity. They report precision, recall, and F1 scores for background, kidney, and tumor on a 10% test split, with a mean kidney/tumor F1 of 0.6758 after volumetric validation, and they note that this score is computed on preprocessed images and annotations rather than original-format data.","tokens_in":5457,"tokens_out":4705,"duration_ms":50936,"significance":"If the evaluation were sound, this would be a useful and reproducible baseline for 2D U-net ensembles with volumetric post-processing on KiTS19. The manuscript explicitly provides code, a full description of the hyperparameters, and a clear rule-based post-processing stage, which are strengths. However, the quantitative claim (mean F1 = 0.6758) currently rests on an evaluation protocol that is insufficiently specified or potentially biased: the training/test split is not stated at the patient level, the metrics are computed on preprocessed ground truth, and the volumetric validation thresholds appear to be empirically tuned on the same dataset. These issues materially affect the confidence in the reported score as a measure of generalization to new patients.","major_comments":[{"comment":"The description of the data split states only that \"an 80:10:10 split was performed to separate data into training, validation, and test sets\" for each model, without specifying whether the split unit is a patient or an individual frame. Because the dataset consists of 45,424 frames from 210 patients, a per-frame split would place highly correlated adjacent slices from the same patient into both training and test sets, and tumor-bearing frames for a given case form contiguous blocks. Such leakage would inflate the reported test-set F1 values in Table 1 and would not reflect generalization to new patients, which is the stated clinical goal and the KiTS19 benchmark setting. The authors must specify the split granularity; if the split was per-frame, they must re-evaluate with a patient-exclusive split and report the resulting scores.","section":"§2.3"},{"comment":"The evaluation is explicitly computed on \"pre-processed ground truth labels from the KiTS19 dataset,\" and the abstract states that restoring to the original image format reduces the score. Since the preprocessing pipeline resizes, crops, and zooms images, the reported metrics do not directly quantify segmentation quality on the native KiTS19 volumes. The headline F1 score of 0.6758 is therefore not comparable to other published KiTS19 results without additional information. The authors should report the quantitative score after mapping predictions back to the original image format, or clearly state the magnitude of the reduction mentioned in the abstract.","section":"§2.5"},{"comment":"The volumetric validation parameters (kidney volume above 19,000 normalized units, center-of-mass between 20% and 80% depth, tumor volume above 350, tumor sphericity above 0.29, and axis-length thresholds) are described as derived from \"basic anatomical knowledge and empirical evidence.\" If these thresholds were selected using the same test split on which the final F1 is computed, the reported scores incorporate tuning-on-the-test-set bias. The manuscript should describe how the thresholds were chosen (e.g., from the validation split) and confirm that they were fixed before evaluating the test set.","section":"§2.4"}],"minor_comments":[{"comment":"The statement that each model was trained for \"ten iterations\" is ambiguous; it should be clarified as ten epochs, because deep U-net training for only ten epochs is an unusual choice that may affect reproducibility.","section":"§2.3"},{"comment":"The reported sample size of 19,479 images for the strict kidney-and-tumor model is inconsistent with the stated 4:1 ratio: combining 16,356 kidney-containing frames with one quarter of the 29,068 non-kidney frames would give approximately 23,623 images. Please reconcile the numbers or clarify the sampling scheme.","section":"§2.3"},{"comment":"The units used for volumetric and morphological thresholds (e.g., \"19,000 normalized units,\" \"350 normalized units,\" \"major axis length > 10\") are not defined. The authors should state whether these refer to voxel counts and lengths at the 256x256 resolution, and whether slice spacing is taken into account.","section":"§2.4"},{"comment":"Equations (1) and (2) contain the typo \"Flase Positive\" in the denominator; this should be corrected to \"False Positive.\"","section":"§2.5"},{"comment":"The discussion references \"Figure 5\" when describing the relationship between volume and performance, but the corresponding plots appear in Figure 4 (a-c). The cross-reference should be corrected.","section":"§4"},{"comment":"The paper should explicitly state the definition of the \"average F1 score\": from Table 1, the reported 0.6758 is the mean of the kidney F1 (0.8497) and tumor F1 (0.5019). This computation should be stated in the text.","section":"§2.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a challenge-participation report with a modest technical contribution. The central quantitative claim is not yet established because of the unspecified split granularity and the use of preprocessed ground truth in evaluation. If the split was indeed per-frame, the reported F1 should be treated as an upper bound rather than a generalizable estimate. The authors should be asked to re-analyze at the patient level and to report scores in the original image format before this can be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a straightforward KiTS19 challenge entry, not a methods breakthrough. The new bit is a logical AND ensemble of three 2D U-nets plus hand-crafted volumetric rules, and the paper is transparent about its limits. The main concern is that the reported F1 may not generalize to new patients because the data split is not specified as patient-level.\n\nWhat the paper does well: the method is described in enough detail to reproduce (code on GitHub), the preprocessing is standard but clear, and the authors honestly state that performance drops when mapping back to original format and that the result is below the clinical bar. The per-class precision/recall numbers, especially tumor precision ~0.40 versus recall ~0.86, give a useful picture of where such a pipeline fails.\n\nThe soft spots, in order of importance. First, split granularity is the load-bearing question. The dataset is 210 patients / 45,424 frames. The text says an 80:10:10 split was performed without saying whether randomization was per patient or per frame. Since training is on individual frames, a per-frame split puts slices from the same patient in both train and test. Adjacent slices are highly correlated, and tumor-positive slices form small contiguous blocks (only 5,696 frames total). Even mild patient-specific leakage can inflate kidney F1 and especially tumor F1. This is not a pedantic point; it changes the meaning of the headline number. The authors need to either confirm a per-case split or rerun the evaluation.\n\nSecond, the volumetric-validation thresholds (volume > 19,000, sphericity > 0.29, etc.) are described as empirical and derived from the same data. That is threshold tuning on the test distribution, though the effect is likely smaller than the split issue. Third, evaluation in preprocessed space is another caveat, but the authors flag it themselves, so I weight that less.\n\nBottom line: for someone building on KiTS19, this is a useful baseline with honest reporting; for a clinical claim, it is not. I would not desk-reject it, but I would send it to peer review with a requirement that the split be clarified. If the split was per-frame, the evaluation needs to be redone at patient level before the numbers can be trusted.","headline":"A clean, reproducible KiTS19 challenge write-up whose headline F1 depends on an unspecified train/test split that may leak same-patient slices into the test set.","tokens_in":5927,"tokens_out":2512,"would_cite":false,"duration_ms":27461,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A logical ensemble of three U-nets plus volumetric checks segments kidneys and tumors with average F1 0.6758.","keywords":["kidney tumor segmentation","U-net","logical ensemble","volumetric validation","KiTS19","CT imaging","fully convolutional network","medical image segmentation"],"falsifier":"Map predicted masks back through the inverse of the preprocessing pipeline and recompute precision, recall, and F1 against the original KiTS19 ground-truth segmentations; since the paper reports the score drops on restoration, the size of that drop directly tests whether the 0.6758 average is an artifact of evaluation space.","tokens_in":4983,"feed_emoji":"🩻","tokens_out":4679,"duration_ms":45989,"temperature":0.7,"pith_summary":"This paper tries to show that a modest, fully 2D segmentation pipeline can produce clinically useful kidney and tumor outlines from CT scans without any 3D network. Three binary U-net models are trained separately—two for kidneys-and-tumors with different strictness, one for tumors only—and their outputs are combined with bitwise AND operations before a volumetric validation step cleans up slices stacked in depth. On a held-out 10 percent test split of the KiTS19 data, the authors report an average F1 score of 0.6758 across kidney and tumor pixel classes. Kidneys are segmented with precision and recall above 0.8, while tumors are detected with high recall (0.86) but low precision (0.40), a trade-off the authors argue suits a screening setting where a human verifies flagged regions. The authors state the score is measured on preprocessed images and annotations, and that restoring the original image format lowers it.","feed_headline":"Three U-nets plus 3D checks segment kidneys at F1 0.68","feed_subtitle":"On KiTS19 CT scans, the ensemble finds kidneys reliably and favors over-detecting tumors for screening.","key_machinery":"The load-bearing mechanism is a logical ensemble of three 2D U-nets followed by volumetric validation. U-net, a fully convolutional encoder–decoder with skip connections, is used as the base segmenter; the ensemble combines two kidney-and-tumor models (one lax, one strict) with a tumor-only model using bitwise AND, so a pixel is labeled tumor only if all relevant models agree. Volumetric validation then stacks the 2D masks per patient and applies hand-set thresholds on volume, position, morphology, and sphericity to reject or relabel false regions, using the assumption that kidney and tumor anatomy is consistent across nearby slices.","core_discovery":"The central claim is that logical ensembling of 2D U-nets with a post hoc 3D validation step yields automatic kidney and tumor segmentations whose average F1 is 0.6758 on the KiTS19 test split. After volumetric validation, the reported per-class metrics are: background precision 0.9992/recall 0.9975/F1 0.9983; kidney precision 0.8365/recall 0.8723/F1 0.8497; tumor precision 0.3964/recall 0.8623/F1 0.5019. The ensemble works by AND-ing the outputs of a lax and a strict kidney-and-tumor model, then AND-ing with a tumor-only model, and adding the masks so tumors take value 2 and kidneys value 1. Volumetric validation then applies empirical inclusion criteria—kidney volume above 19,000 normalized units, tumor volume above 350 units, sphericity above 0.29, position limits, and multi-frame consistency—to remove or relabel implausible regions. The paper argues that over-predicting tumors at the cost of precision is acceptable for cancer screening because false negatives are the more dangerous error, but it concedes the performance is below clinical deployment level.","pith_inferences":["If the preprocessing-inverse drop is large, the volumetric thresholds may have been tuned to the evaluation space; testing on an external CT cohort would show whether the 0.6758 average holds on native scans.","The authors leave the human-review cost unmeasured; a concrete next experiment would time radiologists on flagged volumes to see whether the screening workflow actually saves effort.","The logical AND of three binary models means tumor labels require agreement across models; a single multi-class network could produce a different precision–recall trade-off and might be compared directly on the same test split."],"forward_implications":["Volumetric validation raises average segmentation performance compared with simply stacking 2D predictions, despite the tumor class losing some recall.","Kidney regions are located with precision and recall above 0.8, suggesting the pipeline is reliable at the organ level even where tumor boundaries are uncertain.","The high-recall, low-precision tumor profile is appropriate for a screening pipeline in which a radiologist reviews flagged candidate regions rather than scanning the full volume.","Further gains would have to come from better artifact removal, more training data, or a multi-class network, since the paper finds no clear volume–performance relationship to target."],"supporting_citations":[{"why":"Supplies the KiTS19 dataset of 210 patients with CT scans and ground-truth kidney/tumor labels used for training, validation, and testing.","marker":"[1]"},{"why":"Defines the U-net architecture on which the three segmentation models are based.","marker":"[2]"},{"why":"Provides the U-net implementation code the authors adapted for the three binary models.","marker":"[3]"}],"fun_headline_variants":["Three U-nets AND-ed and volume-checked score F1 0.68 on CTs","Logical ensemble of U-nets with 3D validation hits F1 0.68","AND-ing U-nets, then 3D checks: kidney/tumor F1 0.68 on KiTS19","Tumor over-prediction preferred: U-net ensemble F1 0.68","Ensemble of U-nets with volumetric validation: F1 0.68"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline F1 is computed on preprocessed images and ground-truth labels transformed by the same pipeline; the paper itself says restoring the original format lowers the score, so the claim assumes the transformed evaluation reflects true performance on native CT scans.","fun_headline_variants_meta":{"raw":{"variants":["Three U-nets AND-ed and volume-checked score F1 0.68 on CTs","Logical ensemble of U-nets with 3D validation hits F1 0.68","AND-ing U-nets, then 3D checks: kidney/tumor F1 0.68 on KiTS19","Tumor over-prediction preferred: U-net ensemble F1 0.68","Ensemble of U-nets with volumetric validation: F1 0.68"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":2157,"prompt_tokens":1018,"completion_tokens":1139,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1017}},"tokens_in":634,"tokens_out":1139,"duration_ms":10092,"temperature":1.0,"reasoning_tokens":1017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:38:54.317038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Map predicted masks back through the inverse of the preprocessing pipeline and recompute precision, recall, and F1 against the original KiTS19 ground-truth segmentations; since the paper reports the score drops on restoration, the size of that drop directly tests whether the 0.6758 average is an artifact of evaluation space.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the KiTS19 dataset of 210 patients with CT scans and ground-truth kidney/tumor labels used for training, validation, and testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the U-net implementation code the authors adapted for the three binary models."}],"review_version":1}