{"id":"7af8d85b-0a6c-45ae-ad0e-c0d587653cb6","arxiv_id":"1908.02182","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A plain 3D U-Net and its residual variants performed almost identically on KiTS2019, and the residual version won the challenge by a small margin.","lead":"The authors entered the 2019 Kidney and Kidney Tumor Segmentation Challenge with a plain 3D U-Net and two residual variants, and won with a Composite Dice score of 91.23. The paper's main finding is that architectural upgrades over a basic U-Net produced barely measurable gains, casting doubt on the value of complex architecture engineering in medical segmentation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The challenge win is externally validated, but the paper's central scientific conclusion—that residual modifications yield only minor improvements—rests on a confounded, significance-free comparison of architectures that differ in nonlinearity, width, and block structure.","rationale":"The reader's weakest_assumption correctly identifies the confounded cross-validation comparison as the fragile premise, and the authors' own admission of the LReLU/ReLU implementation error supports this. I partially agree because the self-generated labels for cases 15 and 37 introduce an additional circularity in the tumor Dice comparison that the reader mentions only in passing. The challenge-win claim itself is independently verified by the public KiTS2019 leaderboard, so no substantive objection applies to that central claim; the concern is confined to the secondary interpretive claim about architecture. The paper is unusually honest about its limitations, including the absence of statistical significance testing and the possibility of implementation error, which makes a conditional verdict appropriate but does not require changing the reader's conclusion. The concrete test above would settle whether the confound actually matters by isolating the residual connection from the other simultaneous changes.","tokens_in":5645,"tokens_out":2920,"duration_ms":37680,"concrete_test":"Run a controlled five-fold cross-validation on the original (unmodified) KiTS training labels with two additional architectures: (A) the plain U-Net re-implemented with the residual network's exact settings (ReLU, 24 initial feature maps, same per-level block counts), and (B) the residual network re-implemented with the plain U-Net's settings (leaky ReLU, 30 initial feature maps, original block pattern). Report per-case paired Composite Dice with bootstrap confidence intervals. If the 0.35-point Residual-minus-Plain gap in Table 1 does not reproduce in both controlled comparisons, the claimed negligible improvement cannot be attributed to residual connections.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main interpretive claim is that architectural modifications to the 3D U-Net produce only 'very minor, barely measurable improvements' (Abstract; Discussion). This claim rests entirely on the five-fold cross-validation comparison in Table 1: Plain 3D U-Net Composite Dice 91.19, Residual 91.54, Pre-activation Residual 91.25, ensemble 91.50. But the three architectures differ simultaneously in at least four uncontrolled ways: the plain U-Net uses leaky ReLU while the residual models use ReLU (an admitted implementation error, Discussion); the initial feature map count is 30 for the plain model and 24 for the residual models; the block repetition pattern differs (two conv-instnorm-LReLU blocks per resolution vs. increasing numbers of residual blocks and a single decoder convolution); and the loss surface is trained with different nonlinearities throughout. The observed Composite Dice differences are within the range one would expect from random fold variation, and the authors explicitly state their conclusions 'were not tested for statistical significance.' Furthermore, the cross-validation scores were computed on a modified dataset: four cases were excluded and two cases had their reference annotations replaced by the authors' own network predictions (Section 2.4). This creates a circularity for the tumor Dice component of the comparison: the residual network's higher tumor Dice in Table 1 could partly reflect that the labels it helped generate are easier for it to predict, not a genuine architectural advantage. None of this undermines the externally validated challenge win—the test set result is independent and the leaderboard is public—but it does undermine the paper's central negative result about architecture. The Discussion's assertion that the LReLU/ReLU discrepancy is not expected to impact results is an unsupported claim, not a control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports a 3D U-Net baseline and two residual variants (residual and pre-activation residual) for kidney and kidney tumor segmentation in the KiTS2019 challenge. On a five-fold cross-validation of a slightly modified training set, the three architectures achieve nearly identical composite Dice scores (91.19, 91.54, and 91.25), which the authors interpret as evidence that residual modifications yield only very minor, barely measurable improvements. The authors submit the residual U-Net for test-set prediction and report a composite Dice of 91.23, placing first on the official leaderboard among 106 teams. The paper's central interpretive claim is that architectural modifications to the 3D U-Net do not substantially improve segmentation accuracy, and it explicitly acknowledges that the comparison was not tested for statistical significance.","tokens_in":6151,"tokens_out":4359,"duration_ms":48521,"significance":"If the central negative result were well supported, the paper would provide a useful, externally validated data point against overvaluing residual modifications in medical segmentation, especially because the test-set win is confirmed by the official challenge leaderboard. The authors are transparent about several limitations, including the absence of significance testing and the leaky ReLU implementation error. However, the cross-validation comparison that motivates the main claim does not isolate the effect of residual connections, so the negative result is not yet established. The lasting contribution is the strong and independently verified test-set performance, but the scientific conclusion about residual blocks needs substantially stronger support.","major_comments":[{"comment":"The comparison that supports the paper's main claim is confounded: the plain U-Net uses leaky ReLU, 30 initial feature maps, and two conv-instnorm-LReLU blocks per resolution, while both residual models use ReLU, 24 initial feature maps, and different block and decoder structures. Because the plain and residual models differ simultaneously in nonlinearity, width, block repetition, and decoder depth, the observed differences in Table 1 (Composite Dice 91.19, 91.54, 91.25) cannot be attributed to the residual connections. The leaky ReLU discrepancy is admitted in the Discussion, and the feature-map reduction is stated in Section 2.2. At minimum, the conclusion that architectural modifications yield 'very minor, barely measurable improvements' should be reworded to describe differences between the specific implemented configurations, or the authors should report a controlled comparison with matched nonlinearity and initial feature-map count.","section":"Section 2.2, Table 1"},{"comment":"The cross-validation scores are computed on a modified training set in which four cases were excluded and the reference labels of cases 15 and 37 were replaced by segmentations generated by the authors' own networks. Evaluating models against labels that were themselves produced by the same family of models introduces a form of circularity, and the effect may differ across the architectures being compared, potentially biasing the tumor Dice component of the ranking in Table 1. The authors should report the per-case contribution of cases 15 and 37 to the mean tumor Dice, repeat the comparison with these cases excluded, or otherwise demonstrate that the replaced labels do not change the conclusions.","section":"Section 2.4, Table 1"},{"comment":"No statistical significance testing is reported, and the composite Dice differences between architectures (0.06 to 0.35 points) are of a magnitude that could easily arise from fold-to-fold variation. The authors' statement that they are not comfortable declaring a winner is appropriately cautious, but the stronger accompanying claims that the improvements are 'very minor' and 'barely measurable' require support from per-fold paired results, confidence intervals, or a formal significance test. Without such analysis, the central negative claim is not verifiable from Table 1.","section":"Section 3, Table 1, Discussion"},{"comment":"The statement that 'even ensembling ... did not yield an improvement' is not supported by the data as presented. Table 1 reports a single cross-validation mean for the ensemble (Composite Dice 91.50) that is numerically lower than the residual U-Net single-model score (91.54), but no measure of variability or per-fold paired comparison is given. With only one ensemble configuration and no error bars, the claim that ensembling did not help cannot be evaluated; the manuscript should either provide per-fold ensemble and single-model scores with paired comparisons or soften the claim to a statement about the specific mean values observed.","section":"Section 3, ensemble result"}],"minor_comments":[{"comment":"In the paragraph describing the plain 3D U-Net, the sentence 'This number is is doubled' contains a duplicated word and should be corrected.","section":"Section 2.2"},{"comment":"The voxel spacing is printed as '3 .22 × 1.62 × 1.62 mm' with a space after the decimal point; this should be '3.22 × 1.62 × 1.62 mm'.","section":"Section 2.1"},{"comment":"The table label appears as 'T able 1' with a stray space; the formatting should be corrected.","section":"Table 1"},{"comment":"It would be helpful to state explicitly whether the 'previous iterations of our networks' used to generate replacement labels for cases 15 and 37 were of the same architectures (plain, residual, or pre-activation residual) that are compared in Table 1, because this affects how much circularity is introduced for those cases.","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The test-set win is real and externally validated, but the paper's scientific value depends on the negative result about residual blocks, which is currently not adequately supported due to the confounded architecture comparison and the absence of significance testing. I would encourage the authors to either add controlled experiments with matched nonlinearity and width, or explicitly downgrade the conclusion to an observation about the specific implemented configurations. The paper fits a challenge-report format, but for a regular journal the analysis needs tightening and the claims need to be scaled to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the challenge win is real: 91.23 composite Dice, 0.6 ahead of the second place team, leaderboard is public. Second, the paper's main scientific message—residual blocks do not beat a plain 3D U-Net—is much weaker than it looks, and the authors know it.\n\nWhat is actually new is the empirical comparison on KiTS2019: three U-Net variants trained with the same pipeline, cross-validated, and a test-set outcome. The architectures are standard, but the comparison is a useful datapoint for medical segmentation. The paper is refreshingly honest. It explicitly says the results are inconclusive, admits the leaky ReLU/ReLU discrepancy is an implementation error, states that no significance testing was done, and lists possible causes for the null result—bad implementation, poor hyperparameter choices, and so on. That is more candor than most papers of this type.\n\nThe soft spots are real but mostly acknowledged. The cross-validation table compares three architectures that differ in at least four ways at once: nonlinearity, initial feature map count, block structure, and decoder depth. The authors choose “what fits in 12 GB GPU memory” as the constraint, which is a reasonable challenge constraint, but it means you cannot attribute the observed Dice differences to residual connections per se. The differences (91.19 vs 91.54 vs 91.25) are within fold noise. The dataset modifications are the only part that bothers me more than the authors let on. Excluding four cases is fine, but replacing the reference annotations of cases 15 and 37 with the authors' own prior network predictions is a mild circular step. It is explicitly permitted and does not affect the test set, but it does contaminate the tumor-dice column of the cross-validation, which is the column that tips the model choice toward the residual network. The paper notes the dataset was modified, but it does not discuss the circularity for those two cases.\n\nThe framing is careful—the paper calls the comparison inconclusive—so this is not a load-bearing flaw in the challenge result. It is a flaw in the secondary claim, and the paper itself hedges that claim.\n\nWho should read it: medical-imaging researchers working on segmentation, especially challenge-oriented groups. The lesson is not \"residuals don't matter\" but \"a strong baseline built from nnU-Net will take you far in a typical CT segmentation challenge.\" It deserves peer review; I would send it out. The authors' own last paragraph points toward the more rigorous follow-up that would settle the question.","headline":"A real KiTS19 win, and an honest but confounded architecture comparison—treat the null result as evidence the baseline is strong, not that residual blocks are useless.","tokens_in":6529,"tokens_out":3204,"would_cite":true,"duration_ms":31274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding residual blocks to a 3D U-Net barely improved cross-validation kidney-tumor segmentation, yet the residual variant won the KiTS 2019 challenge with a composite dice of 91.23.","keywords":["3D U-Net","residual blocks","pre-activation residual blocks","kidney tumor segmentation","CT imaging","KiTS 2019","cross-validation","composite dice score"],"falsifier":"Run the same five-fold cross-validation on the same data with the activation functions, starting channel counts, and block schedules made identical across all three architectures; if the plain U-Net then reaches or surpasses the residual U-Net's composite dice, the claimed marginal benefit of residual blocks is not supported.","tokens_in":5489,"feed_emoji":"🏆","tokens_out":8360,"duration_ms":79497,"temperature":0.7,"pith_summary":"The paper asks whether augmenting the standard 3D U-Net with residual or pre-activation residual blocks improves kidney and kidney-tumor segmentation in CT scans, and it answers in the negative for practical purposes. Under a fixed 12 GB GPU budget, three U-Net variants trained on the KiTS 2019 challenge data produced nearly identical five-fold cross-validation composite dice scores: 91.19 for the plain U-Net, 91.54 for the residual U-Net, and 91.25 for the pre-activation residual U-Net, with an ensemble at 91.50. The paper reports that the differences are so small that the authors do not feel comfortable declaring a winner, and the residual U-Net is submitted only because the challenge forces a choice. It wins the challenge with a test composite dice of 91.23, outperforming 105 competing teams by a margin of 0.6 over second place. The broader point is that the segmentation problem appears well-behaved and a well-tuned plain U-Net is a hard baseline to beat.","feed_headline":"Residual U-Net wins kidney challenge; plain U-Net nearly matches","feed_subtitle":"Cross-validation showed residual blocks added only 0.35 dice points; the winning margin was 0.6.","key_machinery":"The object that carries the argument is the U-Net itself—an encoder–decoder convolutional network with skip connections that is the standard baseline for medical image segmentation—instantiated in three variants: a plain 3D U-Net with two convolution-instance-normalization-leaky-ReLU blocks per resolution and 30 initial feature maps; a residual 3D U-Net with residual blocks in the encoder (24 initial feature maps); and a pre-activation residual 3D U-Net in which normalization and activation precede the convolution. The comparison is run under a fixed hardware constraint of a 12 GB GPU, with identical preprocessing (resampling to $3.22 \\times 1.62 \\times 1.62$ mm, intensity clipping to $[-79, 304]$), a fixed patch size of $80 \\times 160 \\times 160$, and five-fold cross-validation on a slightly modified training set. The metric is composite dice, the geometric mean of kidney and tumor dice, which is also the challenge's ranking criterion.","core_discovery":"The central claim is that architectural sophistication—swapping ordinary convolutional blocks for residual or pre-activation residual blocks—yields only marginal, barely measurable gains on the KiTS 2019 kidney tumor segmentation task, so the winning result is best read as evidence for the strength of the U-Net baseline and the robustness of the problem rather than for any particular modification. The residual 3D U-Net, chosen because its cross-validation composite dice (91.54) and tumor dice (85.73) were the highest, scored 91.23 composite dice on the held-out test set and ranked first among 106 teams. The same table that motivates the submission shows the plain U-Net at 91.19 composite dice and the pre-activation variant at 91.25, with an ensemble at 91.50, and the authors state they cannot declare one model better than the others. They also note the comparison was not tested for statistical significance and that a decisive test would require per-architecture hyperparameter optimization.","pith_inferences":["A decisive test of the paper's interpretation would hold all settings fixed except for the residual connection itself; the admitted activation-function mismatch means the current comparison cannot isolate the architectural effect.","If the plain U-Net is this strong, challenge organizers and practitioners may gain more from better training-data curation, such as correcting mislabeled cases, than from novel block designs.","The hardware-constrained comparison suggests a norm shift: architecture improvements should be evaluated under a fixed memory and time budget, and under that norm many published gains from adding residual or dense connections may shrink.","The 0.6-point winning margin is comparable to the cross-validation spread among the paper's own models, so a challenge win at this level should be interpreted as evidence of a strong baseline, not as proof that the submitted architecture is objectively best."],"forward_implications":["A plain 3D U-Net without residual connections is within 0.35 composite dice points of the winning variant on this task, so future U-Net-based submissions should treat it as a serious baseline rather than a starting point that needs architectural upgrades.","Ensembling the three architectures did not beat the best single model in cross-validation (91.50 versus 91.54), which cautions against assuming ensembles always help in segmentation challenges.","Because varying the resampling spacing and patch size within reasonable ranges gave basically identical cross-validation results, the KiTS 2019 problem is likely well-behaved and insensitive to typical hyperparameter choices.","The top of the KiTS 2019 leaderboard is very tight: the winning margin over second place was 0.6 composite dice, meaning small differences in preprocessing, training-data curation, or random seeds can determine challenge rank."],"supporting_citations":[{"why":"Supplies the base implementation pipeline that all three architectures start from.","marker":"[7]"},{"why":"Defines the U-Net encoder-decoder architecture being modified.","marker":"[13]"},{"why":"Extends the U-Net to 3D volumes, grounding the volumetric approach.","marker":"[2]"},{"why":"Provides the residual block construction used in the residual 3D U-Net.","marker":"[4]"},{"why":"Provides the pre-activation residual block construction used in the third variant.","marker":"[5]"},{"why":"Provides the KiTS 2019 dataset, annotations, and challenge evaluation used for the comparison.","marker":"[6]"}],"fun_headline_variants":["Residual blocks add almost nothing, U-Net still wins","Kidney challenge: residual U-Net wins by a hair","U-Net tweaks barely matter; residual version wins","0.04 dice gain: residual U-Net tops plain in KiTS","Residual U-Net wins, but plain U-Net nearly ties"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the tiny score differences reflect the residual connections themselves rather than accidental differences among the models, which included a different activation function, a different number of starting channels, and a different block schedule, and the paper admits that the activation difference was an implementation error.","fun_headline_variants_meta":{"raw":{"variants":["Residual blocks add almost nothing, U-Net still wins","Kidney challenge: residual U-Net wins by a hair","U-Net tweaks barely matter; residual version wins","0.04 dice gain: residual U-Net tops plain in KiTS","Residual U-Net wins, but plain U-Net nearly ties"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1106,"prompt_tokens":851,"completion_tokens":255,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":165}},"tokens_in":467,"tokens_out":255,"duration_ms":3401,"temperature":1.0,"reasoning_tokens":165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:50:44.861126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five-fold cross-validation on the same data with the activation functions, starting channel counts, and block schedules made identical across all three architectures; if the plain U-Net then reaches or surpasses the residual U-Net's composite dice, the claimed marginal benefit of residual blocks is not supported.","supporting_citations":[],"review_version":1}