{"id":"3c29dc2d-9dbe-468d-bb92-f9383bee11a5","arxiv_id":"2411.17213","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A modified nnU-Net ResEnc L with larger patches, deeper topology, no left/right mirroring, and class-wise postprocessing took first place in the ToothFairy2 CBCT segmentation challenge.","lead":"This paper describes a tuned nnU-Net configuration that won the ToothFairy2 challenge for dental CBCT segmentation, reaching a mean Dice of 0.9253 on the hidden test set. The win came from engineering choices such as larger patches, a deeper network, disabling left/right mirroring, and postprocessing that removes small false positives.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Postprocessing cutoffs appear overfit: test HD95 (18.47) is ~3x the F-case CV value (6.07), so the PP component's generalization is unverified and may not have helped on the test set.","rationale":"Good faith reading: the paper is an honest challenge report from the nnU-Net authors. The official leaderboard provides strong external evidence for the first-place claim, and the source code is released. The main stress point is not the leaderboard position but the scientific interpretation of the pipeline's components. The weakest assumption—that F-case-derived PP cutoffs transfer to the hidden test set—is identified by the authors themselves and matches the reader's assessment. I make it concrete with a numeric inconsistency: F-case PP HD95 is 6.069, test HD95 is 18.472, a ~3x gap despite the claimed FOV match and only a small Dice drop. This suggests the PP stage may be overfit to training F-cases and could have hurt test HD95. The missing no-PP test metrics prevent deciding between domain shift and PP failure. The proposed nested-CV check would estimate cutoff optimism without needing the hidden test set. Since the first place is externally verified, this concern does not overturn the result; it strengthens the need for the conditional verdict already issued.","tokens_in":4977,"tokens_out":7031,"duration_ms":68481,"concrete_test":"Perform a nested five-fold analysis on the 63 F-cases from the training set: for each fold, optimize the class-wise postprocessing cutoffs on the remaining four folds and evaluate both the PP and no-PP HD95/Dice on the held-out fold. If the held-out PP HD95 exceeds the no-PP HD95, or if the mean held-out PP HD95 is much larger than the within-fold-optimized 6.069 reported in Table 1, the cutoff transferability assumption fails and the reported test HD95 is explained by PP overfitting. If held-out PP HD95 is consistently below no-PP, the postprocessing generalizes and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (first place, mean rank 4.6) is externally supported, but the method's postprocessing component carries a hidden load. Section 2.2 says class-wise removal cutoffs were optimized on five-fold CV predictions from F-cases only, and Section 3.1 admits these postprocessed metrics 'may be optimistic' and that cutoff generalizability could not be verified. That caveat is not merely formal. For the final configuration, Table 1 reports F-case PP HD95=6.069 and no-PP HD95=10.140; Table 2 reports test-set HD95=18.472 even though test cases are claimed to share the F-case field of view. Dice drops only from 0.9344 to 0.9253, so the 3x HD95 gap points to the PP cutoffs discarding small true structures on the test distribution, inflating HD95. Since no test-set no-PP numbers are reported, we cannot tell whether postprocessing helped, hurt, or was neutral on the hidden test set. With a mean-rank margin of only 0.2 over second place, this unresolved transfer question is load-bearing for the claim that the described pipeline, including PP, is a robust SOTA baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the authors' entry to the ToothFairy2 CBCT segmentation challenge. The method is based on the nnU-Net ResEnc L configuration and introduces a larger patch size (160x320x320), a deeper network (7 resolution stages), disabled left/right mirroring augmentation, longer training (1500 epochs), and class-wise postprocessing cutoffs that discard small predicted regions. On the challenge test set (n=50), the method achieves a mean Dice of 0.9253 and HD95 of 18.472, with a mean rank of 4.6, which placed first ahead of the runner-up at 4.8. The paper also reports five-fold cross-validation ablations on the training set and qualitative examples, and states that source code is publicly available.","tokens_in":5196,"tokens_out":2827,"duration_ms":27394,"significance":"If the reported test-set result is reliable, the contribution is a strong, reproducible baseline for multi-structure dental CBCT segmentation and a useful reference for applying nnU-Net to high-resolution 3D medical images with many classes. The paper is transparent about its design choices and failures, provides public code, and its central claim (first place in a challenge) is externally anchored by the official leaderboard. However, the paper's internal quantitative evidence is weakened by the circularity of the postprocessed cross-validation metrics and by the lack of test-set results without postprocessing, which leaves the contribution of the postprocessing component to the final ranking unverified.","major_comments":[{"comment":"The cross-validation Dice and HD95 values for the final configuration are computed on the same fold predictions that were used to optimize the class-wise postprocessing cutoffs (Section 2.2). The authors acknowledge this in the text, but Table 1 is nevertheless used to support the main ablations: for example, the improvement of the final model over the no-mirror/1500-epoch model is reported as HD95 6.069 vs 6.262 for F-cases, a margin smaller than the likely optimism in the postprocessed numbers. An unbiased estimate, obtained by optimizing cutoffs on a separate split or via nested cross-validation, is needed to support the claimed ordering of configurations.","section":"Section 3.1, Table 1"},{"comment":"The test-set HD95 is 18.472, approximately three times the F-case cross-validated HD95 of 6.069, while Dice drops only modestly from 0.9344 to 0.9253. The paper attributes this drop to domain shift, but no test-set metrics without postprocessing are reported. Since the postprocessing cutoffs were optimized on F-case training predictions, the reader cannot determine whether postprocessing helped, hurt, or was neutral on the hidden test distribution. The authors note that cutoff generalizability could not be verified, but this unresolved question is load-bearing for the claim that the described pipeline, including the postprocessing component, is a robust state-of-the-art baseline. Reporting test Dice and HD95 with and without postprocessing would resolve this.","section":"Section 3.4, Table 2"},{"comment":"The first-place claim rests on a mean rank of 4.6 versus 4.8 for the runner-up. No confidence intervals, standard deviations, or significance tests are reported for the mean rank or for the underlying 84 per-class/metric ranks. Given the test set size (n=50) and the narrow margin, the ranking difference may be within noise. The paper should at least report the distribution of ranks or otherwise quantify the uncertainty of the mean rank.","section":"Section 3.4"}],"minor_comments":[{"comment":"The sentence \"The cutoffs for discarding predictions are optimized on the training set predictions obtained via a five-fold cross-validation\" is clear, but the subsequent sentence about choosing the smaller cutoff between Dice and HD95 is ambiguous: specify whether the cutoff is chosen per class independently for each metric and then the minimum is taken, as stated, or if some joint optimization is performed.","section":"Section 2.2"},{"comment":"The table formatting for the row with pretraining is corrupted in the manuscript (\"pretr0.9119 9.2580.9320 5.875\"), which makes the reported values hard to read; the table should be typeset with clear column separation.","section":"Section 3.1, Table 1"},{"comment":"The paper states that two models are trained from scratch on all available training data and ensembled, but it does not report whether the final test predictions come from the ensemble or a single model; this should be stated explicitly, as the cross-validation results are for single models.","section":"Section 3.3"},{"comment":"The claim that test cases have the same field of view as F cases is referenced to the challenge FAQ, but the exact number of F and P cases in the training set is not given; stating this split would help the reader interpret the F-case-focused optimization.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a challenge-report style manuscript by the nnU-Net developers. The central test-set result is externally validated by the leaderboard, but the internal ablation evidence for the postprocessing component is circular and the test-set transfer of the cutoffs is unverified. The recommended revisions are feasible within the manuscript's scope and do not undermine the challenge result, but they are necessary for the paper to support the robustness claims it makes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a clean, honest challenge report from a group that knows nnU-Net better than anyone, and the first-place claim is externally backed by the ToothFairy2 leaderboard. The genuinely new bits are small but real: disabling left/right mirroring helps for CBCT of the jaw, a 7-stage residual encoder with larger patch helps, and a class-wise postprocessing cutoff scheme tuned on cross-validation. The paper is refreshingly candid about what did not work (pretraining, full mirroring off, tooth instance segmentation) and about the optimistic nature of its postprocessed CV numbers.\n\nWhere it gets soft: the postprocessing cutoffs are fitted to the same CV predictions they are then evaluated on, and the test-set HD95 is about 3x the F-case CV value (18.47 vs 6.07). The authors admit they could not verify cutoff generalizability. Because the margin over second place is 0.2 mean rank (4.6 vs 4.8), we don't know whether the postprocessing helped, hurt, or was neutral on the hidden test set—no test-set no-PP numbers are reported. That makes the 'described pipeline as robust SOTA baseline' claim heavier than the evidence supports. The stress-test note is on point. The other soft spot is statistical: a single test set, no confidence intervals, and ranking margins that are probably within noise across reruns.\n\nBut the central result—first place—is solid as a benchmark outcome. The code is public, the cross-validation is standard 5-fold, and the external leaderboard supports the rank. This is not a methods paper proposing a new algorithm; it's a reproducible configuration report. As such, it deserves to be in the record.\n\nWho is this for: people working on dental CBCT segmentation, and anyone using nnU-Net in structured challenges who wants to know which knobs matter. It's a useful baseline paper, not a breakthrough.\n\nRecommendation: send it to peer review. A serious referee should ask for (a) confidence intervals or repeated runs, (b) test-set results with and without postprocessing, or at least an honest statement that the PP transfer is unverified, and (c) a small ablation isolating the PP contribution. None of that kills the paper; it makes the claims match the evidence.","headline":"Honest, externally backed first-place challenge report; the postprocessing cutoffs are the load-bearing weakness and need to be handled head-on.","tokens_in":5733,"tokens_out":2491,"would_cite":true,"duration_ms":21446,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that a scaled nnU-Net configuration achieves state-of-the-art dental CBCT segmentation, winning the ToothFairy2 challenge with a mean Dice of 0.9253 and mean rank 4.6.","keywords":["CBCT segmentation","nnU-Net","ToothFairy2 Challenge","dental imaging","deep learning","data augmentation","postprocessing","FDI tooth labeling"],"falsifier":"Run the same final pipeline with and without the postprocessing cutoffs on a held-out set of F-case CBCT scans that was never used to choose the cutoffs. If the postprocessed Dice and HD95 do not improve over the raw predictions—or if applying the cutoffs to a set with slightly different field of view or contrast reduces HD95—the reported gains would be overfitting to the training distribution rather than a generalizable improvement.","tokens_in":4787,"feed_emoji":"🦷","tokens_out":6862,"duration_ms":56674,"temperature":0.7,"pith_summary":"This paper reports a scaled configuration of the nnU-Net segmentation framework that wins the ToothFairy2 dental CBCT challenge. The authors enlarge the training patch to 160x320x320 voxels, deepen the network to seven resolution stages, disable left/right mirroring during training, extend training to 1500 epochs, and apply per-class postprocessing cutoffs that remove small predicted regions. On the hidden test set the method achieves a mean Dice coefficient (voxel overlap) of 0.9253 and HD95 (a 95th-percentile boundary-distance error) of 18.472, with a mean rank of 4.6 and first place among challenge teams. The paper also reports which modifications contribute most: turning off left/right mirroring gives the largest single gain, while pretraining and a separate tooth-instance segmentation stage do not beat the final pipeline once postprocessing is applied.","feed_headline":"Scaled nnU-Net wins ToothFairy2 at Dice 0.9253","feed_subtitle":"Disabling left-right mirroring and tuning per-class cutoffs lands first place in dental CBCT segmentation.","key_machinery":"The central mechanism is the nnU-Net ResEnc L framework—a residual-encoder U-Net with self-configuring patch size, batch size, and topology—modified in four ways: a patch of 160x320x320 voxels so the network sees enough spatial context to assign FDI tooth labels; seven resolution stages (input stride 32x64x64 in the bottleneck) instead of six; mirroring augmentation disabled only on the left/right axis because dental CBCT offers few cues for left/right disambiguation; and 1500 training epochs instead of 1000. A second load-bearing component is the postprocessing scheme: class-wise cutoffs that discard small predicted regions by turning them into background, tuned on five-fold cross-validation predictions of F cases and applied separately to Dice and HD95. The argument is that these modifications address the specific failure modes of dental CBCT—crown artifacts, fragmented teeth, and FDI misclassification—rather than requiring a new network family.","core_discovery":"On its own terms the contribution is a recipe, not a new architecture: the authors show that the existing nnU-Net ResEnc L model, adjusted to the geometry of dental CBCT data, wins the ToothFairy2 segmentation challenge. The central results are the measured gains from each modification—left/right mirroring off raises Dice on F-cases from 0.7445 to 0.9083 before postprocessing; adding the larger patch, longer training, and the extra resolution stage pushes postprocessed Dice to 0.9344 on F-cases, and 0.9271 on all cases in cross-validation; the final ensemble of two from-scratch models scores 0.9253 Dice and 18.472 HD95 on the test set with mean rank 4.6. The postprocessing step exploits the challenge evaluation by replacing small, likely false-positive predictions with background, with per-class cutoffs optimized separately for Dice and HD95 on F-case predictions from the five-fold cross-validation.","pith_inferences":["If the cutoff transfer holds beyond this challenge, the same recipe of per-class, per-metric cutoff optimization could be applied to any segmentation benchmark whose evaluation rewards true negatives, turning postprocessing into a tunable component rather than a fixed step.","The mirroring ablation suggests that other deployments on lateralized structures should consider disabling augmentation along only the left/right axis; testing this on independent dental CBCT datasets would show whether the result is specific to ToothFairy2.","The authors' explanation that reduced-field-of-view P cases interfere with tooth-instance segmentations could be tested by running that two-stage pipeline on F-only cases, where all teeth lie fully inside the image.","A nested validation, with cutoffs tuned on one fold and evaluated on another, would quantify how much of the reported cross-validation gain is optimism from tuning and evaluating on the same cases."],"forward_implications":["The winning ToothFairy2 configuration is a ready-made baseline for future dental CBCT segmentation tasks involving 42 jaw, nerve, sinus, implant, and tooth classes.","Disabling only the left/right mirroring axis is a simple change with a large measured benefit for lateralized anatomies whose image statistics carry few left/right cues.","Metric-aware postprocessing that converts small predictions to background can improve Dice and HD95 substantially under a scoring scheme that rewards true negatives, so challenge rankings and absolute numbers should be interpreted with that scoring in mind.","Training the same model twice from scratch without fixing seeds and averaging the two outputs is a straightforward way to close the gap between single-model development scores and final test-set performance.","Because pretraining and two-stage instance segmentation did not beat the streamlined semantic pipeline after postprocessing, the main leverage for this dataset sits in configuration and postprocessing rather than in representation learning."],"supporting_citations":[{"why":"Defines the base nnU-Net self-configuring segmentation method that the paper scales.","marker":"[3]"},{"why":"Introduces the ResEnc L configuration with residual encoder and increased VRAM budget that serves as the starting point.","marker":"[4]"},{"why":"Supplies the canonical U-Net architecture that the residual encoder replaces.","marker":"[6]"},{"why":"Provide the ToothFairy2 challenge and related dental CBCT datasets and annotations used for training and evaluation.","marker":"[5,2,1]"},{"why":"The MultiTalent pretraining strategy tested in ablations, found equivalent to training from scratch after postprocessing.","marker":"[7]"}],"fun_headline_variants":["nnU-Net scaling wins ToothFairy2 with 0.9253 Dice","ToothFairy2 first: nnU-Net tweaks hit 0.9253 Dice","CBCT champion: scaled nnU-Net scores 0.9253 Dice","Dental CBCT: nnU-Net recipe wins at Dice 0.9253","Tweaked nnU-Net tops ToothFairy2, 0.9253 Dice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-class postprocessing cutoffs, optimized on F-case predictions from the training set's five-fold cross-validation, transfer to the hidden test set, whose field of view is assumed to match the F cases; the authors note they could not independently verify this.","fun_headline_variants_meta":{"raw":{"variants":["nnU-Net scaling wins ToothFairy2 with 0.9253 Dice","ToothFairy2 first: nnU-Net tweaks hit 0.9253 Dice","CBCT champion: scaled nnU-Net scores 0.9253 Dice","Dental CBCT: nnU-Net recipe wins at Dice 0.9253","Tweaked nnU-Net tops ToothFairy2, 0.9253 Dice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2972,"prompt_tokens":865,"completion_tokens":2107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1991}},"tokens_in":481,"tokens_out":2107,"duration_ms":14778,"temperature":1.0,"reasoning_tokens":1991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:21:50.330343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same final pipeline with and without the postprocessing cutoffs on a held-out set of F-case CBCT scans that was never used to choose the cutoffs. If the postprocessed Dice and HD95 do not improve over the raw predictions—or if applying the cutoffs to a set with slightly different field of view or contrast reduces HD95—the reported gains would be overfitting to the training distribution rather than a generalizable improvement.","supporting_citations":[{"cited_title":"Ronneberger, P","cited_arxiv_id":null,"evidence_quote":"Supplies the canonical U-Net architecture that the residual encoder replaces."},{"cited_title":"Ulrich, F","cited_arxiv_id":null,"evidence_quote":"The MultiTalent pretraining strategy tested in ablations, found equivalent to training from scratch after postprocessing."},{"cited_title":"Isensee, P","cited_arxiv_id":null,"evidence_quote":"Defines the base nnU-Net self-configuring segmentation method that the paper scales."}],"review_version":1}