{"id":"82a8d025-8e6b-4bfe-9648-4683f416a444","arxiv_id":"2510.11047","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"A curated 1,029-scan laryngeal CT benchmark with standardized 1-mm crops, six benchmarked 3D deep-learning classifiers, and open preprocessing code and metadata.","lead":"Researchers assembled 1,029 laryngeal CT scans from six public archives into a standardized benchmark, LaryngealCT, and tested six 3D deep-learning models on early-versus-advanced and T4-versus-non-T4 staging. The benchmark gives AI developers a reproducible, anatomy-focused testbed for laryngeal cancer staging, though T4 detection remains unreliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth T-stage labels from TCIA are unverified; label noise could invalidate the benchmark's core value.","rationale":"The reader's weakest assumption (unverified ground-truth T-stage labels) is indeed the most load-bearing concern. Every downstream result—model ranking, reported AUC, the benchmark's stated utility—depends on the labels being correct. The circular cropping validation is a real methodological flaw, but it is mitigated by independent expert review of 100 cases; the best-fold selection affects reported model performance but not the dataset's foundational validity. Label noise, however, directly undermines the benchmark itself. The proposed test is a concrete, feasible check: re-stage a sample and measure agreement. Since the reader already identified this concern and issued CONDITIONAL, my read does not change the verdict; it reinforces it. The paper must either provide label validation or clearly caveat all results as conditional on unverified metadata.","tokens_in":21641,"tokens_out":4123,"duration_ms":39836,"concrete_test":"Select a stratified random sample of ~150 cases (oversampling T4) from LaryngealCT. Have two blinded head-and-neck radiologists independently re-stage each case from the CT images using a standardized system (e.g., AJCC 8th), and, where available, compare with pathology/clinical T-stage in the original TCIA metadata. Compute Cohen's kappa between original LaryngealCT labels and each re-adjudicated label, and between radiologists. If kappa < 0.8 or >5% of T4 labels change, label noise is material and the benchmark's reported performance should be re-evaluated after correcting or excluding ambiguous cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LaryngealCT's central claim is to be a reliable benchmark for laryngeal cancer CT staging. This rests entirely on the T-stage labels harvested from six TCIA collections (Table 2, Methods). The paper does not re-adjudicate any labels by central imaging review or pathology correlation; the only expert validation covers cropping (Table 3), not staging. TCIA datasets are known to contain clinical-record labels that may reflect different AJCC editions, mixed clinical/pathological staging, or entry errors. With T4 representing only 84/1,029 cases (8.2%), even a handful of mislabeled T4 cases would shift the T4-vs-non-T4 results. The abstract reports Accuracy 0.854 / F1-macro 0.841 while Table 5 reports 0.840 / 0.821, suggesting internal inconsistency that compounds the label issue. If ground truth is noisy, all reported AUCs and conclusions about model ranking are unreliable, and the benchmark's utility for future research is compromised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LaryngealCT, a curated benchmark of 1,029 laryngeal CT volumes aggregated from six TCIA collections, with standardized 1-mm isotropic laryngeal VOIs produced by a parameter-search cropping pipeline. Six 3D CNN architectures (custom 3D CNN, ResNet18/50/101, DenseNet121, MedicalNet-pretrained ResNet50) are benchmarked on two tasks: early (Tis–T2) vs advanced (T3–T4) staging, and T4 vs non-T4 classification. The authors report that the lightweight 3D CNN performs best in cross-validation and test for the first task, while ResNet18 shows the best calibrated per-class T4 recall in the second. The paper also includes calibration, DeLong tests, perturbation-based explainability, and an open-source reproducibility pipeline.","tokens_in":21850,"tokens_out":5980,"duration_ms":52100,"significance":"If the underlying labels and cropping are reliable, LaryngealCT would be a valuable public benchmark for a disease area that lacks standardized CT datasets. The paper is unusually transparent: it reports confidence intervals, per-class metrics, Brier scores, calibration curves, and openly provides cropping parameters, metadata, code, and trained checkpoints. These reproducibility strengths are real. However, the central quantitative validation of the cropping pipeline is circular, the ground-truth T-stage labels are inherited from TCIA without adjudication, and the final test evaluation selects the best of five validation folds. These issues directly affect the benchmark's credibility and the reported performance numbers, so the contribution is not yet ready in its present form.","major_comments":[{"comment":"The quantitative crop validation is circular. The parameter-search procedure selects each bounding box by minimizing MSE against the expert manual crop; the validation step then reports Dice=0.995/IoU=0.990 by comparing those same automated crops to the same manual crops. This is a restatement of the fitting objective, not an independent accuracy check. The 100-case expert Likert review is independent but subjective and covers qualitative criteria only. Please add a held-out validation (e.g., withhold manual crops from the parameter search, compare against independent anatomical landmark annotations, or measure downstream classification sensitivity to crop perturbations) before claiming automated crops are 'essentially indistinguishable from expert manual crops.'","section":"Methods — Parameter Search for Reproducible Cropping; Methods — Validation of Cropping"},{"comment":"The final test evaluation selects the best-performing of the 5 cross-validation folds based on validation macro-F1. With five chances, the selected fold's test performance is optimistically biased relative to the expected performance of the training pipeline. This is especially problematic for the T4 task, where the test set contains only 17 positives (Table 8). Please report test metrics for all folds, an ensemble of folds, or a prespecified fold-selection rule; at minimum, quantify the selection effect.","section":"Methods — Training Protocol; Results — Test Performance"},{"comment":"The benchmark's ground truth consists of T-stage labels inherited from six TCIA collections without central imaging review, pathology correlation, or adjudication. Because LaryngealCT's primary contribution is as a reliable benchmark, label provenance is load-bearing: the same TCIA collections may use different AJCC editions, clinical vs pathological staging, and may contain entry errors. With only 84 T4 cases (8.2%), even a few mislabeled T4s would materially change AUC and sensitivity estimates. Please document the original staging source for each collection, report any cleaning or audit performed, and include a sensitivity analysis (e.g., excluding or relabeling uncertain cases) to show that the conclusions are robust.","section":"Methods — Dataset Acquisition and Curation; Table 2"},{"comment":"There are internal reporting inconsistencies that must be corrected. The abstract reports Accuracy 0.854 and F1-macro 0.841 for the 3D CNN on the early-vs-advanced task, while Table 5 reports 0.840 and 0.821 for the same entry. In Table 7, DenseNet121's Accuracy CI is 0.833 (0.835–0.927) — the lower bound exceeds the point estimate — and several F1-macro CIs are inverted (e.g., 0.607 with CI 0.114–0.465; 0.646 with CI 0.077–0.581). These issues undermine the reliability of the reported uncertainty intervals. Please regenerate all CIs and ensure abstract/table consistency.","section":"Abstract; Tables 5 and 7"}],"minor_comments":[{"comment":"Table 7 has formatting errors in the DenseNet121 row and in several F1-macro CIs; Table 8's class labels say 'Class 0 - Early Stage' and 'Class 1 – Advanced Stage' for the T4 task, which should read 'non-T4' and 'T4'. The calibrated confusion matrix for the pretrained ResNet50 row is garbled.","section":"Tables 7 and 8"},{"comment":"Notation is inconsistent: 'GradCAMpp' (abstract) vs 'Grad-CAM++' (text), and 'Delong' vs 'DeLong'. Please standardize.","section":"Throughout"},{"comment":"The CAM-mask overlap fraction is 1.4±1.3% for T4 and 2.8±1.9% for non-T4, with enrichment ratios below 1. The text nonetheless concludes that decisions are 'anatomically and causally coherent with radiological evidence.' This conclusion is stronger than the quantitative evidence supports; please temper it or add a statistical comparison against random activation maps.","section":"Results — Model Explainability"},{"comment":"The voxel-wise error metrics for crop validation (MAE = 0.99, MSE = 1.9) are reported without units or a description of the intensity scale (e.g., after z-score normalization or in HU). Please clarify so the numbers are interpretable.","section":"Methods — Data Preprocessing"},{"comment":"The discussion of LODO validation is helpful, but the phrase 'statistically significant variability between lightweight and deeper networks' is vague after the DeLong tests; please specify which model pairs were significantly different in each task.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"This is a useful dataset contribution with genuinely valuable open-science elements, but the current version has a circular cropping validation, unadjudicated label provenance, and a model-selection protocol that may inflate test numbers. These are fixable within the scope of a major revision. I would not recommend rejection, because the dataset and reproducibility infrastructure are likely to be useful once the validation and reporting issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best read as a resource rather than a methods paper. LaryngealCT is the first public ROI-focused laryngeal cancer CT benchmark, and that alone is worth something: 1,029 scans from six TCIA collections, standardized to 1-mm isotropic VOIs, with cropping parameters, metadata, splits, and code all on GitHub. The benchmarking of six standard 3D CNNs is not novel, but it is honest—they report CIs, per-class metrics, calibration, and admit T4 sensitivity is too low for clinical use. That level of transparency is rare.\n\nThe soft spots are real but fixable. The quantitative validation of the cropping pipeline is circular: the parameter search was fit by minimizing MSE against the manual expert crops, and then the same manual crops are used as the reference to compute Dice 0.995 and IoU 0.990. That number is a restatement of the fitting objective, not an independent check. The expert review covers only 100 manual crops, not the automated pipeline.\n\nThe abstract in the version I saw reports Accuracy 0.854 and F1-macro 0.841 for the 3D CNN on early vs advanced, while Table 5 gives 0.840 and 0.821. That needs reconciling before publication. Similarly, the test metrics are based on the best of five folds, selected by validation macro-F1; that conditions the results on a favorable fold and should be reported transparently (and ideally averaged over folds).\n\nThe bigger limitation is the ground truth. The T-stage labels are taken directly from six TCIA collections and were not re-adjudicated by central imaging review or pathology. With only 84 T4 cases, a small number of mislabeled T4s could shift the results. The paper should state this as a limitation explicitly and, if possible, include a sensitivity analysis or at least a discussion of label noise.\n\nNone of this sinks the contribution. The dataset and pipeline are still valuable for the head-and-neck imaging community, and the code/data release makes it reproducible. The paper needs a serious referee and a major revision, but a desk rejection would be wrong.","headline":"A genuinely useful laryngeal CT benchmark with fixable reporting problems, but the cropping validation is circular and the ground-truth labels are unverified.","tokens_in":22413,"tokens_out":4502,"would_cite":true,"duration_ms":37841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces LaryngealCT, a public benchmark of 1,029 CT scans for laryngeal cancer staging, and shows that a lightweight 3D CNN outperforms deeper networks on early-versus-advanced classification.","keywords":["laryngeal cancer","CT imaging","benchmark dataset","deep learning","T-stage classification","early vs advanced","T4 detection","explainability"],"falsifier":"Take a random sample of, say, 150 of the 1,029 volumes, oversampling T4, and have a blinded head-and-neck radiologist re-stage each from the CT; compare the re-adjudicated stages with the source metadata. If disagreement on T4 reaches roughly 20% or more, the reported sensitivities and rankings cannot be regarded as measuring the intended staging task.","tokens_in":21474,"feed_emoji":"🩻","tokens_out":5644,"duration_ms":50968,"temperature":0.7,"pith_summary":"This paper is trying to establish that laryngeal cancer staging can be studied reproducibly with deep learning once the imaging input is standardised. To that end it introduces LaryngealCT, a benchmark of 1,029 CT scans of laryngeal cancer cases, each reduced to a uniform 1 mm isotropic volume of interest around the larynx. On that benchmark, a small custom 3D CNN outperforms deeper standard architectures for distinguishing early (Tis–T2) from advanced (T3–T4) disease, reaching about 0.85 accuracy and 0.84 macro-F1. The harder clinical question—detecting T4 disease—remains unsolved: most models reach AU-ROC above 0.82 but miss most T4 cases at default thresholds. A sympathetic reader would care because the field has lacked a fixed, publicly reproducible test bed, and this dataset and protocol are meant to be that test bed.","feed_headline":"1029 larynx CT scans set the first public AI staging benchmark","feed_subtitle":"A small 3D CNN wins early-vs-advanced staging; T4 recall stays low, exposing the real bottleneck.","key_machinery":"The load-bearing object is the laryngeal volume-of-interest (VOI) protocol: a parameter-search procedure that finds, for each scan, the bounding box that best matches an expert crop, then stores those coordinates so anyone with access to the source scans can reconstruct identical 1 mm isotropic laryngeal sub-volumes. Alongside it, the custom five-layer 3D CNN (about 1.5 million parameters) serves as the baseline classifier that outperforms deeper residual, densely connected, and transfer-learning variants. The VOI protocol is what makes the benchmark reproducible; the lightweight CNN is what the benchmark currently endorses.","core_discovery":"The paper's central claim is that the combination of a curated, anatomy-standardised laryngeal CT benchmark and a lightweight 3D CNN establishes a reproducible baseline for AI-based laryngeal cancer staging. The dataset aggregates 1,029 scans from six public collections, with laryngeal volumes cropped at 1 mm isotropic resolution using a weakly supervised parameter search; expert review and quantitative comparison show automated crops are essentially identical to manual expert crops (mean Dice 0.995). On the independent test set the custom 3D CNN achieves the strongest overall early-vs-advanced performance (accuracy 0.840, F1-macro 0.821, AUC 0.881 in the detailed tables; 0.854 and 0.841 in","pith_inferences":["If the source-collection T-stage labels are noisier than expert re-adjudication would be, the benchmark's difficulty and the reported ranking could shift; publishing a small re-adjudicated validation subset would harden the benchmark.","The cropping-parameter trick—releasing bounding-box coordinates instead of derived images—could be reused for other organ-focused benchmarks built from restricted public archives.","The 'simple model wins' result may be a mid-data-regime phenomenon; as the benchmark grows or synthetic T4 samples are added, deeper architectures might close the gap, so the benchmark should track model scaling, not just a leaderboard.","If T4 false negatives are the main clinical risk, the benchmark's next testable extension is a cost-sensitive or anatomy-guided training objective that explicitly penalises missing invasion cues; the reported explainability pattern suggests such cues are currently under-weighted."],"forward_implications":["If the benchmark is adopted, future staging models can be compared on identical anatomy-standardised inputs, removing cropping variability as a confound.","The success of the small 3D CNN implies that on mid-sized, anatomy-focused datasets, architectural depth is not the main performance driver, and resources may be better spent on cropping, calibration, and class-imbalance handling.","The consistently low T4 sensitivity means the benchmark currently exposes a real clinical gap: no model is ready to rule in or rule out thyroid-cartilage invasion on its own.","The open cropping parameters allow independent teams to reconstruct the same dataset from the original public scans, making the benchmark a practical common test bed.","Calibration and threshold optimisation change rankings and per-class trade-offs, so the paper's dual reporting of raw and calibrated metrics sets a reporting norm that could carry over to other medical-imaging benchmarks."],"fun_headline_variants":["First public larynx CT benchmark for AI staging","1029 CT scans fuel reproducible AI larynx staging","LaryngealCT: open benchmark, small CNN wins staging","Simple 3D CNN tops larynx cancer staging benchmark","AI still stumbles on T4 larynx cancer in new benchmark"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the T-stage labels taken from the six public source collections are correct enough to serve as ground truth; they were not re-adjudicated for this study, and the T4 class has only 84 cases.","fun_headline_variants_meta":{"raw":{"variants":["First public larynx CT benchmark for AI staging","1029 CT scans fuel reproducible AI larynx staging","LaryngealCT: open benchmark, small CNN wins staging","Simple 3D CNN tops larynx cancer staging benchmark","AI still stumbles on T4 larynx cancer in new benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":2928,"prompt_tokens":844,"completion_tokens":2084,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2004}},"tokens_in":588,"tokens_out":2084,"duration_ms":12507,"temperature":1.0,"reasoning_tokens":2004,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:11:04.726425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 150 of the 1,029 volumes, oversampling T4, and have a blinded head-and-neck radiologist re-stage each from the CT; compare the re-adjudicated stages with the source metadata. If disagreement on T4 reaches roughly 20% or more, the reported sensitivities and rankings cannot be regarded as measuring the intended staging task.","supporting_citations":[],"review_version":1}