REVIEW 4 major objections 5 minor 35 references
Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Dataset
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read The paper introduces LaryngealCT, a public benchmark of 1,029 CT scans for laryngeal cancer staging, and shows that a lightweight 3D CNN outperforms deeper networks on early-versus-advanced classification.
desk verdict A genuinely useful laryngeal CT benchmark with fixable reporting problems, but the cropping validation is circular and the ground-truth labels are unverified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the laryngeal volume-of-interest (VOI) protocol: a parameter-search procedure that finds, for each scan, the bounding box that best matches an expert crop, then stores those coordinates so anyone with access to the source scans can reconstruct identical 1 mm isotropic laryngeal sub-volumes. Alongside it, the custom five-layer 3D CNN (about 1.5 million parameters) serves as the baseline classifier that outperforms deeper residual, densely connected, and transfer-learning variants. The VOI protocol is what makes the benchmark reproducible; the lightweight CNN is what the benchmark currently endorses.
What would settle it
Take a random sample of, say, 150 of the 1,029 volumes, oversampling T4, and have a blinded head-and-neck radiologist re-stage each from the CT; compare the re-adjudicated stages with the source metadata. If disagreement on T4 reaches roughly 20% or more, the reported sensitivities and rankings cannot be regarded as measuring the intended staging task.
Extended reading notes
Core claim
The paper's central claim is that the combination of a curated, anatomy-standardised laryngeal CT benchmark and a lightweight 3D CNN establishes a reproducible baseline for AI-based laryngeal cancer staging. The dataset aggregates 1,029 scans from six public collections, with laryngeal volumes cropped at 1 mm isotropic resolution using a weakly supervised parameter search; expert review and quantitative comparison show automated crops are essentially identical to manual expert crops (mean Dice 0.995). On the independent test set the custom 3D CNN achieves the strongest overall early-vs-advanced performance (accuracy 0.840, F1-macro 0.821, AUC 0.881 in the detailed tables; 0.854 and 0.841 in
Load-bearing premise
The load-bearing premise is that the T-stage labels taken from the six public source collections are correct enough to serve as ground truth; they were not re-adjudicated for this study, and the T4 class has only 84 cases.
Editorial extensions
If this is right
- If the benchmark is adopted, future staging models can be compared on identical anatomy-standardised inputs, removing cropping variability as a confound.
- The success of the small 3D CNN implies that on mid-sized, anatomy-focused datasets, architectural depth is not the main performance driver, and resources may be better spent on cropping, calibration, and class-imbalance handling.
- The consistently low T4 sensitivity means the benchmark currently exposes a real clinical gap: no model is ready to rule in or rule out thyroid-cartilage invasion on its own.
- The open cropping parameters allow independent teams to reconstruct the same dataset from the original public scans, making the benchmark a practical common test bed.
- Calibration and threshold optimisation change rankings and per-class trade-offs, so the paper's dual reporting of raw and calibrated metrics sets a reporting norm that could carry over to other medical-imaging benchmarks.
Reading between the lines
- If the source-collection T-stage labels are noisier than expert re-adjudication would be, the benchmark's difficulty and the reported ranking could shift; publishing a small re-adjudicated validation subset would harden the benchmark.
- The cropping-parameter trick—releasing bounding-box coordinates instead of derived images—could be reused for other organ-focused benchmarks built from restricted public archives.
- The 'simple model wins' result may be a mid-data-regime phenomenon; as the benchmark grows or synthetic T4 samples are added, deeper architectures might close the gap, so the benchmark should track model scaling, not just a leaderboard.
- If T4 false negatives are the main clinical risk, the benchmark's next testable extension is a cost-sensitive or anatomy-guided training objective that explicitly penalises missing invasion cues; the reported explainability pattern suggests such cues are currently under-weighted.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LaryngealCT, a curated benchmark of 1,029 laryngeal CT volumes aggregated from six TCIA collections, with standardized 1-mm isotropic laryngeal VOIs produced by a parameter-search cropping pipeline. Six 3D CNN architectures (custom 3D CNN, ResNet18/50/101, DenseNet121, MedicalNet-pretrained ResNet50) are benchmarked on two tasks: early (Tis–T2) vs advanced (T3–T4) staging, and T4 vs non-T4 classification. The authors report that the lightweight 3D CNN performs best in cross-validation and test for the first task, while ResNet18 shows the best calibrated per-class T4 recall in the second. The paper also includes calibration, DeLong tests, perturbation-based explainability, and an open-source reproducibility pipeline.
Significance. If the underlying labels and cropping are reliable, LaryngealCT would be a valuable public benchmark for a disease area that lacks standardized CT datasets. The paper is unusually transparent: it reports confidence intervals, per-class metrics, Brier scores, calibration curves, and openly provides cropping parameters, metadata, code, and trained checkpoints. These reproducibility strengths are real. However, the central quantitative validation of the cropping pipeline is circular, the ground-truth T-stage labels are inherited from TCIA without adjudication, and the final test evaluation selects the best of five validation folds. These issues directly affect the benchmark's credibility and the reported performance numbers, so the contribution is not yet ready in its present form.
major comments (4)
- [Methods — Parameter Search for Reproducible Cropping; Methods — Validation of Cropping] The quantitative crop validation is circular. The parameter-search procedure selects each bounding box by minimizing MSE against the expert manual crop; the validation step then reports Dice=0.995/IoU=0.990 by comparing those same automated crops to the same manual crops. This is a restatement of the fitting objective, not an independent accuracy check. The 100-case expert Likert review is independent but subjective and covers qualitative criteria only. Please add a held-out validation (e.g., withhold manual crops from the parameter search, compare against independent anatomical landmark annotations, or measure downstream classification sensitivity to crop perturbations) before claiming automated crops are 'essentially indistinguishable from expert manual crops.'
- [Methods — Training Protocol; Results — Test Performance] The final test evaluation selects the best-performing of the 5 cross-validation folds based on validation macro-F1. With five chances, the selected fold's test performance is optimistically biased relative to the expected performance of the training pipeline. This is especially problematic for the T4 task, where the test set contains only 17 positives (Table 8). Please report test metrics for all folds, an ensemble of folds, or a prespecified fold-selection rule; at minimum, quantify the selection effect.
- [Methods — Dataset Acquisition and Curation; Table 2] The benchmark's ground truth consists of T-stage labels inherited from six TCIA collections without central imaging review, pathology correlation, or adjudication. Because LaryngealCT's primary contribution is as a reliable benchmark, label provenance is load-bearing: the same TCIA collections may use different AJCC editions, clinical vs pathological staging, and may contain entry errors. With only 84 T4 cases (8.2%), even a few mislabeled T4s would materially change AUC and sensitivity estimates. Please document the original staging source for each collection, report any cleaning or audit performed, and include a sensitivity analysis (e.g., excluding or relabeling uncertain cases) to show that the conclusions are robust.
- [Abstract; Tables 5 and 7] There are internal reporting inconsistencies that must be corrected. The abstract reports Accuracy 0.854 and F1-macro 0.841 for the 3D CNN on the early-vs-advanced task, while Table 5 reports 0.840 and 0.821 for the same entry. In Table 7, DenseNet121's Accuracy CI is 0.833 (0.835–0.927) — the lower bound exceeds the point estimate — and several F1-macro CIs are inverted (e.g., 0.607 with CI 0.114–0.465; 0.646 with CI 0.077–0.581). These issues undermine the reliability of the reported uncertainty intervals. Please regenerate all CIs and ensure abstract/table consistency.
minor comments (5)
- [Tables 7 and 8] Table 7 has formatting errors in the DenseNet121 row and in several F1-macro CIs; Table 8's class labels say 'Class 0 - Early Stage' and 'Class 1 – Advanced Stage' for the T4 task, which should read 'non-T4' and 'T4'. The calibrated confusion matrix for the pretrained ResNet50 row is garbled.
- [Throughout] Notation is inconsistent: 'GradCAMpp' (abstract) vs 'Grad-CAM++' (text), and 'Delong' vs 'DeLong'. Please standardize.
- [Results — Model Explainability] The CAM-mask overlap fraction is 1.4±1.3% for T4 and 2.8±1.9% for non-T4, with enrichment ratios below 1. The text nonetheless concludes that decisions are 'anatomically and causally coherent with radiological evidence.' This conclusion is stronger than the quantitative evidence supports; please temper it or add a statistical comparison against random activation maps.
- [Methods — Data Preprocessing] The voxel-wise error metrics for crop validation (MAE = 0.99, MSE = 1.9) are reported without units or a description of the intensity scale (e.g., after z-score normalization or in HU). Please clarify so the numbers are interpretable.
- [Discussion] The discussion of LODO validation is helpful, but the phrase 'statistically significant variability between lightweight and deeper networks' is vague after the DeLong tests; please specify which model pairs were significantly different in each task.
Circularity Check
The quantitative crop-fidelity claim (Dice=0.995/IoU=0.990) restates the MSE fitting objective used to define the bounding boxes; the DL benchmark itself uses held-out test data and is not circular.
-
self definitional
[Methods 'Parameter Search for Reproducible Cropping'; Results 'Validation of Cropping' (quantitative validation paragraph)]
"Each candidate crop was compared to the manually extracted volume using a similarity metric, mean-squared error (MSE), to identify the best-matching region... For 1,025 cases (99.6%), automated crops exhibited near-perfect concordance with manual volumes, achieving a mean Dice similarity coefficient of 0.995 (SD 0.0013) and mean Intersection-over-Union of 0.990 (SD 0.0026)."
The per-case bounding box was fit by exhaustive parameter search minimizing MSE against the expert manual crop. The same manual crops are then used as the reference for the Dice/IoU 'quantitative validation.' Because the search objective is to maximize agreement with those references, near-perfect Dice/IoU is a restatement of the fitting objective, not an independent accuracy check. The independent expert validation covers only 100 cases and uses Likert ratings; it does not produce the 0.995/0.990 figures. Thus the paper's quantitative claim that automated crops are 'essentially indistinguishable from expert manual crops' is forced by construction for the fitted boxes and cannot by itself validate the cropping pipeline.
full rationale
LaryngealCT's core preprocessing step is cropping each volume to a standardized laryngeal VOI. The paper's quantitative evidence that this cropping is accurate (Dice=0.995, IoU=0.990 over n=1,029) is not independent: the per-case bounding box parameters were obtained by a search that minimizes MSE between candidate crops and the expert manual crops, and the same expert manual crops are then used as the reference for the Dice/IoU computation. High overlap is therefore a by-construction property of the fitting procedure, so this validation metric cannot by itself establish clinical fidelity. The separate 100-case Likert review with kappa=0.90-1.00 is genuinely independent expert evidence and is not circular. The DL benchmark is also not circular: models are evaluated on a held-out test split with labels derived from external TCIA staging metadata, not from model predictions. The unverified nature of the TCIA T-stage labels is a data-quality/correctness concern rather than a circularity concern. Overall, one central quantitative validation claim reduces to its fitting objective, so partial circularity; score 6.
Assumptions & free parameters
free parameters (9)
- Focal loss alpha (early vs advanced) =
0.75
- Focal loss alpha (T4 vs non-T4) =
8.0
- Focal loss gamma =
2.0
- Temperature scaling T (per model, per task) =
e.g., 0.202–14.332 in Tables 6/8
- Decision threshold t_opt (per model, per task) =
e.g., 0.183–0.519 in Tables 6/8
- Bounding box parameters (origin, size) per case =
stored in bbox3d.txt
- Cropped volume resolution =
32x96x96
- HU clipping range =
[-300, 300]
- Augmentation hyperparameters =
flip p=0.5, affine p=0.3, gamma p=0.2, noise p=0.15
assumptions (5)
- domain assumption T-stage labels (Tis–T4) in the TCIA source metadata are accurate enough to serve as ground truth for classification.
- domain assumption The expert manual laryngeal crops in 3D Slicer define the correct anatomical ROI.
- domain assumption Resampling to 1-mm isotropic and HU clipping to [-300,300] preserve task-relevant information.
- domain assumption The stratified 80/20 split with patient-level separation yields a valid independent test estimate.
- standard math DeLong test assumptions for paired AUC comparison hold.
Cite this review
Pith. "Pith review of Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Dataset." pith.science (2026). https://pith.science/paper/IJ5YX4XX
@misc{pith2026251011047,
author = {Pith},
title = {Pith review of: Benchmarking Deep Learning Models for Laryngeal Cancer Staging Using the LaryngealCT Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/IJ5YX4XX}},
note = {Machine review of arXiv:2510.11047}
}
read the original abstract
Laryngeal cancer imaging research lacks standardised public datasets to enable reproducible deep learning (DL) model development. We present LaryngealCT, a curated benchmark of 1,029 computed tomography (CT) scans aggregated from six collections from The Cancer Imaging Archive (TCIA). Uniform 1 mm isotropic volumes of interest encompassing the larynx were extracted using a weakly supervised parameter search framework validated by clinical experts. Six 3D DL architectures (custom 3D CNN, ResNet18,50,101, DenseNet121 and MedicalNet-pretrained ResNet50) were benchmarked on (i) early (Tis,T1,T2) vs. advanced (T3,T4) and (ii) T4 vs. non-T4 classification tasks. On the independent test set, the 3D CNN achieved the strongest overall performance across global and per-class metrics (Accuracy 0.854, F1-macro 0.841) in early vs. advanced classification. In the T4 task, AU-ROC values exceeded 0.82 for most models, but sensitivity for T4 disease remained limited (less than or equal to 0.412), with ResNet101 showing the most promising calibrated T4 recall (0.706. Model explainability assessed using GradCAMpp with thyroid cartilage overlays for T4 classification task revealed anatomically plausible peri-cartilage activations, although spatial overlap was modest. Through open-source data, pretrained models, and integrated explainability tools, LaryngealCT offers a reproducible foundation for AI-driven research to support future clinical decision-making in laryngeal oncology.
Reference graph
Works this paper leans on
-
[1]
& Moore, M
Igissin, N., Zatonskikh, V., Telmanova, Z., Tulebaev, R. & Moore, M. Laryngeal cancer: Epidemiology, etiology, and prevention: A narrative review. Iran. J. Public Health 52, 2248– 2259 (2023)
2023
-
[2]
& Song, Y
Lin, C., Cheng, W., Liu, X., Li, H. & Song, Y. The global, regional, national burden of laryngeal cancer and its attributable risk factors (1990-2019) and predictions to 2035. Eur. J. Cancer Care (Engl.) 31, e13689 (2022)
1990
-
[3]
Benazzo, M. et al. Imaging accuracy in preoperative staging of T3 -T4 laryngeal cancers. Cancers (Basel) 12, 1074 (2020)
2020
-
[4]
B., Barbu, A
Itamura, K., Hsue, V. B., Barbu, A. M. & Chen, M. M. Diagnostic assessment (imaging) and staging of laryngeal cancer. Otolaryngol. Clin. North Am. 56, 215–231 (2023)
2023
-
[5]
Chen, J. et al. Medical image translation with deep learning: Advances, datasets and perspectives. Med. Image Anal. 103, 103605 (2025)
2025
-
[6]
& Kumar, S
Rebinth, A. & Kumar, S. Importance of Manual Image Annotation Tools and Free Datasets for Medical Research. Journal of Advanced Research in Dynamical and Control Systems 10, 1880–1885 (2019)
2019
-
[7]
Andrearczyk, V. et al. Overview of the HECKTOR challenge at MICCAI 2021: Automatic Head and Neck TumOR segmentation and outcome prediction in PET/CT images. arXiv [eess.IV] (2022) doi:10.48550/ARXIV.2201.04138
work page Pith review arXiv doi:10.48550/arxiv.2201.04138 2021
-
[8]
Head and Neck Cancer Imaging
Hermans, R. Head and Neck Cancer Imaging . (Springer International Publishing, Cham, 2020)
2020
Show all 35 references
-
[9]
& Vrtovec, T
Podobnik, G., Strojan, P ., Peterlin, P., Ibragimov, B. & Vrtovec, T. HaN-Seg: The head and neck organ-at-risk CT and MR segmentation dataset. Med. Phys. 50, 1917–1927 (2023)
1917
-
[10]
Santin, M. et al. Detecting abnormal thyroid cartilages on CT using deep learning. Diagn. Interv. Imaging 100, 251–257 (2019)
2019
-
[11]
Guo, R. et al. CT-based radiomics features in the prediction of thyroid cartilage invasion from laryngeal and hypopharyngeal squamous cell carcinoma. Cancer Imaging 20, 81 (2020)
2020
-
[12]
Rajgor, A. D. et al. Computed tomography-based radiomic markers are independent prognosticators of survival in advanced laryngeal cancer: a pilot study. J. Laryngol. Otol. 138, 685–691 (2024)
2024
-
[13]
Hao, Y. et al. Deep learning-based prediction of thyroid cartilage invasion: Analysis on CT images in laryngeal and hypopharyngeal squamous cell carcinoma. J. Radiat. Res. Appl. Sci. 17, 100974 (2024)
2024
-
[14]
& Jagannatha, V
Rao, D., Singh, R., Koteshwara, P. & Jagannatha, V. Prediction of Tumour Site in Larynx Contrast CT Images by Radiomic Feature Analysis. International Journal on Engineering Applications (IREA) 11, 196–204 (2023)
2023
-
[15]
& Vijayananda, J
Rao, D., Singh, R., Koteshwara, P . & Vijayananda, J. Exploring the impact of model complexity on laryngeal cancer detection. Indian J. Otolaryngol. Head Neck Surg. 76, 4036– 4042 (2024)
2024
-
[16]
Takano, Y. et al. Diagnosis of thyroid cartilage invasion by laryngeal and hypopharyngeal cancers based on CT with deep learning. Eur. J. Radiol. 189, 112168 (2025)
2025
-
[17]
K., Mishra, S., Panigrahi, R., Bhoi, A
Sahoo, P . K., Mishra, S., Panigrahi, R., Bhoi, A. K. & Barsocchi, P. An improvised deep- learning-based Mask R-CNN model for laryngeal cancer detection using CT images. Sensors (Basel) 22, 8834 (2022)
2022
-
[18]
Chen, X. et al. A combined model integrating radiomics and deep learning based on contrast-enhanced CT for preoperative staging of laryngeal carcinoma. Acad. Radiol. 30, 3022–3031 (2023)
2023
-
[19]
& Jagannatha, V
Rao, D., Koteshwara, P., Singh, R. & Jagannatha, V. Exploring radiomics for classification of supraglottic tumors: A pilot study in a tertiary care center. Indian J. Otolaryngol. Head Neck Surg. 75, 433–439 (2023)
2023
-
[20]
& Sun, J
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, 2016)
2016
-
[21]
& Weinberger, K
Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2017)
2017
-
[22]
& Zheng, Y
Chen, S., Ma, K. & Zheng, Y. Med3D: Transfer learning for 3D medical image analysis. arXiv [cs.CV] (2019)
2019
-
[23]
Clark, K. et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013)
2013
-
[24]
Welch, M. L. et al. Computed tomography images from large head and neck cohort (RADCURE). The Cancer Imaging Archive (2023)
2023
-
[25]
Vallières, M. et al. Data from head-neck-PET-CT. The Cancer Imaging Archive (2017)
2017
-
[26]
& Dekker, A
Wee, L. & Dekker, A. Data from HEAD-NECK-RADIOMICS-HN1. The Cancer Imaging Archive (2019)
2019
-
[27]
Grossberg, A. et al. HNSCC. The Cancer Imaging Archive (2020)
2020
-
[28]
Bejarano, T., Ornelas -Couto, M. D. & Mihaylov, I. Head -and-neck squamous cell carcinoma patients with CT taken during pre -treatment, mid-treatment, and post -treatment Dataset (HNSCC -3DCT-RT). Preprint at https://doi.org/10.7937/K9/TCIA.2018.13UPR2XF (2018)
2018 doi
-
[29]
Beichel, R. R. et al. Data From QIN -HEADNECK. Preprint at https://doi.org/10.7937/K9/TCIA.2015.K0F5CGLI (2015)
2015 doi
-
[30]
Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977)
1977
-
[31]
& Aydin, M
Kuş, Z. & Aydin, M. MedSegBench: A comprehensive benchmark for medical image segmentation in diverse data modalities. Sci. Data 11, 1283 (2024)
2024
- [32]
-
[33]
Fedorov, A. et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 30, 1323–1341 (2012)
2012
-
[34]
P M, D. A. & Munirathinam, S. Deep Learning -Based Classification of Laryngeal Abnormalities Using DenseNet Architectures. in 2025 3rd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) 1235–1239 (2025)
2025
-
[35]
Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations
Kocak, B. Evaluation metrics in medical imaging AI: fundamentals, pitfalls, misapplications, and recommendations. European Journal of Radiology Artificial Intelligence 3, (2025). Acknowledgments We thank TCIA for granting access to the restricted datasets used in this study. T...
2025
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.