Pith. sign in

REVIEW 3 major objections 4 minor 1 references

Overview of the Head and Neck Tumor Segmentation for Magnetic Resonance Guided Applications (HNTS-MRG) 2024 Challenge

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The HNTS-MRG 2024 challenge shows that top AI algorithms segment head and neck tumors on MRI above clinician interobserver variability benchmarks on both pre-radiotherapy and mid-radiotherapy scans, with multi-timepoint input driving the…

desk verdict A genuinely useful multi-timepoint MRI benchmark, but the 'AI surpasses clinicians' headline is not supported by the evaluation design. read the letter →

arxiv 2411.18585 v2 pith:YZA3UDDQ submitted 2024-11-27 physics.med-ph

Kareem A. Wahid , Cem Dede , Dina M. El-Habashy , Serageldin Kamel , Michael K. Rooney , Yomna Khamis , Moamen R. A. Abdelaal , Sara Ahmed
show 16 more authors
Kelsey L. Corrigan Enoch Chang Stephanie O. Dudzinski Travis C. Salzillo Brigid A. McDonald Samuel L. Mulder Lucas McCullum Qusai Alakayleh Carlos Sjogreen Renjie He Abdallah S.R. Mohamed Stephen Y. Lai John P. Christodouleas Andrew J. Schaefer Mohamed A. Naser Clifton D. Fuller
This is my paper · ORCID
classification physics.med-ph
keywords headandneckcancermagneticresonanceimagingdatachallengesegmentationcontouringradiotherapydeeplearninginterobservervariability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports the design and outcome of a crowdsourced challenge on automatic segmentation of head and neck tumors from T2-weighted MRI scans, using 150 training and 50 test cases each with a pre-radiotherapy and a mid-radiotherapy scan. Its central claim is that top-performing deep learning methods achieved a mean aggregated Dice Similarity Coefficient of 0.825 on the pre-radiotherapy task and 0.733 on the mid-radiotherapy task, both above the clinician interobserver variability benchmarks of 0.806 and 0.714 that the organizers measured from multiple expert annotations. If this comparison holds, automated contouring has reached or passed the level of agreement among clinicians for these structures, which matters because manual tumor segmentation is a bottleneck in MRI-guided adaptive radiotherapy. The paper also shows that the mid-treatment task is harder than the pre-treatment one, that primary tumors are harder than lymph nodes, and that the winning mid-treatment methods all used the registered pre-treatment image and its contour as additional input.

What carries the argument

The argument is carried by two linked methodological objects. First, the aggregated Dice similarity coefficient (DSCagg), which computes Dice per case and then aggregates over the whole test set, serving as the official ranking metric and as the basis for the clinician interobserver variability comparison. Second, a ground-truth construction in which three to four independent expert annotators segmented every scan and were combined with a consensus algorithm, with pairwise comparisons between annotators defining the clinical performance bar that AI had to beat. A self-configuring deep learning segmentation pipeline (nnU-Net) trained from scratch provided the reference baseline (scoring 0.817 on Task 1 and 0.633 on Task 2), and a null algorithm that simply propagates the pre-treatment contours served as the floor for the mid-treatment task.

What would settle it

Recalculate the interobserver variability after re-including the set-aside cases where only the senior physician produced the final segmentation; if the resulting benchmark drops below the top AI scores (0.825 pre-RT, 0.733 mid-RT), the paper's headline comparison fails. A prospective test would be to have fresh clinicians independently re-contour the 50 test scans and compare their pairwise agreement directly against the submitted AI predictions on the same cases.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that state-of-the-art automated segmentation of gross tumor volume (GTVp) and metastatic lymph nodes (GTVn) from T2-weighted MRI can match or exceed the measured level of agreement between expert clinician annotators. Top methods scored a mean aggregated Dice (DSCagg-mean) of 0.825 for pre-radiotherapy segmentation and 0.733 for mid-radiotherapy segmentation, against interobserver variability benchmarks of 0.806 and 0.714 respectively. The mid-radiotherapy task proved substantially harder, with only four of fifteen teams surpassing the interobserver benchmark and only the winning team crossing it for the primary tumor sub-structure, whose clinician agreement was low (DSCagg around 0.60). The paper further reports that leveraging the registered pre-radiotherapy scan and its segmentation was the distinguishing feature of the strongest mid-treatment solutions, whereas for pre-treatment segmentation a strong self-configuring baseline already rivaled the top submissions.

Load-bearing premise

The clinician agreement benchmark was computed after setting aside the most difficult cases, in which only the senior physician could produce a final contour, so that benchmark may be higher than real clinical agreement and the claim that AI surpasses clinicians may not hold on those hardest cases.

Editorial extensions

If this is right

  • If the scores generalize beyond this single institution, deep learning auto-segmentation is clinically viable for pre-radiotherapy head and neck tumor contouring, and can substantially reduce manual contouring workload in adaptive MRI-guided radiotherapy workflows.
  • For mid-radiotherapy scans, giving the model the registered pre-radiotherapy image and its contour is a reproducible recipe: nearly every top team used it, and all but one test submission beat the null propagation baseline of 0.601.
  • The primary tumor (GTVp) remains the weak spot, with clinician agreement itself around 0.60 aggregated Dice at mid-treatment; this is the area where both humans and algorithms need improvement before reliable adaptive contouring.
  • Because a self-configuring baseline reached within 0.008 of the winning pre-treatment score, further gains on the single-timepoint task are more likely to come from data, preprocessing, and ensembling than from new network architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported interobserver benchmark excludes exactly the hardest cases, where annotators disagreed so strongly that a single senior physician had to define the final contour; including those cases would likely lower the clinical bar, meaning the 'AI surpasses clinicians' headline may be conservative in an uneven way: on the hardest cases there is no clinician agreement to compare against.
  • The same multi-timepoint recipe that won the mid-treatment task could be extended to more frequent imaging, such as weekly intra-treatment MRI, which would let the model track tumor response instead of just one pre-to-mid step.
  • A single-institution dataset with standardized immobilization and consistent fat-suppression pairing likely makes these scores optimistic for deployment across hospitals; a multi-institution test set is the natural next experiment to bound that optimism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports the organization and results of the HNTS-MRG 2024 challenge, a MICCAI satellite event for segmentation of primary gross tumor volume (GTVp) and nodal gross tumor volume (GTVn) on pre-RT (Task 1) and mid-RT (Task 2) T2-weighted MRI. The challenge provided 150 training cases and 50 test cases, used Docker-based submissions on grand-challenge.org, and evaluated 18 Task 1 and 15 Task 2 submissions with the aggregated Dice Similarity Coefficient (DSCagg). The top-performing algorithms achieved DSCagg-mean scores of 0.825 (Task 1) and 0.733 (Task 2), and the abstract states that these results surpassed clinician interobserver variability (IOV) benchmarks of 0.806 and 0.714. The paper describes the dataset, annotation protocol, evaluation metric, baseline models, participant methods, and rankings, and it discusses limitations including cohort size, single-institution data, and high IOV.

Significance. The challenge is a useful community resource: it releases a publicly available MR-specific adaptive radiotherapy dataset, follows BIAS reporting guidelines, provides Docker evaluation infrastructure, and includes detailed descriptions of many independent methods. The reproducibility artifacts (Zenodo data, GitHub examples, Docker framework) are concrete strengths. If the superiority claim were properly supported, the finding that automated segmentation can match or exceed clinician agreement in a difficult head-and-neck MRI task would be clinically meaningful. However, the central claim that algorithms surpassed clinician IOV is not currently supported by the comparison as analyzed. The challenge is still well designed for ranking algorithms, but the headline claim needs to be either rigorously substantiated or reframed.

major comments (3)
  1. [Section 2.3, Section 2.4, Abstract] The headline claim that top AI scores (0.825 for Task 1, 0.733 for Task 2) surpassed clinician interobserver variability (0.806 and 0.714) compares two non-commensurate quantities. Algorithm scores in Section 2.4 are DSCagg between each prediction and the STAPLE consensus ground truth, while the IOV values in Section 2.3 are pairwise DSCagg between individual annotators. A STAPLE consensus is a weighted combination of the annotator set, so an individual annotator will in general agree more with that consensus than with another individual annotator. The appropriate clinician reference for an algorithm scored against STAPLE is a clinician-versus-consensus score (e.g., each annotator versus a leave-one-out STAPLE of the remaining annotators), not a pairwise clinician-clinician score. The paper should compute that benchmark or explicitly restrict the claim to comparing algorithms with pairwise human agreement. As written, the abstract's 'surpassed clinician interobserver variability benchmarks' is not supported by the reported analysis.
  2. [Section 2.3 and Section 4.1] The IOV computation excludes patient cases where only the senior faculty member contributed a final segmentation because of significant annotator disagreement. The authors acknowledge in Section 4.1 that this 'may be slightly inflated,' but they do not report how many cases were excluded or what the IOV values become when those cases are included. Because the IOV benchmark is the reference for the paper's main claim, this is a load-bearing sensitivity issue. The manuscript should state the number of excluded cases and provide a sensitivity analysis, for example by including those cases with the senior faculty segmentation treated as one annotator or as the reference.
  3. [Sections 3.2 and 3.3] The reported test-set DSCagg values are single aggregate numbers with no confidence intervals, and the differences between the top scores and the IOV benchmark are small (Task 1: 0.825 vs 0.806; Task 2: 0.733 vs 0.714). Without bootstrap intervals, paired case-level analysis, or a stated uncertainty on the IOV estimate, the text 'surpassed' is too strong. This is especially relevant because Table 2 shows several teams within 0.01 of the IOV, and Section 4.1 itself notes that only one algorithm crossed GTVp IOV in Task 2. The paper should add uncertainty quantification or soften the claim to 'comparable to or exceeding' where statistically supported.
minor comments (4)
  1. [Section 2.3] The text states that 3 to 4 expert physicians independently segmented each case and also lists 13 unique annotators, but the distribution of the number of annotators per case is not reported; this is relevant to understanding the pairwise IOV computation.
  2. [Section 2.3] Please state explicitly whether the IOV values were computed on the training set, the 50-case test set, or the full 202-case dataset, since the algorithm scores are on the 50-case test set and the comparison should be interpretable with respect to case overlap.
  3. [Section 3.2] The text says 'The top 9 teams (top 50%) achieved DSCagg-mean results higher than interobserver variability (0.806),' but Table 2 shows the 9th-place team scored exactly 0.806, equal to the IOV value; the wording should be 'higher than or equal to' or the team list should be adjusted.
  4. [Section 2.4, Eq. (1)] The displayed equation for DSCagg appears typeset incorrectly, with missing summation indices and unclear absolute-value expressions; please correct it for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the challenge reports empirical algorithm scores against STAPLE consensus and compares them with independently measured clinician interobserver variability; no prediction reduces to its inputs by construction.

full rationale

This paper is an empirical challenge report, not a derivation. Algorithm performance is measured by DSCagg against STAPLE consensus ground truth, while the clinician interobserver variability benchmark is computed from pairwise annotator agreement. Neither quantity is defined in terms of the other, and no fitted parameter is renamed as a prediction. The only self-citations occur in methodological justifications, such as citing the authors' prior work for the use of at least three annotators with STAPLE (Section 2.3, AnnotationCharacteristics); these are not load-bearing for the central claim. The authors also explicitly disclose in Section 4.1 that their measured IOV may be slightly inflated because particularly challenging cases were excluded, which is an honest limitation rather than a circular move. The potential concern that comparing algorithm scores against STAPLE consensus with pairwise clinician IOV is apples-to-oranges is a validity or interpretation issue, not circularity: it does not reduce any result to its own inputs by construction. The central claim that top algorithms surpassed clinician IOV is empirically contingent and could fail under a different benchmark definition, but that does not make the analysis circular. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters, entities, or fitted coefficients. It relies on domain assumptions about the annotation consensus process, the representativeness of the IOV benchmark, and the registration quality. These assumptions are stated or cited, and the most fragile one (IOV representativeness) is acknowledged by the authors.

assumptions (3)
  • domain assumption STAPLE consensus of at least three annotators yields acceptable ground truth segmentations.
    Section 2.3 Annotation Characteristics: the authors rely on their prior findings (refs 19 and 20) that a minimum of three annotators is needed, then combine with STAPLE. This is a background assumption from prior literature, not proven in this paper.
  • domain assumption Interobserver variability computed on the subset of cases with multiple annotators is representative of clinician performance for the comparison.
    Section 2.3 and Section 4.1: cases where only the senior faculty member contributed segmentations were excluded from IOV calculation. The authors state this may inflate the IOV benchmark, directly acknowledging the fragility of this assumption.
  • domain assumption Deformable registration using Elastix parameter map 23 produces sufficiently accurate alignment for the mid-RT task inputs.
    Section 2.3 Data Preprocessing Methods: the registration pipeline is selected based on prior benchmarking (ref 27) and applied without per-case quality assessment except for manual fallback on excessive warping. The quality of fusion of pre-RT and mid-RT images is assumed adequate for all teams.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Overview of the Head and Neck Tumor Segmentation for Magnetic Resonance Guided Applications (HNTS-MRG) 2024 Challenge." pith.science (2026). https://pith.science/paper/YZA3UDDQ

@misc{pith2026241118585,
  author       = {Pith},
  title        = {Pith review of: Overview of the Head and Neck Tumor Segmentation for Magnetic Resonance Guided Applications (HNTS-MRG) 2024 Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZA3UDDQ}},
  note         = {Machine review of arXiv:2411.18585}
}
read the original abstract

Magnetic resonance (MR)-guided radiation therapy (RT) is enhancing head and neck cancer (HNC) treatment through superior soft tissue contrast and longitudinal imaging capabilities. However, manual tumor segmentation remains a significant challenge, spurring interest in artificial intelligence (AI)-driven automation. To accelerate innovation in this field, we present the Head and Neck Tumor Segmentation for MR-Guided Applications (HNTS-MRG) 2024 Challenge, a satellite event of the 27th International Conference on Medical Image Computing and Computer Assisted Intervention. This challenge addresses the scarcity of large, publicly available AI-ready adaptive RT datasets in HNC and explores the potential of incorporating multi-timepoint data to enhance RT auto-segmentation performance. Participants tackled two HNC segmentation tasks: automatic delineation of primary gross tumor volume (GTVp) and gross metastatic regional lymph nodes (GTVn) on pre-RT (Task 1) and mid-RT (Task 2) T2-weighted scans. The challenge provided 150 HNC cases for training and 50 for testing, hosted on Grand Challenge using a Docker submission framework. In total, 19 independent teams from across the world qualified by submitting both their algorithms and corresponding papers, resulting in 18 submissions for Task 1 and 15 submissions for Task 2. Evaluation using the mean aggregated Dice Similarity Coefficient showed top-performing AI methods achieved scores of 0.825 in Task 1 and 0.733 in Task 2. These results surpassed clinician interobserver variability benchmarks, marking significant strides in automated tumor segmentation for MR-guided RT applications in HNC.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 linked inside Pith

  1. [1]

    AprèsMois,LeDéluge

    1. Pollard,J.M.,Wen,Z.,Sadagopan,R.,Wang,J.,Ibbott,G.S.:Thefutureofimage-guidedradiotherapywillbeMRguided.Br.J.Radiol.90,20160667(2017).2. Mulder,S.L.,Heukelom,J.,McDonald,B.A.,VanDijk,L.,Wahid,K.A.,Sanders,K.,Salzillo,T.C.,Hemmati,M.,Schaefer,A.,Fuller,C.D.:MR-guidedadaptiveradiotherapyforOARsparinginheadandneckcancers.Cancers.14,1909(2022).3. Salzillo,T...

    arXiv doi:10.48550/arxiv.2312.09576 2017

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.