REVIEW 4 major objections 4 minor 1 cited by
ACM Multimedia Grand Challenge on ENT Endoscopy Analysis
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper introduces ENTRep as the first benchmark to jointly evaluate fine-grained ENT endoscopy classification with image-to-image and text-to-image retrieval under bilingual Vietnamese-English clinical supervision.
desk verdict A useful new ENT benchmark with a statistically underpowered retrieval leaderboard and some fixable documentation gaps; give it a serious, revision-minded peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ENTRep dataset and its evaluation protocol. The dataset is a set of endoscopic images from routine clinical practice, each carrying a seven-way anatomical label (Ear Right/Left, Nose Right/Left, Throat, VC-open/VC-closed), a normal/abnormal flag, and, for a subset, paired Vietnamese-English narrative descriptions. The three tasks — classification, image-to-image retrieval, and text-to-image retrieval — are evaluated server-side on public (645 images) and private (646 images) test splits that mirror the training distribution, with retrieval scored by Recall@1 and Mean Reciprocal Rank where relevance is defined as a ground-truth image match. This machinery conver
What would settle it
Take a random subsample of about 150 ENTRep images and have two independent ENT clinicians re-annotate the seven-way anatomical label and the normal/abnormal flag; if agreement falls well below the level implied by the top models' roughly 95% accuracy (say, below 95%), the ground truth is too noisy to support the claimed separation of models. A complementary check: paraphrase the 71 English text queries into a second set of 71 and re-run the leading text-to-image models — the paper's own public-to-private drops (top public team falling from 95.11% to 85.56% Recall@1) predict that rankings unde
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a single expert-annotated resource can carry all three clinical workflows at once, and that a modest, class-imbalanced collection of endoscopy frames is enough to rank vision-language approaches in a way the authors can defend. Each ENTRep image is labeled with one of seven anatomical categories — Ear Right, Ear Left, Nose Right, Nose Left, Throat, VC-open, VC-closed — plus a normal/abnormal flag, and, where available, matched Vietnamese and English narrative descriptions; retrieval relevance is defined by ground-truth image pairings (141 training pairs, 139 per public and private test split, 71 text queries per split). The paper reports that t
Load-bearing premise
The benchmark's conclusions depend on the assumption that a single-institution image set — roughly 645 public and 646 private test images for classification, only 139 image pairs and 71 text queries per retrieval split, with expert annotations whose agreement is unreported — is large and reliable enough that the measured rankings and generalization gaps are stable and not noise; the paper itself, in its outlook, acknowledges that expanding beyond a single institution would in
Editorial extensions
If this is right
- Classification on this benchmark is nearly saturated: the top entries cluster at 94-96% accuracy on both splits, so the paper's framework implies further progress must come from the specific error modes it names — left/right laterality and vocal-fold open/closed states — rather than from generic accuracy gains.
- Image-to-image retrieval reaches 92-94% Recall@1 for the best systems, with the paper attributing early-precision gains to symmetric pairing losses, hard-negative mining, and anatomical gating of the gallery; these ingredients become the default recipe for clinical case retrieval.
- Text-to-image retrieval is the least stable task: the leading public-set entry drops about 10 points in Recall@1 on the private set, which the paper reads as sensitivity to description phrasing and coverage, pointing toward synonym normalization and stronger language encoders as the next required step.
- The small public-to-private degradation of the strongest models supports the paper's claim that the server-side protocol limits leaderboard overfitting, making ENTRep usable as a standardized comparison basis for future ENT endoscopy vision-language research.
- Anatomy-aware augmentation — label-swapping flips for left/right classes, state-preserving flips for throat and vocal folds — recurs across winning submissions, so the paper's results imply that generic augmentation schemes are a measurable liability for fine-grained anatomical classification.
Reading between the lines
- My read: with only 139 ground-truth pairs per retrieval split, Recall@1 is a high-variance statistic — a handful of pairs moving between ranks shifts the metric by about a point — so the tight leaderboard gaps (e.g., 92.09 vs 88.79 in private image retrieval) may not be robust; enlarging the paired-query pool or reporting confidence intervals would likely reorder the field. The paper does not addr
- My read: the Vietnamese half of the supervision is an untapped test asset — the tasks only query in English, so one could directly probe cross-lingual retrieval by querying in Vietnamese and checking whether the same ground-truth images are retrieved; the paper neither promises nor tests this.
- My read: the single-institution acquisition leaves device- and operator-induced domain shift unmeasured, so the benchmark's implicit promise that its rankings predict deployment would be best tested by replicating the three tasks on a second hospital's endoscopy archive of comparable size.
- My read: the instability of text-to-image rankings across splits suggests a metric design choice — treating each 71-query set as a fixed evaluation rather than sampling over paraphrase variants — and formalizing stability under paraphrase as part of the task would make the text-to-image leaderboard more meaningful.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ENTRep, the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, and positions it as the first benchmark combining fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual (Vietnamese/English) clinical supervision. The dataset consists of expert-annotated ENT endoscopic images from a single hospital, with labels for anatomical region and normal/abnormal status, plus paired bilingual descriptions. Three tasks are defined: seven-class image classification, image-to-image retrieval, and text-to-image retrieval. The paper specifies evaluation metrics (weighted accuracy/precision/recall/F1 for Task 1; Recall@1 and MRR for Tasks 2 and 3), describes public/private test splits and server-side scoring, and reports the top participating teams' results on both splits, followed by a qualitative discussion of what worked.
Significance. If adequately documented, ENTRep would fill a genuine gap: public ENT endoscopy benchmarks are scarce, and combining classification with bimodal retrieval under bilingual supervision is clinically useful. The organizers deserve credit for committing to a transparent evaluation protocol (server-side scoring, public/private splits, a unified output schema) and for reporting results from multiple independent teams, which the community can reuse. The dataset and challenge infrastructure are likely to lower the barrier for future work on ENT vision-language models. However, the current manuscript does not yet provide the statistical and methodological support needed for the 'standardized benchmark' claim: retrieval test sets are very small, no baselines are shown, and annotation reliability is unreported. The value of the resource is real, but the paper's stronger comparative claims are not yet supported.
major comments (4)
- [§4.1 and Tables 2–3] The retrieval evaluation is built on very small test sets: n=139 query images per split for Task 2 and n=71 text queries per split for Task 3. For a binary per-query outcome with n=71, the 95% Wilson interval for a Recall@1 of about 92.6% is roughly [82.7%, 98.1%]; for n=139 it is roughly ±5 percentage points. The top-team differences in Table 3 (e.g., 95.11 vs. 94.96 on public; 92.64 vs. 90.77 on private) and in Table 2 (e.g., 92.09 vs. 88.79 on private) are the same order as, or smaller than, this noise floor. Section 5 nonetheless interprets these differences as evidence of model strength and generalization gaps, and the private-test ranking determines challenge winners. Please report confidence intervals or bootstrap/McNemar-style significance tests, and explicitly state the statistical uncertainty around the leaderboard. Without this, the 'standardized comparison' claim is not suppo
- [§4.1] The training set size is never stated. The text says 'The public and private test sets closely mirror the class distributions and sample sizes of the training set (645 and 646 images, respectively),' which is ambiguous: are 645 and 646 the training sizes or the test sizes? The abstract and later text imply the test splits contain 645 and 646 images, but the exact composition of the training split is essential for reproducing the challenge and for interpreting the public-to-private generalization statements. Please provide explicit counts per split, per class, and per retrieval task, together with a data-access URL or a clear statement of where the dataset can be obtained.
- [§1 and §5 (Tables 1–3)] The introduction says the paper presents 'baseline results,' but no baselines are actually reported. Tables 1–3 list only challenge participant teams. Without a majority-class classifier, a random-retrieval baseline, a zero-shot CLIP/ViT baseline, or an organizer-run model trained on the provided training set, the reader cannot judge task difficulty or calibrate the 'tightly clustered' statements in Section 5. Adding at least simple baselines is necessary to support the benchmark's purpose of standardized comparison and to make the reported leaderboard differences interpretable.
- [§4.1] Annotation reliability is not reported. The paper states that 'Clinical experts annotate each image' and that descriptions were provided, but it does not specify the number of annotators, their experience levels, the adjudication procedure, or any inter-annotator agreement (e.g., Cohen's or Fleiss' kappa). Since classification includes fine-grained distinctions (laterality, vocal-fold open/closed) and since retrieval ground truth depends on expert-curated image pairs and text descriptions, agreement is a load-bearing quality indicator. Please report it, or explicitly justify why it is not needed for the challenge's conclusions.
minor comments (4)
- [§7] The conclusion contains a typo: 'clinically benchmark' should be 'clinical benchmark.'
- [§4.1] The text says the dataset is 'partitioned into three subsets, as illustrated in Fig. 1,' but Fig. 1 illustrates Task 1 classes. The partition diagram appears to be Fig. 4; please fix the cross-reference.
- [§4.2, Eq. (2)] The indicator function is typeset as '⊮[·]', which is nonstandard and easily confused with a negation symbol. Use \mathbb{1}[·] or bold 1. Also, N is used in Eq. (2) but defined only immediately before MRR; please make the definitions explicit for both R@1 and MRR.
- [§4.1] The word cloud in Fig. 5 is descriptive but not used in any quantitative analysis. Consider replacing it with a query-length histogram or a table of the most frequent terms, which would be more informative for retrieval tasks.
Circularity Check
No significant circularity: the paper reports a dataset and third-party challenge results, with no fitted-parameter or self-citation chain that reduces to its own inputs.
full rationale
The paper's central contribution is the ENTRep dataset and the organization of three benchmark tasks. It does not derive quantitative predictions from fitted parameters; instead, it reports leaderboard metrics submitted by independent challenge teams (Tables 1–3). The dataset statistics, evaluation metrics (Eqs. 1–2), and task definitions are standard benchmark specifications, not results obtained by fitting the data. The authors cite their own prior work (e.g., VoFoCD-related references [3,4,5,12]) only as background on the organizing team's experience and related medical-imaging efforts; these citations are not used to justify the central claim of being the first benchmark to jointly evaluate fine-grained ENT classification with intra-modal and cross-modal retrieval. The 'to the best of our knowledge' first-benchmark claim is stated directly rather than derived from a self-citation. No equation in the paper reproduces its inputs by construction, and no fitted parameter is relabeled as a prediction. Therefore, the paper is self-contained with respect to circularity concerns, and the appropriate score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Expert annotations of anatomical region and normal/abnormal status are treated as ground truth without inter-annotator agreement or adjudication details.
- domain assumption The relevance criterion for retrieval tasks is a ground-truth match to the query's annotated image(s).
- domain assumption Stratified sampling of test splits mirrors training class distributions, implying the private test is representative.
Cite this review
Pith. "Pith review of ACM Multimedia Grand Challenge on ENT Endoscopy Analysis." pith.science (2026). https://pith.science/paper/GTWN5J5G
@misc{pith2026250804801,
author = {Pith},
title = {Pith review of: ACM Multimedia Grand Challenge on ENT Endoscopy Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/GTWN5J5G}},
note = {Machine review of arXiv:2508.04801}
}
read the original abstract
Automated analysis of endoscopic imagery is a critical yet underdeveloped component of ENT (ear, nose, and throat) care, hindered by variability in devices and operators, subtle and localized findings, and fine-grained distinctions such as laterality and vocal-fold state. In addition to classification, clinicians require reliable retrieval of similar cases, both visually and through concise textual descriptions. These capabilities are rarely supported by existing public benchmarks. To this end, we introduce ENTRep, the ACM Multimedia 2025 Grand Challenge on ENT endoscopy analysis, which integrates fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual (Vietnamese and English) clinical supervision. Specifically, the dataset comprises expert-annotated images, labeled for anatomical region and normal or abnormal status, and accompanied by dual-language narrative descriptions. In addition, we define three benchmark tasks, standardize the submission protocol, and evaluate performance on public and private test splits using server-side scoring. Moreover, we report results from the top-performing teams and provide an insight discussion.
Figures
Forward citations
Cited by 1 Pith paper
-
Multi-Level CLS Token Fusion for Contrastive Learning in Endoscopy Image Classification
A multi-task CLIP model with LoRA, multi-level CLS fusion, and spherical feature interpolation reports 95% accuracy and strong retrieval scores on the ENTRep endoscopy benchmark.
Reference graph
Works this paper leans on
-
[1]
Rohaid Ali, Oliver Y Tang, Ian D Connolly, Hael F Abdulrazeq, Fatima N Mirza, Rachel K Lim, Benjamin R Johnston, Michael W Groff, Theresa Williamson, Konstantina Svokos, et al. 2024. Demographic representation in 3 leading artificial intelligence text-to-image generators. JAMA surgery 159, 1 (2024), 87–95
work page 2024
-
[2]
Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, and Zhiwei Xiong. 2024. Bimcv-r: A landmark dataset for 3d ct text-image retrieval. In International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 124–134
work page 2024
-
[3]
Thao Thi Phuong Dao, Tuan-Luc Huynh, Minh-Khoi Pham, Trung-Nghia Le, Tan- Cong Nguyen, Quang-Thuc Nguyen, Bich Anh Tran, Boi Ngoc Van, Chanh Cong Ha, and Minh-Triet Tran. 2024. Improving Laryngoscopy Image Analysis Through Integration of Global Information and Local Features in VoFoCD Dataset.Journal of Imaging Informatics in Medicine 37, 6 (2024), 2794–2809
work page 2024
-
[4]
Thao Thi Phuong Dao, Tan-Cong Nguyen, Viet-Tham Huynh, Xuan-Hai Bui, Trung-Nghia Le, and Minh-Triet Tran. 2024. Artificial intelligence for laryn- goscopy in vocal fold diseases: a review of dataset, technology, and ethics.Machine Learning 113, 11 (2024), 8749–8767
work page 2024
-
[5]
Thao Thi Phuong Dao, Minh-Khoi Pham, Mai-Khiem Tran, Chanh Cong Ha, Boi Ngoc Van, Bich Anh Tran, and Minh-Triet Tran. 2024. Vision-Based Assistance for Vocal Fold Identification in Laryngoscopy with Knowledge Distillation. In MEDINFO 2023—The Future Is Accessible . IOS Press, 946–950
work page 2024
-
[6]
Emre Demir, Burak Numan Uğurlu, Gülay Aktar Uğurlu, and Gülçin Aydoğdu
-
[7]
Brian Hu, Bhavan Vasu, and Anthony Hoogs. 2022. X-mir: Explainable medical image retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 440–450
work page 2022
-
[8]
Viet-Tham Huynh, Trong-Thuan Nguyen, Thao Thi-Phuong Dao, Tam V Nguyen, and Minh-Triet Tran. 2024. DermAI: A Chatbot Assistant for Skin Lesion Di- agnosis Using Vision and Large Language Models. In Proceedings of the Asian Conference on Computer Vision . 287–301
work page 2024
Show all 15 references
-
[9]
Hai Le, Trong-Thuan Nguyen, and Minh-Triet Tran. 2024. MedGraph-RPE: Graph- Based Medical Segmentation Enhanced by Novel Relative Positioning Encod- ing. In International Symposium on Information and Communication Technology . Springer, 27–42
2024
-
[10]
Trong-Thuan Nguyen, Tam V Nguyen, and Minh-Triet Tran. 2023. Collaborative consultation doctors model: Unifying cnn and vit for covid-19 diagnostic. IEEE Access 11 (2023), 95346–95357
2023
-
[11]
Kim Anh Phung, Thuan Trong Nguyen, Nileshkumar Wangad, Samah Baraheem, Nguyen D Vo, and Khang Nguyen. 2022. Disease recognition in X-ray images with doctor consultation-inspired model. Journal of Imaging 8, 12 (2022), 323
2022
-
[12]
Bich Anh Tran, Thao Thi Phuong Dao, Ho Dang Quy Dung, Ngoc Boi Van, Chanh Cong Ha, Nam Hoang Pham, Tu Cong Huyen Ton Nu Cam, Tan-Cong Nguyen, Minh-Khoi Pham, Mai-Khiem Tran, et al. 2023. Support of deep learn- ing to classify vocal fold images in flexible laryngoscopy. America...
2023
-
[13]
Trong-Nghia Tran, Minh-Nhat Nguyen, Minh-Triet Tran, and Trong-Le Do. 2025. Enhancing Medical Chatbot Reliability: A Multi-Step Verification Approach to Prevent Hallucinations. In Proceedings of the 2nd Workshop on Security-Centric Strategies for Combating Information Disorder . 1–12
2025
-
[14]
Qingwu Wu, Xinyue Wang, Guixian Liang, Xin Luo, Min Zhou, Huiyi Deng, Yana Zhang, Xuekun Huang, and Qintai Yang. 2023. Advances in image-based artificial intelligence in otorhinolaryngology–head and neck surgery: a systematic review. Otolaryngology–Head and Neck Surgery 169, 5...
2023
-
[2025]
European Archives of Oto-Rhino-Laryngology (2025), 1–11
Artificial intelligence in otorhinolaryngology: current trends and applica- tion areas. European Archives of Oto-Rhino-Laryngology (2025), 1–11
2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.