Pith. sign in

REVIEW 5 major objections 6 minor 20 references

Lesion-Aware Visual-Language Fusion for Automated Image Captioning of Ulcerative Colitis Endoscopic Examinations

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that feeding Grad-CAM lesion heatmaps and structured clinical prompts into a ResNet-CBAM-T5 pipeline improves both ulcerative colitis captioning and MES severity classification over the prior state of the art on the same d

desk verdict Genuine but unverified incremental gain in UC endoscopic captioning, with an inflated statistical claim that needs per-sample baseline data. read the letter →

arxiv 2509.03011 v1 pith:SPW4QANK submitted 2025-09-03 cs.CV cs.AI

Alexis Ivan Lopez Escamilla , Gilberto Ochoa , Sharib Al This is my paper · ORCID
classification cs.CVcs.AI
keywords ulcerativecolitisimagecaptioningGrad-CAMCBAMT5lesion-awareattentionMESclassificationvisual-languagefusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that an automated ulcerative colitis endoscopy captioner produces better reports when it is explicitly made lesion-aware: Grad-CAM heatmaps from a severity-classification branch are multiplied into the visual features, CBAM reweights those features, and structured clinical metadata is fed to a T5 decoder as a prompt. On 2,355 annotated endoscopic images, it reports 84.7% MES classification accuracy, BLEU-4 of 0.87, and ROUGE-L of 0.85, versus 77.8%, 0.77, and 0.72 for the previous system on the same dataset, with a paired-bootstrap p-value below 0.01. If true, the practical payoff is a single pipeline that writes structured, interpretable endoscopy reports and grades disease severity at the same time, with built-in visual explanation of where the model is looking. The qualitative examples show the model sometimes names findings—friability, erythema, superficial ulcers—that are absent from the expert reference captions, which the paper reads as evidence that spatial lesion grounding extracts relevant detail beyond the original annotations.

What carries the argument

The mechanism that carries the argument is the Grad-CAM-modulated feature map used as a spatial gate: F' = CBAM(F) ⊙ (1 + αM), where M is the upsampled, normalized Grad-CAM heatmap, α is a learnable scalar, and ⊙ is elementwise multiplication. This turns a post hoc explanation tool into an explicit attention mask during training and inference, so lesion-relevant pixels are amplified before the visual features enter the language decoder. The CBAM module (a Convolutional Block Attention Module) adds channel and spatial attention; the T5 decoder (a pretrained text-to-text transformer) adds cross-attention over the projected visual features; and the clinical metadata prompt conditions the decode

What would settle it

Replace the Grad-CAM heatmap with a uniform spatial mask of the same smoothness, keeping all other components fixed; if MES accuracy stays at 84.7% and BLEU-4/ROUGE-L stay at 0.87/0.85, the lesion-specific content of the heatmap is not what produces the gain. Separately, re-running the prior baseline on the exact test split and references used here would reveal whether the margin is method or evaluation setup.

Watch

Extended reading notes

Core claim

The central claim is that spatial lesion grounding is not just an explanation layer but a training signal that improves both the vision and language sides of the model. The architecture passes the endoscopic image through two ResNet-50 branches; one produces the MES severity class and its Grad-CAM heatmap, the other extracts features refined by CBAM. Those features are modulated by the Grad-CAM map through F' = CBAM(F) ⊙ (1 + αM), so regions the classifier treats as pathological get amplified before projection into the T5 decoder. The decoder also receives clinical metadata (MES grade, vascular pattern, bleeding, erythema, friability, ulceration) as a natural-language prompt, and training co

Load-bearing premise

The load-bearing assumption is that the comparison with the prior baseline is fair: the same test split, the same reference captions, and the same MES labels were used for both systems.

Editorial extensions

If this is right

  • If the claim holds, removing any of the three designed components—CBAM, Grad-CAM spatial supervision, or clinical prompts—should cost measurable performance, exactly as the ablation table shows.
  • A single model can replace separate classification and reporting tools: the same forward pass yields MES grade, structured lesion tags, a free-text caption, and a Grad-CAM heatmap for clinicians to inspect.
  • Because generated captions sometimes contain findings absent from reference captions, lesion-aware captioning may surface clinically relevant details that human annotation workflows miss or leave implicit.
  • At roughly 1.45 seconds per image on a modest GPU, the pipeline is near the threshold for batch or near-real-time use in endoscopic reporting workflows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 'classifier heatmap as a training mask' recipe is a natural transfer to other tasks with small, localized lesions and ordinal severity labels, such as Crohn's disease endoscopic indices or polyp surveillance; the paper does not test this.
  • The metric question is left open: if generated captions add clinically valid findings not in the reference, BLEU and ROUGE will systematically under-reward them, so a clinician-preference study would be the sharper test of usefulness.
  • The global learned scalar α could be made spatial or per-channel to let lesion emphasis vary by severity; the paper's fixed global scaling is one simple choice among several possible designs.
  • The dual-loss setup presumes MES classification and captioning share visual features; an experiment that removes the classification branch and compares attention maps would reveal how much of the benefit is from joint training rather than from the Grad-CAM mask.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a lesion-aware image captioning framework for ulcerative colitis endoscopy. The architecture uses a dual-branch ResNet-50: one branch performs MES classification and produces Grad-CAM heatmaps, while the other extracts visual features that are refined by CBAM and multiplied by a Grad-CAM-derived spatial mask. A T5 decoder with visual cross-attention generates captions, optionally conditioned on clinical metadata formatted as natural-language prompts. The authors report MES classification accuracy of 84.7%, BLEU-4 of 0.87, and ROUGE-L of 0.85, compared with 77.8%, 0.77, and 0.72 for Valencia et al. (2023), and claim p<0.01 from a paired bootstrap. They also present ablations removing CBAM, Grad-CAM, clinical prompts, and attention fusion, as well as qualitative examples across MES grades.

Significance. If the reported results are reproducible, the combination of Grad-CAM spatial supervision, CBAM feature enhancement, and clinical prompt conditioning is a plausible, practical contribution to automated endoscopic reporting. The paper includes a useful ablation matrix and reports concrete quantitative gains. However, the central comparison with the prior state of the art is not currently verifiable: the paired-bootstrap significance claim cannot be supported by the aggregate baseline statistics quoted in Table 1, and the test-split comparability with Valencia et al. is not established. The absence of training and metric-computation details further limits reproducibility. The paper's contribution is therefore promising but not yet demonstrated to the standard required for acceptance.

major comments (5)
  1. [§4.1] The paired bootstrap significance claim is unsupported by the evidence presented. A paired bootstrap with 1,000 iterations requires per-sample outputs from both models on the identical test items. The manuscript quotes only aggregate baseline numbers from Valencia et al. (77.8%, 0.77, 0.72) and does not state that the baseline was re-run on the same 15% split. As written, the reported p<0.01 cannot be computed from the information in the paper. Please provide either (a) per-sample outputs from a re-implemented baseline evaluated on the exact same test split, with the bootstrap procedure fully specified, or (b) revise the claim to a non-statistical comparison.
  2. [§3.1 and Table 1] The fair-comparison assumption is not established. The dataset is described as introduced in [15] and split 70/15/15 with MES balance, but the paper does not state that this test split matches the evaluation split used by Valencia et al., nor that the caption references and MES labels are identical. If the split or preprocessing differs, the reported improvements in Table 1 may reflect evaluation setup rather than the method. Please specify the exact split indices or a formal justification that the splits coincide, and wherever possible re-run the baseline under the same protocol.
  3. [§3.4–3.5] Reproducibility details are insufficient for the central experiments. The paper does not specify the exact T5-base configuration, optimizer, learning rate schedule, batch size, number of training steps, warmup, max sequence length, or random seeds. The value λ=0.2 in Eq. (2) is said to be 'selected through validation' but no search range or criterion is given. These details are needed to assess whether the reported metrics are stable and to allow independent replication.
  4. [§3.4 and §4.1] There is a potential circularity involving MES labels. The clinical prompt is exemplified as 'MES-2; bleeding: yes', and MES classification accuracy is reported as a headline result. If the ground-truth MES label is used to form the prompt at test time, the 84.7% accuracy is not an independent model prediction. Please clarify whether the prompts are formed from ground-truth labels or from the model's own classifier, and evaluate the setting where prompts are predicted rather than oracle-provided.
  5. [§4.2] A 'T5-based post-processor' or 'grammar-aware refinement' stage is introduced only in the qualitative results and is not described in the methodology. It is unclear whether the BLEU-4 and ROUGE-L scores in Table 1 are computed on the raw model outputs or on the refined captions. If the refined outputs are scored, this post-processing component must be part of the method description and ablated separately; if it is not scored, its role in the reported numbers should be stated.
minor comments (6)
  1. [Eq. (1)] The dimensions in F' = CBAM(F) ⊙ (1 + αM) are inconsistent as written: M is R^{1×H×W} and F is R^{C×H×W}. Please clarify how the Grad-CAM mask is broadcast or replicated across channels.
  2. [§3.6] The metrics 'alignment score' and 'token-level precision' are mentioned but never defined or reported. Either define them and include results, or remove the mention.
  3. [§3.4] The phrase 'modified T5 decoder incorporated visual cross-attention at each decoding layer' is vague. T5 already has cross-attention to encoder outputs; please specify how the visual features are injected relative to the prompt tokens, and whether the encoder also receives text or is bypassed.
  4. [§4.3] Ablation results are presented without error bars or significance tests. Given that the headline comparison uses p<0.01, the ablations should be accompanied by the same kind of statistical assessment or at least a statement of variance across seeds.
  5. [§3.1] The MES balance of the split is asserted but not quantified. Please report class frequencies in train/validation/test and state whether the balance was stratified by MES or by patient.
  6. [References] Several references are incomplete, e.g., 'Valencia, H.e.a.' and 'Jiang, X.e.a.' lack full author lists and venue details, making it difficult for readers to locate the baseline work.

Circularity Check

1 steps flagged · score 6.0 of 10

Captioning gains are partially circular because the model is prompted with the ground-truth MES label and then evaluated on caption quality.

  1. self definitional [Section 3.4 ('Language Decoder and Cross-Attention Integration') and Section 4.1 ('Quantitative Results')]
    "We experimented with conditioning the T5 encoder on structured metadata (e.g., MES-2; bleeding: yes; friability: moderate) formatted as a prompt. ... Structured clinical prompts (e.g., 'MES-2; bleeding: yes') guide T5 toward diagnostic phrasing."

    The captioning model is given the ground-truth MES score and associated findings as prompt tokens. The reference captions and BLEU/ROUGE metrics reward the same label-derived terminology. Therefore the reported captioning improvement over Valencia et al. is partly forced by construction: the model is told the diagnostic label it is supposed to describe, rather than predicting it from the image. The MES classification accuracy (84.7%) is measured on a separate branch and is not affected, so the circularity is partial rather than total.

full rationale

The paper's classification claim (84.7% vs 77.8% MES accuracy) is an independent evaluation on a held-out split and uses a ResNet-50 trained with cross-entropy; that part is not circular. However, the captioning comparison is confounded: the T5 decoder receives structured clinical metadata—including the MES score and findings (bleeding, erythema, etc.)—as natural-language prompts, and the human reference captions describe exactly those findings. Thus the BLEU-4/ROUGE-L gains partly reflect the model echoing the supplied label, not a learned image-to-text mapping. The paper never states whether prompts come from the classifier's prediction or from ground-truth annotations; the qualitative examples are organized by known MES grades, indicating oracle prompts. This is a self-definitional step: the output caption is conditioned on the target variable it is evaluated against. The p<0.01 paired-bootstrap claim cannot be checked from aggregate baseline numbers, but that is a reporting/statistical issue rather than circularity. Ablations show attention fusion also contributes independently, so the central method retains some independent content. Score 6 reflects one prediction (captioning) partially reducing by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new theoretical entities or parameters beyond standard architecture components. The free parameters are the loss weight lambda and the learnable alpha. The axioms are all domain assumptions about the dataset, metrics, and evaluation comparability. No new particles, forces, or entities are postulated.

free parameters (2)
  • lambda (loss weighting) = 0.2
    The weight balancing caption loss and MES classification loss is selected through validation, as stated in Section 3.5. It is a hyperparameter chosen on data, influencing the reported accuracy/fluency trade-off.
  • alpha (learnable scalar in feature enhancement) = not reported
    Section 3.3 defines the enhanced features as F' = CBAM(F) * (1 + alpha * M) with alpha a learnable scalar. Its fitted value is not reported; it is learned from data.
assumptions (4)
  • domain assumption The dataset from [15] is representative and its MES labels and caption annotations are reliable ground truth.
    Section 3.1 states the dataset was introduced in [15] and validated by experts. All reported metrics are computed against these labels, so the entire evaluation depends on this.
  • domain assumption BLEU-4, ROUGE-L, and MES accuracy are valid and meaningful measures for evaluating clinical caption quality.
    Section 3.6 lists these metrics without justifying their clinical adequacy. They are standard text metrics but may not capture clinical correctness.
  • domain assumption The test split used by the authors is comparable to the evaluation split used by Valencia et al. [15].
    Section 3.1 describes the split but does not show it matches the baseline's split. The state-of-the-art comparison in Section 4.1 depends on this assumption.
  • domain assumption Grad-CAM heatmaps from a ResNet-50 trained for MES classification provide useful spatial supervision for captioning.
    This is the core design premise, described in Section 3.2 and used in Section 3.3. It is plausible but not independently established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lesion-Aware Visual-Language Fusion for Automated Image Captioning of Ulcerative Colitis Endoscopic Examinations." pith.science (2026). https://pith.science/paper/SPW4QANK

@misc{pith2026250903011,
  author       = {Pith},
  title        = {Pith review of: Lesion-Aware Visual-Language Fusion for Automated Image Captioning of Ulcerative Colitis Endoscopic Examinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPW4QANK}},
  note         = {Machine review of arXiv:2509.03011}
}
read the original abstract

We present a lesion-aware image captioning framework for ulcerative colitis (UC). The model integrates ResNet embeddings, Grad-CAM heatmaps, and CBAM-enhanced attention with a T5 decoder. Clinical metadata (MES score 0-3, vascular pattern, bleeding, erythema, friability, ulceration) is injected as natural-language prompts to guide caption generation. The system produces structured, interpretable descriptions aligned with clinical practice and provides MES classification and lesion tags. Compared with baselines, our approach improves caption quality and MES classification accuracy, supporting reliable endoscopic reporting.

Figures

Figures reproduced from arXiv: 2509.03011 by the authors.

Figure 1
Figure 1. Clinical Workflow for Lesion-Aware Image Captioning. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Top level architecture for Lesion-Aware Image Captioning. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qualitative examples of lesion-aware caption generation across MES [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

  1. [15]

    In: Proceedings of the MICCAI (2023)

    Valencia, H.e.a.: Lesion-aware image captioning and mes prediction in ulcerative colitis. In: Proceedings of the MICCAI (2023)

    work page 2023

  2. [1]

    Abacha, A.B., Shivade, C., Hasan, S.A., Datla, V., Liu, J.D., Demner-Fushman, D.: Vqa-med: Overview of the medical visual question answering task at imageclef

    work page

  3. [2]

    European Journal of Gastroenterology & Hepatology (2021) 10 López Escamilla et al

    Bhambhvani, H., Zamora, A.: Deep learning model for automated mayo endoscopic subscore classification. European Journal of Gastroenterology & Hepatology (2021) 10 López Escamilla et al

    work page 2021

  4. [3]

    Proceedings of the Machine Learning for Healthcare Conference pp

    Boag, W., Lou, J., Zech, J.R., Hughes, M.C., McDermott, M.B.A., Ghassemi, M.: Baselines for chest x-ray report generation. Proceedings of the Machine Learning for Healthcare Conference pp. 371–385 (2020)

    work page 2020

  5. [4]

    In: European Conference on Computer Vision (ECCV) (2020)

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. In: European Conference on Computer Vision (ECCV) (2020)

    2020

  6. [5]

    Medical Image Analysis65, 101786 (2020)

    Chen, X., Zhang, Z., Kalpathy-Cramer, J., Xing, E.: Generating radiology reports via memory-driven transformer. Medical Image Analysis65, 101786 (2020)

    work page 2020

  7. [6]

    Neurocomputing378, 303–312 (2020)

    Huang, X., Wang, R., Xu, L., Gao, Z.: Biomedical image captioning with multi- level attention. Neurocomputing378, 303–312 (2020)

    work page 2020

  8. [7]

    Journal of Gastroenterology (2022)

    Jiang, X.e.a.: Deep learning-based endoscopic assessment of ulcerative colitis sever- ity. Journal of Gastroenterology (2022)

    work page 2022

Show all 20 references
  1. [8]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)

    Jing,B.,Xie,P.,Xing,E.:Show,describeandconclude:Onexploitingthestructure information of chest x-ray reports. In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)

    2019

  2. [9]

    Medical Image Analysis73, 102196 (2022)

    Li, X., Zhang, Z., Huang, X., Zhao, Q., Metaxas, D.N.: Medical visual question answering via modular co-attention learning. Medical Image Analysis73, 102196 (2022)

    2022

  3. [10]

    arXiv preprint arXiv:1904.02633 (2019)

    Liu, F., Jin, D., Liu, T., Yu, H.: Clinically accurate chest x-ray report generation. arXiv preprint arXiv:1904.02633 (2019)

    1904 arXiv

  4. [11]

    Medical Image Analysis (2022)

    Lo, C.e.a.: Automatic grading of ulcerative colitis from endoscopic images using efficientnet. Medical Image Analysis (2022)

    2022

  5. [12]

    Journal of Machine Learning Research21(140), 1–67 (2020)

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research21(140), 1–67 (2020)

    2020

  6. [13]

    In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

    2017

  7. [14]

    Shin, H.C., Roberts, K., Lu, L., Demner-Fushman, D., Yao, J., Summers, R.M.: Learningtoreadchestx-rays:Recurrentneuralcascademodelforautomatedimage annotation (2016)

    2016

  8. [16]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    2015

  9. [17]

    In: International Conference on Machine Learning (ICML) (2015)

    Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: International Conference on Machine Learning (ICML) (2015)

    2015

  10. [18]

    Advances in Neural Information Processing Systems35 (2022)

    Yuan, Y., Chen, X., Wang, J.: Hrformer: High-resolution vision transformer for dense predictive tasks. Advances in Neural Information Processing Systems35 (2022)

    2022

  11. [19]

    In: Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2021)

    Zhang, Z., Liu, T., Chen, Y.: Hierarchical medical image understanding through multi-scale and multi-modal transformer networks. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2021)

    2021

  12. [2019]

    CEUR Workshop Proceedings (2019)

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.