REVIEW 3 major objections 3 minor 16 references
Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read PathGenIC claims top HistGen report scores by adding retrieved examples, guidelines, and feedback to a vision-language model.
desk verdict Clear writing and a sensible ablation, but the SOTA claim depends on an evaluation that gives PathGenIC context no baseline gets; the measured gains are small and unquantified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is multimodal in-context learning with three context signals: nearest-neighbor retrieval, where the test WSI's contextualized tokens are concatenated with the tokens of the most similar training WSI and its ground-truth report; category guidelines, where GPT-4o summarizes report-writing habits from the top-K nearest disease category; and feedback, where GPT-4o critiques base-model drafts against truth and the critiques of the K nearest training cases are fed in. These context cues are injected through the text prompt and token stream of a LoRA-adapted Quilt-LLaVA, whose visual front end is HistGen's DINOv2 ViT-L with learnable query tokens.
What would settle it
Run the Table 1 comparison again but append the same top-1 nearest-neighbor ground-truth report to the prompt of every baseline, including HistGen and the base model, while keeping all other settings fixed; if PathGenIC's BLEU-4 advantage over HistGen shrinks to roughly zero, the state-of-the-art claim is explained by the added retrieval context rather than by the method itself.
Extended reading notes
Core claim
The central claim is that in-context learning over the training set, rather than a larger architecture or more parameters, is what lifts report generation quality. Starting from Quilt-LLaVA, the authors replace its visual encoder with HistGen's DINOv2 ViT-L, add learnable query tokens and transformer blocks to make fixed-length WSI tokens, and LoRA-tune the VLM. On top of this base, three context signals are appended: the image tokens and report of the most cosine-similar training WSI, a GPT-4o-generated guideline distilled from up to 20 reports in the nearest disease category, and GPT-4o feedback that compares base-generated training reports against ground truth. Each component alone helps; together they produce the reported state-of-the-art numbers, and the paper shows the gain is not a quirk of report length or disease category.
Load-bearing premise
The comparison is only meaningful if giving PathGenIC retrieved example reports, GPT-4o disease guidelines, and GPT-4o feedback at test time does not hand it an advantage that the comparison models were never offered.
Editorial extensions
If this is right
- If the reported numbers hold, any report generation system on HistGen can be improved by retrieving similar cases at test time without retraining the visual encoder.
- The K=3 retrieval setting gives the best tradeoff, so the method is robust to the number of retrieved neighbors rather than requiring exact tuning.
- The per-category analysis suggests the gains are consistent across disease types, so the method is not riding on one easy subset.
- A direct corollary of the ablations is that each context signal contributes incrementally, with the full combination giving the largest improvement.
- Because the base model already outperforms HistGen, part of the state-of-the-art result comes from the improved backbone, while the in-context components provide the additional gains in Table 3.
Reading between the lines
- Editorial inference: The reported comparison is not apples-to-apples, because PathGenIC receives GPT-4o-generated text at test time while the baselines receive only the test image; a control that gives every baseline the same retrieved report would determine how much of the gain comes from retrieval versus from GPT-4o's prompt-writing ability.
- Editorial inference: The same recipe could transfer to other dense-report domains such as radiology, dermatopathology, or surgical pathology notes, provided a retrieval index and a source of supervisory feedback exist.
- Editorial inference: The category guideline prompt asks GPT-4o to expose report-writing biases, which suggests the method could amplify systematic wording or coding biases present in a hospital's historical reports; measuring entity-level calibration rather than only BLEU overlap would test this.
- Editorial inference: Because evaluation truncates reports at 100 tokens, the metric gains may not reflect full-report clinical completeness; a clinician-rated study on full-length reports would be the natural next test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PathGenIC, a framework for histopathology report generation from whole slide images. The method builds on Quilt-LLaVA with HistGen's DINOv2 ViT-L feature extractor and introduces three in-context learning components: nearest-neighbor retrieval of WSI-report pairs, GPT-4o-generated category guidelines, and GPT-4o-generated corrective feedback derived by comparing base-model outputs against ground-truth training reports. The framework is evaluated on the HistGen benchmark, and the paper claims state-of-the-art results, reporting BLEU-1 0.431, BLEU-4 0.196, METEOR 0.197, ROUGE-L 0.357, and factENT 0.462 for the full PathGenIC model, with ablations over components, the number of retrieved neighbors, report length, and disease categories.
Significance. If the reported results are robust, the paper makes a useful empirical contribution: it demonstrates that retrieval-based in-context learning, category-level guidelines, and feedback derived from training-set comparisons can improve vision-language report generation on whole slide images. The clinical motivation is well grounded, the method description is clear, and the appendices provide the exact prompts used. The ablations and the analysis across disease categories are also valuable. However, the core claim of state-of-the-art performance is weakened by an asymmetric evaluation protocol and the absence of statistical significance measures. PathGenIC is the only method receiving retrieved reports, GPT-4o-generated guidelines, and feedback derived from ground-truth comparisons, so the reported gains may partly reflect additional test-time information rather than the method itself. The improvement margins over HistGen are small, and no error bars or significance tests are reported. These issues prevent the paper from currently supporting its strongest claim, but they are addressable with additional controls and analysis.
major comments (3)
- [§3.2, Eq. (5), Table 1] The comparison in Table 1 is not apples-to-apples. PathGenIC is the only system that receives (i) retrieved WSI tokens and their associated reports, (ii) GPT-4o-generated category guidelines, and (iii) GPT-4o feedback produced by comparing the base model's output with the ground-truth training reports (Eq. (5)). None of the baselines is given an equivalent retrieval/reference/feedback prompt, so the reported improvements (BLEU-1 0.431 vs 0.413, BLEU-4 0.196 vs 0.184) may come from the extra test-time information rather than from the ICL mechanism. In particular, feedback B_j is derived from ground-truth reports of the retrieved training WSIs and can state which entities the generated report lacks; exposing this to the VLM at test time is a form of privileged information. I do not claim mathematical circularity, since no test ground truth is used, but the 'state-of-the-art' claim is only valid under this asymmetric protocol. Please add controls where the same retrieved reports, guidelines, and feedback are provided to the baselines (e.g., HistGen with identical ICL context), and include a random-retrieval condition to quantify the contribution of retrieval quality.
- [§4.1, §4.2, Table 1 and Table 3] No measure of variability is reported. All numbers appear to be from single runs; the margins over HistGen are small (BLEU-1 +0.018, BLEU-4 +0.012, METEOR +0.015, ROUGE-L +0.013), and Table 3 shows that adding the guideline and feedback components to nearest-neighbor retrieval changes BLEU-1 by 0.003 and ROUGE-L by 0.003. Without multiple seeds, bootstrap confidence intervals, or significance tests, the claim of 'significant improvements across BLEU, METEOR, and ROUGE-L' in the abstract is unsupported. Additionally, factENT is given only for the two proposed models, with the text noting that values for other methods are 'not available' in §4.2, so the factual-completeness advantage cannot be verified. Please provide standard deviations or significance tests and either obtain factENT for the baselines or remove the implied comparative claim.
- [§4.2, Figure 3 and Table 1] The paper states that the results in Table 1 are obtained 'based on only the first 100 tokens of the generated reports.' It is unclear whether this truncation was applied to the reported baseline numbers taken from HistGen or only to the proposed models. If the baselines were evaluated on full generated reports, the comparison is biased because BLEU, METEOR, and ROUGE-L are sensitive to report length. Please clarify explicitly whether the same truncation was applied to every method, and ideally report results on full generated reports as well.
minor comments (3)
- [§3.2 and Appendix A] The category guideline procedure is described as using the K most similar WSIs to determine the majority category, but the prompt in Appendix A lists 'Report 1 ... Report 20'. The relationship between K and the number of reports used to generate the guideline is not described; please clarify how the 20 reports are selected.
- [Figure 4] The bar plot in Figure 4 shows BLEU scores across 32 disease categories, but no error bars or sample sizes per category are provided, which makes it hard to determine which cross-category differences are meaningful.
- [References and Reproducibility] The motivation cites 'Doe and Smith, 2023' with a webpage reference; a more standard peer-reviewed source on pathology peer review would strengthen the introduction. In addition, the paper does not report the GPT-4o sampling parameters or the computational cost of the feedback and guideline generation, which would aid reproducibility.
Circularity Check
No significant circularity: PathGenIC's test-time context comes from training data only, so the SOTA claim may be an evaluation-fairness concern but not a self-referential derivation.
full rationale
The paper's derivation chain is: (1) take Quilt-LLaVA with HistGen's frozen ViT-L; (2) fine-tune transformer/projector/LoRA on HistGen training reports with cross-entropy; (3) at test time retrieve the nearest training WSI-report pair by cosine similarity to H_test (Eq. 3), gather GPT-4o category guidelines from training reports (Eq. 4), and retrieve GPT-4o feedback generated from base-model outputs and ground-truth reports on training WSIs (Eq. 5). None of these inputs uses the test ground-truth report: retrieval is H_test vs H_train, guidelines are from category C of nearest training WSIs, and feedback B_i is precomputed on training data. Thus the generated test report is not definitionally forced by the target; the prediction is not a fitted input renamed as a prediction. The paper includes no uniqueness theorem and no load-bearing self-citation; all cited methods (HistGen, Quilt-LLaVA, R2Gen, etc.) are external. The only flagged weaknesses are in the evaluation protocol: Table 1 reports factENT only for the proposed methods ('the values of other methods are not available'), and baselines are not given the retrieved prompts; these affect whether the SOTA claim is an apples-to-apples comparison, but they do not make the derivation circular. Section 4.3's admission that key entities are still missed is a limitation statement, not a circular step. Verdict: no significant circularity.
Assumptions & free parameters
free parameters (4)
- Number of query tokens m =
not stated
- K for nearest-neighbor retrieval =
3
- Training hyperparameters =
batch size 8, 20 epochs, lr 1e-4, cosine schedule
- Report length cutoff =
first 100 tokens
assumptions (3)
- domain assumption The retrieved nearest-neighbor reports are genuinely informative for generating the target report.
- domain assumption GPT-4o generated guidelines and feedback are accurate and transferable to the target WSI.
- domain assumption Lexical overlap metrics (BLEU, METEOR, ROUGE-L) correlate with report quality on this task.
invented entities (1)
-
None
Cite this review
Pith. "Pith review of Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning." pith.science (2026). https://pith.science/paper/2XZFJ5DN
@misc{pith2026250617645,
author = {Pith},
title = {Pith review of: Histopathology Image Report Generation by Vision Language Model with Multimodal In-Context Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/2XZFJ5DN}},
note = {Machine review of arXiv:2506.17645}
}
read the original abstract
Automating medical report generation from histopathology images is a critical challenge requiring effective visual representations and domain-specific knowledge. Inspired by the common practices of human experts, we propose an in-context learning framework called PathGenIC that integrates context derived from the training set with a multimodal in-context learning (ICL) mechanism. Our method dynamically retrieves semantically similar whole slide image (WSI)-report pairs and incorporates adaptive feedback to enhance contextual relevance and generation quality. Evaluated on the HistGen benchmark, the framework achieves state-of-the-art results, with significant improvements across BLEU, METEOR, and ROUGE-L metrics, and demonstrates robustness across diverse report lengths and disease categories. By maximizing training data utility and bridging vision and language with ICL, our work offers a solution for AI-driven histopathology reporting, setting a strong foundation for future advancements in multimodal clinical applications.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Bottom-up and top-down attention for image captioning and visual question answering
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6077--6086, 2017
work page 2017
-
[2]
Improving diagnostic accuracy through feedback: The diagnosis learning cycle
Carolina Fernandez Branson, Michelle Williams, Teresa M Chan, Mark L Graber, Kathleen P Lane, Skip Grieser, Zach Landis-Lewis, James Cooke, Divvy K Upadhyay, Shawn Mondoux, Hardeep Singh, Laura Zwaan, Charles Friedman, and Andrew P J Olson. Improving diagnostic accuracy through feedback: The diagnosis learning cycle. BMJ Quality & Safety, 30 0 (12): 0 100...
work page 2021
-
[3]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...
work page 1901
-
[4]
Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images
Pingyi Chen, Honglin Li, Chenglu Zhu, Sunyi Zheng, Zhongyi Shui, and Lin Yang. Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images. In Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention, pages 546--556, 2024
work page 2024
-
[5]
Generating radiology reports via memory-driven transformer
Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. In Proceedings of Conference on Empirical Methods in Natural Language Processing, pages 1439--1449, 2020
work page 2020
-
[6]
Cross-modal memory networks for radiology report generation
Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. Cross-modal memory networks for radiology report generation. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pages 5904--5914, 2021
work page 2021
-
[7]
Meshed-memory transformer for image captioning
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578--10587, 2020
work page 2020
-
[8]
Anatomic pathology quality assurance through peer review
John Doe and Alice Smith. Anatomic pathology quality assurance through peer review. PathologyOutlines, 2023. https://www.pathologyoutlines.com/topic/managementlabAPQA.html
work page 2023
Show all 16 references
-
[9]
Histgen: A local-global encoding framework for pathology report generation
Zhengrui Guo, Jiabo Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, and Hao Chen. Histgen: A local-global encoding framework for pathology report generation. In Proceedings of Medical Image Computing and Computer Assisted Intervention, pages 189--199, 2024
2024
-
[10]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of International Conference on Learning Representations, 2022
2022
-
[11]
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of Con...
2020
-
[12]
Llava-med: Training a large language-and-vision assistant for biomedicine in one day
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Proceedings of Conference on Neural Information Processing ...
2023
-
[13]
Improving factual completeness and consistency of image-to-text radiology report generation
Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. Improving factual completeness and consistency of image-to-text radiology report generation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguist...
2021
-
[14]
Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro
Mehmet Saygin Seyfioglu, Wisdom O. Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. arXiv:2312.04746v3, 2025
2025 arXiv
-
[15]
Gomez, Lukasz Kaiser, and Illia Polosukhin
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Conference on Neural Information Processing Systems, 2017
2017
-
[16]
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3156--3164, 2015
2015
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.