ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images

Jongseong Jang; Sangwook Kim; Soonyoung Lee

REVIEW 4 major objections 4 minor 19 references

ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The authors claim expert-level diagnosis from whole-slide pathology images, with 62.9 percent of answers accepted by an AI evaluator.

desk verdict Useful data pipeline, but the headline acceptance rate rests on an evaluator the authors themselves admit may be untrustworthy—and the published evaluator prompt contradicts the claimed CoT reasoning. read the letter →

arxiv 2504.13023 v1 pith:S2VFMCJU submitted 2025-04-17 cs.CL cs.CV

Sangwook Kim , Soonyoung Lee , Jongseong Jang This is my paper · ORCID

classification cs.CLcs.CV

keywords ChatEXAONEPathmultimodallargelanguagemodelhistopathologywholeslideimagespan-cancerretrieval-augmentedgenerationAI-basedevaluationTCGA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ChatEXAONEPath, a multimodal large language model that takes a whole-slide histopathology image and answers text questions about the diagnosis. The authors show that a two-phase training scheme, vision-language alignment followed by LoRA instruction tuning, can run with 10,094 WSI-report pairs and still produce answers that a separate LLM evaluator accepts in 62.9 percent of 1,134 test cases. Along the way they introduce RAIDER, a retrieval-augmented pipeline that expands original reports into more instruction-tuning examples, and an AI evaluation protocol that scores answers on seven criteria. The central claim is that a slide-level multimodal LLM trained on modest public data can understand pan-cancer morphology and clinical context well enough to generate diagnostic text, making automated assistance in pathology more practical.

What carries the argument

The load-bearing components are a frozen patch encoder (EXAONEPath) that embeds 256x256 patches; a CLAM-based patch aggregator (CBPA) with gated attention that learns which patches matter and outputs a single WSI embedding; a vision projector with attention pooling and linear layers that maps the 512-dimensional WSI embedding to 4096-dimensional text space; LLaMA2-7B-Chat as the language generator, fine-tuned with LoRA; and RAIDER, a retrieval-augmented instruction-data generator that OCRs TCGA reports, stores chunks in a vector database, retrieves relevant chunks by cosine similarity, and uses Llama3.1:70B-instruct to write short diagnostic answers. The evaluation mechanism is an LLM-based evaluator that uses chain-of-thought reasoning and seven criteria to accept or reject the best of ten generated answers.

What would settle it

Run the same 1,134 test pairs through a second, independently constructed LLM evaluator, or have a panel of pathologists score the answers, and compare acceptance rates; a large swing would show the 62.9 percent figure reflects the evaluator, not the model's diagnostic ability.

Watch

Extended reading notes

Core claim

ChatEXAONEPath is a LLaVA-style model whose vision tower combines a frozen EXAONEPath patch encoder with a CLAM-based attention aggregator to turn thousands of patches into one slide-level embedding; a vision projector aligns that embedding with LLaMA2-7B-Chat, and phase-2 training uses LoRA. The authors report that the best configuration (v3) reaches a 62.87 percent acceptance rate on 1,134 WSI-report pairs, with the primary measurement being an instruction-tuned Llama3.1:70B evaluator that first selects the best of 10 generated answers and then accepts or rejects it using seven criteria: accuracy, relevance, completeness, clarity, appropriateness, consistency, and presentation. They also report that scaling the dataset by text-only augmentation (v2, 69,544 pairs) did not beat the smaller v1 dataset, and that the AI evaluator sometimes produces incorrect rationales, so the acceptance rate may not be trustworthy. The claim is that a modestly sized WSI-level multimodal LLM can diagnose across cancer types and that an interpretable AI evaluation protocol can stand in for human judgment.

Load-bearing premise

The load-bearing premise is that the AI evaluator's accept and reject decisions reflect genuine diagnostic quality; the paper itself warns that the evaluator can make inconsistent decisions, so the 62.9 percent acceptance rate may not measure real diagnostic accuracy.

Editorial extensions

If this is right

A WSI-level pathology multimodal LLM can be trained from about ten thousand WSI-report pairs rather than millions, and still answer diagnostic questions in a way that an AI evaluator accepts most of the time.
Text-only augmentation of the same underlying pairs does not reliably improve the model; the v2 dataset with 69,544 pairs performed worse than the v1 dataset with 10,094 pairs, suggesting that alignment between slides and text matters more than quantity.
An LLM evaluator that explains its accept/reject decisions through chain-of-thought can yield interpretable evaluations, but those rationales are sometimes inaccurate, so the evaluation protocol itself needs scrutiny before deployment.
Because the model is pan-cancer, the same system can be applied across multiple cancer types seen in TCGA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the evaluation were repeated with pathologist raters, the 62.9 percent acceptance rate would likely change; the number should be read as accepted by this LLM evaluator, not as a clinically validated diagnostic accuracy.
The success of the frozen patch encoder and attention aggregator suggests the slide embedding produced by the vision tower could be reused for other slide-level tasks, such as survival prediction or biomarker subtyping, which the paper does not test.
The RAIDER result implies that simply paraphrasing existing reports does not create new visual supervision; future gains would likely require new WSI-report pairs or enrichment with genomic or clinical data.
A multimodal evaluator that also sees the WSI, rather than only text answers and references, might make more consistent decisions than the text-only LLM evaluator used here; the paper names this as future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here is the short version: this paper has a genuinely useful data-generation pipeline and a candid limitations section, but its headline number—62.9% acceptance—rests on an evaluator the authors themselves say may be untrustworthy. The appendix prompt for the evaluator asks only for the words "accept" or "reject," which contradicts the paper's claim that chain-of-thought reasoning was elicited. That is a real problem, not a quibble.

What is new: the RAIDER pipeline, which uses retrieval over OCR'd TCGA reports to build instruction-tuning pairs for WSI-level MLLMs. That is a practical idea, and the paper makes a genuine attempt to move from patch-level to whole-slide understanding. The architecture—CLAM aggregator plus LLaVA-style projector—is not original, but the combination is reasonable. The Table 1 comparison of prior WSI models is useful. Credit also for openly reporting that the augmented dataset (v2) did not beat v1; that kind of negative result is worth knowing.

Now the soft spots. The evaluation section says the evaluator is "sometimes inaccurate," makes "incorrect or manipulated reasons," and "in the worst case, the acceptance rate may be non-trustworthy." That is the authors' own text. There is no human-pathologist agreement, no baseline model, no error bars, and no patient-level split description. The appendix prompt tells the evaluator to respond only with accept or reject, while the main text claims CoT was used. Either the prompt is not the one used, or the CoT claim is false. Both options undermine the metric. The title says "expert-level," but nothing in the paper measures diagnostic accuracy against human-anchored ground truth. The 62.9% is an LLM's opinion filtered through an unstable judge.

In proportion: the method and pipeline are worth discussing, but the evaluation is load-bearing and broken. The paper does not support its central claim. I would still take it for peer review—there is enough substance in RAIDER and the honest negative result that a careful reviewer could push the authors toward a proper human study. But it is not close to acceptable in current form. I would not cite it yet. For a reading group, it is a good case study in LLM-as-judge pitfalls.

Referee Report

4 major / 4 minor

Summary. The paper presents ChatEXAONEPath, a whole-slide-image (WSI) multimodal large language model for histopathology. The model uses an EXAONEPath patch encoder, a CLAM-based patch aggregator, and a LLaVA-style architecture with LLaMA2:7B as the language backbone. The authors introduce RAIDER, a retrieval-augmented pipeline that generates instruction-tuning data from 10,094 TCGA WSI-report pairs, scaling to 69,544 instruction pairs. They also propose an AI-based evaluation protocol in which LLaMA3.1:70b selects the best of 10 generated answers and then accepts or rejects it. The main reported result is an acceptance rate of 62.9% (713/1,134) for the best model variant, on the basis of which the abstract claims expert-level diagnostic ability.

Significance. If the evaluation were reliable, the paper would offer a useful practical recipe for building WSI-level pathology MLLMs from public TCGA data and for generating instruction datasets without manual annotation. The RAIDER pipeline and the comparison of three model variants are potentially informative. However, the central quantitative claim rests entirely on an LLM evaluator that the paper itself concedes is unstable and possibly non-trustworthy. No human expert validation, no baseline model comparison, no error bars or repeated-evaluation variance, and no patient-level split analysis are provided. As a result, the headline acceptance rate cannot support the paper's 'expert-level' diagnostic claim.

major comments (4)

[Evaluation, Table 3] The acceptance rate is the only quantitative evidence for the model's diagnostic ability, but the paper itself states that the evaluator's 'responses and interpretations are somewhat unstable, resulting in incorrect decision making' and that 'in the worst case, the acceptance rate may be non-trustworthy - leaving the whole AI-based evaluation system incorrect.' No human-pathologist agreement study, no repeated-evaluation variance estimate, and no calibration against expert ground truth is reported. The 62.9% figure therefore cannot be interpreted as diagnostic accuracy or as supporting 'expert-level' performance.
[Appendix Table A2] The evaluation prompt instructs the evaluator to 'provide only the words accept or reject as your response,' which directly contradicts the Evaluation section's claim that chain-of-thought prompting was used to elicit reasoning steps. Either the published prompt is not the one actually used, making the protocol unreproducible, or CoT reasoning was not actually elicited. Either way, the description of the evaluation protocol is internally inconsistent and this inconsistency bears directly on the validity of the headline metric.
[Experiments, Dataset and Table 3] The manuscript does not clarify whether the 1,134 test pairs were excluded from the RAIDER generation for dataset-v2. If the 69,544 training instruction pairs for v2/v3 were generated from all 10,094 reports, including the test reports, then the reported acceptance rates for v2/v3 are inflated by train/test overlap. This must be stated explicitly, and the evaluation should be re-run on a properly held-out split.
[Experiments, Evaluation] No baseline model is evaluated under the same protocol, and no human pathologist comparison is provided. The paper compares only its own v1/v2/v3 variants. Without a baseline (e.g., an existing pathology MLLM or an untrained LLaVA) and without human expert agreement, the claim that the model is 'expert-level' and can 'comprehensively understand' WSIs is unsupported even setting aside the evaluator-reliability concern.

minor comments (4)

[Dataset-v1 section] The word 'reprsentative' should be 'representative.'
[Table A1] The question 'What is the major diagnosis?' appears twice (question 1 and question 7); one instance is presumably redundant.
[Evaluation section] The phrase 'innately making the whole evaluation process a multi-task performer' is unclear and should be reworded.
[Experiments, Evaluation section] The statement that the acceptance rate was calculated 'based on the test datasets from dataset-v1' is ambiguous for models trained on dataset-v2; clarify whether the same 1,134 test pairs were used for all three variants and whether those pairs were excluded from dataset-v2 generation.

Circularity Check

1 steps flagged · score 6.0 of 10

The 62.9% acceptance rate is a closed LLM-evaluates-LLM loop: GPT-4o/LLaMA3.1-generated references serve as both training labels and evaluation ground truth, and the paper itself concedes the metric may be non-trustworthy.

self definitional [Evaluation section ('Quantitative evaluation'), Dataset-v1 paragraph, and Table A2 evaluation prompt]
"The acceptance rate was calculated based on the test datasets from dataset-v1, which ask for the primary diagnosis or representative diagnostic evidence from the given images. ... This implies that in the worst case, the acceptance rate may be non-trustworthy - leaving the whole AI-based evaluation system incorrect."

Dataset-v1 is explicitly 'LLM-generated responses using GPT-4o' from TCGA reports, so the evaluator's 'reference answer' is itself LLM text rather than independent human ground truth. ChatEXAONEPath was trained on the same LLM-generated instruction data (v1 uses GPT-4o captions; v2/v3 use LLaMA3.1-generated RAIDER data), and the evaluator is another LLaMA3.1 model instructed to accept or reject candidate answers against that LLM reference. The reported 62.9% acceptance rate therefore measures agreement between the candidate model and LLM-generated reference summaries, not pathologist-verified diagnostic accuracy.

full rationale

The training pipeline is not circular: RAIDER builds instruction data from TCGA reports, LLaMA is fine-tuned with standard cross-entropy, and the architecture choices (EXAONEPath encoder, CLAM aggregator, LoRA) are ordinary engineering components that do not define the evaluation metric. The central circularity is in the evaluation claim. The test-set reference answers come from dataset-v1, which is explicitly 'LLM-generated responses using GPT-4o' from TCGA reports; the candidate models were trained on either those GPT-4o responses (v1) or LLaMA3.1-generated RAIDER data (v2/v3); and the evaluator is a LLaMA3.1 model instructed to accept or reject the candidate answer by checking consistency with the LLM-generated reference. Thus the 62.9% acceptance rate is by construction an LLM-consistency score, not a measure of pathologist-level diagnostic correctness. The paper itself admits that 'in the worst case, the acceptance rate may be non-trustworthy - leaving the whole AI-based evaluation system incorrect.' Additionally, the published evaluation prompt in Table A2 instructs the evaluator to output only 'accept' or 'reject', contradicting the Evaluation section's claim that chain-of-thought reasoning was elicited; this is a protocol inconsistency that further undermines the validity of the interpretability claim. The self-citations to EXAONEPath and the WGCNA gene-set work are present but are not the source of the circularity: they are component choices with external training data, and no uniqueness claim is derived from them. Overall, the core diagnostic claim reduces to an LLM-evaluates-LLM loop, so a score of 6 is appropriate rather than a lower value.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on LLM-generated labels and an LLM-as-judge metric. Both are self-referential and not independently validated, while the data split procedure is under-specified for a pan-cancer cohort.

free parameters (1)

number of candidate answers per question = 10
The evaluator picks the best of 10 generated answers before accepting or rejecting. This choice affects the reported acceptance rate and is set by hand without sensitivity analysis.

assumptions (3)

domain assumption TCGA histopathology reports are accurate ground truth for the corresponding WSIs.
The RAIDER pipeline uses these reports to generate labels; any inaccuracies propagate into training and evaluation references.
domain assumption The LLaMA3.1 evaluator reliably judges diagnostic quality using the seven criteria.
The central metric depends on this judge, but the paper itself states the evaluator can be unstable and produce incorrect decisions (Evaluation section).
domain assumption The train/test split does not leak information across slides from the same patient.
TCGA often contains multiple WSIs per patient. The paper does not state that the split is patient-level, so slides from the same patient may appear in both train and test, inflating acceptance rate.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images." pith.science (2026). https://pith.science/paper/S2VFMCJU

@misc{pith2026250413023,
  author       = {Pith},
  title        = {Pith review of: ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S2VFMCJU}},
  note         = {Machine review of arXiv:2504.13023}
}

read the original abstract

Recent studies have made significant progress in developing large language models (LLMs) in the medical domain, which can answer expert-level questions and demonstrate the potential to assist clinicians in real-world clinical scenarios. Studies have also witnessed the importance of integrating various modalities with the existing LLMs for a better understanding of complex clinical contexts, which are innately multi-faceted by nature. Although studies have demonstrated the ability of multimodal LLMs in histopathology to answer questions from given images, they lack in understanding of thorough clinical context due to the patch-level data with limited information from public datasets. Thus, developing WSI-level MLLMs is significant in terms of the scalability and applicability of MLLMs in histopathology. In this study, we introduce an expert-level MLLM for histopathology using WSIs, dubbed as ChatEXAONEPath. We present a retrieval-based data generation pipeline using 10,094 pairs of WSIs and histopathology reports from The Cancer Genome Atlas (TCGA). We also showcase an AI-based evaluation protocol for a comprehensive understanding of the medical context from given multimodal information and evaluate generated answers compared to the original histopathology reports. We demonstrate the ability of diagnosing the given histopathology images using ChatEXAONEPath with the acceptance rate of 62.9% from 1,134 pairs of WSIs and reports. Our proposed model can understand pan-cancer WSIs and clinical context from various cancer types. We argue that our proposed model has the potential to assist clinicians by comprehensively understanding complex morphology of WSIs for cancer diagnosis through the integration of multiple modalities.

Figures

Figures reproduced from arXiv: 2504.13023 by the authors.

**Figure 2.** The overview of the structure of ChatEXAONEPath for training phase 1, vision-language alignment of whole slide images and corresponding captions. bedding feature, we kept the original structure of the attention pooler. We observed that the attention pooler in the vision projector can smoothly project the image embeddings by weighting the vision embedding using self-attention. LLaMA2 Large Language Model Meta AI … view at source ↗

**Figure 4.** Overview of AI-based evaluation process. After then end of the training, we generated 10 answers per each question, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 16 canonical work pages

[1]

What is a major diagnosis?

work page
[2]

What is the crucial diagnosis?

work page
[3]

What is the key diagnosis?

work page
[4]

NEJM AI, 1(3): AIoa2300138

Towards generalist biomedical AI. NEJM AI, 1(3): AIoa2300138. Yang, L.; Xu, S.; Sellergren, A.; Kohlberger, T.; Zhou, Y .; Ktena, I.; Kiraly, A.; Ahmed, F.; Hormozdiari, F.; Jaroensri, T.; et al. 2024. Advancing multimodal medical capabilities of Gemini. arXiv preprint arXiv:2405.03162. Yu, J.; Wang, Z.; Vasudevan, V .; Yeung, L.; Seyedhos- seini, M.; and...

arXiv 2024
[5]

What is the key histopathological feature observed?

work page
[6]

What is the main diagnosis?

work page
[7]

What is the major diagnosis?

work page
[8]

What is the primary diagnosis?

work page

Show all 19 references

[12]

accept” or “reject

What is the most likely diagnosis? Table A1: List of questions used in the instruction datasets for ChatEXAONEPath to generate the diagnosis based on the given whole slide images. Figure A1: A qualitative result and evaluation from the test dataset. ChatEXAONEPath-v3 answered ...
[13]

- Check for any factual errors, misleading information, or incorrect interpretations that deviate from estab- lished medical knowledge

Accuracy: - Verify whether the AI-generated answer accurately reflects the medical facts and knowledge provided in the reference answer. - Check for any factual errors, misleading information, or incorrect interpretations that deviate from estab- lished medical knowledge
[14]

- Ensure that the answer remains focused on the core topic without deviating into unrelated areas or includ- ing extraneous information

Relevance: - Assess whether the AI-generated answer directly addresses the specific question asked. - Ensure that the answer remains focused on the core topic without deviating into unrelated areas or includ- ing extraneous information
[15]

- Determine if any critical elements or important details are missing from the response, and whether the answer provides a comprehensive view

Completeness: - Evaluate whether the AI-generated answer thoroughly covers all aspects of the question as addressed in the reference answer. - Determine if any critical elements or important details are missing from the response, and whether the answer provides a comprehensive view
[16]

- Ensure that the language used is precise, easy to understand, and free of jargon or ambiguous terms that could confuse the reader

Clarity: - Judge the clarity and coherence of the AI-generated answer. - Ensure that the language used is precise, easy to understand, and free of jargon or ambiguous terms that could confuse the reader
[17]

- Confirm that the response is suitable for a professional medical context, with a respectful and appropriate style for the audience

Appropriateness: - Assess the professionalism and tone of the AI-generated answer. - Confirm that the response is suitable for a professional medical context, with a respectful and appropriate style for the audience
[18]

- Evaluate whether the AI’s response aligns with standard medical practices and terminologies

Consistency: - Compare the AI-generated answer with the reference answer for consistency in content and style. - Evaluate whether the AI’s response aligns with standard medical practices and terminologies
[19]

- Ensure that the answer is well-organized and visually easy to follow, contributing to an overall polished appearance

Presentation: - Consider the overall presentation of the AI-generated answer, including formatting, grammar, and punctu- ation. - Ensure that the answer is well-organized and visually easy to follow, contributing to an overall polished appearance. Based on your thorough evalua...
[2022]

Nature Medicine, 28(9): 1773–1784

Multimodal biomedical AI. Nature Medicine, 28(9): 1773–1784. Ahmed, F.; Sellergren, A.; Yang, L.; Xu, S.; Babenko, B.; Ward, A.; Olson, N.; Mohtashamian, A.; Matias, Y .; Cor- rado, G. S.; et al. 2024. PathAlign: A vision-language model for whole slide images in histopathology...

2024 arXiv
[2023]

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 19764–19775

Visual language pretrained multiple instance zero- shot transfer for histopathology images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 19764–19775. Lu, M. Y .; Williamson, D. F.; Chen, T. Y .; Chen, R. J.; Bar- bieri, M.; and Mahmood,...

2021 arXiv
[2024]

arXiv preprint arXiv:2407.05603

WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering. arXiv preprint arXiv:2407.05603. Dai, D.; Zhang, Y .; Xu, L.; Yang, Q.; Shen, X.; Xia, S.; and Wang, G. 2024. PA-LLaV A: A Large Language-Vision As- sistant for Human Pathology Image Understandin...

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

What is a major diagnosis?

work page

[2] [2]

What is the crucial diagnosis?

work page

[3] [3]

What is the key diagnosis?

work page

[4] [4]

NEJM AI, 1(3): AIoa2300138

Towards generalist biomedical AI. NEJM AI, 1(3): AIoa2300138. Yang, L.; Xu, S.; Sellergren, A.; Kohlberger, T.; Zhou, Y .; Ktena, I.; Kiraly, A.; Ahmed, F.; Hormozdiari, F.; Jaroensri, T.; et al. 2024. Advancing multimodal medical capabilities of Gemini. arXiv preprint arXiv:2405.03162. Yu, J.; Wang, Z.; Vasudevan, V .; Yeung, L.; Seyedhos- seini, M.; and...

arXiv 2024

[5] [5]

What is the key histopathological feature observed?

work page

[6] [6]

What is the main diagnosis?

work page

[7] [7]

What is the major diagnosis?

work page

[8] [8]

What is the primary diagnosis?

work page