Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PathFinder, a four-agent system that navigates whole-slide histopathology images like a pathologist, claims 74% accuracy on the M-Path melanoma task—8 points above the best prior AI model and 9 points above the average pathologist.

desk verdict Clever system, unsupported headline claim: the reported 74% is tuned on the test set and compared against a human baseline from a different distribution. read the letter →

arxiv 2502.08916 v1 pith:UYN6KIJB submitted 2025-02-13 cs.CV cs.AIcs.CLcs.MA

Fatemeh Ghezloo , Mehmet Saygin Seyfioglu , Rustin Soraki , Wisdom O. Ikezogwo , Beibin Li , Tejoram Vivekanandan , Joann G. Elmore , Ranjay Krishna
show 1 more author
Linda Shapiro
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.MA
keywords wholeslideimageanalysishistopathologymulti-agentsystemmelanomadiagnosispathologistnavigationvision-languagemodelexplainableAImedicalimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PathFinder is a four-agent AI system for diagnosing whole-slide histopathology images: a Triage Agent labels the slide benign or risky, a Navigation Agent and a Description Agent iteratively pick and describe diagnostically relevant patches, and a Diagnosis Agent reads all the descriptions and outputs a class. The paper's central claim is that on the M-Path skin biopsy melanoma task this emulation of a pathologist's workflow reaches 74% accuracy, 8 points above the best prior model and 9 points above the 65% average accuracy of the pathologists in the original M-Path study. The authors further claim this is the first AI system to surpass the average pathologist on that task, and that its natural-language patch descriptions make the diagnosis inherently explainable. A small double-blind survey by two pathologists rated the Description Agent's outputs as comparable to GPT-4o while using a 7B model.

What carries the argument

The load-bearing mechanism is the text-conditioned visual navigator. At each iteration it runs a lightweight U-Net over the whole slide image (with already-visited patches masked out) conditioned on the average T5 text embedding of all previous patch descriptions, producing an importance map over a 16 × 16 grid; the next patch is sampled proportionally to that map, described by an instruction-tuned Quilt-LLaVA model, and the new description is folded back into the conditioning for the next step. Ten patches are visited per trajectory, and five trajectories per case are combined by majority vote. This description-feedback loop is what the ablation studies credit for the accuracy gain, with the Diagnosis Agent (a GPT-2 LLM with LoRA and a classification head) synthesizing the ten descriptions into the final class.

What would settle it

Re-run PathFinder with the number of trajectories and patches chosen only on validation data, then measure accuracy on the 35-case test set, and have the same pathologist panel read exactly those 35 WSIs; if PathFinder no longer beats the same-subset human average, the headline claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that decomposing whole-slide diagnosis into triage, iterative text-conditioned navigation, patch description, and language-based synthesis outperforms both attention-based multiple-instance learning and static transformer classifiers on the M-Path melanoma dataset. The key result is 74% accuracy with majority voting over five diagnostic trajectories, versus 66% for the best ABMIL baseline and 65% for the average pathologist in the underlying M-Path study. PathFinder's accuracy depends on the feedback loop: a text-conditioned U-Net navigator that sees accumulated descriptions of previously visited patches beats a vision-only navigator (74% vs 64%), exhaustive search (68%), and attention-based top-patch selection (54%). The authors interpret this as evidence that iterative, description-guided sampling, not just more computation, is what makes the system accurate and interpretable.

Load-bearing premise

The headline comparison assumes the balanced 35-case test subset fairly represents the 238-case dataset (where the 65% human average was measured), and that the reported 74% was not inflated by tuning trajectory counts and lengths on the test set rather than the validation set.

Editorial extensions

If this is right

  • If the result holds, AI-assisted pathology can offer a per-patch evidence trail for each diagnosis, making it feasible for clinicians to audit the model's reasoning.
  • The 8-point margin over the best baseline suggests that modeling the search process itself—where to look next, conditioned on what has been seen—adds signal that static attention pooling does not capture.
  • The ablation results imply that description quality is the bottleneck for navigation: the T5-conditioned navigator only beats the CLIP-conditioned one when the Description Agent is fine-tuned for concise, accurate outputs.
  • The better performance of selective sampling over exhaustive search (74% vs 68%) indicates that focusing on a few diagnostically relevant regions, rather than scanning everything, reduces confusion from irrelevant tissue.
  • The Triage Agent is load-bearing: removing it drops the best variant below baselines, so the risk-first filter is what prevents the description model's benign-class bias from dominating.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Although the paper does not test this, the trajectory-ensemble trick—a stochastic navigator plus majority voting—could be bolted onto existing WSI classifiers to trade compute for accuracy.
  • A direct same-subset comparison—having the same pathologist panel read the exact 35 test WSIs—would test the headline claim more cleanly than comparing to the 65% figure from the full 238-case study.
  • The navigation loop could transfer to other gigapixel-scanning domains such as satellite imagery or radiology, where gaze is conditioned on prior findings; only histopathology is demonstrated here.
  • Since the Diagnosis Agent sees only text, occasional hallucinated patch features would propagate; adding a visual verification step is a testable robustness improvement the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents PathFinder, a multi-agent system for whole-slide image (WSI) classification in skin melanoma diagnosis. The pipeline comprises a Triage Agent that separates benign from risky cases, a text-conditioned Navigation Agent (a U-Net) that iteratively selects diagnostically relevant patches, a Description Agent (fine-tuned Quilt-LLaVA) that produces concise natural-language patch descriptions, and a Diagnosis Agent (GPT-2 with a classification head) that aggregates the descriptions into one of four diagnostic classes. The authors report 74% accuracy on the M-Path dataset, claiming an 8% improvement over the best baseline and a 9% improvement over the average pathologist accuracy of 65%, and state that this is the first AI-based system to surpass average pathologist performance on this task. The paper also includes a double-blind pathologist preference study for the generated descriptions, showing that PathFinder's Description Agent performs comparably to GPT-4o.

Significance. PathFinder's architecture is interesting and potentially useful: separating triage, navigation, description, and diagnosis mirrors the real workflow of pathologists, and the system produces interpretable natural-language evidence for each case. The release of code, data, and models is a strength that should facilitate reproducibility. The qualitative description-quality study, with two pathologist raters, is a valuable contribution. However, the central comparative claims—that PathFinder surpasses both state-of-the-art baselines and average pathologist performance—are currently not well supported by the evaluation protocol. The headline numbers are based on a test set that was also used to select the system's operating configuration, and the human benchmark was measured on a different, larger cohort with a different class distribution. Because the claimed effects are small relative to the test-set size, these evaluation issues are load-bearing rather than cosmetic.

major comments (4)
  1. [Section 4.4, Figure 3, Section 5.2] The trajectory configuration (5 trajectories, 10 patches per trajectory) is selected using an ablation (Figure 3) that is explicitly run on the test set. Section 4.4 states: 'For the test set, we extracted additional trajectories (n = 20) to assess the effect of trajectory number on diagnosis results.' The final accuracy of 74% is then reported on that same test set. This is a form of test-set tuning, which optimistically biases the reported estimate of generalization performance. The authors should select hyperparameters on the validation set only, or pre-specify a configuration before evaluation, and should report confidence intervals or a significance test for the test accuracy.
  2. [Table 1 and Section 5.2] The comparison to the best baselines is internally inconsistent. The text states that 'The evaluation of the baseline models are similarly done using the majority voting over 10 runs,' while the Table 1 footnote says '* ABMIL result is based on a single run and does not use majority voting.' These statements cannot both be true. If the baselines were not evaluated under the same ensemble/majority-voting protocol as PathFinder, then the claimed 8% improvement over ABMIL w/ CONCH and w/ UNI2-h may be partly attributable to the voting ensemble rather than to the multi-agent method itself. The authors must clarify the protocol and re-evaluate all baselines with the identical voting scheme.
  3. [Section 5.2 and Section 3] The human benchmark is not directly comparable to the reported PathFinder accuracy. The 65% human accuracy from [9] was measured on the full 238-case M-Path cohort with its natural class distribution, whereas PathFinder is evaluated on a rebalanced test subset whose size and composition are not specified ('We balanced the testing dataset to ensure that each diagnostic class is represented by an equal number of samples'). Accuracy on a balanced subset is not directly comparable to accuracy on the natural cohort unless per-class accuracies are equal, and the paper does not report per-class accuracy for either system. Furthermore, with a test set of only 35 cases, the difference between 74% and 65% corresponds to roughly three correct cases; the exact binomial 95% confidence interval for 26/35 correct spans approximately 56–87%, which includes 65%. No significance test or confidence interval is reported. The claim that PathFinder is the first AI-based system to surpass average pathologist performance on this task is therefore not supported by the evidence.
  4. [Section 4.2 and Section 3] The construction of the Navigation Agent's ground-truth importance maps from the M-Path viewport data is not described in detail, and the paper does not state explicitly whether the viewport data from the test-set WSIs were used in training the U-Net navigator. If importance maps derived from test-case viewport data are used for model development, this would constitute a form of label leakage. The authors should specify how the importance maps are computed from the viewport recordings and confirm that no information from the test cases is used in training any component of PathFinder.
minor comments (6)
  1. [Abstract and Introduction] The phrase 'fail short' should be 'fall short'.
  2. [Throughout] The model name 'LLaVA' is typeset with a space ('LLaV A') in several places; this should be corrected.
  3. [Section 4.2] The word 'conditined' should be 'conditioned'.
  4. [Table 1 caption] The caption says 'coverage here is the percent of patches used across all trajectories,' but the table as printed has no coverage column; this appears to be a leftover from an earlier version.
  5. [Section 4.4] The resampling of the 168 training cases to 20,000 cases (100,000 trajectories) is a very aggressive oversampling; the authors should discuss potential overfitting and justify this choice, since the model might memorize the original training trajectories.
  6. [Section 5.2] The paper states that in the balanced test set, 'micro-averaged F1 score, precision, and recall are equivalent to the accuracy.' This is only true if the test set is exactly balanced, so the exact number of cases per class used for evaluation should be reported.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PathFinder's reported accuracy is a measured empirical result on an external benchmark, not a derivation that reduces to its own inputs.

full rationale

PathFinder is an empirical systems paper. The reported 74% accuracy is the measured performance of a trained multi-agent pipeline on a held-out 35-case test set from the M-Path cohort, and the 65% human benchmark is taken from the published external M-Path study [9], not generated by the present model. The authors' prior work (Quilt-1M and Quilt-LLaVA) is used as concrete released components and is evaluated against external models (GPT-4o and LLaVA-Med) in the description-quality survey, so these self-citations are not load-bearing in the sense of substituting for evidence. No equation in the paper defines a prediction in terms of a fitted parameter, and no uniqueness theorem or ansatz is imported from the authors' prior papers to force the design. The main statistical weakness, selecting the number of trajectories and trajectory length via test-set ablations in Figure 3 and then reporting accuracy on the same test set, is a real validity concern, but it is a form of selection bias rather than circularity: the reported number does not reduce by construction to a fitted constant, and the underlying comparison to baselines and human experts remains an empirical measurement. Accordingly, no specific circular step can be quoted, and the paper should not receive a circularity score above 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

PathFinder rests on standard supervised learning machinery. The main free parameters are design choices (trajectory count, patch count, text encoder) that were selected based on the test set, which introduces optimistic bias. The axioms are domain-level assumptions about ground truth and representation, all reasonable for computational pathology but worth stating.

free parameters (5)
  • Number of trajectories for majority voting (n=5) = 5
    Selected based on the ablation in Figure 3, which evaluates accuracy for 1 to 20 trajectories. If this selection is done on the test set, it is a form of test-set tuning.
  • Patches per trajectory (10) = 10
    Also from Figure 3, where trajectory length is varied. Chosen as the point with optimal accuracy.
  • T5 text encoder over CLIP = T5
    The paper reports T5-based navigation gives 74% vs 62% for CLIP. This is a design choice, not a fitted value, but it affects the reported performance.
  • GPT-2 as Diagnosis Agent base LLM = GPT-2
    Chosen due to resource constraints. The paper speculates larger LLMs could improve results, but the headlined result uses GPT-2.
  • Triage Agent operating threshold
    The Triage Agent is described as binary (benign vs risky), but the decision threshold is not reported. It is an implicit free parameter affecting which cases go to the full pipeline.
assumptions (4)
  • domain assumption The consensus reference diagnoses of the three expert dermatopathologists are the ground truth for all 238 M-Path cases.
    Invoked in Section 3 as the basis for labels. If the consensus is wrong, all evaluations, including the human baseline, inherit that error.
  • domain assumption Patch-level descriptions at 10x magnification preserve sufficient diagnostic information for the 4-class melanoma classification.
    The whole system is built on describing small patches rather than examining the full gigapixel image at high zoom. This assumption is implicit in the architecture and is tested only indirectly through the final accuracy.
  • domain assumption The pathologist viewport data from the M-Path study provides valid ground truth importance maps for training the Navigation Agent.
    Navigation is trained to reproduce where pathologists looked. This assumes viewport patterns are a good proxy for diagnostic relevance.
  • ad hoc to paper Majority voting over five independently sampled trajectories is a valid aggregation method that reflects a stable diagnosis.
    This is a design choice specific to PathFinder. The paper shows accuracy varies with the number of trajectories, but the justification for five is based on an ablation, not a principled argument.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology." pith.science (2026). https://pith.science/paper/UYN6KIJB

@misc{pith2026250208916,
  author       = {Pith},
  title        = {Pith review of: PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYN6KIJB}},
  note         = {Machine review of arXiv:2502.08916}
}
read the original abstract

Diagnosing diseases through histopathology whole slide images (WSIs) is fundamental in modern pathology but is challenged by the gigapixel scale and complexity of WSIs. Trained histopathologists overcome this challenge by navigating the WSI, looking for relevant patches, taking notes, and compiling them to produce a final holistic diagnostic. Traditional AI approaches, such as multiple instance learning and transformer-based models, fail short of such a holistic, iterative, multi-scale diagnostic procedure, limiting their adoption in the real-world. We introduce PathFinder, a multi-modal, multi-agent framework that emulates the decision-making process of expert pathologists. PathFinder integrates four AI agents, the Triage Agent, Navigation Agent, Description Agent, and Diagnosis Agent, that collaboratively navigate WSIs, gather evidence, and provide comprehensive diagnoses with natural language explanations. The Triage Agent classifies the WSI as benign or risky; if risky, the Navigation and Description Agents iteratively focus on significant regions, generating importance maps and descriptive insights of sampled patches. Finally, the Diagnosis Agent synthesizes the findings to determine the patient's diagnostic classification. Our Experiments show that PathFinder outperforms state-of-the-art methods in skin melanoma diagnosis by 8% while offering inherent explainability through natural language descriptions of diagnostically relevant patches. Qualitative analysis by pathologists shows that the Description Agent's outputs are of high quality and comparable to GPT-4o. PathFinder is also the first AI-based system to surpass the average performance of pathologists in this challenging melanoma classification task by 9%, setting a new record for efficient, accurate, and interpretable AI-assisted diagnostics in pathology. Data, code and models available at https://pathfinder-dx.github.io/

Figures

Figures reproduced from arXiv: 2502.08916 by the authors.

Figure 1
Figure 1. We propose PathFinder, a system capable of navigating image patches in a whole slide image, describe each patch to collect evidence, and produce a diagnosis. Pathfinder’s process is interpretable and reminiscent of pathologists. Our system consists of multiple steps carried out by multi-modal agents: 1) Initial Assessment by Triage Agent; 2) Evidence Collection by Navigation and Description Agents; and 3) Integrated… view at source ↗
Figure 2
Figure 2. The left panel illustrates the Navigation Agent, as outlined in Section [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation results. We ran 10 experiments, and plotted both the mean and standard deviation. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Expert human pathologist preferences for each model in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 1
Figure 1. Figure 1: Overview of the Triage Agent architecture. Definitions of [PITH_FULL_IMAGE:figures/full_fig_p013_1.png]
Figure 2
Figure 2. Figure 2: GPT-4 prompt to generate instruction-tuning dataset for the Description Agent. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png]
Figure 3
Figure 3. Figure 3: Expert human pathologist preferences for each model, segmented by the reasons for their choices. Each subplot corresponds to [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology

    cs.CV 2025-12 conditional novelty 5.0 of 10

    Splitting slide captions into random sentence subcaptions and aligning them with region features via text-conditioned attention improves whole-slide classification, retrieval, captioning and VQA in computational pathology.

  2. CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering

    cs.CV 2025-05 conditional novelty 5.0 of 10

    CT-Agent combines an LLM planner, region-specific LoRA adapters, and global/local token compression to improve 3D chest CT report generation and question answering on CT-RATE and RadGenome-ChestCT.

  3. MAARTA:Multi-Agentic Adaptive Radiology Teaching Assistant

    cs.CY 2025-06 conditional novelty 4.0 of 10

    MAARTA, a multi-agent LLM framework comparing expert and student gaze graphs, reports higher accuracy than single-agent baselines on simulated perceptual errors in chest X-ray interpretation.

Reference graph

Works this paper leans on

66 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [9]

    Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and repro- ducibility study

    Joann G Elmore, Raymond L Barnhill, David E Elder, Gary M Longton, Margaret S Pepe, Lisa M Reisch, Patri- cia A Carney, Linda J Titus, Heidi D Nelson, Tracy Onega, et al. Pathologists’ diagnosis of invasive melanoma and melanocytic proliferations: observer accuracy and repro- ducibility study. bmj, 357, 2017. 2, 3, 8

    work page 2017

  2. [1]

    Pathalign: A vision-language model for whole slide images in histopathology

    Faruk Ahmed, Andrew Sellergren, Lin Yang, Shawn Xu, Boris Babenko, Abbi Ward, Niels Olson, Arash Mohtashamian, Yossi Matias, Greg S Corrado, et al. Pathalign: A vision-language model for whole slide images in histopathology. arXiv preprint arXiv:2406.19578, 2024. 2

    arXiv 2024

  3. [2]

    Achieving consensus for the histopathologic diagnosis of melanocytic lesions: use of the modified delphi method

    Patricia A Carney, Lisa M Reisch, Michael W Piepkorn, Raymond L Barnhill, David E Elder, Stevan Knezevich, Berta M Geller, Gary Longton, and Joann G Elmore. Achieving consensus for the histopathologic diagnosis of melanocytic lesions: use of the modified delphi method. Journal of cutaneous pathology, 43(10):830–837, 2016. 3

    work page 2016

  4. [3]

    Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

    Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144–16155, 2022. 2

    work page 2022

  5. [4]

    Towards a general-purpose foundation model for computational pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850–862,

  6. [5]

    Slidechat: A large vision-language assistant for whole-slide pathology image understanding

    Ying Chen, Guoan Wang, Yuanfeng Ji, Yanjun Li, Jin Ye, Tianbin Li, Bin Zhang, Nana Pei, Rongshan Yu, Yu Qiao, et al. Slidechat: A large vision-language assistant for whole-slide pathology image understanding. arXiv preprint arXiv:2410.11761, 2024. 2

    arXiv

  7. [6]

    Improving factuality and reason- ing in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenen- baum, and Igor Mordatch. Improving factuality and reason- ing in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. 2

    arXiv 2023

  8. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

Show all 66 references
  1. [8]

    An interactive agent foundation model

    Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, et al. An interactive agent foundation model. arXiv preprint arXiv:2402.05929, 2024. 2

    2024 arXiv

  2. [10]

    Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis

    Zhihao Fan, Jialong Tang, Wei Chen, Siyuan Wang, Zhongyu Wei, Jun Xi, Fei Huang, and Jingren Zhou. Ai hospital: Interactive evaluation and collaboration of llms as intern doctors for clinical diagnosis. arXiv preprint arXiv:2402.09742, 2024. 2

    2024 arXiv

  3. [11]

    An analysis of pathologists’ viewing processes as they diagnose whole slide digital images

    Fatemeh Ghezloo, Pin-Chieh Wang, Kathleen F Kerr, Tad T Bruny´e, Trafton Drew, Oliver H Chang, Lisa M Reisch, Linda G Shapiro, and Joann G Elmore. An analysis of pathologists’ viewing processes as they diagnose whole slide digital images. Journal of Pathology Informatics, 13:100104,

  4. [12]

    Chang, Stevan R

    Fatemeh Ghezloo, Oliver H. Chang, Stevan R. Knezevich, Kristin C. Shaw, Kia Gianni Thigpen, Lisa M. Reisch, Linda G. Shapiro, and Joann G. Elmore. Robust roi detec- tion in whole slide images guided by pathologists’ viewing patterns. Journal of Imaging Informatics in Medicine,...

    2024

  5. [13]

    Aug- menting pathologists with navipath: design and evaluation of 9 a human-ai collaborative navigation system

    Hongyan Gu, Chunxu Yang, Mohammad Haeri, Jing Wang, Shirley Tang, Wenzhong Yan, Shujin He, Christopher Kazu Williams, Shino Magaki, and Xiang’Anthony’ Chen. Aug- menting pathologists with navipath: design and evaluation of 9 a human-ai collaborative navigation system. In Proce...

    2023

  6. [14]

    Higt: Hierarchical interaction graph-transformer for whole slide image analysis

    Ziyu Guo, Weiqin Zhao, Shujun Wang, and Lequan Yu. Higt: Hierarchical interaction graph-transformer for whole slide image analysis. In International Conference on Medical Im- age Computing and Computer-Assisted Intervention , pages 755–764. Springer, 2023. 2

    2023

  7. [15]

    Llm multi-agent sys- tems: Challenges and open problems

    Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Llm multi-agent sys- tems: Challenges and open problems. arXiv preprint arXiv:2402.03578, 2024. 2

    2024 arXiv

  8. [16]

    Llm-based multi-agent systems for software engineering: Vision and the road ahead

    Junda He, Christoph Treude, and David Lo. Llm-based multi-agent systems for software engineering: Vision and the road ahead. arXiv preprint arXiv:2404.04834, 2024. 2

    2024 arXiv

  9. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 6

    2021

  10. [18]

    Squeeze-and-excitation net- works

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. 2018. 4

    2018

  11. [19]

    How far are we on the decision-making of llms? evaluating llms’ gaming ability in multi-agent environments

    Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. How far are we on the decision-making of llms? evaluating llms’ gaming ability in multi-agent environments. arXiv preprint arXiv:2403.11807,

    arXiv

  12. [20]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6

    2024 arXiv

  13. [21]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems, 36, 2024. 5

    2024

  14. [22]

    Multi-modal masked autoencoders learn compositional histopathological representations

    Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, and Linda Shapiro. Multi-modal masked autoencoders learn compositional histopathological representations. arXiv preprint arXiv:2209.01534, 2022. 2

    2022 arXiv

  15. [23]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fate- meh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. arXiv preprint arXiv:2306.11207, 2023. 2, 4, 7, 8

    2023 arXiv

  16. [24]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Inter- national conference on machine learning, pages 2127–2136. PMLR, 2018. 7, 8

    2018

  17. [25]

    Tomczak, and Max Welling

    Maximilian Ilse, Jakub M. Tomczak, and Max Welling. Attention-based deep multiple instance learning, 2018. 1

    2018

  18. [26]

    Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning

    Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318–14328, 2021. 2

    2021

  19. [27]

    Eliceiri

    Bin Li, Yin Li, and Kevin W. Eliceiri. Dual-stream multiple instance learning network for whole slide image classifica- tion with self-supervised contrastive learning, 2021. 1

    2021

  20. [28]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024. 6

    2024

  21. [29]

    Camel: communicative agents for” mind” explo- ration of large scale language model society.(2023)

    G Li, HAAK Hammoud, H Itani, D Khizbullin, and B Ghanem. Camel: communicative agents for” mind” explo- ration of large scale language model society.(2023). arXiv preprint arXiv:2303.17760, 2023. 2

    2023 arXiv

  22. [30]

    Agent hospi- tal: A simulacrum of hospital with evolvable medical agents

    Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospi- tal: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957, 2024. 2

    2024 arXiv

  23. [31]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 4, 1

    2023 arXiv

  24. [32]

    Semantics-aware attention guidance for diagnosing whole slide images

    Kechun Liu, Wenjun Wu, Joann G Elmore, and Linda G Shapiro. Semantics-aware attention guidance for diagnosing whole slide images. In MICCAI, 2024. 2, 7, 8

    2024

  25. [33]

    A visual- language foundation model for computational pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual- language foundation model for computational pathology. Nature Medicine, 30(3):863–874, 2024. 7, 8

    2024

  26. [34]

    A multimodal gen- erative ai copilot for human pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal gen- erative ai copilot for human pathology. Nature, pages 1–3,

  27. [35]

    End-to-end diagnosis of breast biopsy images with trans- formers

    Sachin Mehta, Ximing Lu, Wenjun Wu, Donald Weaver, Hannaneh Hajishirzi, Joann G Elmore, and Linda G Shapiro. End-to-end diagnosis of breast biopsy images with trans- formers. Medical image analysis, 79:102466, 2022. 2

    2022

  28. [36]

    Characterizing diagnostic search patterns in digital breast pathology: scanners and drillers

    Ezgi Mercan, Linda G Shapiro, Tad T Bruny ´e, Donald L Weaver, and Joann G Elmore. Characterizing diagnostic search patterns in digital breast pathology: scanners and drillers. Journal of digital imaging, 31:32–41, 2018. 2

    2018

  29. [37]

    Slide navigation patterns among patholo- gists with long experience of digital review

    Jesper Molin, Morten Fjeld, Claudia Mello-Thoms, and Claes Lundstr¨om. Slide navigation patterns among patholo- gists with long experience of digital review. Histopathology, 67(2):185–192, 2015. 2

    2015

  30. [38]

    Accuracy of digital pathologic analysis vs tra- ditional microscopy in the interpretation of melanocytic le- sions

    Tracy Onega, Raymond L Barnhill, Michael W Piepkorn, Gary M Longton, David E Elder, Martin A Weinstock, Ste- van R Knezevich, Lisa M Reisch, Patricia A Carney, Heidi D Nelson, et al. Accuracy of digital pathologic analysis vs tra- ditional microscopy in the interpretation of m...

    2018

  31. [39]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsuper- vised multitask learners. 2019. 6

    2019

  32. [40]

    Exploring the limits of transfer learning with a 10 unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a 10 unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 5

    2020

  33. [41]

    An experimental study of pathologist’s navigation patterns in virtual microscopy

    Lucia Roa-Pe ˜na, Francisco G ´omez, and Eduardo Romero. An experimental study of pathologist’s navigation patterns in virtual microscopy. Diagnostic pathology, 5:1–11, 2010. 2

    2010

  34. [42]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Pa...

    2015

  35. [43]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510–4520, 2018. 7

    2018

  36. [44]

    Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos

    Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open-source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  37. [45]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification

    Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136–2147, 2021. 2, 4, 1

    2021

  38. [46]

    Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collabora- tion

    Yuxuan Sun, Yunlong Zhang, Yixuan Si, Chenglu Zhu, Zhongyi Shui, Kai Zhang, Jingxiong Li, Xingheng Lyu, Tao Lin, and Lin Yang. Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collabora- tion. arXiv preprint arXiv:2407.00203, 2024. 2

    2024 arXiv

  39. [47]

    High-performance medicine: the convergence of human and artificial intelligence

    Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence. Nature medicine, 25(1): 44–56, 2019. 2

    2019

  40. [48]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  41. [49]

    A foundation model for clinical-grade computational pathology and rare cancers detection

    Eugene V orontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine ...

    2024

  42. [50]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. 2024. 2

    2024

  43. [51]

    Scale-aware transformers for diagnosing melanocytic lesions

    Wenjun Wu, Sachin Mehta, Shima Nofallah, Stevan Kneze- vich, Caitlin J May, Oliver H Chang, Joann G Elmore, and Linda G Shapiro. Scale-aware transformers for diagnosing melanocytic lesions. IEEE Access, 9:163526–163541, 2021. 2

    2021

  44. [52]

    Scale-aware transformers for diagnosing melanocytic lesions

    Wenjun Wu, Sachin Mehta, Shima Nofallah, Stevan Kneze- vich, Caitlin J May, Oliver H Chang, Joann G Elmore, and Linda G Shapiro. Scale-aware transformers for diagnosing melanocytic lesions. IEEE Access, 9:163526–163541, 2021. 2, 7, 8

    2021

  45. [53]

    A whole-slide foundation model for digital pathology from real-world data

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz ´alez, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, pages 1–8, 2024. 2

    2024

  46. [54]

    A foundation model for generalizable cancer diagnosis and survival prediction from histopathological images

    Zhaochang Yang, Ting Wei, Ying Liang, Xin Yuan, Ruitian Gao, Yujia Xia, Jie Zhou, Yue Zhang, and Zhangsheng Yu. A foundation model for generalizable cancer diagnosis and survival prediction from histopathological images. bioRxiv, pages 2024–05, 2024. 2

    2024

  47. [55]

    A graph-transformer for whole slide image classifica- tion

    Yi Zheng, Rushin H Gindra, Emily J Green, Eric J Burks, Margrit Betke, Jennifer E Beane, and Vijaya B Kolacha- lama. A graph-transformer for whole slide image classifica- tion. IEEE transactions on medical imaging , 41(11):3003– 3015, 2022. 2

    2022

  48. [56]

    Pathm3: A multimodal multi-task multiple instance learning framework for whole slide image classification and captioning

    Qifeng Zhou, Wenliang Zhong, Yuzhi Guo, Michael Xiao, Hehuan Ma, and Junzhou Huang. Pathm3: A multimodal multi-task multiple instance learning framework for whole slide image classification and captioning. arXiv preprint arXiv:2403.08967, 2024. 2 11 PathFinder: A Multi-Modal M...

    2024 arXiv

  49. [57]

    To evaluate its effectiveness, we compared the performance of the Triage Agent against three MIL-based benchmark meth- ods [25, 27, 45] for detecting Class 1 vs

    Triage agent Figure 1 illustrates the architecture of the Triage Agent. To evaluate its effectiveness, we compared the performance of the Triage Agent against three MIL-based benchmark meth- ods [25, 27, 45] for detecting Class 1 vs. Non-Class 1 cases in the M-Path dataset (de...

  50. [58]

    This design aimed to enable direct reason- ing over image latents through an LLM

    VLM-based Navigation Agent Our initial approach to designing the Navigator Agent ex- plored a multi-modal architecture based on the LLaV A framework [31]. This design aimed to enable direct reason- ing over image latents through an LLM. The architecture consisted of two main c...

  51. [59]

    A U-Net encoder [42] pre-trained on pathologist view- ing behavior data (M-Path, details in Section 3), which served as the image encoder

  52. [60]

    Training Process and Architecture We first trained a complete U-Net on the M-Path dataset to learn meaningful representations of WSIs

    The LLaMA-7B language model [48], which acted as the reasoning component 2.1. Training Process and Architecture We first trained a complete U-Net on the M-Path dataset to learn meaningful representations of WSIs. For the Navi- gator implementation, we removed the U-Net’s decod...

  53. [61]

    Data Scarcity: The available navigation training dataset proved insufficient for the model to learn robust region selection strategies

  54. [62]

    Architectural Pivot These limitations led us to revise our approach fundamen- tally

    Overfitting Patterns: The model exhibited clear signs of overfitting: • Consistently selecting patches from the central regions of WSIs, regardless of input • Generating repetitive patch selections • Failing to generalize to novel slide patterns 2.3. Architectural Pivot These ...

  55. [63]

    Figure 2 illustrates the prompt used and a sample of the data generated for fine- tuning the Description Agent

    Description agent We generated fine-tuning data for the Description Agent by prompting GPT-4 to extract short and concise histopathol- ogy findings from provided text. Figure 2 illustrates the prompt used and a sample of the data generated for fine- tuning the Description Agent

  56. [64]

    Evaluation and experiments This section provides details on the qualitative analysis conducted by pathologists and the prompt for our LLM- prompting experiments. 4.1. Qualitative Analysis of Descriptions Assessed by Pathologists To evaluate the quality of the descriptions gene...

  57. [65]

    (Options: Model A, Model B, Model C)

    Selection: Please select the description that you believe best matches the content of the image. (Options: Model A, Model B, Model C)

  58. [66]

    You may select more than one option if applicable

    Reason for Preference: Please choose the primary rea- son for your preference. You may select more than one option if applicable. If Other, please specify. • Correctness: The description accurately reflects the features of the image. • Detail: The description provides a compre...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.