Pith. sign in

REVIEW 4 major objections 5 minor 41 references

PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PathMR writes the diagnosis and draws the cell masks.

desk verdict Useful first pathology pixel-reasoning benchmark, but closed-loop labels and uneven category results undercut the 'consistent outperformance' claim. read the letter →

arxiv 2508.20851 v1 pith:ORWE6YJX submitted 2025-08-28 cs.CV

Ye Zhang , Yu Zhou , Jingwen Qi , Yongbing Zhang , Simon Puettmann , Finn Wichmann , Larissa Pereira Ferreira , Lara Sichward
show 6 more authors
Julius Keyl Sylvia Hartmann Shuo Zhao Hongxiao Wang Xiaowei Xu Jianxu Chen
This is my paper
classification cs.CV
keywords multimodalvisualreasoningpathologydiagnosiscell-levelsegmentationgastricadenocarcinomainterpretableAInucleiquestionansweringGADVRdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PathMR is a multimodal visual reasoning model that, given an H&E-stained pathology image and a text question, produces both an expert-style diagnostic answer and pixel-level masks showing which nuclei are tumor, inflammatory, connective, or epithelial cells. The paper's claim is that this joint output is not only possible but better than what existing visual reasoning models deliver: on the paper's new GADVR dataset and on PathGen, PathMR reports higher text quality, cleaner segmentation, and tighter text-mask alignment across reasoning segmentation, referring segmentation, and conversation tasks. The motivating problem is that current attention-based pathology interpretability is patch-level and silent: it says where the model looked, not what it sees. PathMR's two added mechanisms, classification supervision and a spatial consistency constraint, suppress the fragmented and noisy masks that the authors identify as the main failure of prior pixel-level reasoning models. A sympathetic reader would take the paper as establishing that cell-level visual reasoning is a viable route to traceable AI pathology diagnosis.

What carries the argument

The load-bearing object is the [SEG]-token interface: the LLM emits a diagnostic phrase interrupted by [SEG] tokens, special output tokens that signal "segment the region described here," and the hidden states of those tokens are projected into a SAM-based vision decoder to produce masks. The paper's new contribution is a dual constraint placed on that interface. Classification supervision modifies the mask loss so that predicting the wrong cell category as the current one is penalized, and a consistency loss forces each pixel's predicted class probability to agree with its four immediate neighbors. Together these mechanisms suppress isolated noisy predictions and keep adjacent pixels in the

What would settle it

Take a held-out set of GADVR patches, have pathologists independently create the nuclei masks and answer the diagnostic questions from scratch, and rerun the PathMR-versus-baseline comparison against those expert annotations. If PathMR's lead over MMR and LISA shrinks or disappears under expert-corrected labels, the claimed improvements in segmentation and interpretability are artifacts of matching HoverNet and GPT-4o labels rather than clinical ground truth; if the lead survives, the case for cell-level visual reasoning is substantially stronger.

Watch

Extended reading notes

Core claim

The central discovery is that a single end-to-end model can tie each [SEG] token in a generated diagnostic sentence to a specific cell-class segmentation map, and that two small constraints make those maps reliable enough to beat prior methods. PathMR builds on the LISA/MMR-style architecture: a SAM vision encoder and decoder supply segmentation features, a multimodal LLM generates text containing [SEG] tokens, and the filtered token features drive mask prediction. The paper adds classification supervision, which applies extra penalty weights in the BCE/Dice loss when the model mislabels one cell category as the currently predicted category during sequential [SEG] decoding, and a consistency

Load-bearing premise

The load-bearing premise is that the ground-truth labels are valid clinical references: nuclei masks produced by a pretrained HoverNet segmentation model plus GPT-4o-generated question-answer pairs, with only limited pathologist correction and rating, so a systematic error in either labeler would make the reported metrics look better than the true interpretability.

Editorial extensions

If this is right

  • A single model can provide the two things pathologists ask for when evaluating a diagnosis, where the abnormal cells are and why they are abnormal, in one forward pass.
  • Cell categories that are rare or visually subtle, such as epithelial cells, become segmentable where earlier models often failed or returned empty masks.
  • The GADVR dataset gives the community a standardized pixel-level visual reasoning benchmark for pathology, with over 547,000 image-text pairs and pathologist-assessed labels.
  • The same architecture transfers to pan-cancer patches from PathGen, suggesting the visual reasoning setup is not specific to gastric adenocarcinoma.
  • Training on 8 A100 GPUs for about one day for the 7B model makes the approach reproducible at a moderate compute budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely consequence the paper does not explore: because the consistency loss forces neighboring pixels to agree, it may also erase thin or scattered single-cell structures; checking mask recall on isolated tumor cells would test this.
  • The reported gains are measured against labels produced by HoverNet and GPT-4o. If those automated labels carry a systematic bias, the method's advantage may partly reflect imitation of the labeler rather than clinical truth; re-validating on fully expert-corrected annotations would settle that.
  • Because the framework uses generic SAM and LLM components, the same dual-constraint recipe could be applied to other cell-level diagnostic tasks, such as breast or colorectal nuclei, without redesign, but the paper only demonstrates gastric and pan-cancer patches.
  • A sharper test of the claimed cross-modal alignment would be to perturb the image, for example by removing or relocating a cell cluster, and checking whether the generated text and masks change consistently; that would separate genuine visual grounding from language priors learned from GPT-4o answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. PathMR proposes a cell-level multimodal visual reasoning framework for histopathology. The model combines an LLM (LLaVA-7B/13B) with a SAM-based visual perception module, and introduces two training components: classification supervision and a consistency constraint intended to reduce fragmented segmentation. To benchmark, the authors construct GADVR, a gastric adenocarcinoma dataset whose nuclei masks are produced by HoverNet and whose question-answer pairs are generated by GPT-4o from those masks. They evaluate on GADVR and on PathGen (with HoverNet-generated masks) against prior visual reasoning models, reporting gIoU/cIoU for segmentation and BLEU-4/F1 for conversation, and they claim consistent state-of-the-art text generation, segmentation accuracy, and cross-modal alignment.

Significance. If the claims hold, the contribution would be a useful step toward interpretable pathology AI: the dataset is large, the task formulation is clinically motivated, and the architectural modifications (classification supervision plus consistency loss) are simple and plausible. The authors also state that code and data will be publicly released, which supports reproducibility. However, the significance is curtailed by two issues: the reported results do not uniformly support the headline claim of consistent outperformance, and the evaluation labels are themselves generated by automated models (HoverNet and GPT-4o), making the measured 'clinical interpretability' partly a measure of agreement with the label generators.

major comments (4)
  1. [Table I, Section IV-D1] The abstract and Section IV-D1 state that PathMR consistently outperforms state-of-the-art methods, but Table I contradicts this at the category level. PathMR-13B is below MMR-13B on neoplastic gIoU (0.694 vs 0.709) and inflammatory cIoU (0.737 vs 0.739); PathMR-7B is below MMR-7B on inflammatory gIoU (0.529 vs 0.536) and inflammatory cIoU (0.533 vs 0.548). The claim of 'consistent' superiority should be qualified to overall or selected metrics, or the category-level deficits should be analyzed and explained.
  2. [Section II-B, Section IV-C] The evaluation is a closed loop. Ground-truth segmentation masks are generated by a pretrained HoverNet model, and ground-truth QA answers are generated by GPT-4o from those masks. PathMR is trained on those same labels and then evaluated against them (Table I-III, BLEU-4/F1 against GPT-4o references). The 200-sample pathologist check in Section II-D provides only a coarse overall sanity check; it does not validate the full test set or show inter-observer agreement. Consequently, high scores may reflect imitation of HoverNet's mask style and GPT-4o's phrasing rather than clinically accurate diagnosis. The paper should report independent expert evaluation on a held-out sample, or clearly frame the results as agreement with automated labelers.
  3. [Tables I-III, Section IV-E] No error bars, confidence intervals, or significance tests are reported. Many of the claimed improvements are small (e.g., overall test gIoU 0.633 vs 0.613 in Table I; ablation gains of 0.602 to 0.609 and 0.605 in Fig. 8). Without repeated runs and statistical testing, the headline 'consistently outperforms' is not supported. Please report variance across at least three seeds and test whether the differences are statistically reliable.
  4. [Section IV-A] The PathGen experiment is described as a generalization test, but the segmentation labels for PathGen are generated with the same HoverNet pipeline used for GADVR (Section IV-A: 'we generate segmentation labels using the same pipeline'). This means the PathGen results do not escape the label-bias problem: a model that overfits to HoverNet's annotation style can score well on both datasets. Independent labels or a manual evaluation subset are needed to support the generalization claim.
minor comments (5)
  1. [General] There are several typos and inconsistencies: 'reaoning' (Section II), 'muclei' (Section II), 'Backone' (Section III-A), 'TFLPOs' in Table I header, and inconsistent use of 'pixel-level' versus 'cell-level' throughout (the masks are nuclei/cell-level).
  2. [Table III caption] MMR-7B and MMR-13B are cited as [24] in the caption, but MMR is reference [31]. Please correct.
  3. [Section III-A, Eq. (3)] The penalty weight W is said to be 1.5, but the equation does not specify how W is applied to BCE/DICE losses or what pixel classes receive the penalty. Please clarify the exact weighting scheme.
  4. [Section IV-A] The reported number of image-text pairs is not exactly 3 times the number of patches: training patches 149,928 vs pairs 447,170; validation 14,814 vs 44,080; test 18,754 vs 56,067. Clarify whether some questions were dropped or sampling was not uniform.
  5. [Figure 8] The y-axis labels are unclear; please label the grouped bars explicitly so the reader can see that 'w/o all' corresponds to the MMR baseline, and indicate whether the same hyperparameters are used across ablations.

Circularity Check

2 steps flagged · score 6.0 of 10

Closed-loop labels: text quality is BLEU/F1 against GPT-4o answers and segmentation accuracy is IoU against HoverNet masks, the same automated generators that produced the training targets, so the reported 'expert-level' performance partially reduces to imitation of the labelers.

  1. fitted input called prediction [Section II-B2 (Answer Generation), Eq. (6), Section IV-C (Table IV)]
    "For each image, three questions are randomly sampled from the question pool and answered using GPT-4o. ... both the selected questions and the corresponding nuclei segmentation masks are provided as inputs to the model. ... Ltxt = CE (ˆytxt, ytxt). (6) ... For text conversation, we assess performance using BLEU-4 and F1 scores."

    The training target ytxt in the cross-entropy loss is the GPT-4o-generated answer, and the evaluation metric (BLEU-4/F1) is computed against the same GPT-4o-generated reference distribution. Thus 'text generation quality' is defined as agreement with GPT-4o, the very model that produced the training labels. The pathologist QC in Section II-D covers only 200 samples, not the 56,067-pair test set, so the reported 'expert-level' text quality is not independently validated; it reduces to fidelity to the label generator.

  2. fitted input called prediction [Section II-B1, Eq. (3), Section IV-A/C (Tables I-II)]
    "The remaining high-correlation patches are then used to generate pixel-level nuclei segmentation and classification labels using a pretrained HoverNet model [34]. ... Lmask = λ1BCE( ˆM , M, W) +λ2DICE( ˆM , M, W), (3) ... For segmentation, we use gIoU and cIoU as evaluation metrics, following prior works [24], [31]. ... Since PathGen does not provide cell-level segmentation annotations, we generate segmentation labels using the same pipeline employed for GADVR."

    The ground-truth masks M in the segmentation loss are HoverNet outputs, and the reported gIoU/cIoU scores are computed against the same HoverNet-generated masks. Training minimizes BCE/Dice to HoverNet and testing measures IoU to HoverNet, so 'segmentation accuracy' is agreement with HoverNet by construction. The manual correction of ~200 patches and the 200-sample pathologist QC do not cover the full 18,754-patch test set, and the PathGen labels are generated with the same pipeline, so the generalization results inherit the closed loop.

full rationale

The model architecture and its loss formulation (Eqs. 1-7) are self-contained, and the relative ranking of methods on the published benchmark is not itself circular: all baselines are scored against the same reference labels. The circularity lies in the interpretation of those labels as 'quality' and 'accuracy'. For text, the references are GPT-4o answers, which are also the training targets (Eq. 6); BLEU-4/F1 therefore measures imitation of GPT-4o, not independent clinical correctness. For segmentation, the references are HoverNet masks, which are also the training targets (Eq. 3); gIoU/cIoU therefore measures imitation of HoverNet. The 200-sample pathologist quality control in Section II-D is a stated external check, but it is too small to validate the full test sets (18,754 patches / 56,067 pairs) and no inter-observer agreement statistics are reported. Consequently, the abstract's claims of 'expert-level diagnostic explanations' and consistent superiority in 'text generation quality, segmentation accuracy, and cross-modal alignment' are partially self-referential: the reported numbers certify agreement with the automated labelers that generated the training data. This is a partial circularity rather than a fully forced derivation, so the score is 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests mainly on domain assumptions about the reliability of automatically generated labels. There are no new physical entities, but the dataset itself and its label generation pipeline carry the burden of representing expert clinical ground truth.

free parameters (3)
  • penalty weight W in Lmask = 1.5
    Hand-chosen weight in the BCE loss to penalize incorrect classification predictions, set in Eq. 3.
  • lambda1 and lambda2 in Lmask = 2 and 0.5
    Balancing coefficients for BCE and Dice losses, reported in Section III-A.
  • lambda_mask, lambda_txt, lambda_con = 1, 1, 1
    Default loss balancing weights in Eq. 7, no tuning analysis provided.
assumptions (4)
  • domain assumption PatchGastricADC22 patch labels inherited from WSI-level annotations are reliable enough for gastric adenocarcinoma subtype classification.
    Section II-A uses these labels as the foundation for GADVR without independent validation.
  • domain assumption Pretrained HoverNet segmentation and classification outputs are accurate enough to serve as pixel-level ground truth.
    Section II-B-1 generates all nuclei labels with HoverNet; the model is then trained and evaluated against these labels.
  • domain assumption GPT-4o generated question answer pairs are clinically meaningful and can serve as text reference labels.
    Section II-B-2 uses GPT-4o to generate all QA pairs, and Table IV measures text quality by agreement with these generated answers.
  • domain assumption CLIP-ViT based patch filtering removes weakly correlated patches without introducing selection bias.
    Section II-B-1 relies on this filtering step to build the dataset, but the threshold and bias analysis are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis." pith.science (2026). https://pith.science/paper/ORWE6YJX

@misc{pith2026250820851,
  author       = {Pith},
  title        = {Pith review of: PathMR: Multimodal Visual Reasoning for Interpretable Pathology Diagnosis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ORWE6YJX}},
  note         = {Machine review of arXiv:2508.20851}
}
read the original abstract

Deep learning based automated pathological diagnosis has markedly improved diagnostic efficiency and reduced variability between observers, yet its clinical adoption remains limited by opaque model decisions and a lack of traceable rationale. To address this, recent multimodal visual reasoning architectures provide a unified framework that generates segmentation masks at the pixel level alongside semantically aligned textual explanations. By localizing lesion regions and producing expert style diagnostic narratives, these models deliver the transparent and interpretable insights necessary for dependable AI assisted pathology. Building on these advancements, we propose PathMR, a cell-level Multimodal visual Reasoning framework for Pathological image analysis. Given a pathological image and a textual query, PathMR generates expert-level diagnostic explanations while simultaneously predicting cell distribution patterns. To benchmark its performance, we evaluated our approach on the publicly available PathGen dataset as well as on our newly developed GADVR dataset. Extensive experiments on these two datasets demonstrate that PathMR consistently outperforms state-of-the-art visual reasoning methods in text generation quality, segmentation accuracy, and cross-modal alignment. These results highlight the potential of PathMR for improving interpretability in AI-driven pathological diagnosis. The code will be publicly available in https://github.com/zhangye-zoe/PathMR.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages

  1. [1]

    Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unannotated pathology slides,

    C. Quiros et al., “Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unannotated pathology slides,” Nature Communications, vol. 15, no. 1, p. 4596, 2024

    work page 2024

  2. [2]

    Cancer survival prediction from whole slide images with self-supervised learning and slide consistency,

    L. Fan, A. Sowmya, E. Meijering, and Y . Song, “Cancer survival prediction from whole slide images with self-supervised learning and slide consistency,” IEEE Transactions on Medical Imaging , vol. 42, no. 5, pp. 1401–1412, 2022

    work page 2022

  3. [3]

    Transmil: Transformer based correlated multiple instance learning for whole slide image classification,

    Z. Shao, H. Bian, Y . Chen, Y . Wang, J. Zhang, and Y . Zhang, “Transmil: Transformer based correlated multiple instance learning for whole slide image classification,” Advances in neural information processing systems, vol. 34, pp. 2136–2147, 2021

    work page 2021

  4. [4]

    Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification,

    H. Zhang, Y . Meng, Y . Zhao, Y . Qiao, X. Yang, S. E. Coupland, and Y . Zheng, “Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 18 802–18 812

    work page 2022

  5. [5]

    From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment,

    K. Swanson, E. Wu, A. A. Alizadeh, and J. Zou, “From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatment,” Cell, vol. 186, no. 8, pp. 1772–1791, 2023

    work page 2023

  6. [6]

    A pathology foundation model for cancer diagnosis and prognosis prediction,

    X. Wang, J. Zhao, E. Marostica, W. Yuan, J. Jin, J. Zhang et al. , “A pathology foundation model for cancer diagnosis and prognosis prediction,” Nature, vol. 634, no. 8035, pp. 970–978, 2024

    work page 2024

  7. [7]

    Cross-scale multi-instance learning for pathological image diagnosis,

    R. Deng, C. Cui, L. W. Remedios, S. Bao, R. M. Womick, S. Chiron, J. Li, J. T. Roland, K. S. Lau, Q. Liu et al., “Cross-scale multi-instance learning for pathological image diagnosis,” Medical image analysis , vol. 94, p. 103124, 2024

    work page 2024

  8. [8]

    Dual-stream multiple instance learn- ing network for whole slide image classification with self-supervised contrastive learning,

    B. Li, Y . Li, and K. W. Eliceiri, “Dual-stream multiple instance learn- ing network for whole slide image classification with self-supervised contrastive learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 14 318–14 328

    work page 2021

Show all 41 references
  1. [9]

    Pathologist-level interpretable whole-slide cancer diagnosis with deep learning,

    Z. Zhang, P. Chen, M. McGough, F. Xing, C. Wang, M. Bui, Y . Xie, M. Sapkota, L. Cui, J. Dhillon et al. , “Pathologist-level interpretable whole-slide cancer diagnosis with deep learning,” Nature Machine Intelligence, vol. 1, no. 5, pp. 236–245, 2019

    2019

  2. [10]

    Quantifying explainers of graph neural networks in computational pathology,

    G. Jaume, P. Pati, B. Bozorgtabar, A. Foncubierta, A. M. Anniciello, F. Feroce, T. Rau, J.-P. Thiran, M. Gabrani, and O. Goksel, “Quantifying explainers of graph neural networks in computational pathology,” in Proceedings of the IEEE/CVF conference on computer vision and patte...

    2021

  3. [11]

    Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer,

    N. Zamanitajeddin, M. Jahanifar, M. Bilal, M. Eastwood, and N. Ra- jpoot, “Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer,” Medical Image Analysis , vol. 93, p. 103071, 2024

    2024

  4. [12]

    Wsi-vqa: Interpreting whole slide images by generative visual question answering,

    P. Chen, C. Zhu, S. Zheng, H. Li, and L. Yang, “Wsi-vqa: Interpreting whole slide images by generative visual question answering,” in Euro- pean Conference on Computer Vision . Springer, 2024, pp. 401–417

    2024

  5. [13]

    Wsicaption: Multiple instance generation of pathology reports for gigapixel whole- slide images,

    P. Chen, H. Li, C. Zhu, S. Zheng, Z. Shui, and L. Yang, “Wsicaption: Multiple instance generation of pathology reports for gigapixel whole- slide images,” in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2024, pp. 546–556

    2024

  6. [14]

    Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration,

    Y . Sun, Y . Zhang, Y . Si, C. Zhu, Z. Shui, K. Zhang, J. Li, X. Lyu, T. Lin, and L. Yang, “Pathgen-1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration,” ICLR, 2024

    2024

  7. [15]

    Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction,

    Z. Guo, J. Ma, Y . Xu, Y . Wang, L. Wang, and H. Chen, “Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction,” in International Conference on Med- ical Image Computing and Computer-Assisted Intervention . Springer, 202...

    2024

  8. [16]

    A visual-language foundation model for computational pathology,

    M. Y . Lu, B. Chen, D. F. Williamson, R. J. Chen, I. Liang, T. Ding, G. Jaume, I. Odintsov, L. P. Le, G. Gerber et al. , “A visual-language foundation model for computational pathology,” Nature medicine , vol. 30, no. 3, pp. 863–874, 2024

    2024

  9. [17]

    A foundation model for clinical-grade computational pathology and rare cancers detection,

    E. V orontsov, A. Bozkurt, A. Casson, G. Shaikovski, M. Zelechowski, N. Fusi et al. , “A foundation model for clinical-grade computational pathology and rare cancers detection,” Nature medicine, vol. 30, no. 10, pp. 2924–2935, 2024

    2024

  10. [18]

    A whole-slide foundation model for digital pathology from real-world data,

    H. Xu, N. Usuyama, J. Bagga, S. Zhang, R. Rao, T. Naumann, C. Wong, Z. Gero, J. Gonz ´alez, Y . Guet al., “A whole-slide foundation model for digital pathology from real-world data,” Nature, vol. 630, no. 8015, pp. 181–188, 2024

    2024

  11. [19]

    A multimodal knowledge-enhanced whole-slide pathology foundation model,

    Y . Xu, Y . Wang, F. Zhou, J. Ma, C. Jin, S. Yang, et al., “A multimodal knowledge-enhanced whole-slide pathology foundation model,” arXiv preprint arXiv:2407.15362, 2024

    2024 arXiv

  12. [20]

    Florence-2: Advancing a unified representation for a variety of vision tasks,

    B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y . Lu, M. Zeng, C. Liu, and L. Yuan, “Florence-2: Advancing a unified representation for a variety of vision tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 4818–4829

    2024

  13. [21]

    Groma: Localized visual tokenization for grounding multimodal large language models,

    C. Ma, Y . Jiang, J. Wu, Z. Yuan, and X. Qi, “Groma: Localized visual tokenization for grounding multimodal large language models,” in European Conference on Computer Vision , 2024, pp. 417–435

    2024

  14. [22]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding,

    T. Zhang, X. Li, H. Fei, H. Yuan, S. Wu, S. Ji, C. C. Loy, and S. Yan, “Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding,” Advances in Neural Information Processing Sys- tems, vol. 37, pp. 71 737–71 767, 2024

    2024

  15. [23]

    Lisa: Reasoning segmentation via large language model,

    X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Yuan, S. Liu, and J. Jia, “Lisa: Reasoning segmentation via large language model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9579–9589

    2024

  16. [24]

    Pixellm: Pixel reasoning with large multimodal model,

    Z. Ren, Z. Huang, Y . Wei, Y . Zhao, D. Fu, J. Feng, and X. Jin, “Pixellm: Pixel reasoning with large multimodal model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 26 374–26 383

    2024

  17. [25]

    Glamm: Pixel grounding large multimodal model,

    H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan, “Glamm: Pixel grounding large multimodal model,” in Proceedings of Computer Vision and Pattern Recognition, 2024, pp. 13 009–13 018

    2024

  18. [26]

    Perceptiongpt: Effectively fusing visual perception into llm,

    R. Pi, L. Yao, J. Gao, J. Zhang, and T. Zhang, “Perceptiongpt: Effectively fusing visual perception into llm,” in Proceedings of the IEEE/CVF Conference on CVPR , 2024, pp. 27 124–27 133

    2024

  19. [27]

    Gsva: Generalized segmentation via multimodal large language models,

    Z. Xia, D. Han, Y . Han, X. Pan, S. Song, and G. Huang, “Gsva: Generalized segmentation via multimodal large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 3858–3869

    2024

  20. [28]

    Visual programming for zero-shot open-vocabulary 3d visual grounding,

    Z. Yuan, J. Ren, C.-M. Feng, H. Zhao, S. Cui, and Z. Li, “Visual programming for zero-shot open-vocabulary 3d visual grounding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 20 623–20 633

    2024

  21. [29]

    Prima: Multi-image vision-language models for reasoning segmentation,

    M. Wahed, K. A. Nguyen, A. S. Juvekar, X. Li, X. Zhou, V . Shah, T. Yu, and P. Yanardag, “Prima: Multi-image vision-language models for reasoning segmentation,” arXiv preprint arXiv:2412.15209 , 2024

    2024

  22. [30]

    Geopixel: Pixel grounding large multimodal model in remote sensing,

    A. Shabbir, M. Zumri, M. Bennamoun, F. S. Khan, and S. Khan, “Geopixel: Pixel grounding large multimodal model in remote sensing,” arXiv preprint arXiv:2501.13925 , 2025

    2025 arXiv

  23. [31]

    Mmr: A large- scale benchmark dataset for multi-target and multi-granularity reasoning segmentation,

    D. Jang, Y . Cho, S. Lee, T. Kim, and D. Kim, “Mmr: A large- scale benchmark dataset for multi-target and multi-granularity reasoning segmentation,” in ICLR, 2025

    2025

  24. [32]

    Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification,

    J. Gamper, N. Alemi Koohbanani, K. Benet, A. Khuram, and N. Rajpoot, “Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification,” in Digital Pathology: 15th European Congress, ECDP 2019, Warwick, UK, April 10–13, 2019, Proceedings

    2019

  25. [33]

    Springer, 2019, pp. 11–19

    2019

  26. [34]

    Inference of captions from histopatho- logical patches,

    M. Tsuneki and F. Kanavati, “Inference of captions from histopatho- logical patches,” in International Conference on Medical Imaging with Deep Learning. PMLR, 2022, pp. 1235–1250

    2022

  27. [35]

    Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images,

    S. Graham, Q. D. Vu, S. E. A. Raza, A. Azam, Y . W. Tsang, J. T. Kwak, and N. Rajpoot, “Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images,” Medical image analysis, vol. 58, p. 101563, 2019

    2019

  28. [36]

    Open-vocabulary semantic segmentation with mask-adapted clip,

    F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, “Open-vocabulary semantic segmentation with mask-adapted clip,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 7061–7070

    2023

  29. [37]

    A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities,

    T. Zhao, Y . Gu, J. Yang, N. Usuyama, H. H. Lee, S. Kiblawi, T. Nau- mann, J. Gao, A. Crabtree, J. Abel et al., “A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities,” Nature methods, pp. 1–11, 2024

    2024

  30. [38]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4015–4026

    2023

  31. [39]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning , 2021, pp. 8748–8763

    2021

  32. [40]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems , vol. 36, 2024

    2024

  33. [41]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” Advances in Neural Information Processing Systems, vol. 36, pp. 28 541–28 564, 2023

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.