Pith. sign in

REVIEW 5 major objections 5 minor 54 references

This paper claims that standard lexical metrics (BLEU, ROUGE, METEOR) are weakly aligned with clinical correctness in pathology report generation and often overestimate report quality, and it proposes a structured clinical metric, CRQS, tha

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 15:20 UTC pith:VT3VQART

load-bearing objection Useful standardized benchmark for pathology report generation, but the CRQS metric needs an extraction audit and a corrected normalization before the central claim about lexical metrics is accepted. the 5 major comments →

arxiv 2607.18448 v1 pith:VT3VQART submitted 2026-07-20 cs.CL cs.CV

PathReportEval: A Systematic Benchmark for Pathology Report Generation

Suryakant Singh , Sejuti Majumder , Beatrice Knudsen , Joel Saltz , Prateek Prasanna This is my paper
classification cs.CL cs.CV
keywords pathology report generationwhole-slide imagesclinical fidelity evaluationCRQShallucination detectionbenchmarklexical metricsvisual encoders
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that progress in pathology report generation is being mis-measured. Standard metrics like BLEU, ROUGE, and METEOR reward lexical overlap and can miss clinically consequential errors — a generated report can look fluent while omitting a diagnosis, hallucinating a finding, or contradicting a grade. To fix this, the authors propose the Clinical Report Quality Score (CRQS), which maps both reference and generated reports into structured clinical attributes and scores them on fact coverage, key-information recall, hallucination rate, and clinical discordance. They build a standardized benchmark across three datasets and three pathology visual encoders, showing that lexical metrics and CRQS rank models differently and that encoder choice has clinically meaningful effects that lexical metrics hide. If correct, the field should shift from evaluating surface similarity to evaluating clinical fidelity.

Core claim

The central claim is that clinical correctness in pathology report generation is only weakly aligned with lexical similarity, and that the field's reliance on BLEU, ROUGE, and METEOR therefore overestimates quality and obscures real differences between models. The paper's contribution is CRQS, a structured metric that extracts clinical attributes from reports — diagnosis, histologic type, tumor grade, invasion status, and other dataset-specific fields — and scores four complementary components: clinical fact coverage, key-information recall, hallucination rate, and clinical discordance. A weighted combination gives an overall score normalized to [0,1]. Across four generation methods and thre

What carries the argument

The central object is the Clinical Report Quality Score (CRQS), defined by Eq. (1)–(6). Its load-bearing component is the structured extraction function E(R) = {(f_i, v_i)} that maps a free-text pathology report into a schema of clinical fields with normalized values (synonym resolution, negation handling, clinical equivalence). CRQS then computes four sub-scores — clinical fact coverage (CFC), key-information recall (KIR), hallucination rate (HR), and clinical discordance score (CDS) — and combines them with fixed weights before normalization by 0.7. The metric is designed around synoptic reporting practice, prioritizing fields that are clinically actionable and inferable from the whole-sli

Load-bearing premise

The entire CRQS argument depends on the assumption that the automated extraction of clinical fields from free-text reports is accurate enough — in synonym resolution, negation handling, and clinical-equivalence matching — that the observed disagreements between lexical and clinical scores reflect real clinical differences rather than extraction errors; the paper itself acknowledges this in Appendix A.5 but provides no quantitative validation.

What would settle it

Take a random sample of, say, 100 generated reports and have two pathologists independently annotate the presence and value of each clinical field in the CRQS schema. If pathologist-annotated clinical correctness correlates as strongly with BLEU/ROUGE/METEOR as with CRQS (or if pathologists cannot reproduce the extraction pipeline's field values), then the central misalignment claim would be undermined. Alternatively, if CRQS scores computed on gold manual extractions flip the ranking of models compared to CRQS computed with the automated extraction, the metric is measuring extraction fidelity

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation norms in pathology report generation should shift from lexical overlap metrics to structured clinical fidelity scoring; models that look equivalent under BLEU/ROUGE/METEOR may be ranked differently by CRQS.
  • The specific sub-scores make failure modes actionable: a model with low KIR omits key diagnoses; high HR indicates hallucinated findings; high CDS points to direct contradictions.
  • Encoder choice has clinically meaningful effects that lexical metrics mask, so future studies should report CRQS per encoder rather than collapsing across them.
  • The public benchmark with fixed splits and a modular pipeline allows new methods, datasets, and encoders to be compared under the same protocol, making reported gains attributable to model design.
  • Because CRQS targets structured clinical facts, optimizing for it may yield reports that are more clinically faithful even if they use different phrasing than references.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If CRQS is adopted, the objective for training report generators could shift from imitating reference phrasing to maximizing structured clinical fidelity, potentially changing which architectural choices dominate.
  • The same structured-extraction-plus-scoring scheme could be transferred to adjacent domains (radiology, surgical pathology, discharge summaries) where lexical metrics are equally suspected of masking clinical errors; a cross-domain validation would test the generality of the misalignment finding.
  • The paper's own limitation note (Appendix A.5) indicates the extraction function is not quantitatively validated against a gold standard; a direct validation of E(R) on manually annotated reports would either confirm CRQS's reliability or reveal that some of the observed misalignment is an artifact of extraction noise.
  • One testable consequence: if generated reports are reviewed by practicing pathologists, their clinical quality rankings should correlate more strongly with CRQS than with BLEU/ROUGE/METEOR; a study with a modest sample would settle the claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PathReportEval, a standardized benchmark for pathology report generation from whole-slide images. It evaluates four generation methods (WSI-Caption, HistGen, BiGen, SCOUT) across three datasets (TCGA, HistAI, REG 2025) and three pathology visual encoders (CONCHv1.5, UNI2-h, H-Optimus-1) under a unified preprocessing, feature-extraction, training, and decoding protocol. The central contribution is CRQS, a structured clinical metric that extracts clinical fields from reference and generated reports and computes four sub-scores: clinical fact coverage (CFC), key information recall (KIR), hallucination rate (HR), and clinical discordance score (CDS). The paper claims that standard NLG metrics (BLEU, ROUGE, METEOR) are weakly aligned with clinical correctness and often overestimate report quality, whereas CRQS reveals clinically meaningful differences between models and encoders. The manuscript includes quantitative tables, qualitative examples, and a public framework repository.

Significance. If the central claims hold, the paper addresses a real evaluation gap in pathology report generation: lexical metrics can miss clinically consequential errors, and a standardized benchmark with a clinically grounded metric would be a useful community resource. The authors' effort to standardize preprocessing, feature extraction, model training, decoding, and evaluation across three datasets and three encoders is valuable, and the public plug-and-play framework is a concrete contribution. However, the central empirical claim rests on CRQS, whose validity is not yet established: its LLM-based extraction is not validated against a gold standard, and the metric has internal inconsistencies. With rigorous validation and correction, this could become an important benchmark for the field.

major comments (5)
  1. [§3.4, Eqs. (6)–(8), Table 3] The CRQS weighting scheme is internally inconsistent. In Eq. (6), CRQS_raw = 0.30*CFC + 0.40*KIR + 0.20*(1-HR) + 0.40*(1-CDS). The weights sum to 1.3, so the theoretical maximum raw score is 1.3, not 0.7 as stated. Dividing by 0.7 then maps the maximum to 1.857, contradicting the claim that CRQS ∈ [0,1]. The subtractive form in the Appendix prompt (CRQS = 0.3*CFC + 0.4*KIR - 0.2*HR - 0.4*CDS) would give a maximum of 0.7, but that is not the formula in Eq. (6). Additionally, Table 3 reports a negative CRQS (-0.0714), which is impossible under the claimed [0,1] range with Eq. (6). The authors must correct the formula and recompute all reported CRQS values, or explicitly justify a different normalization.
  2. [§3.4 and Appendix A.5] CRQS is computed entirely from the LLM-based structured extraction function E(R) defined in Eq. (1). All four sub-scores (CFC, KIR, HR, CDS) depend on the accuracy of this extraction, including synonym resolution, negation handling, and clinical-equivalence matching. Appendix A.5 acknowledges that 'the extraction process may still introduce errors or inconsistencies,' but the paper provides no quantitative validation of extraction accuracy against a labeled gold standard or expert-annotated set. Without such an audit, the reported disagreements between lexical metrics and CRQS could be artifacts of extraction noise rather than genuine clinical infidelity. The authors should report field-level precision/recall of extraction on a held-out sample per dataset, ideally with inter-annotator agreement, and show that the main conclusions are robust to extraction errors.
  3. [§4, Figure 3] The central empirical claim—that conventional NLG metrics are 'weakly aligned' with clinical correctness and 'frequently overestimate' report quality—is not supported by quantitative evidence. The paper mentions Figure 3 and a qualitative 'HR ~ 0.5' observation, but it reports no correlation coefficients (e.g., Spearman or Pearson) between BLEU/ROUGE/METEOR and CRQS across datasets, models, or encoders. Figure 3 appears to show scatter plots, but no numerical correlation or regression analysis is given. The claim of weak alignment requires a quantitative measure of agreement (or disagreement) between lexical metrics and CRQS, with confidence intervals, rather than two illustrative examples in Table 3.
  4. [§3.4, Appendix A.7, Tables 7–12] The dataset-specific clinical schemas and key fields are produced by a prompt authored by the authors (A.7), with the assertion that a practicing pathologist was consulted. However, no details are provided about the level of pathologist involvement, the number of cases reviewed, or the agreement between the resulting schema and expert clinical judgment. Because CRQS is the benchmark's evaluation yardstick, the schema derivation must be transparent and reproducible. The authors should document the pathologist's role, provide the actual schema-generation procedure, and ideally include a small validation study showing that the key fields align with independent clinical preferences.
  5. [Tables 2 and 4–6] All experimental comparisons are reported as point estimates without variance, confidence intervals, or significance tests. Many of the CRQS differences that the paper interprets as 'clinically meaningful' are small (e.g., 0.1861 vs. 0.1784 on TCGA in Table 2a), and without repeated seeds or statistical testing it is unclear whether these differences are reliable. The benchmark's conclusions about model and encoder rankings require at least standard deviations or confidence intervals, and preferably paired tests across test samples.
minor comments (5)
  1. [Appendix A.2] Typo: 'available memory per gpu is 32 GB and and a batch size' contains a duplicated 'and'.
  2. [Reference [23]] The author list for FACTent appears to include 'H. Langlotz, C. Langlotz'; likely a formatting error. Please verify and correct.
  3. [Abstract] The abstract says 'CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions'; consider clarifying in the abstract that CRQS is LLM-extraction-based and requires validation, since this is a key assumption.
  4. [Figure 3] The figure caption says 'METEOR vs CRQS across datasets,' but the main text does not describe the axes or whether each point is a single report or a method-average. Clarify the plotting unit and add correlation coefficients or trend lines.
  5. [Appendix A.7] The long prompt is reproduced verbatim; consider summarizing it in the main text and noting that the exact prompt is available in the repository. The current placement is acceptable for reproducibility but makes the appendix lengthy.

Circularity Check

0 steps flagged

No equation-level circularity; CRQS is a proposed rubric rather than a derived quantity, so the benchmark conclusions rest on construct validity rather than a self-referential reduction.

full rationale

Walking the derivation chain from Eq. (1) through Eq. (8), CRQS is defined, not derived: E(R) maps reports to structured fields; CFC, KIR, HR, and CDS are arithmetic fractions over those fields; CRQS is a weighted combination. No component is fitted to the data whose prediction is later presented as an independent result, and no parameter is estimated from benchmark outputs and then renamed as a discovery. The central empirical claim (lexical metrics weakly align with clinical correctness) is a direct comparison of METEOR/BLEU/ROUGE against CRQS; it does not reduce to the definition of CRQS by construction. The dataset-specific clinical fields are generated via an LLM prompt written by the authors (Appendix A.7), and the CRQS weights in Eq. (6)-(8) are author-chosen, making CRQS a constructed rubric rather than a mathematical consequence. Appendix A.5 concedes that 'the extraction process may still introduce errors or inconsistencies,' which is a validation gap, not circularity. I also note the internal inconsistency that the weights in Eq. (6) sum to 1.3 while Eq. (8) divides by 0.7, and Table 3 reports a negative CRQS; this undermines the claimed [0,1] normalization but is a numerical bug, not a circular step. Self-citation exists (SCOUT, ref [11], is from the same group), but the benchmark's central comparison does not rely on a uniqueness theorem or on SCOUT's prior claims. No specific circular reduction is exhibited, so the score is low; the mild self-referential character of one group designing both the metric and the benchmark prevents a score of 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims (lexical metrics misalign with clinical correctness; CRQS reveals meaningful differences) depend on (a) the hand-chosen CRQS weights, (b) the author-constructed dataset-specific schemas, (c) an unvalidated LLM extraction pipeline, and (d) a normalization whose formula is inconsistent between the main text and the appendix prompt. These are the real prices of admission for the paper's conclusions.

free parameters (3)
  • CRQS weights w1, w2, w3, w4 = 0.30, 0.40, 0.20, 0.40
    Chosen by hand in Eq. (7) with the stated rationale that discordance is more dangerous than hallucination. No sensitivity analysis, no clinician-preference elicitation, and no independent validation of the weights, yet they determine the aggregate CRQS that drives all cross-model and cross-encoder conclusions.
  • Normalization constant 0.7 = 0.7
    Computed from the chosen weights; the main-text formula (Eq. 6-7) has max 1.3 while the appendix prompt uses max 0.7, so the normalization is internally inconsistent and affects all reported CRQS values.
  • Dataset-specific clinical schemas and key fields = e.g., TCGA 9 key fields, HistAI 10, REG 5
    The clinical field lists and key-field selections are constructed for each dataset via the authors' prompt (A.7) with a single pathologist consultation. These choices determine which facts count as covered, key, hallucinated, or discordant, and hence every CRQS number.
axioms (4)
  • domain assumption Clinical equivalence matching (≡) between extracted values is reliable after normalization and synonym resolution.
    Invoked in Eqs. (2)-(5); the paper provides no error analysis of the LLM extraction/matching pipeline, only a statement in A.5 that errors are possible.
  • domain assumption The predefined clinical-field schemas are sufficient and appropriate for capturing clinically consequential pathology report content in each dataset.
    Introduced in Sec. 3.4; the entire CRQS decomposition is defined over these fields. If important fields are missing or mis-specified, CRQS can miss consequential errors.
  • domain assumption Pathologist verification of extracted fields is accurate for the full test sets.
    Claimed in Sec. 3.4 and A.5, but no protocol, inter-observer agreement, or sample size is given, so the reader cannot assess how thoroughly verification was done.
  • domain assumption TRIDENT-based preprocessing at 20x magnification with encoder-specific patch configurations is adequate for all three datasets and four methods.
    Invoked in Sec. 3.3 and A.1 as the standardized feature extraction protocol; this choice fixes the visual information available to all models.
invented entities (1)
  • CRQS (Clinical Report Quality Score) no independent evidence
    purpose: A new evaluation metric that maps reports into structured clinical attributes and scores fact coverage, key recall, hallucination, and discordance.
    The metric is introduced and used in this paper; its validation is internal (pathologist-verified extraction and face-valid qualitative examples) with no independent benchmark showing it correlates better than lexical metrics with a ground-truth clinical quality judgment.

pith-pipeline@v1.3.0-alltime-deepseek · 15529 in / 9183 out tokens · 64527 ms · 2026-08-01T15:20:32.411107+00:00 · methodology

0 comments
read the original abstract

Pathology report generation from whole-slide images (WSIs) is a rapidly growing multimodal learning problem, yet progress is difficult to measure because existing studies use heterogeneous datasets, model settings, visual encoders, and evaluation protocols. Moreover, commonly used natural language generation metrics, including BLEU, ROUGE, and METEOR, primarily reward lexical similarity and often fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes. We present a standardized benchmark and evaluation framework for pathology report generation. The benchmark evaluates four representative methods across three datasets (TCGA, HistAI, and REG 2025) using three pathology foundation encoders (CONCHv1.5, UNI2-h, and H-Optimus-1). Our framework standardizes preprocessing, feature extraction, training, decoding, and evaluation, enabling fair comparison across models while providing a modular platform for integrating new methods, datasets, and encoders. A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness. CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance, producing both an overall score and interpretable sub-scores. Experiments demonstrate that conventional language-generation metrics are weakly aligned with clinical correctness and frequently overestimate report quality. In contrast, CRQS reveals clinically meaningful differences between models and encoders that lexical metrics fail to capture. Together, the benchmark, public plug-and-play framework, and CRQS establish a reproducible foundation for rigorous evaluation of pathology report generation.

Figures

Figures reproduced from arXiv: 2607.18448 by Beatrice Knudsen, Joel Saltz, Prateek Prasanna, Sejuti Majumder, Suryakant Singh.

Figure 1
Figure 1. Figure 1: Overview of the proposed pathology report generation benchmark and evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dataset characteristics across the three pathology report generation benchmarks. Left: [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: METEOR vs CRQS across datasets. Encoder Sensitivity [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Vision encoder sensitivity across TCGA, REG 2025, and HistAI. The grouped bars show [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Impact of different vision encoders on Clinical Report Quality Score (CRQS) across [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 5 linked inside Pith

  1. [1]

    Pathology report generation and multimodal representation learning for cutaneous melanocytic lesions,

    R. T. Lucassen, S. P. Moonemans, T. van de Luijtgaarden, G. E. Breimer, W. A. Blokx, and M. Veta, “Pathology report generation and multimodal representation learning for cutaneous melanocytic lesions,” inInternational Conference on Medical Image Computing and Computer- Assisted Intervention. Springer, 2025, pp. 502–511

    2025

  2. [2]

    Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection,

    D. Hu, Z. Jiang, J. Shi, F. Xie, K. Wu, K. Tang, M. Cao, J. Huai, and Y . Zheng, “Pathology report generation from whole slide images with knowledge retrieval and multi-level regional feature selection,”Computer Methods and Programs in Biomedicine, vol. 263, p. 108677, 2025

    2025

  3. [3]

    Grounded report generation for enhancing ophthalmic ultrasound interpretation using vision- language segmentation models,

    K. Jin, Q. Sun, D. Kang, Z. Luo, T. Yu, W. Han, Y . Zhang, M. Wang, D. Shi, and A. Grzybowski, “Grounded report generation for enhancing ophthalmic ultrasound interpretation using vision- language segmentation models,”npj Digital Medicine, 2026

    2026

  4. [4]

    S2d-align: Shallow-to-deep auxiliary learning for anatomically- grounded radiology report generation,

    J. Gao, C. Liu, and Y . Li, “S2d-align: Shallow-to-deep auxiliary learning for anatomically- grounded radiology report generation,”arXiv preprint arXiv:2511.11066, 2025

    arXiv 2025

  5. [5]

    A visual-language foundation model for computational pathology,

    M. Y . Lu, B. Chen, D. F. Williamson, R. J. Chen, I. Liang, T. Ding, G. Jaume, I. Odintsov, L. P. Le, G. Gerberet al., “A visual-language foundation model for computational pathology,”Nature medicine, vol. 30, no. 3, pp. 863–874, 2024

    2024

  6. [6]

    Towards a general-purpose foundation model for computational pathology,

    R. J. Chen, T. Ding, M. Y . Lu, D. F. Williamson, G. Jaume, A. H. Song, B. Chen, A. Zhang, D. Shao, M. Shabanet al., “Towards a general-purpose foundation model for computational pathology,”Nature Medicine, vol. 30, no. 3, pp. 850–862, 2024

    2024

  7. [7]

    H-optimus-1,

    Bioptimus, “H-optimus-1,” 2025. [Online]. Available: https://huggingface.co/bioptimus/ H-optimus-1

    2025

  8. [8]

    Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction,

    Z. Guo, J. Ma, Y . Xu, Y . Wang, L. Wang, and H. Chen, “Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction,” 2024. [Online]. Available: https://arxiv.org/abs/2403.05396

    Pith/arXiv arXiv 2024

  9. [9]

    Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images,

    P. Chen, H. Li, C. Zhu, S. Zheng, Z. Shui, and L. Yang, “Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images,” 2024. [Online]. Available: https://arxiv.org/abs/2311.16480

    Pith/arXiv arXiv 2024

  10. [10]

    Historical report guided bi-modal concurrent learn- ing for pathology report generation,

    L. Zhang, B. Yun, Q. Li, and Y . Wang, “Historical report guided bi-modal concurrent learn- ing for pathology report generation,” inMedical Image Computing and Computer Assisted Intervention – MICCAI 2025, J. C. Gee, D. C. Alexander, J. Hong, J. E. Iglesias, C. H. Sudre, A. Venkataraman, P. Golland, J. H. Kim, and J. Park, Eds. Cham: Springer Nature Switz...

    2025

  11. [11]

    Semantic context-aware modality fusion transformer (scout): A context-aware multimodal transformer for concept-grounded pathology report generation,

    S. Singh, S. Kapse, J. Saltz, and P. Prasanna, “Semantic context-aware modality fusion transformer (scout): A context-aware multimodal transformer for concept-grounded pathology report generation,” 2026. [Online]. Available: https://arxiv.org/abs/2605.01144

    Pith/arXiv arXiv 2026

  12. [12]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  13. [13]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText Summarization Branches Out, 2004, pp. 74–81

    2004

  14. [14]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” inProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65–72

    2005

  15. [15]

    Evaluating progress in automatic chest x-ray radiology report generation,

    F. Yu, M. Endo, R. Krishnan, I. Pan, A. Tsai, E. P. Reis, E. K. U. N. Fonseca, H. M. H. Lee, Z. S. H. Abad, A. Y . Ng, C. P. Langlotz, V . K. Venugopal, and P. Rajpurkar, “Evaluating progress in automatic chest x-ray radiology report generation,”Patterns, vol. 4, no. 9, p. 100802, 2023

    2023

  16. [16]

    Crimson: A clinically-grounded LLM-based metric for genera- tive radiology report evaluation,

    M. Baharoon, T. Heintz, S. Raissi, M. Alabbad, M. Alhammad, H. AlOmaish, S. E. Kim, O. Banerjee, and P. Rajpurkar, “Crimson: A clinically-grounded LLM-based metric for genera- tive radiology report evaluation,”arXiv preprint arXiv:2603.06183, 2026

    arXiv 2026

  17. [17]

    Cancer reporting tools,

    College of American Pathologists, “Cancer reporting tools,” https://www.cap.org/protocols-and- guidelines/cancer-reporting-tools/cancer-protocols, 2023. 11

    2023

  18. [18]

    [Online]

    ——,Protocol for the Examination of Specimens From Patients With Invasive Carcinoma of the Breast, College of American Pathologists, Northfield, IL, 2023, version 4.5.0.0. [Online]. Available: https://www.cap.org/protocols-and-guidelines/cancer-reporting-tools/ cancer-protocol-templates

    2023

  19. [19]

    Multiple instance captioning: Learning representations from histopathology textbooks and articles,

    J. Gamper and N. Rajpoot, “Multiple instance captioning: Learning representations from histopathology textbooks and articles,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 549–16 559

    2021

  20. [20]

    A visual–language foundation model for pathology image analysis using medical twitter,

    Z. Huang, F. Bianchi, M. Yuksekgonul, T. J. Montine, and J. Zou, “A visual–language foundation model for pathology image analysis using medical twitter,”Nat. Med., vol. 29, no. 9, pp. 2307– 2316, 2023

    2023

  21. [21]

    Visual language pretrained multiple instance zero-shot transfer for histopathology images,

    M. Y . Lu, B. Chen, A. Zhang, D. F. Williamson, R. J. Chen, T. Ding, L. P. Le, Y .-S. Chuang, and F. Mahmood, “Visual language pretrained multiple instance zero-shot transfer for histopathology images,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 764–19 775

    2023

  22. [22]

    Radgraph: Extracting clinical entities and relations from radiology reports,

    S. Jain, A. Agrawal, A. Saporta, S. Q. Truong, D. N. Duong, T. Bui, P. Chambon, Y . Zhang, M. P. Lungren, A. Y . Ng, C. P. Langlotz, and P. Rajpurkar, “Radgraph: Extracting clinical entities and relations from radiology reports,” 2021. [Online]. Available: https://arxiv.org/abs/2106.14463

    Pith/arXiv arXiv 2021

  23. [23]

    Improving factual completeness and consistency of image-to-text radiology report generation,

    Y . Miura, Y . Zhang, T.-H. Chen, H. Langlotz, and C. Langlotz, “Improving factual completeness and consistency of image-to-text radiology report generation,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 4488–4504. [Online]. Available: https://ac...

    2021

  24. [24]

    Histai: An efficient and robust whole-slide imaging repository for computational pathology challenges,

    J. Bulte, A. Hering, M. Schmitt, M. Veta, N. Brieu, M. A. Kimm, J. van der Laak, and G. Litjens, “Histai: An efficient and robust whole-slide imaging repository for computational pathology challenges,”Scientific Data, vol. 11, no. 1, p. 543, 2024

    2024

  25. [25]

    Miccai 2025 workshop on computational pathology: Report generation challenge,

    MICCAI COMPAY Workshop Organizers, “Miccai 2025 workshop on computational pathology: Report generation challenge,” 2025, challenge website and dataset description. [Online]. Available: https://reg2025.grand-challenge.org/reg2025

    2025

  26. [26]

    Accelerating data processing and benchmarking of ai models for pathology,

    A. Zhang, G. Jaume, A. Vaidya, T. Ding, and F. Mahmood, “Accelerating data processing and benchmarking of ai models for pathology,” 2025. [Online]. Available: https://arxiv.org/abs/2502.06750

    Pith/arXiv arXiv 2025

  27. [27]

    Trident: Gigapath slide processing and feature extraction toolkit,

    Mahmood Lab, “Trident: Gigapath slide processing and feature extraction toolkit,” GitHub repository, 2025

    2025

  28. [28]

    A multimodal whole-slide foundation model for pathology,

    T. Ding, S. J. Wagner, A. H. Song, R. J. Chen, M. Y . Lu, A. Zhang, A. J. Vaidya, G. Jaume, M. Shaban, A. Kimet al., “A multimodal whole-slide foundation model for pathology,”Nature medicine, pp. 1–13, 2025. 12 A Technical appendices and supplementary material A.1 Benchmark and Evaluation Framework TRIDENT provides a standardized pipeline that performs ti...

    arXiv 2025

  29. [29]

    ground-truth pathology report

  30. [30]

    Build a dataset-specific schema based on the dataset structure and the pathology content

    predicted/generated pathology report Do NOT use a universal schema. Build a dataset-specific schema based on the dataset structure and the pathology content. Important clinical constraint: The fields should prioritize facts that are clinically important AND inferable from the histology slide/WSI whenever possible. Do not over-prioritize metadata fields su...

  31. [31]

    Diagnosis hierarchy - benign_vs_malignant - diagnosis - primary_vs_metastatic - lineage - histologic_type

  32. [32]

    Tumor behavior - in_situ_vs_invasive - in_situ_component

  33. [33]

    Grade / aggressiveness - tumor_grade - differentiation - dataset-specific grades such as Gleason score, WHO grade, nuclear grade, mitotic score, tubule score 15

  34. [34]

    Invasion hierarchy - invasion_status - invasion_depth - organ-specific invasion - lymphovascular_invasion - perineural_invasion

  35. [35]

    Premalignant / epithelial change - dysplasia_grade - epithelial_atypia

  36. [36]

    Associated morphology - necrosis - inflammation_status - calcification - tumor_volume or tumor_extent if visually meaningful For this dataset, do the following step by step: Step 1: Read and summarize the dataset. Explain: - file structure - keys/columns - whether it has target only or target + prediction - number of cases if visible - report style - orga...

  37. [37]

    Diagnosis Hierarchy benign_vs_malignant Benign versus malignant disease classification diagnosis Primary pathologic diagnosis primary_vs_metastatic Distinguishes primary tumor from metastatic disease lineage Cellular/tissue lineage of tumor histologic_type Histologic subtype of lesion/tumor

  38. [38]

    Tumor Behavior in_situ_vs_invasive Whether lesion is in-situ or invasive in_situ_component Presence of in-situ neoplastic component

  39. [39]

    Grade / Aggressiveness tumor_grade Overall tumor grade/aggressiveness differentiation Degree of differentiation gleason_score Prostate Gleason grading score grade_group ISUP prostate grade group nuclear_grade Cytologic/nuclear atypia grade mitotic_score Mitotic activity score/count

  40. [40]

    Invasion Hierarchy invasion_status Presence or absence of invasion invasion_depth Extent/depth of tissue invasion lymphovascular_invasion Tumor invasion into lymphatic or vascular spaces perineural_invasion Tumor invasion involving nerves extraprostatic_extension Tumor extension beyond prostate capsule

  41. [41]

    Premalignant / Epithelial Change dysplasia_grade Severity of epithelial dysplasia epithelial_atypia Presence of epithelial atypia

  42. [42]

    Associated Morphology necrosis Presence of tumor necrosis inflammation_status Presence of inflammatory changes inflammation_activity Degree/activity of inflammation calcification Presence of calcifications tumor_extent Extent of tumor involvement tumor_volume Tumor burden or percentage involvement

  43. [43]

    GI-Specific Mucosal Pathology atrophy Mucosal or glandular atrophy intestinal_metaplasia Presence of intestinal metaplasia metaplasia_type Type/subtype of metaplasia hpylori_status Helicobacter pylori infection status gastritis_type Histologic subtype of gastritis olga_stage OLGA gastritis staging olgim_stage OLGIM intestinal metaplasia staging

  44. [44]

    Polyp / Adenoma Pathology polyp_type Histologic type of polyp serrated_lesion Presence of serrated lesion morphology adenoma_type Histologic subtype of adenoma

  45. [45]

    Multifocal / Bilateral Disease multifocality Presence of multiple tumor foci bilateral_involvement Bilateral organ involvement 19 Table 10: Key Fields for HistAI Dataset Key Field Clinical Importance benign_vs_malignant Fundamental distinction guiding diagnosis and treatment diagnosis Primary pathologic diagnosis with highest clinical relevance histologic...

  46. [46]

    Diagnosis Hierarchy benign_vs_malignant Benign versus malignant disease classification diagnosis Primary pathologic diagnosis primary_vs_metastatic Distinguishes primary tumor from metastatic disease lineage Cellular or tissue lineage of tumor histologic_type Histologic subtype of tumor or lesion

  47. [47]

    Tumor Behavior in_situ_vs_invasive Whether lesion is in-situ or invasive in_situ_component Presence of an in-situ neoplastic component tumor_focality Unifocal versus multifocal tumor distribution

  48. [48]

    Grade / Aggressiveness tumor_grade Overall tumor grade or aggressiveness differentiation Degree of tumor differentiation

  49. [49]

    Invasion Hierarchy invasion_status Presence or absence of invasion lymphovascular_invasion Tumor invasion into lymphatic or vascular spaces perineural_invasion Tumor invasion involving nerves capsular_invasion Tumor invasion through or into capsule organ_specific_invasion Dataset-specific invasion into adjacent organ structures

  50. [50]

    Spread / Involvement metastatic_involvement Tumor involvement of distant or secondary sites lymph_node_involvement Presence of tumor involvement in lymph nodes multifocal_involvement Multiple sites or foci involved by disease

  51. [51]

    Premalignant / Epithelial Change dysplasia_grade Severity of epithelial dysplasia epithelial_atypia Presence of epithelial atypia carcinoma_in_situ Presence of carcinoma in situ intratubular_germ_cell_neoplasia Presence of intratubular germ cell neoplasia

  52. [52]

    Associated Morphology necrosis Presence of tumor necrosis inflammation_status Presence of inflammatory changes calcification Presence of calcifications psammoma_bodies Presence of psammoma bodies hemorrhage Presence of hemorrhage papillary_features Presence of papillary architectural features cystic_change Presence of cystic change

  53. [53]

    Tumor Extent / Burden tumor_size Tumor dimension or size when reported

  54. [54]

    Contextual / Lower-Priority Metadata organ_site Anatomical organ or tissue site specimen_type Type of specimen or sampling procedure margin_status Presence or absence of tumor at surgical margins 21 Table 12: Key Fields for TCGA Dataset Key Field Clinical Importance benign_vs_malignant Fundamental distinction between benign and malignant disease diagnosis...