Pith. sign in

REVIEW 3 major objections 5 minor 40 references

MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current 3D medical multimodal models cannot reliably classify, locate, or correct errors in CT reports, according to a new 40,000-question visual QA benchmark.

desk verdict A genuinely useful 3D CT error-correction benchmark held back by unquantified label quality and a training-data overlap, but worth refereeing. read the letter →

arxiv 2506.19217 v1 pith:G3EGMS2G submitted 2025-06-24 cs.CV cs.AI

Sunggu Kyung , Hyungbin Park , Jinyoung Seo , Jimin Sung , Jihyun Kim , Dongyeong Kim , Wooyoung Jo , Yoojin Nam
show 4 more authors
Sangah Park Taehee Kwon Sang Min Lee Namkug Kim
This is my paper
classification cs.CVcs.AI
keywords medicalvisualquestionansweringCTreporterrordetectionradiologycorrection3DmultimodalLLMbenchmarkevaluationvision-centricerrorszero-shotclinicalAIreliability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MedErr-CT is a new benchmark that tests whether multimodal large language models can spot and fix mistakes in CT radiology reports while looking at the corresponding 3D chest scans. The paper builds roughly 40,000 question-answer pairs in which an original report is left intact or corrupted by one of six error types (omission, insertion, wrong direction, wrong size, wrong unit, or a typo), and asks three escalating questions: is there an error, which sentence contains it, and what is the corrected sentence. When eight current open-source 3D medical MLLMs are evaluated zero-shot, the best model reaches only 0.478 accuracy on classification, 0.276 on detection, and 0.241 on correction. The authors conclude that none of the tested models is reliable enough for clinical deployment, and argue that larger and more diverse instruction-following training data, not architecture alone, is what most helps a model handle this task.

What carries the argument

The machinery is the benchmark itself: a semi-automated pipeline that takes grounded region-level chest CT reports, uses a large language model to inject exactly one error of a defined type into each report, and turns the resulting pairs into VQA prompts at three difficulty levels. The six error categories split into four vision-centric errors (omission, insertion, direction, size) that can only be resolved by inspecting the image, and two lexical errors (unit, typo). Two new detection metrics, Soft Index Matching and Hard Index Matching, score sentence-level localization by matching the model's output to the distorted report; correction is scored with lexical metrics plus BERTScore and GREEN. The design makes the error-level progression—classify, detect, correct—the mechanism that separates shallow recognition from genuine clinical reasoning.

What would settle it

Have a panel of independent radiologists, blinded to the generation process and larger than the two used in construction, re-annotate a sample of the QA pairs and measure inter-rater agreement on whether the intended error is present, unique, clinically realistic, and matched to the image; if agreement is low or many labels are rejected, the benchmark's ground truth and resulting scores are not dependable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a capability gap, not a solution: current 3D medical MLLMs cannot reliably identify or correct errors in CT reports. Across all six error types and all three task levels, performance stays far below clinical thresholds, and the pattern of failures is systematic: several models do well on error-free reports only because they default to “no error,” while others over-detect errors and then fail to correct them. The strongest configurations are those trained on the largest instruction-following datasets derived from real radiology reports, with diverse question formats and multi-turn conversations, and the paper attributes part of their vision-side advantage to higher input resolution. The benchmark's most difficult item is omission correction, where all models score near zero.

Load-bearing premise

The benchmark's labels are trustworthy: the original reports are truly error-free, and each automatically altered report contains exactly one clinically realistic error of the intended type that matches the CT image.

Editorial extensions

If this is right

  • If the benchmark is right, no current 3D medical MLLM should be trusted to flag or fix its own CT report errors without human review.
  • Training-data composition becomes a primary lever: instruction-following scale, question diversity, and multi-turn conversation appear to matter more than model architecture or image-text alignment.
  • The benchmark supplies a reusable roughly 40,000-item test suite spanning six error types and three task levels, so future models can be compared on the same clinical error-checking ability.
  • Omission correction is a bottleneck worth targeting separately, since all tested models score near zero on it despite its clinical importance.
  • Metric choices matter: a model can look strong on correction if it scores well on error-free reports, so reporting error/none-error splits and detection metrics is necessary for honest evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper does not run is a text-only control: if an LLM given the report but no CT image matches the vision-enabled models on vision-centric errors, then those errors may be recoverable from report phrasing alone, and the benchmark's image-grounding requirement would need to be demonstrated rather than assumed.
  • The reported split between error and none-error performance suggests that simple calibration or answer-prior adjustment could raise aggregate scores without improving actual error handling; future users should examine per-class scores before trusting overall numbers.
  • The benchmark's scope is limited to nodules and pleural effusions in chest CT; extending the same corruption-and-VQA recipe to other anatomies, modalities, and lesion types would test whether the capability gap is general or specific to these findings.
  • A human-radiologist baseline on the same 40,000 items would calibrate how hard the benchmark really is and would tell whether the low model scores reflect model weakness or task ambiguity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MedErr-CT, a visual question answering benchmark for 3D chest CT that evaluates multimodal large language models on classifying, detecting, and correcting six types of errors in radiology reports. The dataset is built from RadGenome-Chest CT / CT-RATE reports, with distorted reports generated by LLaMA-3.3-70B under a DSPy pipeline and organized into three task levels: classification, detection, and correction. The authors evaluate eight configurations of five open-source 3D medical MLLMs under zero-shot settings and report suboptimal performance, concluding that none of the tested models is ready for clinical deployment. The benchmark, code, and datasets are publicly released.

Significance. If the label construction is valid, MedErr-CT fills a genuine gap: it is, to my knowledge, the first 3D CT VQA benchmark targeting error identification and correction rather than simple recognition, and its three-level evaluation design is well matched to clinical error-management needs. The inclusion of vision-centric error types beyond lexical errors is a meaningful advance over prior text-only or 2D benchmarks. The paper also contributes reproducible zero-shot evaluations of current 3D medical MLLMs, a concrete set of evaluation metrics (SIM and HIM), and a public data/code release. The significance is currently conditional on the validity of the ground-truth labels and on the cleanliness of the model-comparison setup, both of which need strengthening.

major comments (3)
  1. [3.4] The validity of every classification, detection, and correction label in Table 1 rests on two unverified assumptions: the original RadGenome reports are error-free, and the LLaMA-3.3-70B-generated distorted reports contain exactly one clinically realistic error of the intended type. The only reported safeguard is the sentence 'two radiologists manually validated the QA sets,' with no inter-annotator agreement, no number of discarded or corrected cases, no adjudication procedure, and no description of what was checked. Since the Introduction itself cites a 30% radiology discrepancy rate, treating the source reports as error-free is not self-evident. Label noise propagates into every number in Table 2 and the appendix tables, so the benchmark's central claim to measure error identification and correction is not yet supported. The authors should report a measurable validation protocol (e.g., Cohen's kappa, per-type pass rates, corrected-example counts) before the results can be interpreted.
  2. [5.3] The acknowledged overlap between the benchmark's visual data and the CT-RATE training set is load-bearing because the two best-performing families, CT-CHAT and MedM-VL, were trained on CT-RATE. The manuscript simultaneously says the overlap 'may exhibit biased performance and this potential influence cannot be entirely ruled out' and that 'the unseen nature is definitively confirmed,' which are contradictory. To support the ranking and the Section 5.1 attribution to instruction-following dataset scale, the authors need to quantify the overlap at the patient/volume level and re-report results on a strictly disjoint subset, or show per-model contamination sensitivity. Without this, the central model-comparison conclusion is confounded.
  3. [4.3] The Soft Index Matching metric is underspecified. It is not stated whether the ROUGE-L best match is an exact sentence-index match, whether a threshold is used, how ties are broken, or how 'no-error' reports are scored when there is no erroneous sentence to match. Since detection results in Table 2 are reported entirely through SIM and HIM, the metric needs a formal definition with the exact matching rule and the no-error handling before the detection-level comparisons can be reproduced.
minor comments (5)
  1. [Table 2] The 'All' aggregation is not defined; it should state whether it is sample-weighted, macro-averaged over error types, or averaged over error/no-error groups, because the six error types have unequal sizes in Table 1.
  2. [Figures 3 and 4] Figures 3 and 4 appear to compare only RadFM and CT-CHAT-Mistral, but the captions and text refer to 'M3D = MeD-Llama'; clarify which models are shown and why only two are plotted.
  3. [Section 1] There are several typographical spacing errors in the text, e.g., 'Leeet al.,' 'Patelet al.,' and 'Gertzet al.' in Section 1, and 'an error .' in Section 4.4.
  4. [3.4] The exact prompts used for error generation and the QA templates should be provided in the appendix, since the paper says prompts are predefined but only a figure-level list is shown; full templates would support reproducibility.
  5. [Table 2] The high correction scores on 'none error' reports (e.g., 0.884 for CT-CHAT-Llama) may partly reflect copying the input report rather than correction ability; this should be acknowledged when interpreting the correction-level results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MedErr-CT is a benchmark-construction and evaluation paper, and no load-bearing step reduces to a fitted input, self-citation, or definitional equivalence.

full rationale

The paper's central deliverable is a new VQA benchmark plus an empirical evaluation of eight 3D medical MLLM configurations. There is no derivation chain in which an output quantity is defined in terms of the quantity it is claimed to predict. The ground-truth labels are generated from RadGenome-Chest CT reports with LLaMA-3.3-70B-injected distortions and manually validated by two radiologists (Section 3.4); the model scores in Table 2 are computed by comparing model outputs against these externally constructed labels using standard and proposed metrics. This is not circular, although the unquantified radiologist validation is a legitimate label-validity limitation. The acknowledged overlap between MedErr-CT images and the CT-RATE training set (Section 5.3) is a data-leakage and benchmark-fairness concern, not a circularity: the rankings are empirically measured, and the paper explicitly discloses the overlap. No self-citation is load-bearing; the cited prior work (CT-RATE, RadGenome, CT-CHAT, MedM-VL) is external to the present authors' claims, and no uniqueness theorem or ansatz is imported from the authors' own prior work to force a conclusion. The SIM metric uses ROUGE-L to map free-form model output to a distorted-report sentence before scoring; this is a stated matching heuristic, not a label constructed from the model's own predictions. Therefore, no step exhibits self_definitional, fitted_input_called_prediction, self_citation_load_bearing, uniqueness_imported_from_authors, ansatz_smuggled_in_via_citation, or renaming_known_result circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark does not introduce free parameters or novel entities. The central assumptions concern the validity of automatically generated ground truth and the assumption that original reports are error-free, plus the clinical representativeness of the two lesion types. These are domain assumptions made during dataset construction, not parameters fitted to data.

assumptions (4)
  • domain assumption Original RadGenome-Chest CT reports are error-free and usable as 'no error' gold labels.
    Section 3.1 and Section 3.4: normal cases and undistorted reports are treated as ground-truth 'none error' examples without independent verification of report accuracy.
  • domain assumption The DSPy + LLaMA-3.3-70B prompt-based procedure generates clinically realistic errors of the intended type, and the errors are correctly grounded in the image.
    Section 3.4: distorted reports are generated automatically from prompts, with manual radiologist validation mentioned but no agreement statistics or error-generation success rate reported.
  • domain assumption The six chosen error categories and the two lesion types (nodules and pleural effusions) are sufficient to represent clinically meaningful CT report errors.
    Section 3.2 and Section 5.3: the authors restrict to these two lesions for controlled evaluation and acknowledge the limited lesion diversity as a limitation.
  • domain assumption Metric choices (SIM, HIM, GREEN, BERTScore-F1) provide a valid measurement of detection and correction quality for this task.
    Section 4.3: SIM relies on ROUGE-L matching between model output and distorted report sentences, which may reward outputs similar to the erroneous sentence rather than the corrected one; the paper does not validate the new metrics against human judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports." pith.science (2026). https://pith.science/paper/G3EGMS2G

@misc{pith2026250619217,
  author       = {Pith},
  title        = {Pith review of: MedErr-CT: A Visual Question Answering Benchmark for Identifying and Correcting Errors in CT Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3EGMS2G}},
  note         = {Machine review of arXiv:2506.19217}
}
read the original abstract

Computed Tomography (CT) plays a crucial role in clinical diagnosis, but the growing demand for CT examinations has raised concerns about diagnostic errors. While Multimodal Large Language Models (MLLMs) demonstrate promising comprehension of medical knowledge, their tendency to produce inaccurate information highlights the need for rigorous validation. However, existing medical visual question answering (VQA) benchmarks primarily focus on simple visual recognition tasks, lacking clinical relevance and failing to assess expert-level knowledge. We introduce MedErr-CT, a novel benchmark for evaluating medical MLLMs' ability to identify and correct errors in CT reports through a VQA framework. The benchmark includes six error categories - four vision-centric errors (Omission, Insertion, Direction, Size) and two lexical error types (Unit, Typo) - and is organized into three task levels: classification, detection, and correction. Using this benchmark, we quantitatively assess the performance of state-of-the-art 3D medical MLLMs, revealing substantial variation in their capabilities across different error types. Our benchmark contributes to the development of more reliable and clinically applicable MLLMs, ultimately helping reduce diagnostic errors and improve accuracy in clinical practice. The code and datasets are available at https://github.com/babbu3682/MedErr-CT.

Figures

Figures reproduced from arXiv: 2506.19217 by the authors.

Figure 1
Figure 1. The data generation pipeline of the MedErr-CT Benchmark. We automated data generation utilizing the DSPy framework with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Question prompt list for each error task. These prompts are randomly selected during the execution of each task. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. NaN value rates by model in correction task. (M3D = [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Generated sentence length distribution excluding NaN [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    Computed tomography and magnetic resonance imaging: past, present and future.European Respiratory Journal, 19(35 suppl):3s–12s, 2002

    NL M ¨uller. Computed tomography and magnetic resonance imaging: past, present and future.European Respiratory Journal, 19(35 suppl):3s–12s, 2002. 1

    work page 2002

  2. [2]

    Should we be concerned about the rapid increase in ct usage?Reviews on environmental health, 25(1):63–68, 2010

    David J Brenner. Should we be concerned about the rapid increase in ct usage?Reviews on environmental health, 25(1):63–68, 2010. 1

    work page 2010

  3. [3]

    Cognitive and system factors contribut- ing to diagnostic errors in radiology.American Journal of Roentgenology, 201(3):611–617, 2013

    Cindy S Lee, Paul G Nagy, Sallie J Weaver, and David E Newman-Toker. Cognitive and system factors contribut- ing to diagnostic errors in radiology.American Journal of Roentgenology, 201(3):611–617, 2013. 1

    work page 2013

  4. [4]

    Automated radiology report generation: A review of recent advances.IEEE Reviews in Biomedical Engineer- ing, 2024

    Phillip Sloan, Philip Clatworthy, Edwin Simpson, and Majid Mirmehdi. Automated radiology report generation: A review of recent advances.IEEE Reviews in Biomedical Engineer- ing, 2024. 1

    work page 2024

  5. [5]

    Comparing diagnostic accuracy of radiolo- gists versus gpt-4v and gemini pro vision using image inputs from diagnosis please cases.Radiology, 312(1), July 2024

    Pae Sun Suh, Woo Hyun Shim, Chong Hyun Suh, Hwon Heo, Chae Ri Park, Hye Joung Eom, Kye Jin Park, Jooae Choe, Pyeong Hwa Kim, Hyo Jung Park, Yura Ahn, Ho Young Park, Yoonseok Choi, Chang-Yun Woo, and Hyungjun Park. Comparing diagnostic accuracy of radiolo- gists versus gpt-4v and gemini pro vision using image inputs from diagnosis please cases.Radiology, 3...

    work page 2024

  6. [6]

    Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Dur- rett, Justin F Rousseau, et al. Evaluating large language models on medical evidence summarization.NPJ digital medicine, 6(1):158, 2023. 1

    work page 2023

  7. [7]

    Embracing large language models for medical applications: opportuni- ties and challenges.Cureus, 15(5), 2023

    Mert Karabacak and Konstantinos Margetis. Embracing large language models for medical applications: opportuni- ties and challenges.Cureus, 15(5), 2023. 1

    work page 2023

  8. [8]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018. 1

    work page 2018

Show all 40 references
  1. [9]

    Pathvqa: 30000+ questions for medical vi- sual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical vi- sual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  2. [10]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  3. [11]

    Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing.arXiv preprint arXiv:2305.10415, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Vi- sual instruction tuning for medical visual question answer- ing.arXiv preprint arXiv:2305.10415, 2023

    2023 arXiv

  4. [12]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 22170–22183, 2024

    2024

  5. [13]

    Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing Systems, 37:94327–94427, 2024

    Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai.Advances in Neural Information Processing Systems, 37:94327–94...

    2024

  6. [14]

    Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding.arXiv preprint arXiv:2501.18362, 2025. 1

    2025 arXiv

  7. [15]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms,

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, and Ran He. Mme-survey: A comprehensive survey on evaluation of multimodal llms,

  8. [16]

    Kim and Liem T

    Young W. Kim and Liem T. Mansfield. Fool me twice: Delayed diagnoses in radiology with emphasis on per- petuated errors.American Journal of Roentgenology, 202(3):465–470, March 2014. 2

    2014

  9. [17]

    Recovery at the edge of error: debunking the myth of the infallible expert.Journal of biomedical in- formatics, 44(3):413–424, 2011

    Vimla L Patel, Trevor Cohen, Tripti Murarka, Joanne Olsen, Srujana Kagita, Sahiti Myneni, Timothy Buchman, and Vafa Ghaemmaghami. Recovery at the edge of error: debunking the myth of the infallible expert.Journal of biomedical in- formatics, 44(3):413–424, 2011. 2

    2011

  10. [18]

    Overview of the mediqa-corr 2024 shared task on medical error detection and correction

    Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Fei Xia, and Meliha Yetisgen. Overview of the mediqa-corr 2024 shared task on medical error detection and correction. InProceedings of the 6th Clinical Natural Language Pro- cessing Workshop, page 596–603. Association for Co...

    2024

  11. [19]

    Potential of gpt-4 for detecting errors in radiology reports: implications for report- ing accuracy.Radiology, 311(1):e232714, 2024

    Roman Johannes Gertz, Thomas Dratsch, Alexander Chris- tian Bunck, Simon Lennartz, Andra-Iza Iuga, Martin Gunnar Hellmich, Thorsten Persigehl, Lenhard Pennig, Carsten Her- bert Gietzen, Philipp Fervers, et al. Potential of gpt-4 for detecting errors in radiology reports: impli...

    2024

  12. [20]

    Next token prediction towards multi- modal intelligence: A comprehensive survey.arXiv preprint arXiv:2412.18619, 2024

    Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, et al. Next token prediction towards multi- modal intelligence: A comprehensive survey.arXiv preprint arXiv:2412.18619, 2024. 3

    2024 arXiv

  13. [21]

    3d-ct-gpt: Generating 3d radiology reports through integration of large vision-language models.arXiv preprint arXiv:2409.19330, 2024

    Hao Chen, Wei Zhao, Yingli Li, Tianyang Zhong, Yisong Wang, Youlan Shang, Lei Guo, Junwei Han, Tianming Liu, Jun Liu, et al. 3d-ct-gpt: Generating 3d radiology reports through integration of large vision-language models.arXiv preprint arXiv:2409.19330, 2024. 3

    2024 arXiv

  14. [25]

    Med3dvlm: An efficient vision-language model for 3d med- ical image analysis, 2025

    Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. Med3dvlm: An efficient vision-language model for 3d med- ical image analysis, 2025. 3, 5, 6

    2025

  15. [26]

    Medm-vl: What makes a good medical lvlm?,

    Yiming Shi, Shaoshuai Yang, Xun Zhu, Haoyu Wang, Miao Li, and Ji Wu. Medm-vl: What makes a good medical lvlm?,

  16. [27]

    Medec: A bench- mark for medical error detection and correction in clinical notes.arXiv preprint arXiv:2412.19260, 2024

    Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medec: A bench- mark for medical error detection and correction in clinical notes.arXiv preprint arXiv:2412.19260, 2024. 3, 4

    2024 arXiv

  17. [28]

    Rao, Serena Zhang, Julian N

    Vishwanatha M. Rao, Serena Zhang, Julian N. Acosta, Subathra Adithan, and Pranav Rajpurkar. ReXErr: Synthe- sizing Clinically Meaningful Errors in Diagnostic Radiology Reports, September 2024. arXiv:2409.10829 [cs]. 3

    2024 arXiv

  18. [29]

    Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports.Scientific data, 6(1):317, 2019

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de- identified publicly available database of chest radiographs with free-text reports.Scientific data, 6(1):317, 2019. 3

    2019

  19. [30]

    MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context, July 2024

    Zishan Gu, Changchang Yin, Fenglin Liu, and Ping Zhang. MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context, July 2024. arXiv:2407.02730 [cs]. 4

    2024 arXiv

  20. [31]

    A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero- shot detection of abnormalities.CoRR, 2024

    Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Bastian Wittmann, Enis Sim- sar, Mehmet Simsar, et al. A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-...

    2024

  21. [32]

    Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis. arXiv preprint arXiv:2404.16754, 2024. 4

    2024 arXiv

  22. [33]

    Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric explo- ration of multimodal llms.Advances in Neural Information Process...

    2024

  23. [34]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714,

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714,

    arXiv

  24. [35]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 5

    2024 arXiv

  25. [36]

    Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data, November 2023

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data, November 2023. arXiv:2308.02463 [cs]. 5, 6

    2023 arXiv

  26. [37]

    Meng, and Bo Zhao

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q.-H. Meng, and Bo Zhao. M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models, March 2024. arXiv:2404.00578 [cs]. 5, 6

    2024 arXiv

  27. [38]

    De- veloping Generalist Foundation Models from a Multi- modal Dataset for 3D Computed Tomography, April 2025

    Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Doga, Omer Faruk Durugol, Weicheng Dai, Murong Xu, Muhammed Furkan Dasdelen, Bastian Wittmann, Tamaz Amiranashvili, Enis Simsar, Mehmet Simsar, Em- ine Bensu Erdemir, Ab...

    2025

  28. [39]

    ROUGE: A Package for Automatic Evalu- ation of Summaries

    Chin-Yew Lin. ROUGE: A Package for Automatic Evalu- ation of Summaries. InText Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. 5

    2004

  29. [40]

    Bleu: a Method for Automatic Evaluation of Ma- chine Translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Ma- chine Translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors,Proceedings of the 40th Annual Meet- ing of the Association for Computational Linguistics...

    2002

  30. [41]

    METEOR: An Auto- matic Metric for MT Evaluation with Improved Correlation with Human Judgments

    Satanjeev Banerjee and Alon Lavie. METEOR: An Auto- matic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare V oss, editors,Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Mea- s...

    2005

  31. [42]

    Wein- berger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. BERTScore: Evaluating Text Gen- eration with BERT, February 2020. arXiv:1904.09675 [cs]. 5

    2020 arXiv

  32. [43]

    Chaudhari, and Jean-Benoit Delbrouck

    Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S. Chaudhari, and Jean-Benoit Delbrouck. GREEN: Genera- tive Radiology Report Evaluation and Error Notation. In Finding...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.