Pith. sign in

REVIEW 3 major objections 5 minor 39 references

ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An open-source medical AI judge beats GPT-4-Turbo at grading medical answers.

desk verdict A useful open-source multimodal judge for medical QA, but the headline 'beats GPT-4-Turbo' claim rests on a self-referential comparison against GPT-3.5 labels that also served as training targets. read the letter →

arxiv 2412.11453 v1 pith:ZKRPY5FW submitted 2024-12-16 cs.CL cs.AI

Xiechi Zhang , Shunfan Zheng , Linlin Wang , Gerard de Melo , Zhu Cao , Xiaoling Wang , Liang He This is my paper · ORCID
classification cs.CLcs.AI
keywords multimodalmedicalevaluationLLM-as-a-judgebranch-mergearchitecturedirectpreferenceoptimizationvisualquestionansweringopen-sourceevaluatorrewardtoken
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a small open-source model can automatically judge the quality of medical multimodal AI answers as well as or better than large closed models like GPT-4-Turbo, while offering transparency and reproducibility. It introduces ACE-$M^{3}$, a branch-merge evaluator that first scores responses on three medical criteria (expression, knowledge correctness, relevance), then merges these with the original image and question into a final verdict. Training on an instruction dataset distilled from GPT-3.5-Turbo with reference answers, ACE-$M^{3}$ reaches 82.71 percent accuracy on image-text conclusion scoring, beating GPT-4-Turbo's 78.57 percent. The paper also contributes Efficient-RTDPO, a training trick that freezes lower layers and adds reward tokens to direct preference optimization, cutting training time without significant accuracy loss (77.05 vs 76.22 overall).

What carries the argument

The load-bearing mechanism is the branch-merge pipeline: three frozen-vision, fine-tuned text evaluators each score the two responses against a rubric (Expression, Medical Knowledge Correctness, Patient Question Relevance), and a fourth conclusion model consumes those evaluations together with the image, question, and responses to produce the final 0-5 scores and analysis. The second piece is Efficient-RTDPO: it freezes the first 24 transformer layers to cut compute, then prepends positive ([Good]) or negative ([Bad]) reward tokens to preferred/dispreferred evaluation examples and optimizes a DPO-style loss over the full token sequence, counteracting the accuracy loss from freezing. This combination is what lets a small model match or beat much larger closed judges.

What would settle it

Have a panel of board-certified clinicians independently score a random sample of the test-set response pairs (e.g., 500 pairs) and compute agreement between ACE-$M^{3}$'s relative scores and the clinicians' consensus; if that agreement is not significantly better than GPT-4-Turbo's agreement with the same clinicians, the claim that ACE-$M^{3}$ is a more reliable medical evaluator would be falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an open-source, 7B-parameter medical evaluator can outperform proprietary generalist models at the specific task of comparing two medical answers. ACE-$M^{3}$ processes the question, image, and two responses through three sub-domain evaluation models (Expression, Medical Knowledge Correctness, Patient Question Relevance), each producing per-criterion analyses and 0-5 scores. Those branch outputs are then fed to a conclusion model that writes an overall analysis and assigns final scores to each answer. Trained on an instruction set where GPT-3.5-Turbo produced reference-guided evaluations, the model attains 82.71 accuracy on image-text conclusion scoring, ahead of GPT-4-Turbo's 78.57, and also shows smaller position and symmetry biases. The Efficient-RTDPO strategy—freezing the first 24 layers plus a reward-token-augmented DPO loss—keeps overall accuracy at 77.05 versus 76.22 without reward tokens while reducing training time by roughly 30 percent.

Load-bearing premise

The reliability of GPT-3.5-Turbo as the gold-standard evaluator is assumed: the same model generated the training labels and the test labels, so the reported accuracy measures agreement with that teacher rather than with expert clinicians, and the human spot-checks (100 samples per dataset, 200 total for evaluation) are too small to certify true medical judgment.

Editorial extensions

If this is right

  • Medical MLLM developers can use an open-source, transparent evaluator instead of closed APIs for grading open-ended medical QA, improving reproducibility and privacy.
  • The branch-merge architecture provides detailed per-criterion feedback (expression, medical correctness, relevance), enabling diagnosis of specific failure modes rather than a single score.
  • The Efficient-RTDPO training strategy shows that freezing lower layers plus reward-token DPO can cut training cost (approximately 30 percent time saving) with minimal accuracy loss, a transferable recipe for fine-tuning evaluator models.
  • The evaluator will allow scalable, low-cost benchmarking of new medical MLLMs without human expert annotation at every step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the training labels come from GPT-3.5-Turbo, the reported accuracy may overstate true agreement with expert medical judgment; a head-to-head test against clinician consensus on a held-out set would clarify.
  • The same branch-merge plus RTDPO recipe could be ported to other high-stakes domains (legal, financial) where multi-criteria evaluation matters and reference answers are available.
  • The model's lower position and symmetry bias suggests that small specialized evaluators can be engineered to be fairer arbiters than large black-box models, but the verbosity bias (29.26 percentage-point gap between longer and shorter responses) remains substantial and needs targeted mitigation.
  • A direct testable extension: apply ACE-M^3 to rank responses on a fresh medical VQA benchmark with expert-generated scores, and compare its ranking to GPT-4-Turbo and to BLEU/ROUGE, to measure true expert alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ACE-M^3, an open-source multimodal LLM-based evaluator for medical question-answering MLLMs. The model uses a branch-merge architecture with three sub-domain evaluators (Expression, Medical Knowledge Correctness, Patient Question Relevance) and a conclusion evaluator. Training data is constructed by asking GPT-3.5-Turbo to score pairs of responses according to detailed criteria, with reference answers provided to compensate for the teacher's inability to process images. An Efficient-RTDPO training strategy combines reward-token-based DPO with freezing lower LLM layers to reduce training cost. Experiments report accuracy against GPT-3.5-Turbo labels, claiming that ACE-M^3 surpasses GPT-4-Turbo on image-text data (82.71 vs. 78.57 for conclusion scores) and exhibits lower position and symmetry bias. A human evaluation on 200 samples is presented as external validation.

Significance. If the central claims were established, the paper would make a useful contribution: an open-source, multimodal, medical-domain evaluator with a new instruction dataset, a branch-merge architecture, and a training-efficiency method. The release of the model and data is a concrete asset for the community. However, the reported headline advantage over GPT-4-Turbo rests on agreement with the very teacher that generated the training labels, and the human validation is too limited to break that circularity. The claimed significance therefore is not yet supported by the evidence as presented.

major comments (3)
  1. [§3.3, §2.3.3, Table 6] The primary metric in Section 3.3 is Accuracy against labels produced by GPT-3.5-Turbo (Section 2.3.3). Because ACE-M^3 is trained on those same labels, the image-text conclusion gain over GPT-4-Turbo (82.71 vs. 78.57) measures teacher-student agreement rather than independent evaluation skill. The human spot-check in Table 1 (100 entries per aspect per dataset) validates the quality of the GPT-3.5 labels on a sample, but the test-set labels remain GPT-3.5-generated, and the 200-sample human study in Section 4.3 does not compare ACE-M^3 and GPT-4-Turbo on the same items. To support the claim that ACE-M^3 is a better evaluator than GPT-4-Turbo, the authors need an independent gold standard, or a human evaluation that scores the same responses with both systems and reports per-system agreement, ideally with inter-annotator agreement.
  2. [§2.3.3, Appendix B.2] The training labels for image-text data are produced by text-only GPT-3.5-Turbo, which is explicitly instructed to 'pretend that you can see the image' and is given only the reference answer. The teacher therefore never sees the image, so the supervision signal for the multimodal branch is not actually grounded in visual information. This undermines the claim that ACE-M^3 is a multimodal evaluator with visual understanding, and it may explain why the choice of vision encoder in Section 4.4 has only a small effect. The authors should either demonstrate that reference answers are sufficient to determine correct scores (e.g., by comparing human judgements with and without image access) or use a multimodal teacher, or substantially temper the multimodal claims.
  3. [§2.2.3, Eq. (7)] The construction of negative evaluation samples for RTDPO includes 'score addition' and 'score subtraction' that add or subtract the same two points to both responses. Such perturbations preserve the relative ordering of the two responses, yet the evaluation metric in Section 3.3 is defined solely by consistency of relative magnitude. Consequently, these particular negative samples are not negative under the metric being optimized, which weakens the conceptual justification for the RTDPO contribution. The authors should clarify what learning signal these perturbations provide, or restrict the negative construction to perturbations that change the relative ordering (as score swapping does).
minor comments (5)
  1. [§4.5, Table 10] The verbosity bias difference for ACE-M^3 (29.26) is the largest among the compared models; the text acknowledges the preference for longer responses, but this should be presented more prominently as a limitation of the model rather than as a neutral observation.
  2. [§4.3, Figure 4] The human evaluation description lacks details about the number of annotators, inter-annotator agreement, and the exact items used; also, Figure 4 is referenced but not included in the manuscript text, so the claimed divergence of BLEU and METEOR from human preferences cannot be verified.
  3. [§4.4, Table 8] The accuracy differences among CLIP, PubMedCLIP, and BiomedCLIP are small (e.g., 82.71 vs. 83.29 for Conclusion); the paper reports no significance tests or confidence intervals, so it is unclear whether these differences are meaningful.
  4. [References] Several references are incomplete: 'Li et al.' (Auto-J), 'Chen et al.' (CheXagent), 'Chan et al.' (ChatEval), and 'Dao' (FlashAttention-2) lack full author lists and venues; please provide complete citations.
  5. [Table 14] The dataset name 'MedicalMeadowWikidoc' appears inconsistently as 'MeadowWikidoc' elsewhere in the paper; please unify the naming.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported evaluation accuracy largely measures agreement with the GPT-3.5 teacher that generated both training and test labels, so the headline gain over GPT-4-Turbo is partly forced by construction.

  1. fitted input called prediction [Section 2.3.3 (Evaluation Collection); Section 3.3 (Metrics); Table 6]
    "Following the previous work (Wang et al., 2024; Chan et al.), we adopt the common practice of distilling evaluations from powerful LLMs. Specifically, as shown in Appendix B.2, the prompt is comprised of the evaluation criterion, output format requirements, and the task definition, designed to steer GPT-3.5-Turbo (gpt-3.5-turbo-0125) to conduct evaluations. ... We use Accuracy as the metric for automatic evaluation, which measures the consistency between the relative magnitude of the scoring outcomes generated by ACE-M3 for medical models and the relative magnitude of labeled scores."

    ACE-M3 is trained on the evaluations collected from GPT-3.5-Turbo (Section 2.3.3, training splits in Table 5), and the test-time Accuracy in Table 6 is computed against 'labeled scores' produced by the same GPT-3.5-Turbo model with the same prompt. The reported accuracy therefore quantifies how well the student model imitates the teacher, not an independent measure of evaluation skill. The comparison against GPT-4-Turbo is confounded because GPT-4-Turbo was not trained on this teacher's labels, whereas ACE-M3 was explicitly optimized to reproduce them.

full rationale

The core circularity is that the paper's central metric (Accuracy, Section 3.3) is defined as agreement with 'labeled scores,' and those labels are exactly the GPT-3.5-Turbo evaluations used as the model's training signal (Section 2.3.3). Thus the headline result in Table 6 (ACE-M3 82.71 vs GPT-4-Turbo 78.57 on image-text conclusion scores) can be largely explained by ACE-M3 having been trained to match GPT-3.5's scoring behavior, while GPT-4-Turbo was not. This is a textbook case of a fitted input being called a prediction: the model is optimized to reproduce a teacher's outputs and then evaluated on held-out samples from the same teacher. The human validation is too limited to break this reduction: Table 1's E.Q. rates (89-95%) are based on 100 samples per aspect per dataset, and Section 4.3's human overlap study uses only 200 samples and does not compare GPT-4-Turbo against humans on the same items. No load-bearing self-citation chain or uniqueness argument is present; the circularity arises from the teacher-student construction of the evaluation metric itself. Score 6 reflects that this is a partial but substantial circularity: one or more 'predictions' reduce by construction, while the human spot-checks provide a small amount of independent grounding.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The empirical system depends on the quality of synthetic labels, the choice of evaluation criteria, and several architectural and training assumptions. No external derivation or formal proof is provided; these are the key premises that, if wrong, would invalidate the performance claims.

free parameters (3)
  • Number of frozen layers = 24
    Freezing the first 24 layers was chosen based on Figure 3, which shows diminishing accuracy gains and increasing training time as fewer layers are frozen; this is a hyperparameter selection made after observing results.
  • Learning rate per training stage = 2e-5, 2e-6, 2e-5, 1e-6
    Table 4 lists learning rates for the four training stages; these were chosen without a reported search and affect final accuracy.
  • Batch size = 128
    Batch size of 128 was used for all stages; no rationale is given, but it influences training dynamics and final performance.
assumptions (4)
  • domain assumption GPT-3.5-Turbo, when provided with reference answers, produces reliable evaluations suitable as training targets
    Section 2.3.3 distills evaluations from GPT-3.5-Turbo and relies on a two-step check (format and sampling) to establish reliability; this assumption underpins the entire dataset.
  • domain assumption The three evaluation criteria families (Expression, Medical Knowledge Correctness, Patient Question Relevance) with their 10 sub-criteria capture what makes a good medical answer
    Section 2.1 defines these criteria via three annotators; there is no validation against an external standard and no inter-annotator agreement reported.
  • ad hoc to paper A branch-merge architecture (three sub-models plus a conclusion model) is better than a single model for evaluation
    Section 2.2.1 introduces the architecture; no ablation is run against a single-model variant, so the benefit is assumed.
  • ad hoc to paper Prepending reward tokens and using the RTDPO loss improves evaluation accuracy without hurting efficiency
    Section 2.2.3 proposes RTDPO; the ablation in Table 7 shows gains of less than one point, but no comparison to standard DPO or other variants is made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models." pith.science (2026). https://pith.science/paper/ZKRPY5FW

@misc{pith2026241211453,
  author       = {Pith},
  title        = {Pith review of: ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKRPY5FW}},
  note         = {Machine review of arXiv:2412.11453}
}
abstract

As multimodal large language models (MLLMs) gain prominence in the medical field, the need for precise evaluation methods to assess their effectiveness has become critical. While benchmarks provide a reliable means to evaluate the capabilities of MLLMs, traditional metrics like ROUGE and BLEU employed for open domain evaluation only focus on token overlap and may not align with human judgment. Although human evaluation is more reliable, it is labor-intensive, costly, and not scalable. LLM-based evaluation methods have proven promising, but to date, there is still an urgent need for open-source multimodal LLM-based evaluators in the medical field. To address this issue, we introduce ACE-$M^3$, an open-sourced \textbf{A}utomatic \textbf{C}apability \textbf{E}valuator for \textbf{M}ultimodal \textbf{M}edical \textbf{M}odels specifically designed to assess the question answering abilities of medical MLLMs. It first utilizes a branch-merge architecture to provide both detailed analysis and a concise final score based on standard medical evaluation criteria. Subsequently, a reward token-based direct preference optimization (RTDPO) strategy is incorporated to save training time without compromising performance of our model. Extensive experiments have demonstrated the effectiveness of our ACE-$M^3$ model\footnote{\url{https://huggingface.co/collections/AIUSRTMP/ace-m3-67593297ff391b93e3e5d068}} in evaluating the capabilities of medical MLLMs.

Figures

Figures reproduced from arXiv: 2412.11453 by the authors.

Figure 1
Figure 1. Framework and training details of our multimodal evaluation model ACE- [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Word co-occurrence graph of collected eval [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Influence of frozen layers on time and evalua [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Win rate judged by our ACE-M3 model, hu￾man annotator, and traditional metrics. 4.4 Influences of Vision Encoders We conduct experiments over different encoding techniques such as PubMedCLIP (Eslami et al., 2023) and BiomedCLIP (Zhang et al., 2023) to inspect the influ…
Figure 6
Figure 6. Figure 6: The web-based user interface for conducting the human annotation. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages

  1. [1]

    Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Medbench: A large-scale chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17709--17717

    work page 2024

  2. [2]

    Chateval: Towards better llm-based evaluators through multi-agent debate

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations

  3. [3]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  4. [4]

    Chexagent: Towards a foundation model for chest x-ray interpretation

    Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. In AAAI 2024 Spring Symposium on Clinical Foundation Models

    work page 2024

  5. [5]

    Nancy Chinchor and Beth M Sundheim. 1993. Muc-5 evaluation metrics. In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993

    work page 1993

  6. [6]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations

    work page

  7. [7]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems

    2022

  8. [8]

    Sedigheh Eslami, Christoph Meinel, and Gerard de Melo. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.88 PubMedCLIP : H ow much does CLIP benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181--1193, Dubrovnik, Croatia. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-eacl.88 2023

Show all 39 references
  1. [9]

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander L \"o ser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca--an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247

    2023 arXiv

  2. [10]

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286

    2020 arXiv

  3. [11]

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1--10

    2018

  4. [12]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024 a . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  5. [13]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Pengfei Liu, et al. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations

  6. [14]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6)

    2023

  7. [15]

    Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024 b . https://arxiv.org/abs/2401.07103 Leveraging large language models for nlg evaluation: A survey . Preprint, arXiv:2401.07103

    2024 arXiv

  8. [16]

    Chin Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004)

    2004

  9. [17]

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. https://doi.org/10.1109/ISBI48211.2021.9434010 Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering . In 2021 IEEE 18th International Symposium on Biomedical Imagin...

    2021

  10. [18]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  11. [19]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  12. [20]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023 a . Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353--367. PMLR

    2023

  13. [21]

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023 b . https://proceedings.mlr.press/v225/moor23a.html Med-flamingo: a multimodal medical few-shot learner . In Proceedings of the 3r...

    2023

  14. [22]

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375

    2023 arXiv

  15. [23]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  16. [24]

    Chantal Pellegrini, Ege \"O zsoy, Benjamin Busam, Nassir Navab, and Matthias Keicher. 2023. Radialog: A large vision-language model for radiology report generation and conversational assistance. arXiv preprint arXiv:2311.18681

    2023 arXiv

  17. [25]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  18. [26]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch-solve-merge improves large language model evaluation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin...

    2024

  20. [28]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, pages 1--9

    2023

  21. [29]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. In Proceedings of EMNLP Workshop, page 1

    2023

  22. [30]

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. https://openreview.net/forum?id=5Nn2BLV7SB Panda LM : An automatic evaluation benchmark for LLM in...

    2024

  23. [31]

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023 a . Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6268--6278

    2023

  24. [32]

    Jie Xu, Lu Lu, Sen Yang, Bilin Liang, Xinwei Peng, Jiali Pang, Jinru Ding, Xiaoming Shi, Lingrui Yang, Huan Song, Kang Li, Xin Sun, and Shaoting Zhang. 2023 b . https://arxiv.org/abs/2305.07340 Medgpteval: A dataset and benchmark to evaluate responses of large language models ...

    2023 arXiv

  25. [33]

    Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. 2024. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03162

    2024 arXiv

  26. [34]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, page nwae403

    2024

  27. [35]

    Travis Zack, Eric Lehman, Mirac Suzgun, Jorge A Rodriguez, Leo Anthony Celi, Judy Gichoya, Dan Jurafsky, Peter Szolovits, David W Bates, Raja-Elie E Abdulnour, et al. 2024. Assessing the potential of gpt-4 to perpetuate racial and gender biases in health care: a model evaluati...

    2024

  28. [36]

    Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. 2020. Meddialog: Large-scale medical dialogue datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processi...

    2020

  29. [37]

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. 2023. BiomedCLIP : a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2...

    2023 arXiv

  30. [38]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations

  31. [39]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.