Pith. sign in

REVIEW 3 major objections 4 minor 39 references

AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AutoMedEval, a 13B open-source medical evaluator trained with curriculum instruction tuning and iterative knowledge introspection, claims to match human doctors' judgments more closely than GPT-4 and every baseline, with Spearman 0.6314…

desk verdict AutoMedEval is a useful medical evaluator with a genuinely new training combination, but its central claim depends on a train/test disjointness guarantee the paper never states. read the letter →

arxiv 2505.11887 v1 pith:L2SOHRHT submitted 2025-05-17 cs.CL

Xiechi Zhang , Zetian Ouyang , Linlin Wang , Gerard de Melo , Zhu Cao , Xiaoling Wang , Ya Zhang , Yanfeng Wang
show 1 more author
Liang He
This is my paper · ORCID
classification cs.CL
keywords automaticevaluationmedicalLLMcurriculuminstructiontuningiterativeknowledgeintrospectionGPT-4distillationLLM-as-a-judgeretrieval-augmentedgenerationhuman-aligned
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to supply an open, automatic judge for medical question-answering models that agrees with human doctors closely enough to make large-scale expert evaluation unnecessary. AutoMedEval is a 13B parameter evaluator built on MedLLaMA and trained on a physician-verified instruction set distilled from GPT-4 and ChatGPT. The training recipe combines a three-stage curriculum (pattern recognition, mixed data, high-quality GPT-4 evaluations) with an iterative knowledge-introspection loop in which incorrect evaluations are revised through AI-doctor collaboration and used to fine-tune the model again. On a 1,130-question test set, the paper reports that AutoMedEval's scores correlate with human judgments at Spearman 0.6314, ahead of GPT-4's 0.5128 and every other baseline, which is the paper's central evidence that automatic evaluation can approach human-level medical judgment.

What carries the argument

The carrier of the argument is a three-part training and data machinery. The dynamic knowledge completion chain lets the GPT-4 teacher issue a '[Question]' query when uncertain, retrieve relevant passages from a vector database of medical and evaluation manuals, append them to the prompt, and repeat until the evaluation can be completed; this grounds the instruction data in medical evidence. Curriculum instruction tuning then fine-tunes MedLLaMA-13B sequentially on ChatGPT-generated evaluations (pattern recognition), a mixture of ChatGPT and GPT-4 evaluations, and finally high-quality GPT-4 evaluations, so the model learns evaluation patterns before quality criteria. Iterative knowledge introspection closes the loop: for training cases the model still evaluates incorrectly, retrieval-augmented GPT-4 drafts revision suggestions, standard GPT-4 judges them, and a chief physician arbitrates disagreements; the corrected instructions are used to fine-tune the model again, calibrating it to human standards.

What would settle it

Audit the 958-item Wikidoc and 172-item MedText test questions for exact or near-duplicate overlap with the 9,874 training instructions using string matching and embedding similarity; if any test question appears in training, the reported Spearman advantage cannot be trusted. Alternatively, score a newly collected set of clinical QA pairs that was never used in instruction construction and compare AutoMedEval's Spearman correlation with five doctors' judgments against GPT-4's 0.5128; a drop to that level would falsify the generalization claim.

Watch

Extended reading notes

Core claim

The central claim is that a 13B open-source model can outperform proprietary and general-purpose evaluators at judging medical LLM responses, and that this capability can be obtained with limited high-quality data through the paper's hierarchical training method. AutoMedEval was trained on 9,569 instruction instances constructed by asking GPT-4 to evaluate pairs of responses from ChatDoctor and Baize Healthcare to medical questions, with a dynamic knowledge completion chain that retrieves passages from a medical book and manual vector database whenever the teacher is unsure; two chief physicians then filtered out evaluations failing medical, attribution, or fluency standards. After curriculum instruction tuning on 3,823 ChatGPT-sourced and 4,788 GPT-4-sourced instructions, and two rounds of iterative knowledge introspection with AI-doctor revision suggestions, the model reached Spearman 0.6314, Accuracy2-tuple 74.61%, and Accuracytriple 48.65% on a test set of 958 Wikidoc and 172 MedText questions, all higher than GPT-4 (0.5128, 67.98%, 35.42%) and Gemini (0.5674, 71.68%, 42.63%).

Load-bearing premise

The whole comparison rests on the unstated assumption that the 958-entry Wikidoc test set and the 9,874-entry Wikidoc training set are disjoint, because if a test question also appeared in training the reported human-correlation advantage could be memorized GPT-4 grading style rather than a general evaluation ability.

Editorial extensions

If this is right

  • Medical model developers can compare candidate LLMs on open-ended questions without commissioning a full expert panel for every comparison, because AutoMedEval's case-level and pair-level scores track chief physicians' judgments.
  • Because AutoMedEval is open-source and runs locally, it can be used in privacy-sensitive medical settings where sending patient-style questions to GPT-4 or Gemini is not acceptable.
  • The curriculum plus introspection recipe can bootstrap a domain evaluator from a small verified seed: AutoMedEval used 5,000 high-quality and 4,000 lower-quality instructions to surpass MedLLaMA tuned on 9,000 high-quality GPT-4 instructions by 38.5 percent relative on Accuracytriple.
  • Iterative introspection yields steady gains (Accuracytriple 44.61 to 47.13 to 48.65 across rounds), and the paper's saturation model predicts gains stop around six iterations, giving a practical stopping rule for the human-in-the-loop correction step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: if the human-correlation results survive an explicit train/test overlap audit, the pipeline becomes a template for other label-scarce expert domains such as legal advice, radiology, and pharmacovigilance.
  • Because every training label comes from GPT-4 or ChatGPT, AutoMedEval inherits their blind spots; the paper's own error taxonomy (bias, hallucination, unsupported ratings) suggests physician-written rationales or multi-judge distillation as a next step.
  • A stress test the paper does not run is out-of-distribution transfer to other specialties or languages; strong transfer would indicate a general medical judging skill rather than memorized Wikidoc response styles.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces AutoMedEval, a 13B open-source evaluation model for medical question-answering. The authors construct an instruction dataset from roughly 9,874 medical-meadow-wikidoc question-answer pairs, have GPT-4 and ChatGPT produce evaluations with retrieval-augmented knowledge completion chains, and have physicians verify a subset of the resulting evaluations. The model is trained on MedLLaMA-13B using curriculum instruction tuning followed by iterative knowledge introspection, in which incorrect training cases are revised through an AI-doctor collaborative feedback loop. The central empirical claim is that AutoMedEval achieves higher correlation with human judgments than all tested baselines, including GPT-4, on response-level Spearman/Pearson correlations, case-level accuracy metrics, and a double-blind preference experiment.

Significance. If the central comparative claims hold under scrutiny, AutoMedEval would be a useful open-sourced building block for medical LLM evaluation, addressing a real gap left by proprietary judges and general-domain evaluators. The paper has notable strengths: the instruction dataset is physician-verified at scale, the ablations isolate the contributions of the knowledge completion chain, curriculum instruction tuning, and iterative knowledge introspection, and the evaluation is anchored in human annotations with inter-annotator reliability reported. The proposed iterative introspection loop is a practical contribution that could transfer to other domain-specific evaluators. However, the significance is currently conditional on resolving a train/test overlap concern and on reframing the rationale-evaluation metrics, since both bear directly on whether the reported advantage over GPT-4 reflects generalizable evaluation capability or imitation/memorization of GPT-4's evaluation style.

major comments (3)
  1. [Sections 3.1 and 4.1] The manuscript never states whether the 958-entry Medical Meadow Wikidoc test set is disjoint from the 9,874 training cases drawn from the same source dataset. Because AutoMedEval is trained on GPT-4 and ChatGPT evaluations of wikidoc question-response tuples, any test question that also appears in the instruction-tuning data gives the model direct access to a reference evaluation for that question, which could inflate the reported Spearman 0.6314 and Accuracy-triple 48.65 through memorization rather than generalizable evaluation competence. Partial mitigation from using unseen response generators (MedAlpaca, MedLlama2) does not remove the concern, since question-level memorization can still bias scoring of newly generated responses. The authors should report the exact split procedure, provide question-level deduplication or IDs, and give the 172 MedText results separately as the only clearly external portion of the test set.
  2. [Section 4.3] The Rationale Evaluation uses BERTScore and BARTScore with evaluations generated by GPT-4 as the reference. This is circular for AutoMedEval because the model is explicitly distilled from GPT-4's evaluations on the same type of input; high semantic similarity to GPT-4 rationales demonstrates imitation fidelity, not human-aligned quality. The paper should either present these numbers as a distillation-fidelity check rather than a capability comparison, or replace the GPT-4 reference with a human-authored reference rationale set such as the physician-verified evaluations used elsewhere in the paper.
  3. [Appendix F and Section 4.5.2] The saturation prediction is fitted to only three observed points (Accuracy-triple 44.61, 47.13, 48.65 at iterations 0, 1, 2), yet the paper claims that AutoMedEval will outperform curriculum instruction tuning by 17% and that there will be no further growth after six iterations. Equation (7) fixes a1=1, a2=0.9, a3=0.586, and the exponent constants are not derived from any stated fitting procedure; with three data points and at least two free parameters, the sigmoid model is not identifiable and cannot support the saturation and 52.21% predictions. The claim that iterative introspection stops improving after six iterations should be removed or supported by running more iterations and by reporting model-selection or uncertainty information.
minor comments (4)
  1. [Throughout] There are several typos and grammatical errors, including 'evalution' in Equation (1), 'conducte ablation studies' in Section 4.5.1, 'Resposne 3' in Table 7, and 'oftencing' in the same table; a careful proofread is needed.
  2. [Algorithm 1] The loop notation 'for i ← −1 to |D|' and 'for t ← −1 to T' appears to be off-by-one (likely intended to start at 1), and line 5 'd ← −D_i' is not standard pseudocode; this should be cleaned up for reproducibility.
  3. [Section 4.2] The inter-annotator reliability values (ICC 0.712, Krippendorff's alpha 0.725) are reported as a single pair of numbers, but the manuscript does not state which annotation task (response scoring or evaluation-content scoring) these values refer to; please clarify.
  4. [Appendix B] The quality classifier is trained on only 400 labeled examples (200 high, 200 low) and evaluated on 100 test cases; the 91% accuracy should be accompanied by confidence intervals or a confusion matrix, and the paper should note the limited scale when relying on this classifier to select 4,788 + 3,823 training instructions.

Circularity Check

2 steps flagged · score 2.0 of 10

Central human-correlation result is not circular; a secondary GPT-4-referenced rationale metric and a sigmoid extrapolation are minor self-referential steps, with an unverified same-source train/test split as a validity risk.

  1. self definitional [Section 4.3 (Rationale Evaluation); cf. Sections 3.1-3.2]
    "Rationale Evaluation We use two semantic evaluation metrics, BERTScore (Zhang et al., 2019) and BARTScore (Yuan et al., 2021), to assist evaluation by using evaluations generated by GPT-4 as the reference answer."

    AutoMedEval is trained on GPT-4/ChatGPT evaluations (Section 3.1 distills GPT-4 as evaluative evidence; Section 3.2 trains on R' from GPT-4 and S' from ChatGPT). Measuring its rationale quality against GPT-4 rationales via BERTScore/BARTScore therefore rewards reproduction of the teacher that generated the training labels. The rationale columns are a self-consistency check, not independent evidence of medical evaluation quality. The main human-correlation metrics remain external, so this is a secondary circularity.

  2. fitted input called prediction [Section 4.5.2 and Appendix F (Eqs. 3-7, Table 5)]
    "Using the data of iteration results in Table 5, we calculate the final accuracy estimation as follows. ... When the iteration count is set to 6, the accuracy reaches 52.21%."

    The three Accuracy_triple values in Table 5 (44.61, 47.13, 48.65) are used to fix the sigmoid constants, and the same fitted curve is then used to 'find' that performance saturates and to project 52.21% at iteration 6. The prediction is an extrapolation of the fitted function, not an independent measurement; the conclusion of no further growth after six iterations is an artifact of the sigmoid's asymptotic form. This is peripheral to the main result.

full rationale

AutoMedEval is a supervised evaluator distilled from GPT-4/ChatGPT evaluations and then fine-tuned with physician feedback. The central result - Spearman 0.6314 / Accuracy_triple 48.65 against human judgments - is not circular in its design: the training labels are physician-verified GPT-4 scores, and the main evaluation compares the model's scores to independent doctor annotations, not to the training labels. No load-bearing self-citation or uniqueness import occurs; the Wu et al. 2023 MedLLaMA citation is a base-model choice, not an argumentative premise. I therefore do not score this as a derivation that reduces to its inputs. Two secondary weaknesses keep the score above zero: (1) the BERTScore/BARTScore rationale columns use GPT-4 evaluations as the reference for a model that was trained on GPT-4 evaluations, making those columns a self-similarity measure; (2) the Appendix F saturation 'prediction' is an extrapolation of a sigmoid fitted to the three test-set iteration points, so the claim of no further growth after six iterations is a property of the fitted curve. In addition, the paper does not state that the 958 Medical Meadow Wikidoc test entries are disjoint from the 9,874 training entries from the same source; if they overlap, the main correlation could be inflated by memorization. This is an evidentiary gap rather than a demonstrated circular reduction, so I do not count it as a formal circularity step, but it is a serious validity risk.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on the domain assumption that the wikidoc-derived test set is disjoint from training, and on the reliability of physician-verified GPT-4 evaluations as training labels. The sigmoid saturation prediction introduces hand-fitted parameters. No invented entities are proposed.

free parameters (3)
  • Sigmoid saturation model parameters (a1, a2, a3, rate, offset) = a1=1, a2=0.9, a3=0.586, rate=-0.453, offset=-2.83
    Fit to exactly three test-set accuracy values (Iter 0-2) in Appendix F, then used to claim saturation at iteration 6.
  • Curriculum split sizes = 1,911 (curriculum #1), 2,394 (curriculum #3), remainder (#2)
    Chosen by hand with no sensitivity analysis reported.
  • Number of introspection iterations = 2
    Selected based on measured improvements on the test set (Table 3); no held-out validation is described.
assumptions (5)
  • domain assumption The test set (958 wikidoc + 172 MedText) and the training set (9,874 wikidoc) are disjoint, though never explicitly stated.
    If the same question appears in both, the model could memorize evaluations. Section 4.1 defines the test set without addressing overlap with the training data from Section 3.1.
  • domain assumption Five doctors' averaged ratings are treated as ground truth for response quality.
    Inter-rater reliability is moderate (ICC=0.712, alpha=0.725), so the gold labels are themselves noisy. Section 4.2.
  • domain assumption GPT-4 with retrieval augmentation and physician verification produces high-quality evaluation labels for distillation.
    The entire training set is distilled from GPT-4 outputs; physician verification rejected only 305 cases (Section 3.1).
  • domain assumption The quality classifier (SVM on SimCSE embeddings) accurately identifies high-quality instructions.
    Trained on 400 labels and reported at 91% accuracy on 100 test cases (Appendix B).
  • domain assumption MedLLaMA-13B is an appropriate backbone for medical evaluation.
    Chosen as the base model with no comparison against other backbones (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation." pith.science (2026). https://pith.science/paper/L2SOHRHT

@misc{pith2026250511887,
  author       = {Pith},
  title        = {Pith review of: AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2SOHRHT}},
  note         = {Machine review of arXiv:2505.11887}
}
read the original abstract

With the proliferation of large language models (LLMs) in the medical domain, there is increasing demand for improved evaluation techniques to assess their capabilities. However, traditional metrics like F1 and ROUGE, which rely on token overlaps to measure quality, significantly overlook the importance of medical terminology. While human evaluation tends to be more reliable, it can be very costly and may as well suffer from inaccuracies due to limits in human expertise and motivation. Although there are some evaluation methods based on LLMs, their usability in the medical field is limited due to their proprietary nature or lack of expertise. To tackle these challenges, we present AutoMedEval, an open-sourced automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs. The overarching objective of AutoMedEval is to assess the quality of responses produced by diverse models, aspiring to significantly reduce the dependence on human evaluation. Specifically, we propose a hierarchical training method involving curriculum instruction tuning and an iterative knowledge introspection mechanism, enabling AutoMedEval to acquire professional medical assessment capabilities with limited instructional data. Human evaluations indicate that AutoMedEval surpasses other baselines in terms of correlation with human judgments.

Figures

Figures reproduced from arXiv: 2505.11887 by the authors.

Figure 1
Figure 1. Typical example of medical LLMs’ responses [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Creation of our instruction dataset and automatic evaluation model AutoMedEval. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Collaborative Knowledge Introspection curacies could be that AutoMedEval harbors mis￾conceptions about certain elements of evaluation knowledge. Inspired by the process of humans refining initial drafts with feedback (Flower and Hayes, 1981) and cognitive introspection (Chen et al., 2023; Wang and Zhao, 2023), we propose an iterative knowledge introspection approach to enhance the multi-step medical reasoning accura… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: AutoMedEval and chief physicians’ judgment on every two medical LLMs’ performance. "Win" means [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a) Human assessment results on AutoMedE [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The rationale and score from AutoMedEval [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Topics covered by the dataset. You are an expert in the medical field, and you know the causes and symptoms of most disease. You know what treatment should be used for each disease, the effects and side effects of various medicines and treatment. After reading a conver…
Figure 8
Figure 8. Figure 8: Instruction with diverse factors for GPT-4. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Example instruction for tuning AutoMedEval. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 7 canonical work pages

  1. [1]

    Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of nlg systems. In 11th conference of the european chapter of the association for computational linguistics, pages 313--320

    work page 2006

  2. [2]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109

    arXiv 2023

  3. [3]

    Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, et al. 2023. Introspective tips: Large language model for in-context decision making. arXiv preprint arXiv:2305.11598

    arXiv 2023

  4. [4]

    Nancy Chinchor and Beth M Sundheim. 1993. Muc-5 evaluation metrics. In Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993

    1993

  5. [5]

    Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691

    arXiv 2023

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems

    2022

  7. [7]

    Linda Flower and John R Hayes. 1981. A cognitive process theory of writing. College composition and communication, 32(4):365--387

    1981

  8. [8]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821

    arXiv 2021

Show all 39 references
  1. [9]

    Ben Goertzel. 2017. Toward a formal model of cognitive synergy. arXiv preprint arXiv:1703.04361

    2017 arXiv

  2. [10]

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander L \"o ser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca--an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247

    2023 arXiv

  3. [11]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

  4. [12]

    U Hin Lai, Keng Sam Wu, Ting-Yu Hsu, and Jessie Kai Ching Kan. 2023. Evaluating the performance of chatgpt-4 on the united kingdom medical licensing assessment. Frontiers in Medicine, 10

    2023

  5. [13]

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023 a . https://arxiv.org/abs/2310.05470 Generative judge for evaluating alignment . Preprint, arXiv:2310.05470

    2023 arXiv

  6. [14]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023 b . Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6)

    2023

  7. [15]

    Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024. Leveraging large language models for nlg evaluation: A survey. arXiv preprint arXiv:2401.07103

    2024 arXiv

  8. [16]

    Chin Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004)

    2004

  9. [17]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  10. [18]

    Christopher Manning and Hinrich Schutze. 1999. Foundations of statistical natural language processing. MIT press

    1999

  11. [19]

    Allen Nie, Yuhui Zhang, Atharva Shailesh Amdekar, Chris Piech, Tatsunori B Hashimoto, and Tobias Gerstenberg. 2024. Moca: Measuring human-language model alignment on causal and moral judgment tasks. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375

    2023 arXiv

  13. [21]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

    2022

  14. [22]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation

    2002

  15. [23]

    S Rajbhandari, J Rasley, O Ruwase, and Y He. 2019. Zero: memory optimization towards training a trillion parameter models. arxiv e-prints arxiv: 11910.02054 (2019)

    2019 arXiv

  16. [24]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...

    2020

  17. [25]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023 a . Large language models encode clinical knowledge. Nature, pages 1--9

    2023

  18. [26]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023 b . Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617

    2023 arXiv

  19. [27]

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. 2023. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334

    2023 arXiv

  20. [28]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023 a . Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048

    2023 arXiv

  21. [29]

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023 b . Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087

    2023 arXiv

  22. [30]

    Yuqing Wang and Yun Zhao. 2023. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342

    2023 arXiv

  23. [31]

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454

    2023 arXiv

  24. [32]

    Honglin Xiong, Sheng Wang, Yitao Zhu, Zihao Zhao, Yuxiao Liu, Qian Wang, and Dinggang Shen. 2023. Doctorglm: Fine-tuning your chinese doctor is not a herculean task. arXiv preprint arXiv:2304.01097

    2023 arXiv

  25. [33]

    Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023 a . Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196

    2023 arXiv

  26. [34]

    Jie Xu, Lu Lu, Sen Yang, Bilin Liang, Xinwei Peng, Jiali Pang, Jinru Ding, Xiaoming Shi, Lingrui Yang, Huan Song, Kang Li, Xin Sun, and Shaoting Zhang. 2023 b . https://arxiv.org/abs/2305.07340 Medgpteval: A dataset and benchmark to evaluate responses of large language models ...

    2023 arXiv

  27. [35]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263--27277

    2021

  28. [36]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  29. [37]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.