Collaboration among Multiple Large Language Models for Medical Question Answering

Chia-Hsuan Chang; Christopher C. Yang; Kexin Shang

REVIEW 4 major objections 6 minor 33 references

Collaboration among Multiple Large Language Models for Medical Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sharing reasoning among three medical LLMs lifts each one's accuracy on USMLE questions.

desk verdict A plausible multi-LLM collaboration framework for medical QA, undermined by single-run evaluation and internal numeric inconsistencies. read the letter →

arxiv 2505.16648 v1 pith:DMHTXQHA submitted 2025-05-22 cs.CL cs.AI

Kexin Shang , Chia-Hsuan Chang , Christopher C. Yang This is my paper

classification cs.CLcs.AI

keywords largelanguagemodelsmedicalquestionansweringmulti-agentcollaborationiterativeframeworkself-consistencychain-of-thoughtconfidencemeasurementUSMLE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether three different large language models can improve their medical question-answering by sharing their reasoning with one another. It proposes an iterative collaboration framework in which each model first answers USMLE-style multiple-choice questions alone, then models that disagree exchange a summary of their reasoning and answer again. After two rounds, each of the three models gains about 5 to 7 percentage points in accuracy, and the models reach consensus on far more questions. The paper also defines a model's confidence as its tendency to stick with its original answer when contradicted, and finds that confidence tracks standalone accuracy. If the result holds, pooling reasoning across heterogeneous models—without back-and-forth debate—would be a simple way to strengthen medical QA.

What carries the argument

The load-bearing mechanism is the Iterative Collaboration Framework (ICF), built from two parts. Zero-shot Chain-of-Thought with Self-Consistency (ZS-CoT-SC) prompts each model $n=10$ times to reason step by step and answer, takes a majority vote as the model's answer, and uses an external summarizer model (a separate instance of Mixtral) to condense the reasonings behind the majority vote into one paragraph. The Collaboration Loop then collects each disagreeing question's majority answer plus summarized reasoning from all three models into a transcript, and has each model re-answer the question with a review prompt that asks it to critically examine the teammates' reasonings. The loop repeats until the consensus rate $P_{\mathrm{con}} = |Q_{\mathrm{con}}|/|Q|$ reaches at least 80%, which happened after two iterations in the experiments. The transcript is the key object: it lets each model see the others' reasoning in compact form, so the collaboration is a one-way information flow rather than a multi-round debate.

What would settle it

Re-run the entire ICF pipeline several times on the same dataset with different random seeds, or with temperature set to 0; if the 5–7 point accuracy gains disappear, fluctuate wildly, or reverse sign across seeds, the claimed collaboration benefit cannot be separated from sampling noise.

Watch

Extended reading notes

Core claim

The central discovery is that a unidirectional, iterative exchange of summarized reasoning among three heterogeneous LLMs—Med42, ClinCamel, and Mixtral—improves every participant's accuracy on the USMLE sample-question set. Averaged over exam Steps 1–3, accuracy rises by 5.24 points for Med42, 6.56 for ClinCamel, and 5.91 for Mixtral after the collaboration loop, while the rate at which all three models give the same majority answer climbs from 50.82% to 82.62%. The authors interpret the accuracy gains and consensus convergence as evidence that collaboration strengthens reasoning, and they introduce a confidence measure—the empirical probability that a model insists on its original answer when opponents or supporters are present—which is larger for models that are also more accurate alone. They further report that all three models are far more self-consistent across repeated generations on questions they answer correctly than on questions they miss, with the gap widening after collaboration for the two more confident models.

Load-bearing premise

The load-bearing premise is that the majority-vote answers produced by ten sampled generations are stable enough that a change between collaboration rounds reflects a genuine improvement rather than ordinary run-to-run randomness.

Editorial extensions

If this is right

A simple, one-directional exchange of reasoning summaries is enough to lift each participating LLM's accuracy, so multi-round back-and-forth debate is not required to obtain a collaboration benefit.
Consensus among the three models rises from roughly 51% to 83% after two rounds, meaning the framework sharply reduces answer disagreement as well as improving accuracy.
A model's tendency to insist when contradicted—its measured confidence—is higher for models that are more accurate alone, while the less confident model shows the largest accuracy gain from collaboration.
Self-consistency across repeated generations is markedly higher on questions a model answers correctly than on missed questions, suggesting low consistency could serve as a signal of likely error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step, not taken in the paper, is to weight each model's vote by its measured confidence in a joint ensemble; the three-model sample is too small to establish that such weighting helps.
The 80% consensus stopping rule is a fixed threshold; varying it would trade compute against accuracy, and the paper does not report such a sweep.
The same one-way transcript design could be applied to other multiple-choice domains, but because the paper shows LLMs are prompt-sensitive, the templates would need to be re-tuned for each new setting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper proposes a simple multi-LLM collaboration loop for USMLE multiple-choice QA, reports consistent ~5–7 point accuracy gains across three open-source models, and introduces a quantitative confidence metric based on how often a model resists disagreement. The gains are plausible but the experimental evidence is weaker than the tables suggest.

What is actually new is the specific combination: self-consistency majority votes, an external summarizer (a second Mixtral instance), and a unidirectional exchange loop where each model sees all other models' summarized reasoning and re-answers. That is a clean, computationally light design compared to multi-round debate. The confidence metric—insistence rate conditional on teammate support—is a sensible measure, and the paper is appropriately cautious about the Spearman correlation with only three models.

The load-bearing problem is that the whole pipeline is run once. With n=10 samples at temperature 1.0 for the majority vote, each question's vote is stochastic, and several of the reported deltas are just 2–3 questions out of 87–118. Without repeated seeds or confidence intervals, the observed gains could be sampling noise. The authors also change both the prompt template and the input content between baseline and collaboration, so even a stable gain would not isolate 'exchange of reasoning' as the active ingredient. Separately, there are internal inconsistencies: Section IV-C says Med42's confidence is 0.51 and Mixtral's 0.49, while Table IV reports 0.57 and 0.56; Equation 4 defines consistency as count/n but Table V reports raw average counts. The collaboration loop is supposed to run until consensus >= 80%, but Step 2 stops at 74%. There is also no baseline comparison to an ensemble or a simple majority vote among the three models.

This is a paper for a reader interested in engineering tricks for medical QA and in measuring LLM stubbornness. The ideas are worth a serious referee, but the current evidence is not convincing. A revision with repeated runs, a proper baseline, and the numeric inconsistencies cleaned up would make the central claim testable. I'd send it to review with the expectation of major revision; I wouldn't cite it yet.

Referee Report

4 major / 6 minor

Summary. The paper proposes an Iterative Collaboration Framework (ICF) in which three open-source LLMs (Med42, ClinicalCamel, and Mixtral) first answer USMLE-style medical multiple-choice questions using zero-shot chain-of-thought with self-consistency, and then resolve disagreements by iteratively exchanging summarized reasoning through a collaboration loop. Using a filtered USMLE sample (87, 100, and 118 questions for Steps 1, 2, and 3), the authors report that the consensus rate rises from an average of 50.82% to 82.62% after two collaboration rounds, that each model's accuracy improves on average by 5.24 to 6.56 points, and that a proposed confidence metric, based on a model's tendency to insist on or concede its answer under teammate support or opposition, correlates with accuracy. The paper also reports that self-consistency is higher on correctly answered questions than on incorrectly answered ones, before and after collaboration.

Significance. If the empirical claims held, the paper would offer a computationally light, unidirectional multi-LLM collaboration mechanism for medical question answering and a quantifiable confidence measure for LLMs in such collaborations. The strengths are the use of three heterogeneous open-source models, the directionally consistent accuracy improvements across all models and all three USMLE steps, the explicit pseudocode for the pipeline, and the transparency about the small dataset. However, the central empirical claims are not yet established because the pipeline is run only once without variance estimates, and the manuscript contains internal contradictions in the confidence numbers and in the stated stopping criterion. These issues are correctable in a revision, so the work is a promising preliminary study rather than a definitive demonstration.

major comments (4)

[Section IV.C, Table IV, Section V.A] The confidence values are internally inconsistent. The text in Section IV.C states Med42 has confidence 0.51, Mixtral 0.49, and ClinCamel 0.23, while Table IV reports 0.57, 0.56, and 0.33. Computing Eq. (3) as the average of the S+ and S- components yields the Table IV values (e.g., (45.33+68.10)/2 = 56.72%), so the table appears to be the intended result, but Section V.A then repeats the 0.23/0.49/0.51 numbers when discussing the negative correlation with accuracy improvement. This contradiction affects the paper's confidence-related claims and must be resolved. Furthermore, the reported Spearman correlation rs = -0.5 is not derivable from either set of confidence values together with the Table III improvements: for both sets, the ranking yields a perfect negative Spearman correlation of -1, not -0.5. The authors should recompute and report the correct statistic or remove it.
[Section III-B2, Algorithm 2, Table II] The termination criterion of the collaboration loop is violated in the reported results. The text states that the loop continues until P_con meets the criterion of at least 80%, but Algorithm 2's while condition is written as "P_con <= 80%", which is the opposite inequality. More concretely, Table II shows that after two collaboration rounds Step 2 reaches only 74.00% consensus, below the stated 80% threshold, yet the results are reported as final. The authors need to clarify the actual stopping rule and explain why Step 2 was terminated below the threshold, or revise the threshold and the algorithm pseudocode accordingly.
[Section III-B1, Algorithm 1, Section III-C] The central accuracy-improvement claim is not supported by variance estimates. The whole pipeline is executed once, with n=10 generations at temperature 1.0 and no repeated random seeds. Majority votes at temperature 1.0 are stochastic quantities, and several of the reported gains are small in absolute question counts: Med42 Step 1 gains 3.45 points (3 questions out of 87), and Mixtral Step 3 gains 1.69 points (2 questions out of 118). These deltas are comparable to the run-to-run noise expected from a few question flips under majority voting. Without repeated seeds or confidence intervals, the observed accuracy improvements and the consensus-rate changes could partly reflect sampling variation rather than collaboration benefit. This issue is load-bearing for the paper's main claim and should be addressed with repeated runs, bootstrap intervals, or an equivalent variance analysis.
[Section III-B2, Section III-C, Algorithm 1] The Collaboration condition changes two factors at once: the prompt template (Treasoning_review replaces the sequential Treasoning/Tanswer) and the input content (the integrated answers and reasonings of the other models are appended). Consequently, even a stable accuracy gain would not isolate "exchange of reasoning" as the active ingredient, because the prompt change alone could drive part of the effect. An ablation that applies the Treasoning_review template with only the model's own reasoning, or with a control transcript containing no other model's opinion, would be needed to attribute the improvement to collaboration. This is a secondary attribution issue but should be acknowledged and ideally tested in a revision.

minor comments (6)

[Algorithm 1, line 5] The MajorityVote step does not specify tie-breaking; with n=10 generations, exact ties among letter choices are possible, and the treatment of ties affects both accuracy and the confidence metric.
[Section IV.A] The text contains the typo "ZS-SC-SC" where "ZS-CoT-SC" is intended.
[Equation (3)] The notation for averaging the pinsist vector is garbled ("P pinsist"); please define the vector components and the averaging operation explicitly.
[Section V.A] The Spearman correlation is computed on only three models, so the P-value of 0.67 is unsurprising and the test has almost no power. The authors should present this as a descriptive observation only, or remove the inferential language.
[Abstract and Section VI] The word "proved" overstates the strength of a single-run, small-sample experiment; "suggests" would be more appropriate given the lack of variance estimates.
[Section III.C] The prompt-selection procedure is described only as testing variations on 30 randomly selected questions. Please report the set of candidate prompts, the selection criterion, and whether the same prompts were used for all three models, to support reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: accuracy gains are measured outputs, not fitted quantities; confidence and consensus are observationally coupled but not definitionally forced.

full rationale

The paper's central claim — that two rounds of unidirectional reasoning exchange improve individual LLM accuracy on USMLE-style questions — rests on a direct before/after measurement (Table III) of the same pipeline (Algorithm 1 followed by Algorithm 2). The collaboration condition changes both the prompt template and the input content, so the comparison is an empirical attribution rather than a derivation from the inputs. The confidence metric (Eq. 2-3) is a defined behavioral measurement taken from the same collaboration runs; correlating it with accuracy improvement is an observational association with a mechanical upper bound (a model that never changes answers cannot improve), but it is not a fitted parameter renamed as a prediction, and the paper explicitly labels the n=3 Spearman result non-significant. The consensus convergence result is partly entangled with the P_con >= 80% stopping criterion in Algorithm 2, but the actual table reports Step 2 ending at 74% after a fixed two iterations, so the values are not logically forced by the stopping rule. The only same-group citation ([12]) is used for data preprocessing and is not load-bearing for the accuracy or confidence claims. The 30-question prompt tuning is a test-set contamination concern, not a circularity of the derivation. Overall, no claim reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces new metrics (confidence in Equations 2-3 and consistency in Equation 4) but no new physical or structural entities. The external summarizer is an existing model (Mixtral), so there are no graviton-style invented entities. The main ledger entries are the hand-chosen threshold, repeat count, and prompt selection, all of which shape the reported numbers.

free parameters (4)

Consensus threshold P_con = 80%
Hand-chosen termination criterion for the collaboration loop. The reported results stop after two iterations even though Step 2 reaches only 74%, which is below this threshold.
Self-consistency repeat count n = 10
Hand-chosen number of generations per question. It drives the majority vote, the consensus classification, and the confidence and consistency metrics.
Per-model prompt template = best on 30 random questions
Prompt formats were tested on a randomly selected 30-question subset and the highest accuracy template was chosen. This tunes the system on part of the evaluation data.
Temperature = 1.0
Chosen for all models. It controls the diversity of self-consistency samples and therefore the stability of majority votes.

assumptions (5)

domain assumption The 305 filtered USMLE sample questions are a representative medical QA benchmark.
Section III-A filters USMLE sample exams to text-only multiple choice questions. The paper does not claim generalizability beyond this dataset.
domain assumption Self-consistency majority vote approximates the model's true belief.
Algorithm 1 lines 5-8 select the majority vote as the model's prediction and use it to classify consensus and disagreement. This follows Wang et al. but is assumed without validation.
domain assumption The external summarizer Mixtral preserves reasoning content without bias.
Algorithm 1 line 7 summarizes 10 reasonings into one paragraph. The paper does not check whether summarization changes the reasoning or favors particular answers.
domain assumption Prompt selection on 30 random questions transfers to the full dataset.
Section III-C states prompts were chosen on a 30-question subset. The paper does not validate that these prompts generalize to the remaining 275 questions.
domain assumption Consensus among LLMs is a desirable state.
Section III-B uses consensus rate as the termination criterion and treats convergence as a goal, without arguing why consensus implies correctness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Collaboration among Multiple Large Language Models for Medical Question Answering." pith.science (2026). https://pith.science/paper/DMHTXQHA

@misc{pith2026250516648,
  author       = {Pith},
  title        = {Pith review of: Collaboration among Multiple Large Language Models for Medical Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMHTXQHA}},
  note         = {Machine review of arXiv:2505.16648}
}

read the original abstract

Empowered by vast internal knowledge reservoir, the new generation of large language models (LLMs) demonstrate untapped potential to tackle medical tasks. However, there is insufficient effort made towards summoning up a synergic effect from multiple LLMs' expertise and background. In this study, we propose a multi-LLM collaboration framework tailored on a medical multiple-choice questions dataset. Through post-hoc analysis on 3 pre-trained LLM participants, our framework is proved to boost all LLMs reasoning ability as well as alleviate their divergence among questions. We also measure an LLM's confidence when it confronts with adversary opinions from other LLMs and observe a concurrence between LLM's confidence and prediction accuracy.

Figures

Figures reproduced from arXiv: 2505.16648 by the authors.

**Figure 3.** Prompt templates of summerizer LLM ψ. In this experiment, we deploy Mixtral to summarize repetitive reasoning pathways from self-consistency sampling. The highlighted part is a integrated context of the majority vote and n = 10 generated reasonings from one LLM participant in ICF. Then we intersect Qinsist ϕ with S+ and Srespectively, creating a vector p insist ϕ : p insist ϕ =   |S +∩Qinsist ϕ | |S+| |S −∩Qinsi… view at source ↗

**Figure 2.** Prompt templates of Med42 in ZS-CoT-SC and collab [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 16 canonical work pages

[1]

Med42 – evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,

C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal, B. Kanakiya, C. Chen, N. Vassilieva, B. B. Amor, M. A. Pimentel, and S. Khan, “Med42 – evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,” 2024. [Online]. Available: https://arxiv.org/a...

arXiv 2024
[2]

Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding,

A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang, “Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12031

arXiv 2023
[3]

Meditron-70b: Scaling medical pretraining for large language models,

Z. Chen, A. Hern ´andez-Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. K ¨opf, A. Mohtashami, A. Sallinen, A. Sakhaeirad, V . Swamy, I. Krawczuk, D. Bayazit, A. Marmet, S. Mon- tariol, M.-A. Hartley, M. Jaggi, and A. Bosselut, “Meditron-70b: Scaling medical pretraining for large language models,” 2023

work page 2023
[4]

Medalpaca – an open-source collection of medical conversational ai models and training data,

T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. L ¨oser, D. Truhn, and K. K. Bressem, “Medalpaca – an open-source collection of medical conversational ai models and training data,” 2023. [Online]. Available: https://arxiv.org/abs/2304.08247

arXiv 2023
[5]

ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?

Y . Kumah-Crystal, S. Mankowitz, P. Embi, and C. U. Lehmann, “ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?” Journal of the American Medical Informatics Association , vol. 30, no. 9, pp. 1558–1560, Jun. 2023, eprint: https://academic.oup.com/jamia/article- pdf/30/9/1558/51141429/ocad104.pdf. [...

work page doi:10.1093/jamia/ocad104 2023
[6]

Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,

T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepa ˜no, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo, and V . Tseng, “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,” PLOS Digital Health , vol. 2, no. 2, p. e0000198, Feb. 2023. [Online]. Available: https://dx.plos.org/10...

doi:10.1371/journal.pdig.0000198 2023
[7]

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment,

A. Gilson, C. W. Safranek, T. Huang, V . Socrates, L. Chi, R. A. Taylor, and D. Chartash, “How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment,” JMIR Medical Education , vol. 9, p. e45312, Feb. 2023. [Online]. Available: https://mededu...

work page 2023
[8]

Capabilities of GPT-4 on Medical Challenge Problems,

H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of GPT-4 on Medical Challenge Problems,” Apr. 2023, arXiv:2303.13375 [cs]. [Online]. Available: http://arxiv.org/abs/2303.13375

arXiv 2023

Show all 33 references

[9]

Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary Care,

A. J. Thirunavukarasu, R. Hassan, S. Mahmood, R. Sanghera, K. Barzangi, M. El Mukashfi, and S. Shah, “Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary Care...
[10]

Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?

M. Chowdhury, E. Lim, A. Higham, R. McKinnon, N. Ventoura, Y . He, and N. De Pennington, “Can Large Language Models Safely Address Patient Questions Following Cataract Surgery?” in Proceedings of the 5th Clinical Natural Language Processing Workshop . Toronto, Canada: Associat...

2023
[11]

Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard,

Z. W. Lim, K. Pushpanathan, S. M. E. Yew, Y . Lai, C.-H. Sun, J. S. H. Lam, D. Z. Chen, J. H. L. Goh, M. C. J. Tan, B. Sheng, C.-Y . Cheng, V . T. C. Koh, and Y .-C. Tham, “Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5,...

2023
[12]

Reasoning with large language models for medical question answering,

M. M. Lucas, J. Yang, J. K. Pomeroy, and C. C. Yang, “Reasoning with large language models for medical question answering,” Journal of the American Medical Informatics Association , vol. 31, no. 9, pp. 1964–1975, Sep. 2024. [Online]. Available: https://academic.oup.com/jamia/a...

1964
[13]

LARGE LANGUAGE MODELS CANNOT SELF- CORRECT REASONING YET,

J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “LARGE LANGUAGE MODELS CANNOT SELF- CORRECT REASONING YET,” 2024

2024
[14]

Adaptive Chameleon or Stubborn Sloth:,

J. Xie, K. Zhang, J. Chen, R. Lou, and Y . Su, “Adaptive Chameleon or Stubborn Sloth:,” 2024

2024
[15]

Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration

S. Feng, W. Shi, Y . Wang, W. Ding, V . Balachandran, and Y . Tsvetkov, “Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration.”
[16]

Exploring collaboration mechanisms for LLM agents: A social psychology view,

J. Zhang, X. Xu, N. Zhang, R. Liu, B. Hooi, and S. Deng, “Exploring collaboration mechanisms for LLM agents: A social psychology view,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , L.-W. Ku, A. Martins, an...

2024
[17]

Counterfactual debating with preset stances for hallucination elimination of llms,

Y . Fang, M. Li, W. Wang, H. Lin, and F. Feng, “Counterfactual debating with preset stances for hallucination elimination of llms,”
[18]

One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering,

H. Yang, M. Li, H. Zhou, Y . Xiao, Q. Fang, and R. Zhang, “One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering,” Dec. 2023. [Online]. Available: http://medrxiv.org/lookup/doi/10.1101/2023.12.21.23300380

2023 doi
[19]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

2023 arXiv
[20]

Large language models are zero-shot reasoners,

T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” 2023. [Online]. Available: https://arxiv.org/abs/2205.11916

2023 arXiv
[21]

Multivariable analysis of factors associated with USMLE scores across U.S. medical schools,

A. Ghaffari-Rafi, R. E. Lee, R. Fang, and J. D. Miles, “Multivariable analysis of factors associated with USMLE scores across U.S. medical schools,” BMC Medical Education , vol. 19, no. 1, p. 154, May 2019. [Online]. Available: https://doi.org/10.1186/s12909-019-1605-z

2019 doi
[22]

Mixtral of experts,

A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T...

2024 arXiv
[23]

Llama 2: Open foundation and fine-tuned chat models,

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

2023 arXiv
[24]

Qlora: Efficient finetuning of quantized llms,

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” 2023. [Online]. Available: https://arxiv.org/abs/2305.14314

2023 arXiv
[25]

Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting,

M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting,” Oct. 2023, arXiv:2310.11324 [cs]. [Online]. Available: http://arxiv.org/abs/2310.11324

2023 arXiv
[26]

Improving Factuality and Reasoning in Language Models through Multiagent Debate

Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, “Improving Factuality and Reasoning in Language Models through Multiagent Debate.”
[27]

Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration,

S. Feng, W. Shi, Y . Wang, W. Ding, V . Balachandran, and Y . Tsvetkov, “Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration,” Feb. 2024, arXiv:2402.00367 [cs]. [Online]. Available: http://arxiv.org/abs/2402.00367

2024 arXiv
[28]

Self-consistency improves chain of thought reasoning in language models,

X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” 2023. [Online]. Available: https://arxiv.org/abs/2203.11171

2023 arXiv
[29]

Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View,

J. Zhang, X. Xu, N. Zhang, R. Liu, B. Hooi, and S. Deng, “Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View,” Feb. 2024, arXiv:2310.02124 [cs]. [Online]. Available: http://arxiv.org/abs/2310.02124

2024 arXiv
[30]

J. H. Zar, Spearman Rank Correlation: Overview . John Wiley & Sons, Ltd, 2014. [Online]. Available: https://doi.org/10.1002/9781118445112.stat05964

2014 doi
[31]

Debating with more persuasive llms leads to more truthful answers,

A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rockt ¨aschel, and E. Perez, “Debating with more persuasive llms leads to more truthful answers,” 2024. [Online]. Available: https://arxiv.org/abs/2402.06782

2024 arXiv
[2023]

Available: https://mededu.jmir.org/2023/1/e46599

[Online]. Available: https://mededu.jmir.org/2023/1/e46599

2023
[2024]

Available: https://arxiv.org/abs/2406.11514

[Online]. Available: https://arxiv.org/abs/2406.11514

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Med42 – evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,

C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal, B. Kanakiya, C. Chen, N. Vassilieva, B. B. Amor, M. A. Pimentel, and S. Khan, “Med42 – evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,” 2024. [Online]. Available: https://arxiv.org/a...

arXiv 2024

[2] [2]

Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding,

A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang, “Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12031

arXiv 2023

[3] [3]

Meditron-70b: Scaling medical pretraining for large language models,

Z. Chen, A. Hern ´andez-Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. K ¨opf, A. Mohtashami, A. Sallinen, A. Sakhaeirad, V . Swamy, I. Krawczuk, D. Bayazit, A. Marmet, S. Mon- tariol, M.-A. Hartley, M. Jaggi, and A. Bosselut, “Meditron-70b: Scaling medical pretraining for large language models,” 2023

work page 2023

[4] [4]

Medalpaca – an open-source collection of medical conversational ai models and training data,

T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. L ¨oser, D. Truhn, and K. K. Bressem, “Medalpaca – an open-source collection of medical conversational ai models and training data,” 2023. [Online]. Available: https://arxiv.org/abs/2304.08247

arXiv 2023

[5] [5]

ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?

Y . Kumah-Crystal, S. Mankowitz, P. Embi, and C. U. Lehmann, “ChatGPT and the clinical informatics board examination: the end of unproctored maintenance of certification?” Journal of the American Medical Informatics Association , vol. 30, no. 9, pp. 1558–1560, Jun. 2023, eprint: https://academic.oup.com/jamia/article- pdf/30/9/1558/51141429/ocad104.pdf. [...

work page doi:10.1093/jamia/ocad104 2023

[6] [6]

Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,

T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepa ˜no, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo, and V . Tseng, “Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,” PLOS Digital Health , vol. 2, no. 2, p. e0000198, Feb. 2023. [Online]. Available: https://dx.plos.org/10...

doi:10.1371/journal.pdig.0000198 2023

[7] [7]

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment,

A. Gilson, C. W. Safranek, T. Huang, V . Socrates, L. Chi, R. A. Taylor, and D. Chartash, “How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment,” JMIR Medical Education , vol. 9, p. e45312, Feb. 2023. [Online]. Available: https://mededu...

work page 2023

[8] [8]

Capabilities of GPT-4 on Medical Challenge Problems,

H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of GPT-4 on Medical Challenge Problems,” Apr. 2023, arXiv:2303.13375 [cs]. [Online]. Available: http://arxiv.org/abs/2303.13375

arXiv 2023