Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

A Benchmark for Long-Form Medical Question Answering

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces a publicly available benchmark of 1,077 real-world consumer medical questions with doctor-annotated pairwise evaluations of long-form answers, and reports preliminary results in which the open Llama-3.1-405B-Instruct…

desk verdict Worth reading for the dataset and honest accounting, but the headline ranking claim is undercut by near-chance annotator agreement. read the letter →

arxiv 2411.09834 v2 pith:SXYTBC4D submitted 2024-11-14 cs.CL cs.AI

Pedram Hosseini , Jessica M. Sin , Bing Ren , Bryceton G. Thomas , Elnaz Nouri , Ali Farahanchi , Saeed Hassanpour This is my paper · ORCID
classification cs.CLcs.AI
keywords long-formmedicalquestionansweringconsumerhealthquestionsdoctorannotationLLM-as-a-judgeopenLLMsbenchmarkpairwisecomparisonGPT-4ovsLlama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-form medical question answering has lacked a reusable, doctor-annotated benchmark built from questions real consumers actually ask; existing medical QA benchmarks mostly use multiple-choice items or automatic metrics, and prior human-evaluation efforts kept their labels closed. This paper fills that gap by releasing 1,077 semantically distinct consumer health questions together with pairwise judgments from medical doctors on six criteria: correctness, helpfulness, harmfulness, reasoning, efficiency, and bias. Using those annotations, the authors report preliminary evidence that the open-weights Llama-3.1-405B-Instruct outperforms GPT-4o across all six criteria, that AlpaCare-13B beats BioMistral-7B but trails GPT-4, and that medically pretrained Meditron3-70B does not clearly beat its vanilla Llama-3.1-70B base. They also compare LLM-as-a-judge votes with the doctors' votes and find a real but incomplete alignment. If the benchmark is sound, the field gains an open resource for judging long-form medical answers and a concrete challenge to the assumption that closed commercial models lead in this domain.

What carries the argument

The mechanism is a pairwise long-form answer comparison protocol. Two medical doctors per batch judge which of two randomized responses is better on six criteria—correctness, helpfulness, harmfulness, reasoning, efficiency, and bias—with tie and 'neither' options available. The benchmark questions are produced by a pipeline that filters real user queries with GPT-4, manually cleans them, clusters embeddings with DBSCAN, and keeps one medoid per cluster to obtain 1,077 semantically distinct items. An LLM-as-a-judge variant runs the same pairwise comparisons with position-swapped runs and conservative tie/neither resolution, and its votes are compared with the doctors' votes.

What would settle it

Have a different pair of board-certified physicians independently re-annotate all 400 questions from batches 3 and 4 under the same scheme, then compare aggregate pairwise winners; if the new panel's winner differs on correctness or helpfulness, or if the new pair's observed agreement is within a few points of chance, the claim that Llama-3.1 outperforms GPT-4o across all aspects fails. A concrete statistic: Cohen's kappa below 0.1 on correctness or efficiency for the new panel would falsify the labeling reliability the ranking depends on.

Watch

Extended reading notes

Core claim

The central claim is that a public benchmark of 1,077 real-world consumer medical questions with pairwise doctor annotations can be built, and that on this benchmark the open-weights Llama-3.1-405B-Instruct beats GPT-4o on all six evaluated criteria in its batch. The smaller open medical model AlpaCare-13B outperforms BioMistral-7B but loses to GPT-4, while Meditron3-70B fails to improve over its vanilla Llama-3.1-70B base. The authors further claim that LLM-as-a-judge votes broadly agree with the doctors' votes but leave a clear alignment gap, and that the benchmark's questions are semantically distinct from those in MedRedQA, HealthSearchQA, and MASH-QA. The release includes the 1,077 questions, the generated model answers, and the doctor annotations.

Load-bearing premise

The ranking rests on the assumption that two doctors' pairwise judgments on six fine-grained criteria are reliable enough to serve as ground truth; in later batches, observed agreement on several criteria is at or near chance, so if those labels are effectively noise, the reported model ordering does not follow.

Editorial extensions

If this is right

  • If the benchmark is correct, open-weights models can match or beat closed commercial models on long-form consumer medical answers, which matters for privacy-sensitive deployments where patient data cannot be sent to third-party APIs.
  • The released doctor annotations let other teams reproduce the model rankings, reuse the questions, and build evaluation or training pipelines without repeating the costly annotation effort.
  • The evaluation protocol gives healthcare organizations a concrete template: pairwise judgment on six criteria, with randomized answer order and position-swap checks for LLM judges.
  • The Meditron3-70B result challenges the assumption that additional medical pretraining is necessary for a model to give strong clinical answers.
  • The low semantic overlap with MedRedQA, HealthSearchQA, and MASH-QA indicates the benchmark contributes genuinely new question content rather than re-labeling existing datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the near-chance doctor agreement in later batches is confirmed by a larger panel, the main ranking claim should be treated as hypothesis-generating; a coarser single-dimensional judgment might be more reliable than the six-way scale.
  • Because the authors used GPT-4 to filter and correct the consumer questions, any systematic bias in what GPT-4 counts as a medical question is baked into the benchmark; a human-only filtered subset would quantify that effect.
  • A natural next experiment is retrieval-augmented answer generation on these same 1,077 questions; the doctor labels would then show whether grounding in clinical sources changes the open-versus-closed model ordering.
  • The public labels could also be used to train an annotator-consensus model or to calibrate LLM judges, potentially narrowing the observed gap between LLM-as-a-judge votes and human labels on long-form medical answers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a publicly available benchmark for long-form consumer medical question answering, built from 1,077 real-world queries collected on the authors' own platform. The benchmark includes pairwise evaluations of LLM-generated answers by two medical doctors per batch across six criteria (correctness, helpfulness, harmfulness, reasoning, efficiency, bias), as well as LLM-as-a-judge evaluations using GPT-4o and Claude-3.5-Sonnet. The paper reports that the open model Llama-3.1-405B outperforms the closed model GPT-4o across all evaluated aspects, and that open medical models do not necessarily improve over their general-purpose base models. The authors release the questions, answers, and doctor annotations on GitHub.

Significance. If validated, this benchmark would be a useful community resource: it provides a real-world consumer medical QA dataset, releases expert doctor annotations that are otherwise unavailable, and reports a detailed LLM-as-a-judge robustness analysis. The paper is transparent about its annotation pipeline and reports inter-annotator agreement tables rather than hiding disagreements. However, the central comparative claim (Llama-3.1 beats GPT-4o 'across all aspects') rests on doctor annotations whose observed agreement is at or near chance for several key criteria, and the 'long-form' characterization is undercut by the three-sentence answer cap. The resource itself is valuable, but the strength of the headline conclusions requires revision.

major comments (3)
  1. [Section 4.1, Table 3] The headline conclusion that Llama-3.1-405B outperforms GPT-4o across all aspects is not supported by the reported annotator agreement. In Batch 3 (the GPT-4o vs. Llama-3.1 comparison), observed agreement is near chance for Helpfulness (P=0.23 vs. C=0.19), Reasoning (P=0.17 vs. C=0.16), and Efficiency (P=0.26 vs. C=0.25); in Batch 4, Efficiency is exactly at chance (P=0.11 vs. C=0.11). When annotator labels carry no signal beyond chance, the majority vote used to rank models can be driven by one annotator's arbitrary choices. The paper acknowledges the low agreement in Section 4.1 but still draws the 'across all aspects' conclusion without reporting a chance-corrected agreement measure (e.g., Cohen's kappa), a significance test, or a per-criterion reliability threshold. Please either provide such reliability evidence, restrict the comparative claims to criteria with agreement significantly above chance, or present the model-ranking results with appropriate confidence intervals and caveats.
  2. [Section 3.4] The LLM-as-a-judge analysis uses GPT-4o both as an evaluated candidate model and as one of the two judge models. The paper reports 71% observed agreement and 55% Cohen's kappa between GPT-4o and Claude judges, but this only measures agreement between two LLM judges; it does not calibrate LLM judgments against the human doctor labels. The statement in Section 4 that there is 'general agreement across all batches and criteria' is not quantified against the human labels. Please report agreement (e.g., percentage and kappa) between each LLM judge's votes and the majority human votes, and separately for criteria where human agreement is itself above chance. Without such calibration, the LLM-judge results cannot be used to corroborate the human-based model rankings.
  3. [Section 2.5, Figure 10] The benchmark is described as a 'long-form' medical QA benchmark, but the answer-generation prompt in Figure 10 instructs models to 'Complete answer in three sentences max.' Three-sentence answers are not long-form by typical definitions, and the paper does not justify why this operationalization captures long-form medical responses. This affects the construct validity of the benchmark, since the evaluation criteria (e.g., efficiency, reasoning) may behave differently for three-sentence versus genuinely long-form answers. Please either revise the benchmark description, justify the three-sentence cap as an adequate operationalization of long-form answers, or adopt a more generous answer-length constraint.
minor comments (6)
  1. [General] The manuscript contains several typos and duplicated figure captions: Figures 9-12 in the appendix are all labeled 'Table 3: Caption' or 'Table 4: Caption' in the provided text, and 'efficiency' is misspelled as 'Efficiency' in the Figure 12 prompt template. Please correct these issues.
  2. [Section 2.1.2] The quality-check threshold of 85% for SequenceMatcher similarity is described as determined 'after manually inspecting some samples,' but no detail is given about how many samples were inspected or how the threshold was validated. Please provide this detail or state that the threshold is an arbitrary choice.
  3. [Section 2.2] The DBSCAN threshold and epsilon values (threshold=0.75, eps=0.25) are selected by manual inspection of clusters, but no quantitative evaluation of cluster quality is provided. Please clarify how the manual inspection was conducted and whether alternative thresholds materially change the resulting 1,077 questions.
  4. [Section 2.4] The difficulty-level annotation is performed by GPT-4, but the paper does not report any validation of these labels against human judgment, despite the earlier human-validation exercise for medical-question detection in Section 2.1.1. Please add a validation step or explicitly note that difficulty labels are LLM-generated and unvalidated.
  5. [Section 3.3] The human evaluation used only three medical doctors total, with two per batch, and the paper does not specify whether the same two doctors annotated all batches or whether the pairings changed. Please clarify the annotator assignment and whether any annotator effects (e.g., differing strictness) were examined.
  6. [Section 2.3] The inter-dataset similarity analysis compares a random sample of the same size as the benchmark from each other dataset, but the paper does not report the standard deviation or confidence intervals for the similarity scores. Please include these to support the claim of 'low overlap.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark construction, human annotations, and model rankings are externally grounded; the acknowledged low annotator agreement is a reliability concern, not a circular derivation.

full rationale

The paper's central claims rest on two external sources: a newly collected set of real-world consumer questions from the authors' own platform and pairwise preference labels provided by medical doctors. No equation in the paper fits a parameter to a subset of data and then re-presents that fit as a prediction; the model rankings are tabulated from doctor votes rather than derived from the paper's definitions. The LLM-as-a-judge section is presented as a separate alignment study, and the headline result that 'Llama-3.1 outperforms GPT-4o across all aspects' is drawn from the human doctor evaluations in Section 4, not from the fact that GPT-4o also served as an LLM judge. The cited prior work (Med-PaLM, Zheng et al., WildBench) is used for prompt design or benchmark grounding, not as a self-citation chain that encodes the result. The paper itself flags a substantial limitation: 'for the last two batches, the agreement level is fairly low,' with Table 3 showing observed agreement near chance for several criteria (e.g., Batch 4 Efficiency P=0.11 vs C=0.11), and it concedes that 'more rigorous and comprehensive evaluations are needed to generalize these conclusions.' That is a genuine empirical and statistical weakness in the strength of the ranking claim, but it is not circularity: the annotator labels are external judgments, not consequences of the benchmark's construction or annotations. No circular step can be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark's construction depends on several hand-tuned thresholds (85% similarity, DBSCAN eps=0.25), on GPT-4's question filtering, and on the assumption that two doctors' judgments provide reliable ground truth despite near-chance agreement on some criteria. The three-sentence answer cap is an ad hoc operationalization of 'long-form'. No new physical or mathematical entities are introduced.

free parameters (5)
  • SequenceMatcher similarity threshold = 0.85
    Used in Section 2.1.2 to keep corrected queries similar to originals; chosen after manual inspection, no sensitivity analysis.
  • DBSCAN threshold = 0.75 (cosine similarity), eps = 0.25
    Used in Section 2.2 for semantic deduplication; authors tried multiple thresholds and manually checked clusters, so the choice is hand-tuned.
  • Embedding dimension = 1024
    text-embedding-3-large dimension; a modeling choice that affects deduplication and inter-dataset similarity.
  • Answer length cap = 3 sentences
    Figure 10 prompt caps all generated answers at three sentences, which defines the evaluation as short-form rather than long-form.
  • Batch size = 100 questions
    Annotation batches of 100 questions; chosen for annotation logistics, but no power analysis links this to detectable effect sizes.
assumptions (5)
  • domain assumption GPT-4's medical-question detection and grammar correction are accurate enough to build the question pool.
    Section 2.1 relies on GPT-4 labels after a 337-sample human check; only a subset was validated.
  • domain assumption Doctor pairwise judgments on the six criteria encode valid ground truth for long-form medical QA quality.
    Section 3.3 uses doctor votes as gold labels, yet Table 3 shows several criteria with near-chance agreement.
  • domain assumption User queries from Lavita Medical AI Assist are representative of real-world consumer medical questions.
    Section 2 collects all queries from a single commercial platform; no external sampling frame is used.
  • ad hoc to paper Three-sentence responses are a sufficient operationalization of 'long-form' medical answers.
    Figure 10 limits answers to three sentences while the paper claims long-form evaluation.
  • domain assumption Low cosine similarity between embedding centroids implies the benchmark questions are novel relative to MedRedQA, HealthSearchQA, and MASH-QA.
    Section 2.3 interprets average cosine similarity as novelty; no human judgment of overlap is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Benchmark for Long-Form Medical Question Answering." pith.science (2026). https://pith.science/paper/SXYTBC4D

@misc{pith2026241109834,
  author       = {Pith},
  title        = {Pith review of: A Benchmark for Long-Form Medical Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXYTBC4D}},
  note         = {Machine review of arXiv:2411.09834}
}
read the original abstract

There is a lack of benchmarks for evaluating large language models (LLMs) in long-form medical question answering (QA). Most existing medical QA evaluation benchmarks focus on automatic metrics and multiple-choice questions. While valuable, these benchmarks fail to fully capture or assess the complexities of real-world clinical applications where LLMs are being deployed. Furthermore, existing studies on evaluating long-form answer generation in medical QA are primarily closed-source, lacking access to human medical expert annotations, which makes it difficult to reproduce results and enhance existing baselines. In this work, we introduce a new publicly available benchmark featuring real-world consumer medical questions with long-form answer evaluations annotated by medical doctors. We performed pairwise comparisons of responses from various open and closed-source medical and general-purpose LLMs based on criteria such as correctness, helpfulness, harmfulness, and bias. Additionally, we performed a comprehensive LLM-as-a-judge analysis to study the alignment between human judgments and LLMs. Our preliminary results highlight the strong potential of open LLMs in medical QA compared to leading closed models. Code & Data: https://github.com/lavita-ai/medical-eval-sphere

Figures

Figures reproduced from arXiv: 2411.09834 by the authors.

Figure 1
Figure 1. Overview of our benchmark creation process [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The distribution of queries in conversations (excluding outliers) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Similarity graph of consumer medical question answering datasets [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distribution of votes by medical doctors ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Votes grouped by annotators - Batch 1 A.2 LLM-as-a-Judge Statistics [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Votes grouped by annotators - Batch 2 10 13 72 5 11 52 31 6 GPT-4o Llama-3.1-405b-Instruct Tie Neither 0 20 40 60 13 18 62 14 7 77 5 4 GPT-4o Llama-3.1-405b-Instruct Tie Neither 0 20 40 60 80 5 4 7 84 5 2 7 86 GPT-4o Llama-3.1-405b-Instruct Tie Neither 0 20 40 60 80 4 …
Figure 7
Figure 7. Figure 7: Votes grouped by annotators - Batch 3 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Votes grouped by annotators - Batch 4 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 12
Figure 12. Figure 12: LLM-as-a-judge prompt template 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HIVMedQA: Benchmarking large language models for HIV medical decision support

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The HIVMedQA benchmark finds that Gemini 2.5 Pro leads on most clinical reasoning dimensions, medical fine-tuning does not guarantee gains, and LLM judges are more informative than lexical overlap.

  2. MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MedGUIDE tests whether LLMs follow structured NCCN cancer-care decision trees and finds that even medical LLMs often lag general models on this task.

  3. Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Hybrid RAG over UK public health guidance sharply raises MCQA accuracy and free-form faithfulness, letting smaller open models match larger closed models without retrieval.

Reference graph

Works this paper leans on

28 extracted references · 19 canonical work pages · cited by 3 Pith papers

  1. [1]

    H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al

    Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., et al. (2023). Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079

    arXiv 2023

  2. [2]

    Cochran, W. G. (1977). Sampling techniques. john wiley & sons

    work page 1977

  3. [3]

    Deng, C., Zhao, Y ., Tang, X., Gerstein, M., and Cohan, A. (2023). Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly

    work page 2023

  4. [4]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  5. [5]

    Ester, M., Kriegel, H.-P., Sander, J., Xu, X., et al. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231

    work page 1996

  6. [6]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    arXiv 2020

  7. [7]

    Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. (2021). What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    work page 2021

  8. [8]

    Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. (2019). PubMedQA: A dataset for biomedical research question answering. In Inui, K., Jiang, J., Ng, V ., and Wan, X., editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p...

    work page 2019

Show all 28 references
  1. [9]

    Kim, Y ., Wu, J., Abdulle, Y ., and Wu, H. (2024). MedExQA: Medical question answering bench- mark with multiple explanations. In Demner-Fushman, D., Ananiadou, S., Miwa, M., Roberts, K., and Tsujii, J., editors, Proceedings of the 23rd Workshop on Biomedical Natural Language ...

    2024

  2. [10]

    Labrak, Y ., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., and Dufour, R. (2024). Biomistral: A collection of open-source pretrained large language models for medical domains. arXiv preprint arXiv:2402.10373

    2024 arXiv

  3. [11]

    J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., and Alsentzer, E

    Lehman, E., Hernandez, E., Mahajan, D., Wulff, J., Smith, M. J., Ziegler, Z., Nadler, D., Szolovits, P., Johnson, A., and Alsentzer, E. (2023). Do we still need clinical language models? In Conference on health, inference, and learning, pages 578–597. PMLR

    2023

  4. [12]

    Y ., Deng, Y ., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V ., Dziri, N., Bras, R

    Lin, B. Y ., Deng, Y ., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V ., Dziri, N., Bras, R. L., and Choi, Y . (2024). Wildbench: Benchmarking llms with challenging tasks from real users in the wild. 10

    2024

  5. [13]

    Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    2021 arXiv

  6. [14]

    Manes, I., Ronn, N., Cohen, D., Ilan Ber, R., Horowitz-Kugler, Z., and Stanovsky, G. (2024). K-QA: A real-world medical Q&A benchmark. In Demner-Fushman, D., Ananiadou, S., Miwa, M., Roberts, K., and Tsujii, J., editors, Proceedings of the 23rd Workshop on Biomedical Natu- ral...

    2024

  7. [15]

    Nguyen, V ., Karimi, S., Rybinski, M., and Xing, Z. (2023). MedRedQA for medical consumer question answering: Dataset, tasks, and neural baselines. In Park, J. C., Arase, Y ., Hu, B., Lu, W., Wijaya, D., Purwarianti, A., and Krisnadhi, A. A., editors, Proceedings of the 13th I...

    2023

  8. [16]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. (2022). Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Flores, G., Chen, G. H., Pollard, T., Ho, J. C., and Naumann, T., editors,Proceedings of the Conference on Health, Inferen...

    2022

  9. [17]

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V ., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V ., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Jo...

    2011

  10. [18]

    Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al. (2024). Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416

    2024 arXiv

  11. [19]

    Shi, X., Liu, Z., Du, L., Wang, Y ., Wang, H., Guo, Y ., Ruan, T., Xu, J., Zhang, X., and Zhang, S. (2024). Medical dialogue system: A survey of categories, methods, evaluation and challenges. In Ku, L.-W., Martins, A., and Srikumar, V ., editors,Findings of the Association fo...

    2024

  12. [20]

    S., Wei, J., Chung, H

    Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023a). Large language models encode clinical knowledge. Nature, 620(7972):172–180

    2023

  13. [21]

    Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole- Lewis, H., Neal, D., et al. (2023b). Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617

    2023 arXiv

  14. [22]

    Sviridova, E., Yeginbergen, A., Estarrona, A., Cabrio, E., Villata, S., and Agerri, R. (2024). CasiMedicos-arg: A medical question answering dataset annotated with explanatory argumentative structures. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N., editors, Proceedings of t...

    2024

  15. [23]

    Wang, H., Zhao, Y ., Wu, X., and Zheng, Y . (2024). imapScore: Medical fact evaluation made easy. In Ku, L.-W., Martins, A., and Srikumar, V ., editors, Findings of the Association for Computational Linguistics ACL 2024, pages 10242–10257, Bangkok, Thailand and virtual meeting...

    2024

  16. [24]

    Xiong, G., Jin, Q., Lu, Z., and Zhang, A. (2024). Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178

    2024 arXiv

  17. [25]

    Yang, R., Liu, H., Marrese-Taylor, E., Zeng, Q., Ke, Y ., Li, W., Cheng, L., Chen, Q., Caverlee, J., Matsuo, Y ., and Li, I. (2024). KG-rank: Enhancing large language models for medical QA with knowledge graphs and ranking techniques. In Demner-Fushman, D., Ananiadou, S., Miwa...

    2024

  18. [26]

    Zhang, X., Tian, C., Yang, X., Chen, L., Li, Z., and Petzold, L. R. (2023). Alpacare: Instruction- tuned large language models for medical application. arXiv preprint arXiv:2310.14558. 11

    2023 arXiv

  19. [27]

    Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2024). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    Zhu, M., Ahuja, A., Juan, D.-C., Wei, W., and Reddy, C. K. (2020). Question answering with long multiple-span answers. In Cohn, T., He, Y ., and Liu, Y ., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3840–3849, Online. Association for C...

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.