{"id":"65b1043c-5ab6-48d6-865e-d1d43321b872","arxiv_id":"2411.09834","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new public benchmark of 1,077 consumer medical questions with doctor pairwise annotations, plus a preliminary finding that open Llama 3.1 405B outperforms GPT-4o on all evaluated criteria.","lead":"This paper builds a public benchmark of real consumer medical questions with answers scored by doctors, and uses it to compare several AI chatbots. It reports that an open model, Llama 3.1 405B, beat OpenAI's GPT-4o on every doctor-rated criterion, and that a medical-specialized model did not beat its general-purpose base.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Doctor-pair labels for the GPT-4o vs. Llama-3.1 comparison are near chance on several criteria, so the headline ranking claim is not yet supported; the released benchmark resource can still stand.","rationale":"The reader's weakest_assumption correctly identifies near-chance inter-annotator agreement as the fragile premise. I agree that this is the single most load-bearing concern: the headline result that an open model beats a closed model across all aspects depends on doctor judgments that, for several criteria in the decisive batch, have observed agreement at or below chance. This is not merely a statistical nicety; it means the labels used to rank models are uninformative for those criteria, and without adjudication or a reliability filter the conclusion cannot be accepted as evidence. The paper is transparent about the low agreement, and the benchmark resource itself is a valuable contribution, so a conditional acceptance with the requirement to address reliability and significance is the appropriate outcome. Secondary issues, such as the discrepancy between 1,077 questions in the dataset and 400 questions in the 4 annotated batches, and the 'long-form' framing despite a three-sentence answer cap, are also worth noting but are less central than the label reliability problem. I would keep the reader's CONDITIONAL verdict unchanged, with the condition being that the ranking claims must be re-derived using only reliable annotations or appropriately qualified.","tokens_in":14810,"tokens_out":2888,"duration_ms":31495,"concrete_test":"Compute Cohen's kappa (or Gwet's AC1) with confidence intervals for each criterion in Batch 3, then recompute the Llama-3.1-vs-GPT-4o preference using only items on which both doctors agree, and run a two-sided sign test on the doctor majority. If any kappa confidence interval includes 0 for Helpfulness, Reasoning, or Efficiency, or if the Llama advantage disappears or becomes non-significant on the agreed subset, the 'outperforms across all aspects' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that Llama-3.1-405B outperforms GPT-4o across all evaluation aspects (Section 4, abstract). This claim rests entirely on the two-doctors-per-batch pairwise annotations. For the critical Batch 3 comparison, Table 3 reports observed agreement of P=0.23 vs chance C=0.19 for Helpfulness, P=0.17 vs C=0.16 for Reasoning, and P=0.26 vs C=0.25 for Efficiency. For Batch 4, Efficiency is P=0.11 vs C=0.11, i.e., observed agreement is exactly at chance. These numbers mean the annotator labels for these criteria carry essentially no signal beyond random coin flips; the 'majority' preference between models can be driven by one annotator's arbitrary choices. The paper itself acknowledges the low agreement in Section 4.1, yet still draws the across-all-aspects conclusion without any reliability threshold, kappa statistic, or significance test. This is the most load-bearing weakness because if the labels are noise, the model-ranking conclusion does not follow, even though the benchmark data and annotation release may still be useful to the community.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a publicly available benchmark for long-form consumer medical question answering, built from 1,077 real-world queries collected on the authors' own platform. The benchmark includes pairwise evaluations of LLM-generated answers by two medical doctors per batch across six criteria (correctness, helpfulness, harmfulness, reasoning, efficiency, bias), as well as LLM-as-a-judge evaluations using GPT-4o and Claude-3.5-Sonnet. The paper reports that the open model Llama-3.1-405B outperforms the closed model GPT-4o across all evaluated aspects, and that open medical models do not necessarily improve over their general-purpose base models. The authors release the questions, answers, and doctor annotations on GitHub.","tokens_in":15100,"tokens_out":3023,"duration_ms":30264,"significance":"If validated, this benchmark would be a useful community resource: it provides a real-world consumer medical QA dataset, releases expert doctor annotations that are otherwise unavailable, and reports a detailed LLM-as-a-judge robustness analysis. The paper is transparent about its annotation pipeline and reports inter-annotator agreement tables rather than hiding disagreements. However, the central comparative claim (Llama-3.1 beats GPT-4o 'across all aspects') rests on doctor annotations whose observed agreement is at or near chance for several key criteria, and the 'long-form' characterization is undercut by the three-sentence answer cap. The resource itself is valuable, but the strength of the headline conclusions requires revision.","major_comments":[{"comment":"The headline conclusion that Llama-3.1-405B outperforms GPT-4o across all aspects is not supported by the reported annotator agreement. In Batch 3 (the GPT-4o vs. Llama-3.1 comparison), observed agreement is near chance for Helpfulness (P=0.23 vs. C=0.19), Reasoning (P=0.17 vs. C=0.16), and Efficiency (P=0.26 vs. C=0.25); in Batch 4, Efficiency is exactly at chance (P=0.11 vs. C=0.11). When annotator labels carry no signal beyond chance, the majority vote used to rank models can be driven by one annotator's arbitrary choices. The paper acknowledges the low agreement in Section 4.1 but still draws the 'across all aspects' conclusion without reporting a chance-corrected agreement measure (e.g., Cohen's kappa), a significance test, or a per-criterion reliability threshold. Please either provide such reliability evidence, restrict the comparative claims to criteria with agreement significantly above chance, or present the model-ranking results with appropriate confidence intervals and caveats.","section":"Section 4.1, Table 3"},{"comment":"The LLM-as-a-judge analysis uses GPT-4o both as an evaluated candidate model and as one of the two judge models. The paper reports 71% observed agreement and 55% Cohen's kappa between GPT-4o and Claude judges, but this only measures agreement between two LLM judges; it does not calibrate LLM judgments against the human doctor labels. The statement in Section 4 that there is 'general agreement across all batches and criteria' is not quantified against the human labels. Please report agreement (e.g., percentage and kappa) between each LLM judge's votes and the majority human votes, and separately for criteria where human agreement is itself above chance. Without such calibration, the LLM-judge results cannot be used to corroborate the human-based model rankings.","section":"Section 3.4"},{"comment":"The benchmark is described as a 'long-form' medical QA benchmark, but the answer-generation prompt in Figure 10 instructs models to 'Complete answer in three sentences max.' Three-sentence answers are not long-form by typical definitions, and the paper does not justify why this operationalization captures long-form medical responses. This affects the construct validity of the benchmark, since the evaluation criteria (e.g., efficiency, reasoning) may behave differently for three-sentence versus genuinely long-form answers. Please either revise the benchmark description, justify the three-sentence cap as an adequate operationalization of long-form answers, or adopt a more generous answer-length constraint.","section":"Section 2.5, Figure 10"}],"minor_comments":[{"comment":"The manuscript contains several typos and duplicated figure captions: Figures 9-12 in the appendix are all labeled 'Table 3: Caption' or 'Table 4: Caption' in the provided text, and 'efficiency' is misspelled as 'Efficiency' in the Figure 12 prompt template. Please correct these issues.","section":"General"},{"comment":"The quality-check threshold of 85% for SequenceMatcher similarity is described as determined 'after manually inspecting some samples,' but no detail is given about how many samples were inspected or how the threshold was validated. Please provide this detail or state that the threshold is an arbitrary choice.","section":"Section 2.1.2"},{"comment":"The DBSCAN threshold and epsilon values (threshold=0.75, eps=0.25) are selected by manual inspection of clusters, but no quantitative evaluation of cluster quality is provided. Please clarify how the manual inspection was conducted and whether alternative thresholds materially change the resulting 1,077 questions.","section":"Section 2.2"},{"comment":"The difficulty-level annotation is performed by GPT-4, but the paper does not report any validation of these labels against human judgment, despite the earlier human-validation exercise for medical-question detection in Section 2.1.1. Please add a validation step or explicitly note that difficulty labels are LLM-generated and unvalidated.","section":"Section 2.4"},{"comment":"The human evaluation used only three medical doctors total, with two per batch, and the paper does not specify whether the same two doctors annotated all batches or whether the pairings changed. Please clarify the annotator assignment and whether any annotator effects (e.g., differing strictness) were examined.","section":"Section 3.3"},{"comment":"The inter-dataset similarity analysis compares a random sample of the same size as the benchmark from each other dataset, but the paper does not report the standard deviation or confidence intervals for the similarity scores. Please include these to support the claim of 'low overlap.'","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource itself is a useful contribution and the authors are transparent about their annotation process, so I do not recommend rejection. However, the current manuscript overstates the model-comparison findings given the near-chance inter-annotator agreement on several load-bearing criteria, and the 'long-form' label needs reconciliation with the three-sentence answer cap. These issues are fixable with a revised analysis and a more measured set of claims, so major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen,\n\nTwo things to know about arXiv:2411.09834: the released benchmark—1,077 real consumer medical questions with doctor annotations across six criteria—is a genuinely useful public resource, and the paper is refreshingly transparent about how noisy its own labels are. But that transparency undermines the paper's central claim. The conclusion that Llama-3.1-405B outperforms GPT-4o 'across all aspects' rests on pairwise doctor judgments where observed agreement is at or near chance for several criteria in Batches 3 and 4 (e.g., Batch 4 Efficiency P=0.11 vs C=0.11; Batch 3 Reasoning P=0.17 vs C=0.16). When annotators don't agree beyond coin flips, the majority vote is essentially one person's opinion. No kappa, no significance test, no reliability threshold. The ranking claim doesn't follow.\n\nWhat the paper does well: the dataset is new, real, and public. The authors collected actual consumer queries from Lavita, deduplicated them, checked novelty against MedRedQA, HealthSearchQA, and MASH-QA, and released the doctor annotations. The LLM-as-a-judge robustness checks (positional bias, multiple runs, conservative tie-breaking) are careful. And they openly publish the agreement table and disagreement counts—most benchmark papers wouldn't. Credit for that.\n\nThe soft spots beyond the agreement issue: the 'long-form' framing clashes with the 'three sentences max' prompt used to generate answers. And GPT-4o serves as both an evaluated model and a judge; that's a mild circularity, not a fatal one. The paper also lacks any statistical test for the model comparisons, so even in batches with decent agreement, we don't know if the margins are meaningful.\n\nBottom line: this is a benchmark resource worth having, and the authors are honest about its limitations. But the headline comparative result is not yet supported. The right outcome is a serious peer review that demands reliability analysis, significance testing, and a toned-down conclusion. I'd bring it to reading group—it's a useful case study in why inter-annotator agreement matters for evaluation benchmarks.","headline":"Worth reading for the dataset and honest accounting, but the headline ranking claim is undercut by near-chance annotator agreement.","tokens_in":15633,"tokens_out":2016,"would_cite":true,"duration_ms":19406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a publicly available benchmark of 1,077 real-world consumer medical questions with doctor-annotated pairwise evaluations of long-form answers, and reports preliminary results in which the open Llama-3.1-405B-Instruct…","keywords":["long-form medical question answering","consumer health questions","doctor annotation","LLM-as-a-judge","open LLMs","benchmark","pairwise comparison","GPT-4o vs Llama"],"falsifier":"Have a different pair of board-certified physicians independently re-annotate all 400 questions from batches 3 and 4 under the same scheme, then compare aggregate pairwise winners; if the new panel's winner differs on correctness or helpfulness, or if the new pair's observed agreement is within a few points of chance, the claim that Llama-3.1 outperforms GPT-4o across all aspects fails. A concrete statistic: Cohen's kappa below 0.1 on correctness or efficiency for the new panel would falsify the labeling reliability the ranking depends on.","tokens_in":14632,"feed_emoji":"🩺","tokens_out":7339,"duration_ms":68454,"temperature":0.7,"pith_summary":"Long-form medical question answering has lacked a reusable, doctor-annotated benchmark built from questions real consumers actually ask; existing medical QA benchmarks mostly use multiple-choice items or automatic metrics, and prior human-evaluation efforts kept their labels closed. This paper fills that gap by releasing 1,077 semantically distinct consumer health questions together with pairwise judgments from medical doctors on six criteria: correctness, helpfulness, harmfulness, reasoning, efficiency, and bias. Using those annotations, the authors report preliminary evidence that the open-weights Llama-3.1-405B-Instruct outperforms GPT-4o across all six criteria, that AlpaCare-13B beats BioMistral-7B but trails GPT-4, and that medically pretrained Meditron3-70B does not clearly beat its vanilla Llama-3.1-70B base. They also compare LLM-as-a-judge votes with the doctors' votes and find a real but incomplete alignment. If the benchmark is sound, the field gains an open resource for judging long-form medical answers and a concrete challenge to the assumption that closed commercial models lead in this domain.","feed_headline":"Open Llama-3.1 beats GPT-4o on doctor-rated medical answers","feed_subtitle":"A new public benchmark of 1,077 real consumer health questions backs open-source models against closed rivals in long-form medical QA.","key_machinery":"The mechanism is a pairwise long-form answer comparison protocol. Two medical doctors per batch judge which of two randomized responses is better on six criteria—correctness, helpfulness, harmfulness, reasoning, efficiency, and bias—with tie and 'neither' options available. The benchmark questions are produced by a pipeline that filters real user queries with GPT-4, manually cleans them, clusters embeddings with DBSCAN, and keeps one medoid per cluster to obtain 1,077 semantically distinct items. An LLM-as-a-judge variant runs the same pairwise comparisons with position-swapped runs and conservative tie/neither resolution, and its votes are compared with the doctors' votes.","core_discovery":"The central claim is that a public benchmark of 1,077 real-world consumer medical questions with pairwise doctor annotations can be built, and that on this benchmark the open-weights Llama-3.1-405B-Instruct beats GPT-4o on all six evaluated criteria in its batch. The smaller open medical model AlpaCare-13B outperforms BioMistral-7B but loses to GPT-4, while Meditron3-70B fails to improve over its vanilla Llama-3.1-70B base. The authors further claim that LLM-as-a-judge votes broadly agree with the doctors' votes but leave a clear alignment gap, and that the benchmark's questions are semantically distinct from those in MedRedQA, HealthSearchQA, and MASH-QA. The release includes the 1,077 questions, the generated model answers, and the doctor annotations.","pith_inferences":["If the near-chance doctor agreement in later batches is confirmed by a larger panel, the main ranking claim should be treated as hypothesis-generating; a coarser single-dimensional judgment might be more reliable than the six-way scale.","Because the authors used GPT-4 to filter and correct the consumer questions, any systematic bias in what GPT-4 counts as a medical question is baked into the benchmark; a human-only filtered subset would quantify that effect.","A natural next experiment is retrieval-augmented answer generation on these same 1,077 questions; the doctor labels would then show whether grounding in clinical sources changes the open-versus-closed model ordering.","The public labels could also be used to train an annotator-consensus model or to calibrate LLM judges, potentially narrowing the observed gap between LLM-as-a-judge votes and human labels on long-form medical answers."],"forward_implications":["If the benchmark is correct, open-weights models can match or beat closed commercial models on long-form consumer medical answers, which matters for privacy-sensitive deployments where patient data cannot be sent to third-party APIs.","The released doctor annotations let other teams reproduce the model rankings, reuse the questions, and build evaluation or training pipelines without repeating the costly annotation effort.","The evaluation protocol gives healthcare organizations a concrete template: pairwise judgment on six criteria, with randomized answer order and position-swap checks for LLM judges.","The Meditron3-70B result challenges the assumption that additional medical pretraining is necessary for a model to give strong clinical answers.","The low semantic overlap with MedRedQA, HealthSearchQA, and MASH-QA indicates the benchmark contributes genuinely new question content rather than re-labeling existing datasets."],"supporting_citations":[{"why":"Supplies the Meditron3-70B model whose comparison against its vanilla base supports the claim that medical pretraining did not improve over the base.","marker":"[1]"},{"why":"Describes the Llama 3.1 models that are the open-weights systems under evaluation.","marker":"[4]"},{"why":"Provides the DBSCAN clustering algorithm used to deduplicate questions and select medoid representatives, yielding 1,077 distinct items.","marker":"[5]"},{"why":"Describes BioMistral-7B, the smaller domain-specific model compared against AlpaCare in the smaller-model batch.","marker":"[10]"},{"why":"Supplies the MedRedQA dataset used in the inter-dataset similarity analysis to show benchmark novelty.","marker":"[15]"},{"why":"Provides the HealthSearchQA dataset and the Med-PaLM benchmark used as comparison points and evaluation inspiration.","marker":"[20]"},{"why":"Supplies the long-form answering prompt used for consistency with prior work and the Med-PaLM criteria structure the annotation scheme adapts.","marker":"[21]"},{"why":"Describes the AlpaCare model and the difficulty-level scoring system refined into the benchmark's three-level scheme.","marker":"[26]"},{"why":"Provides the LLM-as-a-judge prompting approach and the conservative tie/neither handling for position-swap disagreements.","marker":"[27]"},{"why":"Supplies the MASH-QA dataset used as a third comparison corpus in the inter-dataset similarity analysis.","marker":"[28]"}],"fun_headline_variants":["Open Llama-3.1 tops GPT-4o in doctor-rated medical QA","New benchmark: open model wins over closed rivals in medical answers","Public benchmark reveals open LLM outperforms GPT-4 on health Q&A","Llama-3.1 beats GPT-4o on 1,077 doctor-evaluated health answers","Open-weights Llama-3.1 edges out GPT-4o in clinical QA test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the assumption that two doctors' pairwise judgments on six fine-grained criteria are reliable enough to serve as ground truth; in later batches, observed agreement on several criteria is at or near chance, so if those labels are effectively noise, the reported model ordering does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Open Llama-3.1 tops GPT-4o in doctor-rated medical QA","New benchmark: open model wins over closed rivals in medical answers","Public benchmark reveals open LLM outperforms GPT-4 on health Q&A","Llama-3.1 beats GPT-4o on 1,077 doctor-evaluated health answers","Open-weights Llama-3.1 edges out GPT-4o in clinical QA test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3522,"prompt_tokens":929,"completion_tokens":2593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2482}},"tokens_in":545,"tokens_out":2593,"duration_ms":19419,"temperature":1.0,"reasoning_tokens":2482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:15:34.365053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a different pair of board-certified physicians independently re-annotate all 400 questions from batches 3 and 4 under the same scheme, then compare aggregate pairwise winners; if the new panel's winner differs on correctness or helpfulness, or if the new pair's observed agreement is within a few points of chance, the claim that Llama-3.1 outperforms GPT-4o across all aspects fails. A concrete statistic: Cohen's kappa below 0.1 on correctness or efficiency for the new panel would falsify the labeling reliability the ranking depends on.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DBSCAN clustering algorithm used to deduplicate questions and select medoid representatives, yielding 1,077 distinct items."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MedRedQA dataset used in the inter-dataset similarity analysis to show benchmark novelty."},{"cited_title":"S., Wei, J., Chung, H","cited_arxiv_id":null,"evidence_quote":"Provides the HealthSearchQA dataset and the Med-PaLM benchmark used as comparison points and evaluation inspiration."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LLM-as-a-judge prompting approach and the conservative tie/neither handling for position-swap disagreements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MASH-QA dataset used as a third comparison corpus in the inter-dataset similarity analysis."}],"review_version":1}