{"id":"a1ad9e87-c707-40c5-ad65-9473393466bd","arxiv_id":"2505.04388","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Aloe Beta, a family of open-weights health LLMs built from Llama 3.1 and Qwen 2.5, matches or exceeds closed medical models on MCQA benchmarks while improving safety via DPO.","lead":"Researchers trained four open-source medical chatbots, Aloe Beta, and released the models and data with a permissive license. The largest model matches or beats GPT-4 and MedPaLM 2 on standard medical multiple-choice exams, and its safety training lowers jailbreak success on big models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MCQA parity claim rests on a decontamination step of unverified efficacy; the synthetic training data is generated directly from the same benchmark training sets used for evaluation.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the decontamination step in §4.1.1 is the sole firewall between the evaluation test sets and the training data, and its efficacy is not demonstrated. The paper's own synthetic data section (§4.1.2) confirms that the training mixture intentionally includes CoT-expanded versions of the same benchmark training sets used in the Table 9 evaluation, making contamination not a hypothetical risk but a concrete one. The released models and datasets are the right kind of evidence—they make this concern directly testable, which is a strength of the paper. Other weaknesses, such as the non-reproduced closed-model numbers and the null human preference result, are secondary because they do not single-handedly determine the central 'competitive with private alternatives' claim; the contamination risk does, since even a small overlap can inflate MCQA accuracy by several points. The proposed concrete test is feasible because all artifacts are public, and it would either confirm contamination (moving the verdict toward rejection or at least stronger conditionality) or clear the concern and strengthen the acceptance case. Since the reader already recommends CONDITIONAL, this stress-test supports that verdict rather than changing it.","tokens_in":45770,"tokens_out":3804,"duration_ms":36795,"concrete_test":"Download the released Aloe Beta SFT datasets (HPAI-BSC/aloe-beta-datasets), specifically medqa_cot_llama31, medmcqa_cot_llama31, pubmedqa_cot_llama31, HeadQA_llama31_cot, and MMLU_medical_llama31_cot, plus the Medprompt RAG database. For each sample, extract the embedded question stem and compare it against every question in the Table 9 test splits (MedQA test, MedMCQA test, PubMedQA test, HeadQA test, MMLU medical test) using exact match and an embedding-similarity threshold calibrated on known duplicates (e.g., ROUGE-L greater than 0.85 or cosine greater than 0.95 with a medical sentence encoder). Report the number of overlapping items. If any are found, retrain the 7B model on the deduplicated set or re-evaluate the 72B model on the non-overlapping test subset and report the accuracy change; a drop of more than 1–2 points would materially weaken the Table 9 parity claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Qwen2.5-Aloe-Beta-72B is competitive with GPT-4 and MedPalm-2 on medical MCQA (Table 9, §7.1)—requires that the test sets of MedQA, MedMCQA, PubMedQA, HeadQA, and MMLU (medical) are absent from the SFT mixture. Yet §4.1.2 constructs 419,938 synthetic CoT samples by prompting Llama-3.1-70B-Instruct with the training splits of exactly those benchmarks (MedQA 10,178; MedMCQA 182,736; PubMedQA 210,257; HeadQA 6,600; MMLU-medical 4,321). The only barrier is the LLM-judge decontamination in §4.1.1, which uses Nous-Hermes-2-Yi-34B to 'remove all instructions that it flags.' The paper reports no judge prompt, no similarity threshold, no number of removed samples, and no validation of the judge's recall. Because the synthetic CoT conversion paraphrases questions and expands answers into long reasoning chains, near-duplicates of test items may evade the judge; the released data permit a direct check. If overlap survives, the 82.54 average for Aloe-Beta-72B in Table 9 is inflated and the 'competitive with closed models' conclusion weakens correspondingly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Aloe Beta is a family of four healthcare-specialized LLMs (Llama-3.1-8B/70B and Qwen2.5-7B/72B) built with a publicly documented recipe: curation of roughly 2M SFT samples (about 60% medical instructions from public datasets, 420K synthetic chain-of-thought samples generated from the training splits of MedQA, MedMCQA, PubMedQA, HeadQA, MMLU-medical, and PolyMed, plus 400K general-domain samples), four-epoch SFT, DARE-TIES merging with the official instruct checkpoints, two-stage DPO alignment with a final red-teaming-only stage, and a Medprompt-style retrieval-augmented inference pipeline. Evaluation covers MCQA (Table 9), open-ended tasks, human expert pairwise preferences, and attack-success-rate safety testing (S-Eval). The paper claims that Qwen2.5-Aloe-Beta-72B is the highest-performing open model examined (82.54 MCQA average), that the largest Aloe models are competitive with GPT-4 and MedPalm-2 on medical MCQA, that DPO alignment substantially improves jailbreak resistance in large models, and that all models, datasets, code, and a healthcare-specific risk assessment are released openly.","tokens_in":46049,"tokens_out":17477,"duration_ms":147263,"significance":"If the central claims hold, this is a strong systems contribution to open healthcare LLMs. The complete release of weights, training data, and RAG components, combined with unusually detailed reporting of data provenance, hyperparameters, compute, and carbon cost, makes the recipe directly reusable. The Qwen-based Aloe models show consistent MCQA gains over their instruct baselines (Table 9: +3.4 for 7B, +2.2 for 72B); the large-model safety results are large and internally consistent (average ASR below 9% in Tables 13-14); and the body text is candid about non-reproduced closed-model numbers and statistically non-significant human preferences. The significance is contingent, however: the MCQA parity claim rests on a decontamination step that is neither specified nor validated (Major Comment 1), the abstract overstates the human-evaluation and closed-model-comparison findings (Major Comments 2-3), and the data-count tables contain internal inconsistencies. With the contamination audit performed and the claims tempered accordingly, this would be a valuable and citable recipe paper.","major_comments":[{"comment":"The central claim that Aloe-Beta-72B is 'competitive with the best private alternatives' (Abstract, §7.1, §8) rests on the assumption that the Table 9 test sets are absent from training. However, §4.1.2 constructs 419,938 synthetic CoT samples by prompting Llama-3.1-70B-Instruct with the training splits of exactly the MCQA task families used in Table 9: MedQA (10,178), MedMCQA (182,736), PubMedQA (210,257), HeadQA (6,600), and MMLU-medical (4,321). The only barrier is the decontamination step in §4.1.1, described in a single sentence: Nous-Hermes-2-Yi-34B is used as a judge and 'all instructions that it flags' are removed. No judge prompt, no threshold, no count of removed samples, and no validation of judge recall are reported; moreover, the judge is a 34B open model while the generator is a 70B instruct model, so its recall on paraphrased items is not assured. Because the conversion to CoT paraphrases each question and appends a long reasoning chain, near-duplicates of test items can plausibly evade the judge, and the same concern applies to the RAG database of §6, which is built from the MedQA and MedMCQA training sets. Since the datasets are released, a direct overlap check is feasible. I request: (a) the full decontamination configuration and per-source removal counts; (b) a recall test in which known test items are spiked into the pipeline; and (c) confirmation of the headline gains and parity on a benchmark not derived from the selected training sources (e.g., the CareQA 2024 items already used in Table 9, or another recent exam set). Until then, the parity claim is not established.","section":"§4.1.1, §4.1.2, Table 9"},{"comment":"The abstract claims that the Aloe models 'are often preferred by healthcare professionals,' but §7.3 reports binomial tests in which no pairwise battle reached statistical significance (p<0.05), and the text concludes from 695 expert responses that the choice of one model over the other 'is relegated to the personal preference of the doctors, with limited statistical relevance.' The abstract should be revised to state that experts found the Aloe models comparable to, rather than preferred over, the instruct baselines.","section":"§7.3, Abstract"},{"comment":"The prose in §7.1 ('Aloe-Beta-72B matches or outperforms GPT-4 and MedPalm-2'; 'the performance gap between open and closed models seems minimal') compares 0-shot Aloe accuracy with closed-model numbers that the table footnote correctly says are 'not reproduced' and that were obtained under the protocols of the Medprompt and Med-PaLM papers. The footnote discloses the non-reproduction, but the surrounding text does not carry the caveat into the abstract, where 'competitive with the best private alternatives' is asserted without protocol qualification. Please either evaluate the closed models under the Aloe protocol, or state explicitly in the abstract and §8 that the comparison is against published numbers obtained with different prompting and ensembling setups.","section":"Table 9, §7.1"}],"minor_comments":[{"comment":"The column for Aloe-Beta-72B lists the base model as 'Llama-3.1-72B', which contradicts the text and all other tables identifying it as Qwen2.5-72B; no Llama-3.1-72B model exists.","section":"Table 5"},{"comment":"The 70B entry for 'Total training time' is 0.98, but the sum of the first-stage (12.76, Table 6) and second-stage (0.98, Table 7) training times is 13.74, which is the value actually used in Appendix D Eq. (D12).","section":"Table 8"},{"comment":"The computation 22.68 kW × 15.30 h × 0.158 kg/kWh equals 54.8 kg CO2, not the stated 61.42 kg; the stated value corresponds to 17.14 h, i.e., the 8B training time. The 7B carbon figures should be recomputed consistently with Table 5.","section":"Appendix D, Eq. (D6)"},{"comment":"Dataset counts and percentages do not reconcile: Table 3 totals 1,622,404 samples while Table A1 totals 1,603,732; the synthetic CoT MCQA share is given as 31.54% in the text of §4.1 but 31.17% in Table 3; and the '1.2M medical + 225K synthetic + 420K synthetic' breakdown in §4.1 does not obviously match either table. Please reconcile the data card.","section":"§4.1, Tables 3 and A1"},{"comment":"The 'Avg.' row averages across columns that include both MultiMedQA and its constituent benchmarks (MedQA, MedMCQA, MMLU, and PubMedQA), so those benchmarks are counted twice in the reported average; please report the composition of the MultiMedQA column and either recompute the average over non-overlapping benchmarks or disclose the double counting.","section":"Table 9"},{"comment":"The Medprompt retrieval database (192,084 examples) is constructed from the training sets of MedQA and MedMCQA; please state explicitly whether this database was decontaminated against the corresponding test sets and how it relates to the synthetic SFT CoT data described in §4.1.2.","section":"§6"},{"comment":"The statement that Aloe Beta is 'significantly better in every aspect' than Aloe Alpha is not supported by any comparison in this manuscript; a brief quantitative comparison against Aloe Alpha would substantiate the claim.","section":"Declarations"}],"recommendation":"major_revision","confidential_remarks":"This is a systems/recipe paper whose primary value is the complete open release of models, data, and code with a transparent training description; its novelty over the authors' own Aloe Alpha is modest, but the release is genuinely useful to the community. The main risk is the contamination question in Major Comment 1: the audit is feasible with the already-released data, and a clean result would largely restore the headline claims. I would also encourage the editors to ask for tempering of the promotional abstract language ('sets a new standard', 'often preferred') regardless of the technical outcome, since the body of the paper itself supports weaker claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Aloe Beta paper is a solid engineering contribution. What is genuinely new: four released models (two sizes each on Llama 3.1 and Qwen 2.5), a 2M-sample curated dataset including synthetic CoT expansions, a RAG pipeline, and a four-part evaluation with safety results. The training details, compute tracking, and risk assessment are unusually transparent, and all artifacts are public. Credit is due for releasing the data and weights, which makes independent verification possible.\n\nThe within-open-model gains look credible. Qwen2.5-Aloe-Beta-72B improves over its instruct baseline by 2.2 points on the MCQA average; the 7B Qwen gains 3.4. Safety results for the large models are strong and internally consistent: attack success rates below 9% on S-Eval across attack styles and topics, with the 70B and 72B clearly better than their instruct bases. Those claims I would trust.\n\nThe load-bearing soft spot is the one the stress-test note flags. The synthetic SFT data in §4.1.2 is generated from the training splits of MedQA, MedMCQA, PubMedQA, HeadQA, and MMLU-medical, and evaluation in Table 9 uses test sets from those same benchmarks. The only barrier is the LLM-judge decontamination in §4.1.1, which reports no judge prompt, no similarity threshold, no number of removed samples, and no validation of recall. Because the synthetic conversion rephrases and expands questions into long CoT answers, near-duplicates could survive the judge easily. If even a few percent of test questions leaked, the 82.54 average for the 72B and the \"competitive with GPT-4/MedPalm-2\" conclusion would be inflated. The authors' own decision to add CareQA (2024 exams) as a contamination-resistant benchmark suggests they knew the older sets are risky. This is fixable: the released data permit a direct overlap check, and the paper should report it.\n\nSmaller issues: the human evaluation shows no statistically significant preference in any pairwise battle, yet the abstract says \"often preferred by healthcare professionals\" — that overstates a null result. The open-ended evaluation is noisy and the authors themselves note the metrics are unreliable. The \"sets a new standard\" phrasing in the abstract and conclusion outruns the evidence.\n\nThe math, code, and citation pattern look fine. No invented entities; self-citations are to their own prior Aloe Alpha and RAG work, which is normal. The carbon accounting is clearly documented.\n\nWho is this for? Anyone building or evaluating open healthcare LLMs, or deciding whether closed APIs are necessary for medical MCQA. It deserves a serious referee, not a desk reject. The referee should ask for decontamination evidence, a softened conclusion, and acknowledgment that the human preference claim is not supported. After those revisions, the engineering value stands.","headline":"A solid open-model engineering report with real artifacts, but the headline MCQA parity claim leans on an unvalidated decontamination step that needs to be checked before the numbers are taken at face value.","tokens_in":46659,"tokens_out":1945,"would_cite":false,"duration_ms":21870,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Aloe Beta recipe produces open healthcare LLMs competitive with private models on medical MCQA benchmarks.","keywords":["healthcare LLMs","open-source models","chain-of-thought","direct preference optimization","model merging","retrieval-augmented generation","jailbreak safety","medical benchmarks"],"falsifier":"Compare the released Aloe Beta training corpus against the official test splits of MedQA, MedMCQA, PubMedQA, MMLU, and CareQA using exact-match, paraphrase, and embedding-similarity methods, and check whether any retained training example matches a test question. A nonzero surviving match rate would inflate the reported parity numbers; zero matches under all methods would support the claim.","tokens_in":45565,"feed_emoji":"🩺","tokens_out":8440,"duration_ms":81519,"temperature":0.7,"pith_summary":"This paper tries to establish that a fully open, permissively licensed recipe for healthcare LLMs can reach competitive performance with closed systems. The authors build four Aloe Beta models on Llama 3.1 and Qwen 2.5 base models, expanding public medical multiple-choice data with synthetic chain-of-thought reasoning, merging with the official instruct versions, and aligning with preference optimization plus a jailbreak-focused red-teaming stage. They report that the 72B model is the highest-performing open model on the medical MCQA benchmarks they evaluate, that its accuracy matches or exceeds reported numbers for private systems, and that the large models resist jailbreaking far better than their instruct baselines. If correct, the practical consequence is that institutions can deploy, inspect, and adapt healthcare LLMs of near-top accuracy without relying on closed APIs. The paper frames its four-way evaluation (close-ended, open-ended, expert preference, and safety) as part of the contribution, and it notes that the closed-model numbers were not reproduced by the authors, so the parity claim rests on the MCQA setting.","feed_headline":"Open medical LLM matches private models on exam benchmarks","feed_subtitle":"A 72B open model with synthetic reasoning data and safety training scores at the level of closed systems.","key_machinery":"The mechanism is the full recipe, not any single model. The supervised fine-tuning corpus contains 2M samples, including about 420K synthetic chain-of-thought solutions generated by Llama-3.1-70B-Instruct from the training splits of MedQA, MedMCQA, PubMedQA, HeadQA, MMLU auxiliary, and PolyMed, converting simple letter-answer data into step-by-step reasoning. The training then uses DARE-TIES merging, which drops low-magnitude parameters and rescales the rest, to combine the Aloe SFT weights with the official instruct weights, and a two-stage DPO where the final stage is trained only on a curated jailbreak dataset built from red-teaming templates to harden safety without eroding accuracy. At inference, a Medprompt-style retrieval-augmented pipeline retrieves five chain-of-thought few-shot examples from a 192K-example database generated from the same benchmark training sets, then pools twenty self-consistency samples with choice shuffling.","core_discovery":"The central claim is that an open training and evaluation recipe can make an open 72B model the best open performer on five medical multiple-choice benchmarks and competitive with the best private alternatives reported in the literature. In the paper's own comparison, Qwen2.5-Aloe-Beta-72B averages 82.54 accuracy across MultiMedQA, MedMCQA, MedQA, MMLU, and CareQA, and on the shared benchmark subsets it matches or outperforms the numbers reported for GPT-4 and MedPalm-2; adding a Medprompt-style retrieval pipeline raises it to 85.68 and closes most of the remaining gap. A second reported result is safety: the two-stage DPO alignment reduces the average attack success rate on the S-Eval jailbreak benchmark to below 9% for both large Aloe models, down from 14-15% in the base instruct models. The human evaluation, based on 49 physicians judging 695 pairs of answers to Reddit health questions, showed no statistically significant preference between the Aloe models and their instruct bases, which the authors read as evidence that current generalist LLMs can give reliable advice on simple primary-care questions.","pith_inferences":["The open-ended evaluation results are noisy and disagree across metrics, so a natural extension beyond the paper is to anchor open-ended assessment in clinician-rated outcome measures rather than n-gram or perplexity scores.","Because the synthetic chain-of-thought data are generated from the same benchmarks used for evaluation, the cleanest test of the parity claim is to rerun training with the official test sets excluded under stricter near-duplicate removal than the reported LLM-judge decontamination.","The same merge-plus-DPO sequence could transfer to other safety-critical domains such as finance or legal advice, but that transfer is not demonstrated in this paper.","The retrieval database is built from benchmark training sets, so if retrieval ever returns a near-duplicate of a test question, the in-context-learning scores could be optimistic; the paper does not quantify this risk."],"forward_implications":["Open healthcare LLMs can reach benchmark parity with closed systems while releasing weights, data, and training details, shifting deployment options for hospitals and researchers.","The two-stage DPO with red-teaming data can cut jailbreak success rates dramatically for large models (from an average above 14% to below 9%) without lowering measured MCQA accuracy in the reported results.","Adding retrieval-augmented prompting boosts small Aloe models by roughly 6 to 9 accuracy points and large models by roughly 3 to 4 points, so one model family can serve both lightweight and high-accuracy deployments.","The reported field-by-field accuracy table, with the 72B model achieving top performance in 13 of 17 medical specialties, can serve as a model-selection guide for specialty-specific applications.","The recipe is stated as transferable to future strong open base models, so the same data, merging, and alignment stages could be reapplied when newer open models appear."],"supporting_citations":[{"why":"Supplies the Medprompt prompting method and the closed-model benchmark numbers used for the parity comparison.","marker":"[1]"},{"why":"Supplies the reported MedPalm-2 results that serve as the private-system baseline.","marker":"[2]"},{"why":"Provides the PubMedQA training split expanded into about 210K synthetic chain-of-thought examples.","marker":"[34]"},{"why":"Provides the MedQA training set used to generate synthetic chain-of-thought answers.","marker":"[35]"},{"why":"Supplies the UltraMedical preference dataset used in the first DPO stage for medical response alignment.","marker":"[39]"},{"why":"Supplies the jailbreak template set used to craft the red-teaming preference data.","marker":"[45]"},{"why":"Introduces the DARE-TIES merging method used to combine SFT weights with the official instruct versions.","marker":"[51]"},{"why":"Introduces Direct Preference Optimization, the alignment algorithm used in both preference stages.","marker":"[58]"},{"why":"Provides the Pareto-optimized context-retrieval configuration used to build the Medprompt-style RAG pipeline.","marker":"[63]"},{"why":"Provides the S-Eval safety benchmark whose attack success rates underlie the reported jailbreak-resistance claims.","marker":"[80]"}],"fun_headline_variants":["Open Aloe models rival private LLMs on medical exams","Aloe recipe: open 72B LLM matches GPT-4 on healthcare benchmarks","Open-source Aloe uses synthetic reasoning and DPO to rival closed LLMs","Aloe Family: open medical LLM with safety training matches private models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on the assumption that the decontamination step, an LLM judge that flags and removes instructions, fully eliminated every overlapping or near-duplicate question between the benchmark training sets used for synthetic chain-of-thought generation and the test sets used in the final MCQA evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Open Aloe models rival private LLMs on medical exams","Aloe recipe: open 72B LLM matches GPT-4 on healthcare benchmarks","Open-source Aloe uses synthetic reasoning and DPO to rival closed LLMs","Aloe Family: open medical LLM with safety training matches private models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3939,"prompt_tokens":1094,"completion_tokens":2845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":2765}},"tokens_in":710,"tokens_out":2845,"duration_ms":21900,"temperature":1.0,"reasoning_tokens":2765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:29:57.667412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the released Aloe Beta training corpus against the official test splits of MedQA, MedMCQA, PubMedQA, MMLU, and CareQA using exact-match, paraphrase, and embedding-similarity methods, and check whether any retained training example matches a test question. A nonzero surviving match rate would inflate the reported parity numbers; zero matches under all methods would support the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Direct Preference Optimization, the alignment algorithm used in both preference stages."}],"review_version":1}