{"id":"a7c10950-5aaa-49fa-99bc-192fc3cbe543","arxiv_id":"2506.12307","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A pure reinforcement learning recipe with mixed rule-based rewards and length control improves Qwen2.5-based models across diverse medical QA formats.","lead":"This paper trains medical language models with pure reinforcement learning, using mixed rewards for multiple-choice, numeric, and free-text answers plus a penalty on reasoning length. It reports gains over larger baselines on several medical QA benchmarks and out-of-distribution tasks, though key training details and statistical rigor are incomplete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The open-text correctness reward is a thresholded lexical-overlap classifier with no validation against medical correctness; reported open-text gains may reflect reward overfitting rather than clinical quality.","rationale":"The reader's weakest assumption and my concern coincide. The paper's central claim is that pure RL with mixed binary rewards yields a unified medical QA model; the most fragile reward component is open-text, because correctness is operationalized as thresholded lexical overlap rather than medical validity. The paper itself provides evidence for such fragility in the BLEU failure case and in its Limitations statement. I agree with the CONDITIONAL verdict: MCQ and numeric results are more verifiable and survive, but the open-text component needs validation against expert judgments or at least a precision/recall analysis of the reward classifier. An additional reproducibility gap (the target length lgold used in Eq. 3 is never specified in Section 3.1) reinforces the need for revision but is secondary to the reward-validity issue. No change to the reader's verdict is needed.","tokens_in":16543,"tokens_out":6985,"duration_ms":85441,"concrete_test":"Sample 200 open-text items from the medical-o1-reasoning-SFT test set. Have three clinicians independently judge whether each model output is medically correct, blinded to references. Compare clinician majority labels against the paper's reward classifier (R_correct=+1 when the Mix score exceeds tau=0.5, plus the Rouge-L-only and EMS-only variants) using Cohen's kappa and balanced accuracy. Also compute the correlation between clinician correctness and Rouge-L/EMS scores, and inspect the subset of clinician-correct answers with Mix score below threshold. If agreement is near chance or a substantial fraction of correct paraphrases are scored -1, the reward proxy cannot support the open-text claims. A supplementary check: retrain Med-U1 on a small subset with clinician-scored rewards and see whether proxy-trained accuracy tracks clinician-rated correctness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 defines R_correct for open-ended generation as a thresholded average of Rouge-L and Exact Match Score (Eq. 2), with thresholds tau=40/50/70 reported in Section 3.1. This converts a continuous lexical-overlap score into a binary reward: a concise, semantically correct answer that does not share enough surface tokens with the reference receives -1, while a verbose wrong answer with partial overlap can receive +1. The paper documents the analogous failure for BLEU in the 'Failure Case of BLEU Reward' but never validates the Rouge-L+EMS classifier against human medical judgments. The Limitations section concedes that 'more fine-grained and clinically grounded reward objectives' are needed. Because the same metric family is used as both training reward and evaluation metric (Table 1 and Table 3), open-text improvements can be achieved by optimizing surface overlap without improving medical correctness. The load-bearing issue is therefore that the open-text component of the unified claim rests on an unvalidated proxy; if the proxy mis-scores common clinical paraphrases, the reported open-text gains are not evidence of medically correct reasoning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Med-U1 proposes a pure reinforcement-learning framework for unified medical question answering, trained with Qwen2.5-3B/7B-Instruct backbones via GRPO. The reward is a mixed rule-based signal covering three output formats: multiple-choice options, numeric values, and open-ended text, with an optional length-matching reward to control reasoning verbosity. The model is trained on five medical datasets and evaluated on in-distribution benchmarks plus the MMLU-Pro health subset. The paper reports consistent gains over SFT baselines and several open-source and proprietary models, along with analyses of reasoning length, reward design, and training dynamics.","tokens_in":16828,"tokens_out":6179,"duration_ms":67760,"significance":"If the results are reproducible and robust, Med-U1 provides a useful data point that pure RL with simple rule-based rewards can unify heterogeneous medical QA formats at 7B scale, while also offering a mechanism for length control. The work's strengths include a clear and simple reward formulation, a unified training recipe across three output types, an explicit OOD evaluation, and a stated commitment to open-sourcing the code. The claims would be substantially strengthened by multi-seed reporting, a validated open-text correctness reward, and a specification of how the length target is constructed.","major_comments":[{"comment":"The open-text correctness reward is a thresholded average of Rouge-L and EMS, and Section 3.1 reports thresholds of 40, 50, and 70 for Rouge-L, Mix, and EMS, respectively, without describing how these thresholds were selected or validated. The paper documents a failure case for BLEU but does not provide analogous evidence that the Rouge-L/EMS threshold correctly identifies medically correct paraphrases; the Limitations section concedes that \"more fine-grained and clinically grounded reward objectives\" are needed. Because the same metrics are used for evaluation in Table 3, the open-text improvements may partly reflect overfitting to lexical overlap rather than medical correctness. Please report the threshold selection procedure (train/validation split vs. test set), a sensitivity analysis, and ideally a correlation of the reward with human judgments on a held-out set.","section":"Section 2.1 and Section 3.1"},{"comment":"No variance or confidence intervals are reported for any experiment, and several reported differences are small or even negative (e.g., EHRNoteQA for Med-U1-3B is 55.17 vs. 52.87 for the SFT baseline, and MMLU-Pro Virology for Med-U1-3B is 36.96 vs. 43.48 for SFT). GRPO training is stochastic, so single-run accuracy is insufficient to support the consistent-gains claim. Please report results over at least three seeds with standard deviations or confidence intervals for the main tables.","section":"Section 3.1 and Tables 1-2"},{"comment":"The text states that improvements on the 3B model \"can reach up to 9 percentage points\" and that Med-U1-7B shows \"approximately 19 percentage points\" gain in select OOD subdomains, but Table 2 shows much larger differences: the largest 3B gain is about 29 points (College Medicine: 47.92 vs. 18.75) and the largest 7B gain is about 23 points (College Medicine: 64.58 vs. 41.67); some subdomains show negative differences. Please reconcile the text with the table and specify whether the 9-point and 19-point figures refer to a particular subdomain, an average, or a different metric.","section":"Section 3.2 and Table 2"},{"comment":"The length-matching reward R_length in Eq. (3) requires a target length lgold for each training question, but the paper never states how lgold is constructed (e.g., from reference answers, a heuristic, or a learned prior) nor whether the main Med-U1 models in Table 1 were trained with R_length active or with the length term omitted. This is essential for reproducing the controllable-length claim and for understanding whether the main results depend on the length reward. Please specify the construction of lgold and the training configuration for the models reported in Table 1.","section":"Section 2.2 and Section 3.1"},{"comment":"The evaluation section does not specify whether the same answer-extraction protocol (parsing <answer></answer> tags via regular expressions) is applied to all baselines. If baseline models are not instructed or trained to use this format, their MCQ and open-text scores may be artificially lowered by the extraction procedure. Please clarify the exact extraction steps for each model family and, if needed, provide an alternative evaluation (e.g., using the full output or the last answer span) to ensure a fair comparison.","section":"Appendix B.3"}],"minor_comments":[{"comment":"There is a typo in the sentence describing numeric value computation: \"in ths category\" should be \"in this category\".","section":"Section 2.1"},{"comment":"The yellow-box text says \"faluire case\" and should read \"failure case\".","section":"Section 2.1"},{"comment":"The threshold values 40, 50, and 70 are presumably percentages (0.40, 0.50, 0.70) for Rouge-L, Mix, and EMS, respectively; please state the units explicitly and confirm whether the thresholds are applied to the continuous scores or to the averaged score for the Mix reward.","section":"Section 3.1"},{"comment":"The labels \"+9 pts\" and \"+5 pts\" in Figure 4 are unclear without a legend or axis explanation; please clarify what comparison these deltas refer to and whether they correspond to the 3B or 7B model.","section":"Figure 4"},{"comment":"The table header appears garbled in the manuscript; please format it with clear columns for the reward type and the evaluation metrics (Rouge-L and EMS) so that the comparison between Rouge-L, EMS, and Mix rewards is readable.","section":"Table 3"},{"comment":"The learning rate is written as \"e-5\"; this should be \"1e-5\".","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the claimed results are plausible, but the missing variance reporting, unvalidated open-text reward thresholds, and internal numeric inconsistencies need to be addressed before the claims can be fully trusted. The evaluation-fairness question regarding baseline answer extraction is also important. I would encourage the editor to request a revision that includes multi-seed results, a clearer description of the threshold and length-target choices, and explicit reconciliation of the reported percentage-point gains with Table 2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read Med-U1 with some care. The core novelty is a real one: taking the pure-RL recipe (GRPO with rule-based binary rewards) that works for math and code and extending it to the heterogeneous output formats of medical QA—multiple choice, numeric values, and free-text—all under one length-controllable objective. Each ingredient has been shown before (MedRLVR for MCQ, L1/budget forcing for length, etc.), but no one had combined them in the medical domain with a mixed reward and a verbosity penalty. The experiments support the claim that this works: Med-U1-7B beats its SFT counterpart on nearly every in-distribution benchmark and is competitive with much larger models on some tasks. The ablations on reward metric selection and reasoning-length sensitivity are genuinely informative, and the appendix case studies are good qualitative evidence.\n\nThe soft spots are real but not fatal. First, there are internal numerical inconsistencies: the \"9 percentage points\" and \"19 percentage points\" claims in Section 3.2 do not line up cleanly with Table 2 or Figure 4. This looks like sloppy reporting rather than a deliberate overstatement, but it needs fixing. Second, there are no variance estimates anywhere—RL runs are noisy, and the reader cannot tell whether the differences between Med-U1 and the baselines are within a standard deviation. That is a serious omission for a paper whose whole evidence is benchmark improvements. Third, the open-text correctness reward is a thresholded average of Rouge-L and EMS, and the thresholds (40/50/70) appear to be chosen by hand on the same benchmarks used for evaluation. The paper candidly documents that BLEU fails, but it never validates that the Rouge-L+EMS proxy agrees with human judgment of medical correctness. Since the same metrics are used for training and evaluation, the open-text gains could partly be reward overfitting. The Limitations section acknowledges the need for clinically grounded rewards, which is honest, but the claim of \"unified medical reasoning\" remains somewhat thinner for the open-text component.\n\nThis is a paper worth refereeing seriously. The methodology is mostly sound, the contribution is distinct enough to matter, and the issues are addressable in a revision. I would ask for variance estimates, a sensitivity analysis on tau, clarity on how lgold is set during training, and a reconciliation of the reported percentage gains. If those are fixed, it would be a strong contribution.\n\nWould I cite it? In the next year, probably yes, given the length-control angle is relevant to my own work. I would bring it to our reading group to discuss the reward design.","headline":"A genuinely useful combination of pure RL, mixed rewards, and length control for medical QA, with solid empirical gains but some reporting sloppiness and an unvalidated open-text reward proxy.","tokens_in":17341,"tokens_out":3257,"would_cite":true,"duration_ms":39614,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single pure reinforcement-learning recipe, with task-specific rule-based rewards and a length penalty, can make a 7B medical QA model outperform much larger open-source and proprietary models across…","keywords":["medical question answering","reinforcement learning","rule-based rewards","GRPO","reasoning length control","large language models","medical benchmarks","out-of-distribution generalization"],"falsifier":"Retrain or re-evaluate Med-U1-7B on open-ended medical questions whose reference answers have several synonymous phrasings with low lexical overlap; if expert clinicians score the outputs as correct while the Rouge-L/EMS reward scores them below threshold, the reward proxy, not medical reasoning, is the binding constraint.","tokens_in":16388,"feed_emoji":"🩺","tokens_out":8907,"duration_ms":95048,"temperature":0.7,"pith_summary":"Med-U1 asks whether a single training recipe—pure reinforcement learning with rule-based rewards and no extra supervised reasoning traces—can make one language model handle the full range of medical question answering. The authors answer yes: starting from Qwen2.5-3B/7B-Instruct, they train with GRPO on a mixed reward that checks output format, verifies the answer in a task-specific way, and optionally penalizes deviation from a requested reasoning length. On five medical benchmarks the 7B model surpasses much larger open-source and proprietary baselines, and on an out-of-distribution health benchmark it beats a supervised fine-tuning baseline by up to nine points. If true, this means a unified, verifiable RL objective can substitute for task-specific supervised data and expensive verifiers in medicine.","feed_headline":"Reinforcement learning only: a 7B medical QA model beats 70B rivals","feed_subtitle":"Med-U1's mixed rule-based rewards cover multiple-choice, numeric, and open-text questions and generalize out of distribution.","key_machinery":"The load-bearing object is the mixed task-specific reward function, made of three terms: a format reward R_format that scores whether the output contains <think></think> and <answer></answer> tags; a correctness reward R_correct whose verification rule depends on the task type, using literal option matching for multiple choice, range or string matching for numeric and short answers, and for open text a thresholded average of Rouge-L (longest-common-subsequence overlap) and Exact Match Score; and an optional length reward R_length that is maximized when the total thinking length matches a user-specified target. These rewards are combined and optimized with GRPO. The open-text thresholded proxy is what lets the RL loop treat free-form medical answers as verifiable, while the length term is what makes reasoning budgets controllable.","core_discovery":"The paper's central claim is that pure large-scale reinforcement learning, guided by a mixed rule-based reward, is enough to produce a unified medical QA system from a 7B instruction-tuned model. The authors report that Med-U1-7B reaches 57.55% accuracy on MedCalc-Bench and 17.74% on MedXpertQA, and that on the open-ended medical-o1-reasoning-SFT task it achieves a 73.06 Exact Match Score, surpassing larger open-source and proprietary baselines on the aggregated comparisons. They also find that the RL-trained model beats an SFT baseline by up to nine percentage points on an out-of-distribution health benchmark. The authors interpret these results as evidence that the RL paradigm itself, not the distribution of supervised reasoning traces, drives robust and generalizable medical reasoning.","pith_inferences":["Inference: the open-text results are only as medically trustworthy as the Rouge-L/EMS threshold; if the threshold mis-scores synonymous phrasings, the model will learn surface mimicry, so an expert-reviewed sample of near-miss outputs would show whether the proxy is sound.","Inference: the bell-shaped length-accuracy curves imply that per-task length targets could be learned automatically from validation data, giving a cheap adaptive inference policy the paper does not implement.","Inference: the same reward decomposition—format, task-specific correctness, and length—could transfer to other domains with heterogeneous outputs, such as legal or scientific QA, where answer formats are similarly mixed.","Inference: because the format reward scores the use of <think>/<answer> tags, part of the EMS gain may come from enforced structure rather than better medical knowledge; ablating the format reward would isolate that effect."],"forward_implications":["A single RL run on mixed-format medical data can replace pipelines that first distill reasoning traces with SFT and then apply RL, cutting the need for human- or model-generated reasoning annotations.","Open-ended medical answers can be trained with fully rule-based rewards once correctness is defined as a thresholded lexical-overlap score, so no learned reward model or LLM judge is needed.","Reasoning length becomes a user-tunable dial: the model can be prompted with a target length, and most medical tasks reach peak accuracy near 512 tokens, so inference cost can be cut without losing accuracy.","Pure RL appears safer than SFT-then-RL for calculation-heavy medical tasks, where prior SFT can interfere with numerical reasoning.","Out-of-distribution gains on a held-out health benchmark suggest the RL-trained model learns a general problem-solving behavior rather than memorizing the five training sets."],"supporting_citations":[{"why":"Supplies the pure-RL-with-rule-rewards template that Med-U1 extends to mixed medical formats.","marker":"Guo et al. (2025)"},{"why":"Provides the GRPO algorithm used for all policy optimization in the paper.","marker":"Shao et al. (2024)"},{"why":"Defines MedCalc-Bench, the numeric computation task and benchmark central to the arithmetic-reasoning results.","marker":"Khandekar et al. (2024)"},{"why":"Contributes the medical-o1-reasoning-SFT open-text dataset and a strong SFT-plus-RL baseline.","marker":"Chen et al. (2024)"},{"why":"Defines Rouge-L, half of the open-text correctness reward used in training.","marker":"Lin (2004)"},{"why":"Defines BLEU, whose failure on short factual answers motivates the EMS-plus-Rouge-L threshold.","marker":"Papineni et al. (2002)"},{"why":"Provides MedXpertQA, the hard multiple-choice benchmark where Med-U1-7B beats most open-source baselines.","marker":"Zuo et al. (2025)"},{"why":"Provides EHRNoteQA, the electronic-health-record-based clinical QA benchmark used for training and evaluation.","marker":"Kweon et al. (2024)"},{"why":"Provides MEDiQ, the clinical reasoning benchmark that tests question-asking and contextual reasoning.","marker":"Li et al. (2024)"},{"why":"Supplies the Qwen2.5-3B and Qwen2.5-7B instruction-tuned backbones that Med-U1 initializes from.","marker":"Yang et al. (2024)"}],"fun_headline_variants":["Pure RL unifies medical QA: 7B model tops 70B rivals","Med-U1: Rule-based RL rewards craft concise medical reasoning","Large-scale RL turns 7B LLM into unified medical QA generalist","Mixed rule-based rewards guide LLM to beat bigger med models","RL-only training helps 7B LLM generalize across medical QA tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes a free-text answer is correct exactly when its Rouge-L/Exact-Match blended score passes a fixed threshold, so medically valid answers phrased differently from the reference are treated as wrong and the model is pushed away from them.","fun_headline_variants_meta":{"raw":{"variants":["Pure RL unifies medical QA: 7B model tops 70B rivals","Med-U1: Rule-based RL rewards craft concise medical reasoning","Large-scale RL turns 7B LLM into unified medical QA generalist","Mixed rule-based rewards guide LLM to beat bigger med models","RL-only training helps 7B LLM generalize across medical QA tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1854,"prompt_tokens":934,"completion_tokens":920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":550,"tokens_out":920,"duration_ms":9974,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:53:04.698970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or re-evaluate Med-U1-7B on open-ended medical questions whose reference answers have several synonymous phrasings with low lexical overlap; if expert clinicians score the outputs as correct while the Rouge-L/EMS reward scores them below threshold, the reward proxy, not medical reasoning, is the binding constraint.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MedCalc-Bench, the numeric computation task and benchmark central to the arithmetic-reasoning results."}],"review_version":1}