{"id":"c2dbe685-e6bf-4ecc-b0ca-93226ec9a0c4","arxiv_id":"2507.23541","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A progressive three-stage RL framework with medical-specific rewards improves retrieval-augmented reasoning on medical QA benchmarks, reportedly surpassing GPT-4o-mini with an 8B model.","lead":"This paper introduces Med-R3, a three-stage reinforcement learning method that jointly trains an LLM's medical reasoning and its ability to retrieve external medical knowledge. It reports that an 8B model trained this way beats GPT-4o-mini on several medical QA benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported margins over GPT-4o-mini rest on DeepSeek-V3 as the answer-correctness judge, which also scores the training rewards; the human meta-eval does not cover GPT-4o-mini or reward-hacked outputs, so the 3.93%/13.53% gains could be a judge artifact.","rationale":"The reader's CONDITIONAL verdict is appropriate. The strongest concern is not that the pipeline is ill-specified—it is well-specified with informative ablations—but that the outcome measure is the same model used as the training reward. This creates a closed loop: the model is selected to maximize DeepSeek-V3's approval, and then DeepSeek-V3's approval is reported as accuracy. The paper's own meta-evaluation is a step in the right direction but is too small and does not clearly cover the GPT-4o-mini comparison. A human blind re-scoring of the main comparison would settle this. I also considered whether the abstract discrepancy is the primary issue; it is a credibility problem and should be corrected, but it does not by itself identify a mechanism by which the full-text numbers are wrong. I considered whether lack of error bars is decisive; for a single-run benchmark comparison, it weakens but does not falsify the claim. Therefore the central claim remains plausible but contingent on independent evaluation, matching CONDITIONAL; no code or data release further supports the need for external verification.","tokens_in":122,"tokens_out":8069,"duration_ms":144820,"concrete_test":"Run a blinded human expert evaluation on a stratified random sample of 200 final answers per condition from Table 2's main comparison (LLaMA3.1-8B-Instruct+Med-R3, Qwen2.5-14B+Med-R3, GPT-4o-mini CoT-RAG, ReSearch* CoT-RAG) across all 8 benchmarks; have two board-certified physicians label correctness using the Appendix C.6.2 rubric. Compare human-verified accuracy and the Med-R3-versus-GPT-4o-mini margin to the DeepSeek-V3-judged numbers. If the human-verified margin is not within sampling error of the reported margin, the superiority claim is a judge artifact. As a complementary check, recompute accuracy on the closed-set benchmarks by exact match against original option letters; if the Med-R3 advantage disappears, the open-ended LLM-judge protocol is the source of the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Med-R3 outperforms GPT-4o-mini and general RAG-RL baselines (Table 2, Section 3.2). That claim requires the evaluation metric to reflect true medical correctness. Yet the metric is LLM-as-Judge with DeepSeek-V3 (Section 3.1), the same model used throughout training: R_answer in Eq. (3), the reasoning rewards in Eqs. (5)-(7) that compare against DeepSeek-R1/V3 reference trajectories, and the evidence-quality scores in Eq. (10). Training therefore optimizes the policy against DeepSeek-V3's scoring function, and the reported accuracy is measured with that same scoring function. The model could be rewarded for adopting the judge's preferred answer phrasing or reasoning style rather than for clinically correct content. The human meta-evaluation (Appendix C.6.2) samples 30 instances per dataset per trained model and reports roughly 97% judge-human agreement, but it is not stated to cover GPT-4o-mini or ReSearch* outputs in Table 2, and 30 instances per dataset cannot rule out a small, systematic bias concentrated at the decision boundary between models. A 3% disagreement rate, if biased toward Med-R3's answer style, is enough to erase the 3.93% average margin over GPT-4o-mini. The metadata abstract's different headline numbers (Qwen3-8B, 12.22%, 16.31%) add uncertainty about which experimental configuration produced the published results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Med-R3, a three-stage progressive reinforcement learning framework for medical retrieval-augmented reasoning. Stage 1 cultivates reasoning over medical questions with rewards for format, answer correctness, a composite reasoning reward (semantic, entity/relation coverage, and logical k-hop path similarity against reference trajectories), and a retrieval-count bonus. Stage 2 optimizes retrieval effectiveness via an evidence-quality reward based on an Evidence-Based Medicine hierarchy and a breadth reward measuring the proportion of retrieved knowledge-graph triples. Stage 3 jointly optimizes end-to-end correctness with format and answer rewards. Training data (about 10.5K instances) is built from MedQA, MedMCQA, and RareArena via DeepSeek-R1 generation and DeepSeek-V3 filtering. Experiments on eight medical benchmark columns with LLaMA3.1-8B-Instruct, Qwen2.5-7B, and Qwen2.5-14B report consistent gains over SFT, R1-Searcher*, and ReSearch*, with the strongest claim being LLaMA3.1-8B-Instruct + Med-R3 outperforming GPT-4o-mini by 3.93% on average and Qwen2.5-14B + Med-R3 gaining 13.53%. Ablations show the value of progressive staging and of individual reward components. The paper also includes a failure-case analysis and a small human meta-evaluation of the LLM-as-Judge pipeline.","tokens_in":35414,"tokens_out":5207,"duration_ms":52307,"significance":"The contribution is potentially significant: it is a concrete, well-specified attempt to jointly optimize retrieval and reasoning for the medical domain, with a richer reward structure than outcome-only RAG-RL baselines. The empirical scope is substantial (three backbones, eight evaluation columns, stage and reward ablations, and a physician-labeled failure analysis), and the detailed prompts and reward equations make the method easy to reproduce in principle. The staged nature of the training pipeline, the use of medical knowledge graphs for reasoning-process rewards, and the EBM-based evidence quality reward are genuine novelties. However, the strength of the central claim depends entirely on the trustworthiness of the evaluation judge, and the paper currently does not establish that independence. If the judge-circularity concern is addressed, this would be a useful contribution to medical LLM training methodology.","major_comments":[{"comment":"The final evaluation metric and the training rewards are both produced by DeepSeek-V3: answer correctness (Eq. (3)), the reasoning rewards (Eqs. (5)-(7)), and the evidence-quality reward (Eq. (10)) all use DeepSeek-V3 as judge or extractor, and Section 3.1 states that benchmark accuracy is computed with the same LLM-as-Judge. Because the policy is optimized against this judge, the reported accuracies, including the 3.93% margin over GPT-4o-mini in Table 2, may reflect stylistic alignment with the judge rather than medical correctness. The human meta-evaluation in Appendix C.6.2 (Table 9) samples 30 answers per dataset from the three trained backbones only; it does not cover GPT-4o-mini or the R1-Searcher/ReSearch baselines, and a 3% disagreement rate could in principle erase the 3.93% margin. The Discussion concedes this bias risk but the paper provides no countermeasure. Please add an independent evaluation (e.g., exact-match on the multiple-choice subsets, a second judge, or clinician scoring of all systems in Table 2) and report agreement separately for every compared system.","section":"§3.1 and §2.2 (Eqs. (3), (5)-(7), (10))"},{"comment":"The abstract reports that Qwen3-8B + Med-R3 surpasses GPT-4o-mini by 12.22% and that Qwen2.5-14B + Med-R3 gains 16.31%, but the full text and Table 2 report LLaMA3.1-8B-Instruct + Med-R3 surpassing GPT-4o-mini by 3.93% and Qwen2.5-14B + Med-R3 gaining 13.53%. Qwen3-8B does not appear anywhere in the experiments. This discrepancy must be corrected, and the exact configuration behind each headline number clarified.","section":"Abstract vs. §3.2 and Table 2"},{"comment":"No variance or significance information is reported for any accuracy number. With an LLM-as-Judge evaluation there is judge stochasticity in addition to sampling noise, so a single-point difference such as the 3.93% average margin over GPT-4o-mini cannot be assessed for significance. Please report bootstrap confidence intervals across items or across repeated judge calls, and state the number of judge calls per answer.","section":"Table 2 and §3.1"},{"comment":"The claim that Stage 3 'directly prioritizes end-to-end performance' with only format and answer-correctness rewards is not fully supported by the ablation. Removing Stage 3 (1→2) gives a 2.88% drop for Qwen2.5-7B, but the difference is small and no significance test is provided. Moreover, because the Stage-3 reward is the same answer-correctness judge used for evaluation, this ablation does not separate the value of joint optimization from the value of additional training on the judge's preferences. Please clarify what Stage 3 adds beyond more epochs of the same reward signal.","section":"§2.2.3 and Table 3"}],"minor_comments":[{"comment":"There is a typo in the library list: 'transfomrers' should be 'transformers'.","section":"Appendix C.5"},{"comment":"The word 'descirbed' should be 'described' in the first sentence.","section":"Appendix C.6"},{"comment":"References [35] and [36] appear to be the same paper (both are 'Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL' by Che Liu et al.); one should be removed or distinguished.","section":"References"},{"comment":"Figure 5 is referenced in the main text ('as depicted in Figure 5') but does not appear in the manuscript; please include the figure or remove the reference.","section":"Figure 5"},{"comment":"The retrieval-count threshold δ=3 and the filtering cutoffs (scores of 2 and 5, n′≤3) are free parameters with no sensitivity analysis; a brief discussion or small sweep would strengthen the robustness of the pipeline.","section":"Eq. (8) and §2.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical scope and a well-specified method, but the evaluation-judge circularity is the central risk: the same DeepSeek-V3 judge is used for training rewards, data construction, and final evaluation, and the human meta-evaluation does not cover the specific systems that the headline comparisons rely on. The abstract/full-text inconsistency about Qwen3-8B is also a correctness issue. I would encourage the authors to add an independent or partially independent evaluation (exact-match for multiple-choice subsets, a second judge, or clinician scoring for the key comparisons) and to fix the abstract inconsistency; after that, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real training recipe—three-stage progressive RL that first cultivates reasoning, then retrieval, then joint optimization, with medical-specific rewards (knowledge-graph coverage, EBM evidence tiers, retrieval breadth). The ablations are thorough and the method looks sound on its own terms. But the empirical headline numbers rest on DeepSeek-V3 as both the training oracle and the final judge, and the paper's two abstracts disagree with each other. I'd send it to review, but I wouldn't build on it until the judge issue is addressed and code/data are out.\n\nThe new thing here is the staged reward design. R1-Searcher and ReSearch use outcome-based rewards; Med-R3 adds intermediate supervision on the reasoning process and on evidence quality. The ablations in Tables 3 and 4 actually earn their keep: removing any stage hurts, changing the order hurts, and each reward component contributes non-trivially. The retrieval-frequency analysis in Table 5 is a nice sanity check that the Stage-1 reward really changes behavior. The masking of retrieved document tokens during loss is a good detail too. The failure case analysis is honest and gives a useful breakdown of retrieval vs. reasoning errors.\n\nThe soft spots. First, the circularity: the model is trained to maximize a reward that DeepSeek-V3 computes, and the reported accuracy is the agreement with DeepSeek-V3's judgment of correctness. The human meta-evaluation only samples 30 instances per dataset per backbone and doesn't cover GPT-4o-mini or the reward-hacked outputs—so a small systematic bias in the judge's preferences could explain a chunk of the 3.93% and 13.53% margins. The paper itself flags this limitation in the Discussion, which is good, but it doesn't fix it. Second, the metadata abstract quotes Qwen3-8B +12.22% and Qwen2.5-14B +16.31%, while the full-text abstract says LLaMA3.1-8B +3.93% and Qwen2.5-14B +13.53%. That's not a typo-level discrepancy; it raises a question about which numbers are the published ones. Third, no error bars, no significance tests, no code or data. The paper is a recipe, not a benchmark winner you can verify.\n\nNet: the method is well-specified and the evidence is broad enough to deserve a serious referee. The right outcome is probably major revision: re-evaluate with a different judge (or a held-out human set), reconcile the abstracts, and release artifacts. For a reading group, this is a good example of how to construct and ablate multi-stage RL, but the judge issue should be discussed.","headline":"A substantive three-stage RL recipe for medical RAG with a real judge-circularity problem at the center; worth reviewing, not worth citing yet.","tokens_in":35979,"tokens_out":2725,"would_cite":false,"duration_ms":26873,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage progressive reinforcement learning pipeline that jointly trains retrieval and reasoning achieves state-of-the-art medical question answering for open-source models.","keywords":["retrieval-augmented reasoning","reinforcement learning","medical question answering","knowledge graph","LLM-as-judge","progressive training","evidence-based medicine","GRPO"],"falsifier":"Have two independent physician panels score a random sample (say 500 questions per dataset) of outputs from Qwen2.5-14B + Med-R3 and GPT-4o-mini, blinded to model identity; the paper's central claim predicts Med-R3 should win by at least the 13.53% margin under human scoring. Alternatively, re-run the same staged RL pipeline with a different frontier judge for all rewards and final evaluation, and check whether the margin over GPT-4o-mini and the ablation ordering (1→2→3 > 1&2&3) survive.","tokens_in":1739,"feed_emoji":"🩺","tokens_out":2151,"duration_ms":70230,"temperature":0.7,"pith_summary":"The paper sets out to show that medical question answering improves most when a model's retrieval and reasoning are trained together, not separately, and not by imitation. It proposes Med-R3, a three-stage reinforcement learning pipeline: first train the model to reason over medical problems, then to issue effective retrieval queries, then to coordinate the two. Rewards are tailored to medicine — reasoning quality is scored via semantic overlap, entity and relation coverage, and logical path structure of an extracted knowledge graph, while retrieval is scored by the evidence level and reasoning contribution of retrieved documents. On seven benchmarks, models trained this way beat supervised fine-tuning and general-domain retrieval-reasoning RL, with LLaMA3.1-8B-Instruct + Med-R3 exceeding GPT-4o-mini by 3.93% and Qwen2.5-14B + Med-R3 by 13.53% on average.","feed_headline":"Progressive RL makes small open models beat GPT-4o-mini on medical QA","feed_subtitle":"Staged reasoning-then-retrieval rewards push 7B–14B open models past a closed commercial model on seven benchmarks.","key_machinery":"The central mechanism is the medical knowledge graph extracted from each reasoning trajectory: every step is parsed into triplets (head entity, relation, tail entity, retrieval flag) where the flag marks whether the fact came from an external document. This representation turns qualitative reasoning quality into measurable quantities: Jaccard overlap of entity and relation sets, weighted Jaccard overlap of j-hop logical paths, and the proportion of retrieved triplets (breadth). These feed the stage-specific rewards, and GRPO (group-relative policy optimization) updates the policy from the group-normalized advantages.","core_discovery":"Med-R3 establishes that progressive RL — reasoner first, retriever second, joint coordination third — with medical-specific reward components yields state-of-the-art medical QA among open-source 7B–14B models. The decisive design choices are the structured reward signals: the reasoning reward decomposes into semantic, entity/relation-coverage, and k-hop logical-path similarities against reference trajectories, and the retrieval reward combines an evidence-quality score from the evidence-based-medicine hierarchy with the fraction of the reasoning trajectory's knowledge-graph triplets that came from retrieved documents. Removing any stage degrades performance, with the reasoner-cultivation stage the most critical (7.30% drop), and merging all rewards at once (1&2&3) produces a 4.77% drop from reward conflicts.","pith_inferences":["The staged recipe likely transfers to other knowledge-intensive domains where reasoning is relational (law, scientific literature, finance), where coverage of key entities and evidence quality are similarly measurable.","Since the judge (DeepSeek-V3) supplies both intermediate rewards and final accuracy labels, the reported margins are partly a function of that judge; a systematic judge-bias study with human panels beyond the 200-step/150-document/30-answers meta-evaluation would clarify how much.","A testable extension: warm-start the policy with a short SFT stage and then apply Med-R3's staged RL, to test whether the generalization gains of RL are compatible with SFT initialization speed.","The failure analysis (42% retrieval failures, 58% reasoning failures) suggests adding a reward that penalizes irrelevant documents specifically (rather than only rewarding high-quality ones) could further improve results."],"forward_implications":["Open-source 7B–14B models trained with Med-R3 exceed GPT-4o-mini's average accuracy on seven medical QA benchmarks, by 3.93% for LLaMA3.1-8B-Instruct and 13.53% for Qwen2.5-14B.","Progressive stage ordering is load-bearing: removing Stage 1 (reasoner cultivation) costs 7.30%, removing Stage 2 costs 5.89%, removing Stage 3 costs 2.88%, and swapping Stages 1 and 2 costs 2.56%.","Optimizing all three reward groups simultaneously is worse than staging them, dropping average accuracy by 4.77% relative to the 1→2→3 order.","Among reasoning rewards, entity/relation coverage (R_statistic) is the most valuable — removing it costs 5.65%; among retrieval rewards, breadth (R_breadth) removal costs 3.70%.","A reward for the number of retrievals in Stage 1 is needed to establish retrieval behavior: without it, average retrieval frequency falls from 3.67 to 0.44 in Stage 1."],"supporting_citations":[{"why":"Supplies the frontier judge for answer correctness, data filtering, reference reasoning trajectories, knowledge-graph extraction, and EBM level scoring.","marker":"[34]"},{"why":"Generates up to 16 rollouts per question for complexity filtering and the reference reasoning trajectories.","marker":"[17]"},{"why":"Provides the data-selection and verifiability-transformation approach adapted for medical RL training data and is a baseline that Med-R3 must beat.","marker":"[3]"},{"why":"General-domain retrieval-augmented reasoning RL baseline re-implemented with the paper's data; main comparison for tailored rewards.","marker":"[5]"},{"why":"General-domain retrieval-augmented reasoning RL baseline re-implemented with the paper's data; main comparison for tailored rewards.","marker":"[48]"},{"why":"MedQA-USMLE and MedQA-MCMLE training and evaluation data; defines in-domain exam questions.","marker":"[28]"},{"why":"MedMCQA training and evaluation data; expands in-domain coverage.","marker":"[39]"},{"why":"RareArena rare-disease dataset with an 8:2 train-test split used for in-domain evaluation.","marker":"[51]"},{"why":"Defines the EBM evidence hierarchy underlying the retrieval quality reward.","marker":"[41]"},{"why":"Provides the GRPO reinforcement learning algorithm that optimizes the policy with group-normalized advantages.","marker":"[45]"}],"fun_headline_variants":["Med-R3: Progressive RL helps 8B open model beat GPT-4o-mini on medical QA","Staged reasoning and retrieval RL lifts 7-14B open models past GPT-4o-mini","Reason-first then retrieve: RL makes small open models top GPT-4o-mini in med QA","Medical QA: Progressive RL with joint retrieval-reasoning rewards beats GPT-4o-mini at 8B ","Open 7-14B models beat GPT-4o-mini on medical QA via Med-R3 progressive RL"],"cache_read_input_tokens":38016,"weakest_assumption_plain":"The whole pipeline treats DeepSeek-V3's judgments — for answer correctness, reasoning quality, evidence levels, and final accuracy — as ground truth, so if that judge is systematically biased in favor of certain phrasings or knowledge, the model would be trained to please the judge rather than to be medically right, and the reported gains over GPT-4o-mini could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Med-R3: Progressive RL helps 8B open model beat GPT-4o-mini on medical QA","Staged reasoning and retrieval RL lifts 7-14B open models past GPT-4o-mini","Reason-first then retrieve: RL makes small open models top GPT-4o-mini in med QA","Medical QA: Progressive RL with joint retrieval-reasoning rewards beats GPT-4o-mini at 8B scale","Open 7-14B models beat GPT-4o-mini on medical QA via Med-R3 progressive RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000476,"raw_usage":{"total_tokens":2399,"prompt_tokens":1021,"completion_tokens":1378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1242}},"tokens_in":637,"tokens_out":1378,"duration_ms":10509,"temperature":1.0,"reasoning_tokens":1242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:37:58.627698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent physician panels score a random sample (say 500 questions per dataset) of outputs from Qwen2.5-14B + Med-R3 and GPT-4o-mini, blinded to model identity; the paper's central claim predicts Med-R3 should win by at least the 13.53% margin under human scoring. Alternatively, re-run the same staged RL pipeline with a different frontier judge for all rewards and final evaluation, and check whether the margin over GPT-4o-mini and the ablation ordering (1→2→3 > 1&2&3) survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MedMCQA training and evaluation data; expands in-domain coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RareArena rare-disease dataset with an 8:2 train-test split used for in-domain evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the EBM evidence hierarchy underlying the retrieval quality reward."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GRPO reinforcement learning algorithm that optimizes the policy with group-normalized advantages."}],"review_version":1}