{"id":"cfd22f50-0154-4a4a-9b74-c8abaf5b0a15","arxiv_id":"2509.16679","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.","lead":"The paper organizes recent research on using reinforcement learning, a reward-based training technique, to improve large language models at every stage of their life cycle. It is useful as a reference map for researchers, but it adds no new experiments or methods.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (5) in §2.1 misstates GRPO's advantage normalization (max instead of mean), contradicting its own prose and the original paper; this is a load-bearing technical error for a survey whose central value is an accurate RL reference.","rationale":"The paper's central claim is a comprehensive, lifecycle-wide synthesis of RL for LLMs, with a special focus on RLVR. A survey only succeeds if its algorithmic descriptions are correct enough to be used as a reference. The GRPO advantage formula is the core of the RLVR methods the survey highlights, and Eq. (5) contradicts both the prose (which correctly says 'intra-group average') and the original DeepSeekMath paper. The reader's weakest assumption concerned Table 1's attribution of gains to RL. That is a genuine issue, but the qualitative conclusion (RL improves reasoning) is robust across many controlled studies; a wrong equation, by contrast, is a definitive factual error that would mislead anyone implementing from the survey. I therefore regard the GRPO misstatement as the single most load-bearing technical flaw. My proposed check—recomputing the advantage for a simple group—settles the matter directly. The verdict should remain CONDITIONAL: the paper's overall organizational contribution is valuable, but this and related accuracy issues need correction before it can serve as a reliable reference.","tokens_in":41710,"tokens_out":7896,"duration_ms":68803,"concrete_test":"Pull Eq. (10) from DeepSeekMath (Shao et al., 2024). Verify whether the advantage is (r_i - mean_group)/std_group. Then evaluate both formulations on reward vector [0,1,2]: GRPO gives [-1.22, 0, 1.22] after standardization; Eq. (5) gives [-2,-1,0]. If the survey's max-based formula yields non-positive advantages, the description is incorrect and must be fixed before publication.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §2.1, Eq. (5) defines the GRPO advantage as (r_i - max_j r_j)/std, i.e., it subtracts the maximum group reward. The surrounding text explicitly says GRPO uses the intra-group average reward as baseline (A_i = R_i - mean), and the original DeepSeekMath paper defines A_i = (r_i - mean)/std. If a reader implements Eq. (5), the best response in the group gets advantage 0 and every other response gets a negative advantage, so no response is positively reinforced. This is not a cosmetic typo; it inverts the learning signal and would prevent the policy update from promoting good outputs. It also indicates a broader reliability problem: the survey claims to give 'clear practical references' and 'in-depth analysis' of RLVR, but a central algorithmic equation is wrong and internally inconsistent. Since GRPO is the backbone of the survey's RLVR emphasis, this weakens the trustworthiness of the entire technical synthesis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a survey of reinforcement learning applied to large language models across the full LLM training lifecycle: pre-training, alignment, and reasoning, with particular emphasis on Reinforcement Learning with Verifiable Rewards (RLVR). It proposes a taxonomy, reviews algorithmic foundations (REINFORCE, PPO, GRPO), surveys recent RLVR-based methods for reasoning, multimodal reasoning, adaptive thinking, agents, and internal feedback, and consolidates datasets, benchmarks, and open-source training frameworks. The central claim is that this lifecycle-wide synthesis, together with the consolidated resources, provides a practical and comprehensive roadmap for researchers and practitioners working on RL-enhanced LLMs. The paper does not present new experiments or derivations; its contribution is compilation and organization.","tokens_in":41971,"tokens_out":5314,"duration_ms":49047,"significance":"If accurate, the survey is a genuinely useful resource: it covers a fast-moving 2024-2025 literature, organizes methods by lifecycle stage, and includes tables of benchmarks and frameworks that practitioners could use as a starting point. It also surfaces important controversies, such as whether RLVR truly expands reasoning or merely re-weights pretrained behavior, and it discusses training-stability and reward-design issues. However, the value of a survey of this type rests entirely on technical correctness and faithful citation. The manuscript currently contains a load-bearing algorithmic error in the GRPO formula, citation mismatches in the datasets section, and a mis-attributed foundational reference. These issues weaken confidence in the survey's reliability as a reference, although they are local and correctable rather than fatal to the overall structure.","major_comments":[{"comment":"The GRPO advantage formula is misstated as \\hat A_{i,t} = (r_i - max_j r_j)/std. The surrounding prose says GRPO uses the intra-group average reward as baseline, and the original DeepSeekMath paper defines A_i = (r_i - mean)/std. With max subtraction, the best response in the group receives zero advantage and all other responses receive negative advantage, so no response is positively reinforced. This is not a cosmetic typo; it inverts the learning signal. Since GRPO is a backbone of the RLVR discussion, this equation must be corrected and the notation (r_i vs R_i) harmonized.","section":"§2.1, Eq. (5)"},{"comment":"Reference [56] (Hendrycks et al., APPS) is used for both APPS and MATH: the text says 'APPS [56]' and later 'MATH [56] introduced 12,500...'. Table 3 lists APPS with citation [32], which is the StepCoder paper, not APPS. This makes the consolidated resource table unreliable. Please map APPS to the correct APPS reference, MATH to the Hendrycks MATH dataset reference, and align Table 3 with the text.","section":"§5.2, Table 3, ref. [56]"},{"comment":"The sentence 'Christiano et al. [129] established a foundational paradigm for modern LLM alignment' cites reference [129], which is Ouyang et al. (InstructGPT/RLHF). Christiano et al.'s deep RL from human preferences is a different paper and does not appear in the reference list. Please correct the attribution and add the appropriate reference, or rephrase to cite Ouyang et al. directly.","section":"§3.2, first paragraph"},{"comment":"The text states that SWE-bench Verified contains 50,000 instances collected from 128 GitHub repositories. SWE-bench Verified is a human-validated subset of 500 instances; the 50,000-instance figure corresponds to the synthetic SWE-smith data in reference [207], not to SWE-bench Verified. This is a factual error in a section whose purpose is to consolidate benchmark resources. Please verify all dataset sizes and descriptions.","section":"§5.2, SWE-bench Verified"},{"comment":"Table 1 attributes the full performance gains (e.g., DeepSeek-R1 vs DeepSeek-V3, +40.6 on AIME2024) to RL, but DeepSeek-R1 includes a cold-start SFT stage and other pipeline changes, so the comparison is not an RL-only ablation. DeepSeek-R1-Zero is a cleaner RL-only comparison. Please add a caveat or decompose the contributions; otherwise the motivating claim overstates the causal effect of RL.","section":"Table 1"}],"minor_comments":[{"comment":"Two distinct methods, 'SRPO [226]' (cross-domain LLM RL) and 'SRPO [164]' (reflection-aware multimodal RL), share the same acronym and are both discussed in the survey. Please disambiguate, e.g., 'cross-domain SRPO' and 'multimodal SRPO', to avoid confusion.","section":"§4.2 and §4.3"},{"comment":"The Math row lists OlympiadBench twice. Remove the duplicate entry.","section":"Table 3"},{"comment":"There is a malformed sentence: 'violating user intent or ethical normswang [41, 121, 122, 168]'. A word or punctuation is missing after 'norms'.","section":"§7.1.4"},{"comment":"The phrase 'Wang, Fu, and Miao et al. [41, 121, 122, 168]' is an awkward grouping; either list the author names separately or write 'several studies'.","section":"§3.3"},{"comment":"Kang et al. [77] is described twice with nearly identical sentences in the same subsection. Consolidate the two descriptions.","section":"§4.6"}],"recommendation":"major_revision","confidential_remarks":"The survey is broad and timely, and the lifecycle organization is a strength. However, the GRPO equation error and the citation/dataset inconsistencies are exactly the kind of defects that undermine a survey's raison d'être as a reliable reference. I would ask for a full reference audit and a careful pass over all equations and benchmark statistics before resubmission. The scope is appropriate for the journal, but the current version is not yet ready."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked for a read on arXiv:2509.16679. The short version: it's a broad, current survey of RL for LLMs with a sensible lifecycle organization, but it has a load-bearing technical error in the GRPO equation and some citation sloppiness that make it unreliable as a reference until fixed.\n\nWhat's genuinely useful: the lifecycle framing (pre-training, alignment, reinforced reasoning) is a reasonable way to organize the space, and the compilation of 2024-2025 RLVR work, datasets, benchmarks, and open-source frameworks is convenient. Figure 1's five-branch taxonomy gives a quick map, and the reference list is rich. If you need a starting point for what's happening in reasoning RL, this covers a lot of ground.\n\nThe soft spots are real. The stress-test note is right: Equation (5) in §2.1 uses max where GRPO uses mean. The surrounding text even says \"intra-group average reward\" and defines A_i = R_i - mean(R_group), so the equation contradicts the prose. If you implement from the equation, the best response gets advantage zero and everyone else gets negative advantage, so no response is positively reinforced. That's not cosmetic; for a survey whose selling point is being an accurate reference, this is a serious defect.\n\nThe citation issues the reader flagged hold up too. Reference [56] is used for both APPS and MATH, while Table 3 lists APPS with [32] (which is actually StepCoder). And §3.2 attributes the foundational RLHF paper to Christiano et al. [129], but [129] is Ouyang et al.'s InstructGPT. These look mechanical, but in a reference work they corrode trust.\n\nThe novelty claim of being the first full-lifecycle survey is asserted rather than demonstrated; Table 2 compares a handful of surveys, mostly from 2024, and the dimensions are coarse. Several 2025 surveys overlap heavily.\n\nThat said, the core organizational idea holds up. With the GRPO equation fixed and citations cleaned, this could become a legitimate reference.\n\nWho it's for: practitioners and newcomers wanting a map of the current RL-for-LLM landscape. It doesn't answer a scientific question, but it saves time. Recommendation: send to peer review with expectation of major revision. I wouldn't cite it in its current form.","headline":"Broad, current survey with a genuinely useful lifecycle organization, but a wrong GRPO equation and citation errors make it unreliable as-is; needs major revision before it earns a cite.","tokens_in":42418,"tokens_out":3501,"would_cite":false,"duration_ms":29001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey systematically maps how reinforcement learning operates across the full large-language-model lifecycle—pre-training, alignment fine-tuning, and reinforced reasoning—and argues that Reinforcement Learning with Verifiable Rewards","keywords":["reinforcement learning","large language models","reinforcement learning with verifiable rewards","RLHF","GRPO","reasoning","alignment","LLM lifecycle"],"falsifier":"Take one of the base models listed in Table 1 and train it with the same data and compute but no RL—for example, supervised fine-tuning on the RL-generated correct rollouts—then evaluate on AIME2024. If the non-RL run reproduces the reported gain, the survey's central attribution of the improvement to RL is empirically falsified.","tokens_in":41647,"feed_emoji":"🧠","tokens_out":5540,"duration_ms":52521,"temperature":0.7,"pith_summary":"This paper is a survey that tries to establish a complete, lifecycle-wide picture of how reinforcement learning (RL) is used to improve large language models (LLMs). It claims that RL now plays a role in every major phase of model development—pre-training, alignment with human preferences, and the newer 'reinforced reasoning' phase that powers state-of-the-art mathematical and coding performance. The survey's central focus is Reinforcement Learning with Verifiable Rewards (RLVR), where a model is rewarded by automated checks such as unit tests or math-answer verification rather than by human judgments. It also consolidates datasets, benchmarks, and open-source training frameworks into a single reference. A reader would care because the paper offers a practical map of when and how to apply RL, and it surfaces the unresolved question of whether RL truly creates new reasoning abilities or merely amplifies what pre-training already encoded.","feed_headline":"RL powers LLMs from pretraining to reasoning","feed_subtitle":"A lifecycle-wide synthesis ties reinforcement learning to alignment, verifiable rewards, and the tools to run it at scale.","key_machinery":"The load-bearing organizational device is the lifecycle taxonomy, which divides RL-for-LLM work into pre-training, alignment, and reinforced reasoning, with a separate branch for RLVR. The central algorithmic object is RLVR—reinforcement learning with verifiable rewards—where the reward signal is produced by an automated oracle (unit tests, math answer keys, formal verification) rather than by a learned human-preference model. The methodological anchor within RLVR is GRPO (group relative policy optimization), which samples several responses per prompt, uses the group-average reward as a baseline in place of a trained value network, and updates the policy to favor above-average responses. The","core_discovery":"The paper's central claim is that reinforcement learning has become a unifying training mechanism across the LLM lifecycle, and that the most consequential recent variant is Reinforcement Learning with Verifiable Rewards (RLVR). In RLVR, reward comes from an objective, automatically checkable source—passing unit tests, matching a known math answer, or satisfying a formal specification—rather than from a learned model of human preference. The paper argues this paradigm, implemented by algorithms in the GRPO family, is what drove the sharp jumps in mathematical and coding reasoning seen in recent models. To support this, it organizes the field into a five-branch taxonomy—pre-training RL, align","pith_inferences":["Editorial inference: the reward-design dimensions the paper surfaces (source, density, dynamics, landscape) read like a checklist for diagnosing failures in RLVR; if a run stagnates, identifying which dimension is shallow—e.g., outcome-only rewards on a long-horizon task—points to the likely fix.","Editorial inference: if the cited evidence that RLVR mostly amplifies base-model sampling holds, then the next bottleneck is not more compute but diversity-preserving exploration; methods that manage entropy or generate new tasks should see disproportionate gains.","Editorial inference: the same lifecycle taxonomy could be turned into a benchmark suite that tests one model across pre-training RL, alignment, and reasoning RL under matched compute, which would directly quantify how much each stage contributes."],"forward_implications":["For tasks with automated correctness checks (math, code, formal logic), RLVR is the current high-leverage training recipe, so new work in those domains should start from the GRPO-style family.","For alignment, classic RLHF and preference-optimization methods remain the relevant toolkit; the paper's organization implies alignment and reasoning are distinct lifecycle phases needing different reward setups.","The consolidated benchmark list means researchers can evaluate an RL method across a standard set: math competitions, live code, general knowledge, logic, and agentic multi-turn tasks.","The open-source framework collection makes large-scale RL experimentation reproducible enough for practitioners to run without building infrastructure from scratch.","The cited controversy over whether RLVR expands reasoning beyond the base model implies that reported gains should be interpreted with care, especially at small sample sizes (pass@k)."],"fun_headline_variants":["RL unifies LLM training from pretraining to reasoning","Verifiable rewards: the engine behind LLM reasoning leaps","How RLVR powers the full LLM lifecycle","From pretraining to reasoning: RL's grand tour","RLVR: the key to LLM reasoning and alignment"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the large benchmark gains in Table 1 are actually caused by reinforcement learning and not by simultaneous changes in data, compute, or supervised fine-tuning; if that attribution fails, the survey's central story loses its foundation.","fun_headline_variants_meta":{"raw":{"variants":["RL unifies LLM training from pretraining to reasoning","Verifiable rewards: the engine behind LLM reasoning leaps","How RLVR powers the full LLM lifecycle","From pretraining to reasoning: RL's grand tour","RLVR: the key to LLM reasoning and alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1488,"prompt_tokens":799,"completion_tokens":689,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":543,"tokens_out":689,"duration_ms":6047,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T16:03:47.544464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the base models listed in Table 1 and train it with the same data and compute but no RL—for example, supervised fine-tuning on the RL-generated correct rollouts—then evaluate on AIME2024. If the non-RL run reproduces the reported gain, the survey's central attribution of the improvement to RL is empirically falsified.","supporting_citations":[],"review_version":1}