{"id":"a0007540-2e2c-4643-a011-cabbd05d3fd4","arxiv_id":"2412.10400","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that catalogs RL-enhanced LLMs and organizes alignment methods into RLHF, RLAIF, and DPO, without presenting new results.","lead":"This paper surveys how reinforcement learning is used to train large language models, covering reward-based methods like RLHF and RLAIF as well as direct preference optimization (DPO). It compiles summaries of many existing models and methods, but contributes no new experiments or theory.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The survey's gap-filling claim is undercut by taxonomy-level errors: Section 5.2 includes LLM-for-RL reward-shaping methods as if they were RLAIF for LLMs, and HelpSteer2 is mislabeled as AI feedback.","rationale":"The reader's weakest assumption singled out the accuracy of one-sentence paraphrases across Sections 3 to 8. My stress-test finds that concern lands, but in a sharper form: the problem is not only possibly inaccurate summaries but observable category errors in the survey's organizing taxonomy. Section 5.2 presents ELLM, RDLM, Eureka, and Text2Reward as RLAIF methods for language models, yet the cited papers use LLMs to shape rewards or goals for non-LLM RL agents; these are 'LLMs for RL' rather than 'RL for LLMs.' Similarly, HelpSteer2 is placed under AI-feedback distillation even though it is a human-annotated reward-modeling dataset. These are not merely stylistic choices: a survey whose central justification is filling a gap and helping researchers navigate the field must at least place each cited method in the correct category. The prior-survey omission identified by the reader is real, but it is less decisive than internal misclassification, because novelty is secondary to utility for a survey; a survey can still be useful even if predecessors exist, whereas a survey that mislocates core methods cannot be relied on for orientation. I also note the paper does contain useful coverage and many accurate descriptions, so the appropriate response is correction and tightening rather than rejection. The reader's CONDITIONAL verdict already accommodates exactly this kind of revision, so I recommend no change to the verdict. My disagreement is only partial because I would shift the emphasis from generic paraphrase risk to the specific, checkable category errors in Section 5.","tokens_in":46427,"tokens_out":3950,"duration_ms":39702,"concrete_test":"Verify a stratified sample of the Section 5 entries against primary sources. Specifically, check the four Section 5.2 entries (ELLM, RDLM, Eureka, Text2Reward) and HelpSteer2: for each, record (a) whether the trained policy is an LLM and (b) whether the feedback signal is AI-generated. If ELLM/RDLM/Eureka/Text2Reward train non-LLM RL agents, or HelpSteer2 uses human annotations, remove them from the RLAIF taxonomy and recount the number of genuine RLAIF-for-LLM methods in Sections 5.2 and 5.3. If the count drops by more than a third, or if the paper's own definition of RLAIF no longer covers the remaining entries, the survey's systematic-review claim is materially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's motivating claim—that no comprehensive survey of RL-enhanced LLMs existed—requires this paper to be an accurate, coherent consolidation of that literature. The weakest load-bearing point is not just that prior surveys were ignored; it is that several core entries are misclassified at the taxonomic level, so a reader using the survey to understand RLAIF would receive a distorted picture. In Section 5.2, ELLM (Du et al., 2023) uses an LLM to propose exploration goals for a separate RL agent in Crafter and Housekeep; Eureka (Ma et al., 2023) and Text2Reward (Xie et al., 2023) generate reward-function code for robot RL tasks; RDLM (Kwon et al., 2023) prompts GPT-3 to produce reward values for RL agents. None of these trains an LLM policy with RL from AI feedback, and none belongs under the paper's stated object of RL-enhanced LLMs. Their inclusion inflates the RLAIF section and conflates 'LLMs as reward designers' with 'RL for LLMs.' Separately, Section 5.1 classifies HelpSteer2 (Wang et al., 2024d) under 'Distilling AI Feedback to Train Reward Model,' although HelpSteer2 is a human-annotated preference dataset; the original paper describes human ratings, not AI-generated labels. These are correctable factual and category errors, but they are exactly the kind of unverified summarization the reader flagged, and they directly weaken the claim that the survey provides a reliable systematic review.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a survey of reinforcement learning (RL) techniques applied to large language models (LLMs). It covers RL basics, a set of popular RL-enhanced LLMs (DeepSeek-R1, Kimi-k1.5, InstructGPT, GPT-4, Gemini, among others), the RLHF pipeline, RLAIF, DPO and its variants, and analyses of challenges such as reward-model out-of-distribution robustness, interpretability, safety, and evaluation. The authors claim to fill a gap left by the absence of a comprehensive survey on RL-enhanced LLMs, and they organize the material by technique and by model, with numerous tables and figures borrowed from primary sources.","tokens_in":46760,"tokens_out":6070,"duration_ms":53858,"significance":"The topic is timely, and the scope is appropriate for a survey: RLHF, RLAIF, and DPO are currently the dominant alignment paradigms, so a reliable consolidation would be valuable to practitioners and newcomers. The paper does not ship code or machine-checked proofs, but it does provide a broad model-level overview (e.g., the multi-stage training pipeline of DeepSeek-R1), a comparison table of models, and discussions of open problems. However, the survey's usefulness depends entirely on accurate classification and faithful summarization of primary sources. In its current form, several core entries are misclassified (Section 5.2) or factually wrong (Table 1, Section 5.1), which compromises the reliability of the consolidation. If these issues are fixed, the survey could serve as a useful entry point; as written, the taxonomic errors are load-bearing because the paper's stated contribution is accurate organization of the field.","major_comments":[{"comment":"The subsection titled 'Prompting LLMs As a Reward Function' sits under 'RLAIF: Reinforcement Learning from AI Feedback' (Section 5), whose opening defines RLAIF as using AI systems 'to provide feedback on the outputs of the LLM being trained.' ELLM (Du et al., 2023) uses an LLM to propose exploration goals for a separate RL agent in Crafter and Housekeep; Eureka (Ma et al., 2023) generates reward-function code for robot RL; RDLM (Kwon et al., 2023) prompts GPT-3 to output reward values for RL agents; and Text2Reward (Xie et al., 2023) generates reward code for RL tasks. None of these methods trains an LLM policy with RL from AI feedback; they are instances of LLMs as reward designers for non-LLM RL agents. Including them in the RLAIF category conflates 'LLMs for RL' with 'RL for LLMs' and would mislead a reader about the RLAIF literature. These entries should be moved to a separate category or explicitly marked as LLM-for-RL methods rather than RL-enhanced-LLM methods.","section":"§5.2"},{"comment":"HelpSteer2 (Wang et al., 2024d) is presented under 'Distilling AI Feedback to Train Reward Model,' with the text saying 'Preference pairs are crafted based on human or automated evaluations.' The original HelpSteer2 paper describes the dataset as constructed from human ratings: annotators rated responses along dimensions such as helpfulness, correctness, coherence, complexity, and verbosity, and those human labels were used to train reward models. There is no AI-generated feedback in the HelpSteer2 construction. Classifying it under AI feedback is a factual misrepresentation of the cited work and gives readers an incorrect picture of the dataset's provenance.","section":"§5.1"},{"comment":"The Zephyr entry lists '141B-A39B' (39B active parameters), and Section 3.8 is titled 'Zephyr 141B-A39B.' However, the cited model card (HuggingFaceH4, 2024, 'zephyr-orpo-141b-A35b-v0.1') states that the model has 141B total and 35B active parameters. The survey is internally consistent but factually wrong relative to the primary source. Since Table 1 is a key reference for readers, this error should be corrected to A35B.","section":"Table 1 and §3.8"},{"comment":"The motivating claim that 'the absence of a comprehensive survey summarizing existing research on RL-enhanced LLMs has limited progress' is stated without engagement with existing surveys of RLHF and preference optimization (e.g., Casper et al., 2023, 'Open Problems and Fundamental Limitations of RLHF,' and the several RLHF surveys published in 2023–2024). The manuscript has no related-work section comparing its scope and coverage with prior surveys, so the novelty claim is unsubstantiated. The authors should either qualify the claim (e.g., 'a survey covering RLHF, RLAIF, and DPO jointly') or cite and differentiate the prior surveys.","section":"Abstract and §1"}],"minor_comments":[{"comment":"The abstract contains the typo 'Depsite' for 'Despite', and the sentence 'Project page of this work can be found at our latest repo' should give the actual URL found later in the full text.","section":"Abstract"},{"comment":"The sentence 'the agent receives a reward tt' should read 'a reward rt', matching the notation of the preceding paragraph.","section":"§2.1.2"},{"comment":"The introduction cites a large number of tangential works (e.g., text classification, sentiment analysis, GPT-NER), several of which are the authors' own papers and do not concern RL-enhanced LLMs; this inflates the reference list and should be trimmed or moved to a less prominent position.","section":"§1"},{"comment":"The Magpie entry describes an instruction-synthesis dataset rather than an RLAIF reward-model training method; if retained, its relevance to 'Distilling AI Feedback to Train Reward Model' should be clarified.","section":"§5.1"},{"comment":"The entry for Quark is described as equipping reward models with unlearning mechanisms, but the original paper is a text-generation method that uses a reward function to guide conditional generation; the description should be aligned with the source terminology.","section":"§6.3"},{"comment":"The sentence 'SPPO reformulates language model optimization as a constant-sum two-player game' is slightly ambiguous; the original method is a two-player constant-sum game formulation of preference optimization, and the phrasing should be adjusted for accuracy.","section":"§8.2"},{"comment":"The claim of 'up to 550% improvement over existing models such as GPT-4o and Claude 3.5' should be stated relative to which baseline and with the exact metric, to avoid overstatement.","section":"§3.2"},{"comment":"Several figures (e.g., Figures 3, 6, 7) contain lengthy verbatim text from the original papers embedded as images; replacing these with original schematic summaries would avoid copyright concerns and would make the survey's expository voice more consistent.","section":"Figures"},{"comment":"The conclusion is a single paragraph that only restates the paper structure; a bulleted summary of the main open problems and the survey's limitations would strengthen the closing.","section":"§9"}],"recommendation":"major_revision","confidential_remarks":"The survey's introduction cites a large number of the authors' own prior papers that are not about RL-enhanced LLMs (e.g., Wang et al., 2023a; 2023b; Sun et al., 2023b; 2023c; 2023d; Zhang et al., 2023b); while self-citation is not disqualifying, the volume here is disproportionate and should be pruned. The manuscript also reproduces large excerpts from other papers as figures, which the editor may wish to check for copyright compliance under the journal's policy. Finally, the paper would benefit from an explicit comparison with existing RLHF surveys so that the claimed gap is demonstrated rather than asserted. The taxonomic errors in Sections 5.1 and 5.2 are factually correctable and do not require new experiments, which is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a broad but uneven survey. It gives a newcomer a quick map of RLHF, RLAIF, and DPO, plus the recent reasoning-model line-up, but it is not reliable enough to be the systematic review it claims to be.\n\nWhat it does well: the coverage is current and genuinely wide. Section 3 on RL-enhanced LLMs includes DeepSeek-R1 and Kimi-k1.5, and the DPO-variant round-up in Section 7 is reasonably up to date. I don't know another single place that lists these post-training recipes side by side. The RL basics in Section 2 are accessible, and the high-level pipeline descriptions are fine for someone entering the area.\n\nNow the soft spots, which are load-bearing rather than cosmetic. Section 5.2, under the RLAIF umbrella, includes ELLM, Eureka, Text2Reward, and RDLM. Those methods use LLMs as reward designers for RL agents in games and robotics; they are not reinforcement learning from AI feedback for LLMs. Including them inflates the RLAIF section and blurs the distinction between 'LLMs in the RL loop' and 'RL for LLMs.' Separately, HelpSteer2 is listed under 'Distilling AI Feedback to Train Reward Model,' but HelpSteer2 is a human-annotated preference dataset, not AI feedback. These are correctable errors, but they matter: a reader relying on this survey to understand RLAIF will come away with a distorted picture.\n\nThere are also smaller but real accuracy issues. Table 1 lists Zephyr as 141B-A39B; the actual model is A35B. And the abstract's claim that no comprehensive survey exists ignores prior RLHF surveys; without a related-work section benchmarking against them, the gap-filling assertion is unsupported. The heavy cluster of self-citations in the introduction is not disqualifying, but it doesn't help.\n\nThe survey introduces no new methods or measurements, so its value is organizational. That can still be real value, but the organizing has to be faithful to the sources. Right now, several one-sentence paraphrases are precisely the kind of unverified summaries the field doesn't need more of.\n\nProportionately, this is not a hopeless paper. It is a solid draft with a few wrong entries. With a careful revision—fix the taxonomy, correct the numbers, add related work—it could serve as a useful entry point for newcomers.\n\nMy recommendation: send it to peer review with major revision. The demand for such a survey is real and the coverage is useful, but it should not be accepted as is.","headline":"Broad but uneven survey: useful orientation for newcomers, yet the RLAIF taxonomy contains real misclassifications and the gap-filling claim is overstated.","tokens_in":47262,"tokens_out":2455,"would_cite":false,"duration_ms":25135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new survey organizes RL-enhanced LLMs into two competing alignment families: reward-model-based RLHF/RLAIF and reward-free DPO.","keywords":["reinforcement learning","large language models","RLHF","RLAIF","direct preference optimization","PPO","model alignment","survey"],"falsifier":"Take any model row in Table 1 and check the official technical report for the stated post-training algorithm; if a model attributed to DPO or RLHF actually used a different preference-optimization method, the survey's organizing taxonomy misrepresents the literature. A systematic mismatch across several rows would show the consolidation is unreliable.","tokens_in":46251,"feed_emoji":"🤖","tokens_out":6439,"duration_ms":56982,"temperature":0.7,"pith_summary":"This survey sets out to consolidate the fast-growing literature on reinforcement learning (RL) as a post-training step for large language models (LLMs), motivated by the success of models like DeepSeek-R1. Its central claim is that RL-enhanced training has become a standard ingredient in state-of-the-art LLMs, and that the field is best understood as two families: reward-model-based methods (RLHF and RLAIF) and reward-model-free preference optimization led by DPO. The paper argues that the field's complexity and the absence of a comprehensive survey have slowed systematic understanding, and it responds by organizing the basics of RL, the RL-enhanced models themselves, the main alignment techniques, and the current challenges. A sympathetic reader would take the paper's contribution to be a map of the territory that helps researchers locate methods and open problems rather than a new algorithm.","feed_headline":"Survey maps the RL methods behind GPT-4, DeepSeek-R1, and Llama 3","feed_subtitle":"Reward-based RLHF and reward-free DPO are the two paths surveyed for aligning models with human preferences.","key_machinery":"The organizing device is the mapping of an LLM into an RL agent, following the InstructGPT framework: the model is the policy, the token sequence is the state, the next token is the action, and a trained reward model supplies the scalar reward. On top of this mapping the survey builds its main taxonomy — reward-model-based alignment (RLHF, RLAIF) versus reward-model-free preference optimization (DPO and its variants) — and uses that split to structure the literature. The taxonomy does the work of the argument: it turns a scattered set of technical reports and method papers into two coherent research programs with distinct costs, failure modes, and open problems.","core_discovery":"The paper's contribution is a systematic organization of how RL is applied to LLMs. It identifies a canonical three-step pipeline — training a reward model, scoring generated responses, and optimizing the policy with an algorithm such as PPO — and then surveys two routes built on it: RLHF, which uses human preference labels, and RLAIF, which substitutes AI feedback for scalability. It then treats Direct Preference Optimization and its variants as a second broad route that eliminates the reward model entirely by framing alignment as a classification-style objective over preference pairs. The survey also catalogs RL-enhanced production models, from InstructGPT and GPT-4 to DeepSeek-R1, Kimi-k1.5, Llama 3, and Qwen2, and collects the main failure modes — out-of-distribution reward models, poor interpretability, reward hacking, and safety concerns — alongside evaluation resources such as RewardBench. The overall claim is that the rapidly growing body of work can be consolidated into this taxonomy, and that doing so clarifies both current challenges and promising directions.","pith_inferences":["If reward-model-free methods keep closing the gap with PPO-based RLHF, the field may converge on hybrid pipelines where DPO-style losses handle most alignment and small online RL stages handle targeted skills; that synthesis is implied by the survey's structure but not stated by the authors.","The survey's reliance on single-sentence summaries of dozens of primary papers means a reader should treat its method attributions (for example, which model used DPO versus RLHF) as pointers to verify against the original technical reports rather than as verified facts.","A testable extension would be to turn Table 1 into a living benchmark that tracks which RL method each released model actually used, since the taxonomy's usefulness degrades as models adopt mixed pipelines like iterative DPO plus online RL.","The boundary between RLAIF and self-rewarding methods suggests a possible convergence: if a model can judge its own outputs well enough, the reward model and the policy may eventually be the same network, which would collapse the survey's two families into one."],"forward_implications":["If the taxonomy holds, the practical choice for aligning a new model is between an expensive reward-model pipeline (RLHF/RLAIF with PPO-style optimization) and a cheaper DPO-style direct preference loss.","The prominence of DeepSeek-R1, Kimi-k1.5, and o1 implies that RL through long chain-of-thought is a mainstream route to reasoning capability, not a one-off experiment.","RLAIF's reliance on AI-generated feedback implies that alignment can scale beyond human annotation, with consistency and cost advantages that the survey claims.","The failure modes the survey collects — OOD reward models, reward hacking, verbosity bias, and safety trade-offs — define a checklist that any RL-enhancement method must address.","The survey's DPO section implies that the reward model may be an implementation detail rather than a conceptual requirement for preference alignment."],"supporting_citations":[{"why":"Supplies the canonical RLHF pipeline — reward model plus PPO — that the survey uses as the baseline framework for RL-enhanced LLMs.","marker":"Ouyang et al., 2022"},{"why":"The motivating state-of-the-art example; its pure-RL reasoning training (DeepSeek-R1) anchors the survey's claim that RL is a standard post-training ingredient.","marker":"DeepSeek-AI et al., 2025"},{"why":"Introduces DPO, the reward-model-free approach whose variants occupy a major section of the survey.","marker":"Rafailov et al., 2024"},{"why":"Provides PPO, the policy optimization algorithm underlying most RLHF/RLAIF pipelines described in the survey.","marker":"Schulman et al., 2017"},{"why":"Constitutional AI is the basis for the RLAIF discussion and for Claude 3's alignment approach as summarized in the survey.","marker":"Bai et al., 2022"},{"why":"Formalizes RLAIF and direct-RLAIF, the AI-feedback route the survey contrasts with RLHF.","marker":"Lee et al., 2023"},{"why":"Self-Rewarding Language Models anchors the survey's self-rewarding subsection, where the model acts as its own reward model.","marker":"Yuan et al., 2024"},{"why":"RewardBench is the evaluation benchmark the survey uses to frame reward-model assessment.","marker":"Lambert et al., 2024"}],"fun_headline_variants":["RL for LLMs: from DeepSeek-R1 to DPO","Survey maps RLHF, RLAIF, and DPO for LLMs","How RL trains LLMs: a survey of techniques","Two routes to align LLMs: reward models or DPO","RL-enhanced LLMs, systematically surveyed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's value depends on the reliability of its compact paraphrases of dozens of cited papers, since it does not itself verify the methods or numbers it reports.","fun_headline_variants_meta":{"raw":{"variants":["RL for LLMs: from DeepSeek-R1 to DPO","Survey maps RLHF, RLAIF, and DPO for LLMs","How RL trains LLMs: a survey of techniques","Two routes to align LLMs: reward models or DPO","RL-enhanced LLMs, systematically surveyed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1723,"prompt_tokens":1031,"completion_tokens":692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":608}},"tokens_in":647,"tokens_out":692,"duration_ms":7448,"temperature":1.0,"reasoning_tokens":608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:32:06.055660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any model row in Table 1 and check the official technical report for the stated post-training algorithm; if a model attributed to DPO or RLHF actually used a different preference-optimization method, the survey's organizing taxonomy misrepresents the literature. A systematic mismatch across several rows would show the consolidation is unreliable.","supporting_citations":[],"review_version":1}