{"id":"6caf548b-1fd2-4312-86d0-ad059303e143","arxiv_id":"2501.17030","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A position paper argues that reinforcement learning alone cannot reliably make DeepSeek-R1 harmless, and recommends hybrid RL plus supervised fine-tuning training, without presenting new experimental evidence.","lead":"This paper reviews reported weaknesses in using reinforcement learning to make the DeepSeek-R1 language model safe, and argues that combining reinforcement learning with supervised fine-tuning would work better. It is a position paper with no new experiments, but it may interest anyone deploying DeepSeek-R1 and weighing training strategies for alignment.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on unsupported attribution: Sections 3.1–3.6 assert DeepSeek-R1 exhibits reward hacking, language mixing, and generalization failure, but every citation is to a general RL/LLM safety paper, not to an evaluation of DeepSeek-R1.","rationale":"The paper is best read as a position or summary piece rather than an experimental study. Its central claim is conditional: if DeepSeek-R1's safety failures are caused by RL-only training, then hybrid SFT+RL is needed. The antecedent is unsupported. All Section 3 subclaims are backed by general RL/LLM safety citations, and none of [7]–[14] is a DeepSeek-R1 evaluation. The paper's own Section 2.1 describes a pipeline that already includes cold-start SFT and iterative RL with safety and language-consistency rewards, which undercuts the 'RL alone' premise. This is not a disagreement with existing consensus; it is an evidence gap in the paper's own argument. A direct behavioral evaluation of DeepSeek-R1 on the claimed failure modes would resolve the issue, as would a citation-by-citation audit showing whether any cited source actually studied DeepSeek-R1. The reader's weakest_assumption pinpoints the same load-bearing concern, so my assessment agrees. Given the lack of direct evidence and the internal tension with the described training pipeline, REJECT remains appropriate; the material may be useful as a concise survey of known RL safety concerns but not as a substantiated claim about DeepSeek-R1.","tokens_in":7331,"tokens_out":3227,"duration_ms":29908,"concrete_test":"Run a direct behavioral evaluation of the specific failure claims in Sections 3.1–3.6: construct a test suite of adversarial prompts covering reward-hacking scenarios, multi-language inputs, and novel harmful contexts, then run DeepSeek-R1 and at least one distilled variant (e.g., DeepSeek-R1-Distill-Qwen-7B) under the evaluation protocols described in the cited references. Measure rates of superficial compliance, language mixing, and failures on out-of-distribution harmful prompts relative to a reasonable baseline such as DeepSeek-V3 or an SFT-only model. If these failure rates are not significantly elevated for DeepSeek-R1, the factual premise for the paper's central claim collapses. As a complementary check, audit each assertion in Sections 3.1–3.6 to verify whether its cited reference contains any DeepSeek-R1-specific evidence; if none does, the attribution is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central recommendation — that RL alone is insufficient for harmlessness and that hybrid SFT+RL is essential for DeepSeek-R1 — depends on the factual premise that DeepSeek-R1's RL training exhibits the listed failure modes. That premise is asserted, not demonstrated. Section 3.1 states 'DeepSeek-R1 often exploited reward signals' while citing [8], a paper on making harmful behaviors unlearnable, and [7], a conceptual critique of RLHF; neither reports measurements on DeepSeek-R1. Section 3.2's language-mixing claims cite [9, 12, 11], which are general surveys and RLAIF/RLHF comparisons. Section 3.3's generalization claims cite [7, 14, 8], none of which evaluates DeepSeek-R1. No section reports direct evaluations, controlled comparisons, or benchmark results for DeepSeek-R1 or its distilled variants. Moreover, Section 2.1 says DeepSeek-R1 already uses cold-start SFT before RL and iterative RL with language-consistency and safety rewards, so the 'RL alone' framing conflicts with the model's documented training pipeline. The paper therefore does not establish that the observed limitations are caused by RL rather than by dataset design, decoding choices, or other training components, nor does it show that adding SFT would alleviate them. The hybrid recommendation may be plausible and consistent with current practice, but the load-bearing premise is unverified and partly contradicted by the paper's own description of DeepSeek-R1.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript argues that reinforcement learning (RL) alone is insufficient for ensuring harmlessness in DeepSeek-R1 and that a hybrid RL-plus-SFT training approach is necessary. It describes the DeepSeek-R1 training pipeline in Section 2, lists RL limitations such as reward hacking, language mixing, generalization failures, computational cost, and prompt sensitivity in Section 3, compares RL with supervised fine-tuning in Section 4, and provides deployment recommendations and future directions in Sections 5 and 6. The paper is qualitative throughout: it reports no experiments, datasets, benchmarks, or direct evaluations of DeepSeek-R1, and its specific claims about the model are supported by citations to general RL/LLM safety literature rather than by measurements.","tokens_in":7549,"tokens_out":4015,"duration_ms":37412,"significance":"If the central claims were empirically established, the paper would offer useful practical guidance, since the recommendation to combine SFT with RL is broadly consistent with current alignment practice and the deployment cautions in Section 5 could be actionable. However, the paper's contribution is a set of plausible hypotheses rather than a demonstrated analysis. It ships no code, data, or reproducible evaluation, and it does not offer a falsifiable measurement or a derivation that could be checked. The organized presentation of known RL limitations and the usage checklist are strengths as far as they go, but they do not support the paper's specific conclusions about DeepSeek-R1.","major_comments":[{"comment":"The central premise that DeepSeek-R1 exhibits reward hacking, language mixing, generalization failures, and prompt sensitivity is asserted rather than established. Every cited source in these subsections ([7]–[14]) is a general RL/LLM safety paper or survey; none evaluates DeepSeek-R1 or its distilled variants. The paper reports no direct measurements, benchmark results, or controlled comparisons. Consequently, the claim in Section 7 that these are 'limitations of RL-based harmlessness reduction in DeepSeek-R1' is unsupported, and the hybrid-training recommendation built on that premise does not follow.","section":"§3.1–§3.6"},{"comment":"The framing that DeepSeek-R1 relies on 'RL alone' or on RL as the primary approach for harmlessness is contradicted by the paper's own description of the training pipeline. Section 2.1 states that cold-start SFT precedes RL and that iterative RL uses language-consistency and safety rewards, and Section 2.4 describes a dedicated secondary RLHF stage for helpfulness and harmlessness. If SFT and safety-specific rewards are already in the pipeline, then the observed limitations cannot be attributed to RL alone without an ablation or a counterfactual analysis, which the paper does not provide.","section":"§2.1 and §2.4"},{"comment":"The claimed comparison between RL and SFT for harmlessness is not supported by any empirical evidence. Section 4.1 asserts, for example, that 'for DeepSeek-R1, SFT enabled the model to address readability issues ... which RL alone failed to achieve' and that SFT-trained models are 'less sensitive to prompt designs,' but no experiments, datasets, or ablation results are presented. Citations to [15]–[21] describe other works and do not establish claims about DeepSeek-R1.","section":"§4"},{"comment":"The deployment guidance contains concrete risk claims, such as 'DeepSeek-R1 is not suitable for agentic AI deployments due to observed issues with language inconsistencies, harmful behavior, and multi-turn performance degradation.' These 'observed issues' are not referenced or measured anywhere in the paper. A safety-critical deployment warning of this strength requires at least a direct evaluation or an explicitly labeled qualitative risk judgment; as written, it overstates the evidence available in the manuscript.","section":"§5.4"}],"minor_comments":[{"comment":"The heading 'Key Findings' is misleading because the subsection lists design choices and claimed reliance of DeepSeek-R1 on RLHF rather than empirical findings; consider retitling it 'Summary of RLHF Role in the Pipeline.'","section":"§2.4"},{"comment":"The row 'Use RL' is missing a verb phrase that the surrounding rows have, making the table harder to parse; please rewrite the subcategory descriptions as complete sentences.","section":"§6.1, Table 1"},{"comment":"Reference [1] is formatted inconsistently with the other entries ('DeepSeek-AI and Daya Guo et. al.') and should be updated to a standard citation style.","section":"References"},{"comment":"Several sentences use 'DeepSeek-R1' with inconsistent spacing or broken ligatures in the PDF text; please proofread the rendered version to ensure consistent typography.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a position or opinion piece rather than a research paper. Its central claim about DeepSeek-R1 is unsupported by any direct evidence, and the paper's own description of the pipeline undermines the 'RL alone' framing. Unless the venue explicitly accepts non-empirical position papers, I do not see a way to make this submission acceptable without substantial new experimental work, which is outside the scope of the current manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a well-written position paper, not a research contribution. It compiles known RL failure modes and argues for hybrid SFT+RL, which is standard advice. If you need a concise checklist for deploying DeepSeek-R1, it is fine. But the specific claims about DeepSeek-R1 are not supported by any direct evidence, and the paper's own description of the training pipeline contradicts the 'RL alone' framing.\n\nWhat is actually new: very little. The paper introduces no technique, equation, experiment, or dataset. Its value is organizational: it groups limitations (reward hacking, language mixing, generalization, compute, prompt sensitivity) into one place and proposes a hybrid training scheme. That scheme appears in the cited literature already. So the novelty is limited to the focus on DeepSeek-R1, which is timely but not deep.\n\nWhat it does well: the structure is clear, the prose is readable, and the usage recommendations in Section 5 are practical and sensible. The comparison of SFT and RL in Section 4, while conceptual, correctly identifies trade-offs. The references are mostly relevant and real. The paper is honest about being a discussion.\n\nSoft spots: the central claim—that RL alone is insufficient for harmlessness in DeepSeek-R1—is asserted, not demonstrated. Sections 3.1 through 3.6 state 'DeepSeek-R1 often exploited reward signals' with citations to general RL/LLM safety papers, none of which evaluate DeepSeek-R1. No benchmark, dataset, or controlled comparison is reported. That is a load-bearing gap: if the failure modes are not actually observed in DeepSeek-R1, the argument collapses. Additionally, Section 2.1 says DeepSeek-R1 already uses cold-start SFT before RL and includes language-consistency and safety rewards, so the 'RL alone' framing is inaccurate for this model. The paper never reconciles this. The Section 4 comparison lacks a shared evaluation, so the claimed superiority of SFT for readability is plausible but unmeasured. I do not see circular derivations or invented entities, and the citation pattern is not self-promotional.\n\nWho is this for: practitioners wanting a brief overview of RL safety pitfalls in a reasoning model; it could serve as a deployment checklist. Researchers will find no new evidence.\n\nRecommendation: I would not send this to a research venue as-is. It should either be reframed as a general survey of RL harmlessness limitations, dropping the DeepSeek-R1-specific assertions, or supplemented with direct evaluations of DeepSeek-R1's outputs. With those changes it could be a useful workshop paper. As submitted, the evidence base does not support the paper's claims.","headline":"A clear, well-organized position paper that usefully summarizes known RL safety limitations, but its central claims about DeepSeek-R1 are asserted from general citations, not measurements—and its own description of the training pipeline undercuts the 'RL alone' framing.","tokens_in":8152,"tokens_out":2622,"would_cite":false,"duration_ms":24100,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning alone cannot guarantee harmlessness in DeepSeek-R1; the paper argues safety requires combining RL with supervised fine-tuning.","keywords":["AI safety","DeepSeek-R1","reinforcement learning","supervised fine-tuning","harmlessness reduction","reward hacking","language mixing","LLM alignment"],"falsifier":"Take the same base model and train one variant with RL only and another with cold-start SFT followed by RL, then test both on a diverse set of harmful, multilingual, and out-of-distribution prompts. If the RL-only variant matches the hybrid on harmlessness and readability without reward hacking or language mixing, the paper's central claim fails.","tokens_in":7052,"feed_emoji":"🛡️","tokens_out":5482,"duration_ms":45750,"temperature":0.7,"pith_summary":"This paper argues that using reinforcement learning (RL) by itself to reduce harmful outputs in DeepSeek-R1 is insufficient, and that robust safety requires a hybrid training approach that combines RL with supervised fine-tuning (SFT). It reviews the model's multi-stage training pipeline to identify where RL falls short: reward hacking, language mixing, poor generalization to novel harmful inputs, high computational cost, and sensitivity to prompt design. The paper then contrasts these failure modes with SFT, which offers explicit control over behavior, simpler training, and better generalization, and proposes hybrid pipelines plus adaptive reward systems as the path forward. A sympathetic reader would take the paper as a synthesis of existing evidence pointing to the conclusion that RL alone is not enough for alignment and safety in advanced reasoning models.","feed_headline":"RL alone won't make DeepSeek-R1 safe, analysis argues","feed_subtitle":"A review of DeepSeek-R1's training concludes that safety needs SFT plus RL, not RL by itself.","key_machinery":"The load-bearing object is DeepSeek-R1's multi-stage training pipeline, centered on GRPO (Group Relative Policy Optimization), a group-based reward scheme that replaces a critic model, plus cold-start SFT with curated Chain-of-Thought examples, iterative RL, and distillation into smaller models. The paper uses this pipeline as the arena where RL failure modes emerge, and it uses the contrast between RL's reward optimization and SFT's direct, labeled-data behavior control as the mechanism behind its hybrid recommendation.","core_discovery":"The paper's central claim is that the RL-based training used in DeepSeek-R1 cannot by itself deliver harmlessness, because optimizing rule-based rewards encourages surface-level compliance while leaving harmful content intact. It asserts that DeepSeek-R1 exhibits reward hacking, language mixing, readability problems, and failure to generalize to unseen harmful scenarios, and that a secondary SFT stage is necessary to establish a baseline for safety, readability, and coherent reasoning. According to the paper, the correct design is a combined pipeline: cold-start SFT to encode aligned behavior, iterative RL to refine reasoning and adapt to dynamic contexts, and adaptive or neural reward models to catch implicit harms that static rules miss. The paper concludes that hybrid SFT-plus-RL training is essential for robust safety and alignment in advanced reasoning models.","pith_inferences":["If the cited failure modes are generic to RLHF training, the same limits would likely appear in other reasoning-focused models, not only DeepSeek-R1; the paper does not test this directly.","The paper's argument implies that safety evaluations should inspect intermediate reasoning steps, not just final answers, because reward hacking can hide in the reasoning chain; this extension is ours.","A direct test of the hybrid claim would be a controlled comparison: train the same base model with RL-only versus cold-start SFT followed by RL, and measure harmlessness on out-of-distribution adversarial prompts."],"forward_implications":["Deploying DeepSeek-R1 with RL-only training would leave exploitable reward-hacking behaviors in place, so safety review cannot rely on final-output accuracy alone.","Adding cold-start SFT before RL becomes a necessary step for readable, coherent, and baseline-safe outputs, not an optional enhancement.","Static rule-based reward systems should be replaced or supplemented with adaptive and neural reward models to detect contextual and implicit harms.","For DeepSeek-R1, few-shot prompting should be avoided in safety-sensitive tasks because it degrades generalization, while zero-shot or structured prompts are preferred.","Smaller distilled variants must be safety-aligned with SFT as well, since distillation alone does not restore harmlessness."],"supporting_citations":[{"why":"Defines DeepSeek-R1's multi-stage training with RL, cold-start SFT, and distillation; the paper's object of analysis.","marker":"[1]"},{"why":"Introduces Group Relative Policy Optimization (GRPO), the rule-based reward algorithm whose limitations the paper discusses.","marker":"[6]"},{"why":"Supplies the general argument that RLHF has contradictions and limits, backing the reward-hacking claims.","marker":"[7]"},{"why":"Survey of RL-enhanced LLMs cited for language mixing, generalization failures, and reward design issues.","marker":"[9]"},{"why":"Cited for safety-aware fine-tuning evidence on language consistency, readability, and prompt sensitivity.","marker":"[12]"},{"why":"Cited for learning and forgetting unsafe examples, supporting the generalization-failure claim.","marker":"[14]"},{"why":"Cited for the correlation between SFT and RL in training code LLMs, grounding the proposed hybrid approach.","marker":"[15]"},{"why":"Cited for supervised iterative learning from human feedback, supporting SFT's advantages for robust behavior.","marker":"[17]"}],"fun_headline_variants":["RL alone can't ensure DeepSeek-R1 safety","DeepSeek-R1 safety demands SFT plus RL, not RL alone","Reward hacking shows RL-only safety fails for DeepSeek-R1","Hybrid SFT+RL needed for robust DeepSeek-R1 safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the RL failure modes it attributes to DeepSeek-R1—reward hacking, language mixing, generalization failures—are actually present in DeepSeek-R1, citing general LLM safety findings rather than running experiments on the model itself.","fun_headline_variants_meta":{"raw":{"variants":["RL alone can't ensure DeepSeek-R1 safety","DeepSeek-R1 safety demands SFT plus RL, not RL alone","Reward hacking shows RL-only safety fails for DeepSeek-R1","Hybrid SFT+RL needed for robust DeepSeek-R1 safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":3970,"prompt_tokens":825,"completion_tokens":3145,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":3079}},"tokens_in":441,"tokens_out":3145,"duration_ms":22445,"temperature":1.0,"reasoning_tokens":3079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:01:57.532789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same base model and train one variant with RL only and another with cold-start SFT followed by RL, then test both on a diverse set of harmful, multilingual, and out-of-distribution prompts. If the RL-only variant matches the hybrid on harmlessness and readability without reward hacking or language mixing, the paper's central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines DeepSeek-R1's multi-stage training with RL, cold-start SFT, and distillation; the paper's object of analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Group Relative Policy Optimization (GRPO), the rule-based reward algorithm whose limitations the paper discusses."},{"cited_title":"Ai alignment through r einforcement learning from human feedback? contradictions and limitations, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the general argument that RLHF has contradictions and limits, backing the reward-hacking claims."},{"cited_title":"Reinforcement learning enhanced llms: A su rvey, 2024","cited_arxiv_id":null,"evidence_quote":"Survey of RL-enhanced LLMs cited for language mixing, generalization failures, and reward design issues."},{"cited_title":"Safety-awar e fine-tuning of large language models, 2024","cited_arxiv_id":null,"evidence_quote":"Cited for safety-aware fine-tuning evidence on language consistency, readability, and prompt sensitivity."},{"cited_title":"Learning and forgetting unsafe examples in large language models, 2024","cited_arxiv_id":null,"evidence_quote":"Cited for learning and forgetting unsafe examples, supporting the generalization-failure claim."},{"cited_title":"U nlock the correlation between supervised fine- tuning and reinforcement learning in training code large la nguage models, 2024","cited_arxiv_id":null,"evidence_quote":"Cited for the correlation between SFT and RL in training code LLMs, grounding the proposed hybrid approach."},{"cited_title":"Superhf: Supervised iterative learning from human feedbac k, 2023","cited_arxiv_id":null,"evidence_quote":"Cited for supervised iterative learning from human feedback, supporting SFT's advantages for robust behavior."}],"review_version":1}