{"id":"4baae4cc-f1a1-4ed7-a7ba-67d061d0fc3b","arxiv_id":"2508.12387","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ReaLM trains small language models to learn from both right and wrong reasoning chains, then fades the chains out so the model reasons independently, improving benchmark accuracy.","lead":"A ByteDance team proposes ReaLM, a reinforcement learning recipe that trains small language models to reason using both correct and incorrect example reasoning chains, then gradually removes those examples so the model reasons alone. The method reports consistent accuracy gains on math, factual, and reasoning benchmarks, plus a proprietary ad relevance task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EAAI's contribution to autonomy is not isolated: no control trains with a constant CoT rate and evaluates without CoTs, so the 2.6% no-CoT gain could come from CoT-augmented RL rather than from the fade schedule.","rationale":"The empirical core is coherent, and the no-CoT results in Tables 1 and 3 show consistent gains over RL with the same base models. The weakest point is the causal attribution to EAAI. The paper frames EAAI as enabling autonomy via asymptotic induction, but the evidence compares conditions that differ in two ways simultaneously: training-time CoT availability and inference-time CoT availability. A control with constant CoT availability during training and no CoTs at inference would separate the fade schedule from CoT-augmented RL; this is a standard curriculum-learning control and it is missing. I would not reject on this basis because the no-CoT inference condition is still meaningful and the gains replicate across two base models. The Appendix C proof is not load-bearing for the empirical claim but should be fixed or removed, and the footnote/appendix mismatch on whether external CoTs contain final answers is a real but secondary concern that mainly affects the with-CoT comparison. The reader's CONDITIONAL verdict is appropriate; the EAAI concern reinforces it rather than changing it.","tokens_in":22179,"tokens_out":10180,"duration_ms":110772,"concrete_test":"Train a control variant of ReaLM in which external CoTs are provided during training at a fixed 50% inclusion probability with no cosine decay, then evaluate it at inference with no external CoTs on GSM8K and GSM8KPlus; compare against the RL baseline and against ReaLM with EAAI. If the fixed-rate control matches or exceeds ReaLM, the fade schedule is not the cause of the no-CoT gains; if it clearly underperforms, EAAI is doing causal work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central autonomy claim is that EAAI's cosine-decay schedule makes the SLM internalize external reasoning, so inference without external CoTs retains most of the benefit. The evidence does not isolate this mechanism. Table 3 compares MRPV+EAAI (evaluated without CoTs) to RL (trained and evaluated without CoTs), but no condition trains with CoTs at a constant rate and then evaluates without them. Without that control, the 2.6% average gain over RL in Table 1 could simply come from CoT-augmented RL training, with the fade schedule contributing little or nothing. The internalization claim also lacks a dynamic check: no checkpoint evaluation shows whether the with-CoT vs. without-CoT accuracy gap shrinks as the threshold decays. Appendix C cannot fill this gap: Eqs. (13)-(19) assume the CoT embeddings are complementary/negatively correlated and that the reward Hessian at the optimum makes cross terms nonnegative; that is the very property MRPV is supposed to learn, not an a priori guarantee, so the proof is circular at the load-bearing point. Additionally, Footnote 3 states external CoTs are 'without final answers,' but Appendix Tables A5/A6 show CoT responses ending in '#### 8'/'#### 9'; if those examples are representative, the with-CoT results include answer leakage, which would make the 4.2% gain partly a selection effect. The 2.6% no-CoT result is less affected, so the overall conclusion is conditionally supported, but the autonomous-reasoning mechanism remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReaLM, a reinforcement learning framework for small language models (SLMs) that combines three components: Multi-Route Process Verification (MRPV), which contrasts positive and negative external chain-of-thought (CoT) paths; Enabling Autonomy via Asymptotic Induction (EAAI), which fades the external CoT input over training; and guided CoT distillation with an industrial benchmark for vertical-domain generalization. The main empirical claim is that ReaLM improves SLM reasoning by 4.2% on average when external CoTs are available at inference and by 2.6% when they are not, while removing the need for external CoTs at test time. Experiments are conducted on GSM8K, BBH, MMLU, MRPC, and their out-of-domain variants, plus a proprietary ad-search relevance dataset, with two base models and three random seeds.","tokens_in":22608,"tokens_out":4871,"duration_ms":49304,"significance":"If the claims hold, ReaLM would be a meaningful step toward SLMs that reason robustly, autonomously, and in domain-specific settings without sacrificing efficiency. The study's strengths include ablations that isolate the process-verification reward, error analysis that examines learning from incorrect CoTs, and evaluation on two base models with multiple seeds. The public-benchmark results are plausible, but the central autonomy mechanism (EAAI) is not cleanly isolated, and the theoretical justification in Appendix C is circular at the load-bearing point. The industrial results lack error bars and are not directly reproducible due to the proprietary dataset.","major_comments":[{"comment":"The autonomy claim for EAAI is not isolated by the reported experiments. No condition trains with external CoTs at a constant rate and then evaluates without them; the comparison in Table 3 is between MRPV+EAAI (trained with a fade schedule, evaluated without CoTs) and RL (trained and evaluated without CoTs). The 2.6% average gain in Table 1 could therefore be explained by CoT-augmented RL training alone, rather than by the fade schedule specifically. Please add a control that keeps the CoT inclusion probability constant (e.g., p=1 or p=0.5) for the full training run and then evaluates without CoTs. In addition, report checkpoint evaluations showing whether the accuracy gap between with-CoT and without-CoT evaluation shrinks as the cosine threshold decays, which would directly test the internalization mechanism.","section":"§3.3, Tables 1 and 3"},{"comment":"The proof sketch assumes the property it sets out to establish. Eq. (19) requires the cross terms δ_i^T H δ_j to be nonnegative, which is justified by the assumption that the CoT deviation vectors are 'mildly negatively correlated' in a negative-definite reward landscape. That is precisely the contrastive-benefit property that MRPV is designed to induce; it is not shown to hold a priori for arbitrary CoT samples. Proposition 2.1.2 similarly posits Δ_gen ≥ 0 as 'additional gains' rather than deriving it from the stated assumptions. Since these assumptions are untested, the theoretical analysis does not provide independent support for MRPV. Please either validate the assumptions empirically (e.g., measure the Hessian-vector products on actual model embeddings) or reposition the discussion as an intuition rather than a theorem.","section":"Appendix C, Propositions 2.1.1 and 2.1.2"},{"comment":"Footnote 3 states that external CoTs are 'without final answers,' but the GPT responses shown in Tables A5 and A6 end with tokens such as '#### 8' and '#### 9' (and similar answer strings). If these CoTs are used in the prompts exactly as shown, the with-CoT results in Table 1 include answer leakage, and the 4.2% gain over CoT-Synthesizer could be partly a selection effect rather than a reasoning improvement. Please clarify whether the final-answer tokens were stripped from the external CoTs before being fed to the SLM. If they were not stripped, rerun the with-CoT experiments with answer tokens removed and report the difference.","section":"Footnote 3 vs. Appendix Tables A5 and A6"},{"comment":"The baseline suite omits RFT (reject-sampling fine-tuning), which is the central comparison in the paper's motivation: the introduction explicitly criticizes RFT for filtering out negative reasoning paths. Without an RFT baseline, the claim that MRPV's contrastive supervision improves over reject-sampling is not directly demonstrated. Please add RFT (or an analogous reject-sampling SFT baseline) to Table 1 and the corresponding analysis.","section":"§3.1, Table 1"},{"comment":"The industrial benchmark results in Table 5 are reported without error bars or statistical significance tests, and the comparison includes only two baselines (SKIntern and CoT-Synthesizer). Because the dataset is proprietary and the expert comments are not publicly available, the reproducibility of this result is limited. Please report multiple seeds with variance, and consider adding a public vertical-domain task or releasing a sanitized subset to support the generalization claim.","section":"§3.1, Table 5"}],"minor_comments":[{"comment":"In Table A1, 'GSK8K' appears to be a typo for 'GSM8K'; the same typo appears in the dataset description in Section 3.1.","section":"Table A1"},{"comment":"The last column header in Table 2, 'ARAI', appears to be a typo for 'EAAI'.","section":"Table 2"},{"comment":"The sentence 'Assuming uniform weights wi = 1/k wi≠1/k' is garbled; it should read 'Assuming uniform weights, i.e., wi = 1/k'.","section":"Appendix C, after Eq. (16)"},{"comment":"The statement that 'MR+EAAI in Tab. 3 significantly outperforms the RL method' uses the word 'significantly' without a reported statistical test; please either provide a significance test or replace 'significantly' with a more neutral phrase.","section":"Section 3.3"},{"comment":"The description 'Only CoTs with sampled probability below this threshold are used' is slightly confusing because 'prob' in Algorithm 1 is a uniform random draw, not a property of the CoT. Clarify that the random draw determines inclusion probability.","section":"Algorithm 1 and Eq. (6)"},{"comment":"The term 'reflection-enhanced' in the abstract is not explicitly defined. The method largely involves contrastive reasoning over provided CoTs and assessing their utility; consider defining 'reflection' in Section 2.1 to avoid ambiguity.","section":"Abstract and Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper reports solid public-benchmark results with multiple seeds, and the central empirical claim is defensible. However, the autonomy mechanism (EAAI) is not isolated, the Appendix C proof is circular at the key step, and the industrial results lack variance reporting. The requested control experiment (constant CoT rate with no-CoT evaluation) and answer-leakage clarification are essential before publication. The proprietary nature of the industrial dataset also limits external verification; the authors should be encouraged to provide additional public evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The training recipe here is real and the main tables are consistent, but the paper overclaims the fade mechanism and the appendix contradicts its own footnote about external CoT answers.\n\nWhat is actually new: the specific combination of MRPV (contrastive multi-CoT process rewards), EAAI (cosine-faded external CoT exposure), and rule-aware CoT distillation is not in the cited prior work. The authors do two base models, multiple seeds for the main result, and ablations for every component. The gains over the tested baselines are consistent across GSM8K, BBH, MMLU, MRPC, and the OOD sets. The error analysis separating SAME/DIFF errors is a nice attempt to argue that the model learns from failures rather than just copying answers.\n\nBut there are serious soft spots, and they land on the central claims. First, the paper states in Footnote 3 that external CoTs are \"without final answers,\" yet the Appendix A5 prompt examples show CoTs ending in \"#### 8\" and \"#### 9\". This is not a suspicion; it is in the paper. The Stage 2 reward r_gain is then not a process reward at all; it is a reward for judging whether the final answer embedded in each CoT is correct. The with-CoT gains of 4.2% are therefore partly a selection effect. The authors need to rerun with CoTs that truly omit final answers, or openly acknowledge that they are doing answer-aware reranking.\n\nSecond, the autonomy claim for EAAI is not isolated. There is no control that trains with a constant CoT inclusion rate and then evaluates without CoTs. The 2.6% no-CoT gain over RL could come simply from CoT-augmented RL training, not from the cosine fade. The paper presents no checkpoint analysis showing the model gradually relies less on the external paths. Without that control, the \"asymptotic induction\" mechanism is unverified.\n\nThird, the Appendix C proofs are decorative. They assume the CoT embeddings are complementary and that the Hessian cross-terms are nonnegative, which is exactly the property MRPV is supposed to learn. Prop 2.1.2 follows from an already-posited \"additional gains\" term. These propositions add no rigor.\n\nFinally, hyperparameters like scale_gain, k, and the rule injection probability are not reported, and neither code nor data is released. The industrial benchmark is proprietary, so those results cannot be independently checked.\n\nThe paper deserves a serious referee, but it is not ready to be accepted as-is. The empirical recipe is plausible and the noise in the evidence is manageable; the missing constant-rate control and the answer-leakage contradiction are fixable. I would send it to review with a clear request for those controls, a rerun without answer-embedded CoTs, and a reduced emphasis on the theoretical propositions.","headline":"ReaLM is a well-executed recipe with a promising training signal, but the autonomy mechanism is not actually isolated and the external CoTs contain answer leakage, so the claims outrun the evidence.","tokens_in":23065,"tokens_out":4078,"would_cite":false,"duration_ms":44871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement learning framework, ReaLM, trains small language models to reason more reliably by learning from both correct and incorrect reasoning traces, and then removes the need for external reasoning prompts at inference.","keywords":["small language models","reinforcement learning","chain-of-thought distillation","process supervision","autonomous reasoning","vertical domains","multi-route reasoning"],"falsifier":"At intermediate training checkpoints, evaluate the model both with and without external CoTs in the prompt while the cosine threshold decays; if the with/without accuracy gap does not shrink as the threshold falls, the fading schedule is not teaching internalized reasoning but simply exposing the model to fewer hints.","tokens_in":21992,"feed_emoji":"🧠","tokens_out":8456,"duration_ms":75460,"temperature":0.7,"pith_summary":"ReaLM is a reinforcement learning framework that aims to give small language models (SLMs, ≤7B parameters) three properties at once: stronger multi-step reasoning, the ability to reason without externally supplied chain-of-thought prompts at inference time, and better generalization to vertical domains with proprietary rules. The central idea is to feed the model a bundle of unfiltered reasoning traces from a large teacher model, including many wrong ones, reward it both for judging each trace and for producing the correct final answer, and then gradually fade the traces out of the training input using a cosine schedule. On nine academic benchmarks and a private industrial ad-relevance set, the paper reports average gains of 4.2% over prior methods when external CoTs are available and 2.6% when they are not. The paper's claim, stated fairly, is that comparative reflection over failed reasoning paths is what teaches a small model to reason, and that fading external guidance converts that skill into autonomous inference.","feed_headline":"Training on wrong answers helps small models reason on their own","feed_subtitle":"ReaLM's contrastive rewards lift small-model accuracy by 4.2% with hints and 2.6% without.","key_machinery":"The load-bearing mechanism is a contrastive two-stage reward inside a GRPO reinforcement loop, coupled with a cosine-decay gate that withdraws external chain-of-thought prompts during training. Stage 1 rewards answer and format correctness; Stage 2, active only when Stage 1 succeeds, rewards agreement between the model's per-chain judgment and the chain's own conclusion ($r_{gain}=\\mathrm{scale}_{gain}\\cdot\\tfrac{1}{k}\\sum_{i} s_i$). EAAI selects which external CoTs enter the context according to threshold $0.5\\,(1+\\cos(\\pi\\,t))$, and disables Stage 2 when the selection probability becomes very low to avoid reward sparsity. The theoretical support is Proposition 2.1.1, a reward bound showing $E[R(C_{out})] \\ge E[R(C_{SLM})]$ when reference CoTs are provided, and Proposition 2.1.2, which decomposes the reward into a learned weighted combination of reference rewards plus a non-negative generation gain.","core_discovery":"The central discovery is that a small model's reasoning improves when reinforcement learning is supervised to contrast whole reasoning routes rather than to imitate only correct ones or to use external chains as static features. MRPV presents the model with k unfiltered teacher-generated chains (correct and incorrect together), asks it to predict both the answer and a per-chain correctness judgment, and gates a process reward r_gain behind a correct-answer reward r_base so that the auxiliary judgment sharpens main-task learning without displacing it. EAAI then withdraws the chains during training by sampling them with probability 0.5(1+cos(πt)), so at inference the model sees only the question; the paper reports that this preserves most of the benefit (e.g., 94.4 vs 96.0 on GSM8K for DeepSeek-R1-7B) while removing the external dependency. The paper also contributes a vertical-domain recipe: guided chain-of-thought distillation that injects expert comments into teacher prompts, validated on a newly introduced industrial ad-search relevance benchmark. Two formal propositions, a reward bound and a reward decomposition, state that referencing diverse CoTs raises the expected reward at least to the level of reasoning from scratch.","pith_inferences":["Going beyond the paper: the EAAI cosine decay is never compared against fixed-probability or abrupt CoT dropping, so the shape of the fade-out curve is not isolated; a direct ablation would test whether smoothness matters.","Going beyond the paper: the SAME/DIFF error analysis could be extended by directly probing how much the final model attends to external traces when they are present, to verify internalization rather than residual dependence.","Going beyond the paper: the theoretical reward bound assumes teacher CoTs are at least as good as the student's unaided reasoning and that their embedding deviations are mildly negatively correlated; a weaker-teacher test would show where the bound breaks.","Going beyond the paper: rule-aware CoT distillation suggests a general recipe for RL training in any vertical domain that has expert annotations, so the industrial-dataset result is a template rather than a one-off."],"forward_implications":["If ReaLM is correct, SLMs can be trained to match or beat GRPO-based RL and distillation baselines on math, commonsense, and factual reasoning while needing no external CoTs at inference.","Performance is highest when 20–80% of the training CoTs are wrong, which empirically argues against reject-sampling-only supervision and supports learning from failure.","The method transfers to a proprietary vertical domain, with ReaLM-R1 reaching 82.61 accuracy on the new ad-search relevance benchmark, suggesting rule-aware CoT distillation is a workable way to inject non-public rules.","The two propositions imply that any SLM trained with contrastive multi-route reference CoTs enjoys an expected-reward bound above unaided reasoning, making MRPV a generally applicable RL supervision layer."],"supporting_citations":[{"why":"Provides the GRPO advantage estimator and the RL baseline that ReaLM compares against.","marker":"Shao et al. 2024"},{"why":"Supplies DeepSeek-R1-7B initialization and the iterative RL training recipe used for ReaLM-R1.","marker":"Guo et al. 2025"},{"why":"Is the CoT-Synthesizer baseline that consumes external CoTs as features, the main method ReaLM's 4.2% gain is measured against.","marker":"Zhang et al. 2025a"},{"why":"Defines the reject-sampling (RFT) supervision that MRPV specifically contrasts by keeping negative paths.","marker":"Yuan et al. 2023"},{"why":"Is the LLMs-as-Instructor baseline and the source of the academic evaluation suite.","marker":"Ying et al. 2024"},{"why":"Provides GSM8K, the in-domain math set for ablations and error-ratio analysis.","marker":"Cobbe et al. 2021"},{"why":"Provides BBH, the in-domain complex-reasoning evaluation.","marker":"Suzgun et al. 2022"},{"why":"Provides MMLU, the in-domain factual-reasoning evaluation.","marker":"Hendrycks et al. 2020"}],"fun_headline_variants":["Contrastive rewards boost small-model reasoning by 4.2%","Learning from wrong answers fades external hints for reasoning","Small models self-reason via contrastive path learning","Autonomy from fading teacher hints in small-model RL","Why small models should train on their mistakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that gradually fading external reasoning traces during training makes the small model internalize the reasoning patterns, so that at test time accuracy without those traces stays close to accuracy with them.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive rewards boost small-model reasoning by 4.2%","Learning from wrong answers fades external hints for reasoning","Small models self-reason via contrastive path learning","Autonomy from fading teacher hints in small-model RL","Why small models should train on their mistakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000615,"raw_usage":{"total_tokens":2904,"prompt_tokens":1037,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1789}},"tokens_in":653,"tokens_out":1867,"duration_ms":15153,"temperature":1.0,"reasoning_tokens":1789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:22:35.405882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At intermediate training checkpoints, evaluate the model both with and without external CoTs in the prompt while the cosine threshold decays; if the with/without accuracy gap does not shrink as the threshold falls, the fading schedule is not teaching internalized reasoning but simply exposing the model to fewer hints.","supporting_citations":[],"review_version":2}