{"id":"f536ecac-1851-4cc4-b8f6-c023af598dc9","arxiv_id":"2506.15710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Injecting the logit gap between a small RL-trained model and its base into a larger base model at decoding time improves reasoning accuracy on math and code benchmarks, sometimes matching or exceeding RL training of the large model.","lead":"A decoding-time method that borrows logit changes from a small RL-trained model and adds them to a larger base model improves math and code reasoning without training the large model. The authors argue RL's probability shifts transfer across model sizes, which would make reasoning enhancement much cheaper.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Size-invariance is only measured on RL-expert trajectories; on the base-model prefixes where RAST actually computes and applies the delta, alignment between small-model and large-model deltas is never shown.","rationale":"The reader's weakest assumption focuses on tokenizer/logit-scale/prompt-matching as prerequisites for the additive correction. I agree those are relevant, but the more fundamental gap is that the paper validates the transfer signal on the wrong distribution. PCR and Figure 4 both use trajectories generated by the RL expert (SimpleRL-32B), whereas RAST operates on prefixes generated by the large base model (plus the delta). If the small-model delta is only aligned with the large-model delta on expert trajectories, the method's mechanism is not the one claimed. The recommended test is cheap because all required models already exist, and it directly targets the invariance hypothesis. This does not overturn the empirical finding that RAST improves accuracy; it questions why. If the test shows high similarity, the hypothesis is strongly supported and the paper's framing is justified. If not, the paper should be reframed as a practical decoding-time transfer method whose mechanism (e.g., reward-model bias or output-length control) remains to be identified. The verdict stays CONDITIONAL, with the condition that the authors either run this distribution-matched invariance check or soften the claim of model-size invariance. Secondary issues (missing error bars in Table 1, greedy-decoding baseline asymmetry, and the admitted prompt mismatch for ΔR1.5B) further support caution but are less directly load-bearing than the missing distribution-matched test.","tokens_in":23630,"tokens_out":9117,"duration_ms":96491,"concrete_test":"Sample 50–100 prefixes from actual RAST decoding runs (e.g., Qwen-2.5-32B + ΔR14B on MATH500) and from greedy Qwen-2.5-32B decoding. On each prefix, compute ΔR14B = logits(SimpleRL-14B) − logits(Qwen-2.5-14B), ΔR7B, and ΔR32B = logits(SimpleRL-32B) − logits(Qwen-2.5-32B). Then compute AvgCosineSim per Eq. 6 for pairs (ΔR14B, ΔR32B) and (ΔR7B, ΔR32B) on these prefixes, and compare with the values reported in Figure 4. If similarity on RAST/base prefixes remains above roughly 0.9, the invariance claim holds in the deployment distribution; if it drops substantially (e.g., below 0.7) while RAST still improves accuracy, the success is not explained by the stated size-invariance hypothesis and the explanation must be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central hypothesis is that RL-induced probability shifts are model-size invariant, and RAST's mechanism is to inject the small-pair delta ΔR = SRL − Sbase into the large base model's logits (Eq. 3). The paper's direct evidence for invariance is (i) PCR in Section 2.1, which measures whether the large base model can continue trajectories generated by SimpleRL-32B, and (ii) cosine similarity in Figure 4, which is computed on the same SimpleRL-32B trajectories. Neither measures what happens on the prefixes RAST actually decodes from: the large base model's own generation prefixes (modified by the injected delta). Because Mbase and SRL have different distributions, the small-model logits can be off-distribution on base/RAST prefixes, making the transferred delta directionally or quantitatively different from the large model's true RL delta. If this is the case, RAST's empirical gains could stem from a fixed output bias of the small RL model (e.g., promoting longer, self-verification-style text) rather than from a size-invariant reasoning-activation signal. This gap is load-bearing because the method's conceptual novelty and the stated hypothesis depend on the delta being transferable where it is actually used, not just on the expert's own paths.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAST, a decoding-time method that augments a large base model's logits with the logit difference ΔR = SRL − Sbase computed from a small RL-trained model and its base counterpart. The authors first present a hypothesis that RL-induced probability shifts are largely model-size invariant, supported by a path-coverage-rate analysis on 50 SimpleRL trajectories. They then evaluate RAST across Qwen-2.5 and Llama-3.1 math benchmarks and a code benchmark, reporting consistent pass@1 gains, a recovery-rate analysis, robustness sweeps over λ and temperature, and an estimated GPU-memory comparison with GRPO.","tokens_in":23796,"tokens_out":6050,"duration_ms":67516,"significance":"If the empirical results hold, RAST is a practical and inexpensive alternative to full-scale RL: it requires no RL training on the large model, only an additional forward pass through a small pair at inference time, and it is simple to implement. The paper's strongest assets are the breadth of the evaluation (three model families, nine benchmarks, multiple pass@k levels), the fixed λ=1.0 with a robustness sweep, and honest limitations. However, the paper's explanatory claim about size invariance is not yet supported by the evidence, because all alignment measurements are made on RL-expert trajectories rather than on the prefixes where RAST actually applies the delta. If the empirical gains are robust, the method could still be valuable even if the hypothesis is refined or replaced; the main contribution would then be empirical rather than mechanistic.","major_comments":[{"comment":"The size-invariance evidence is computed only on 50 trajectories generated by SimpleRL-32B. PCR measures whether Mbase can continue the RL expert's path, and AvgCosineSim is computed by feeding those same RL trajectories through Mbase and MRL. Neither measures the alignment between small-model and large-model deltas on the prefixes that RAST actually decodes from, namely prefixes generated by Mbase or by the augmented model in Eq. (3). Because SRL and Sbase may be off-distribution on those prefixes, the transferred ΔR could in principle be a fixed output bias (e.g., for longer or self-verification text) rather than a scale-invariant reasoning signal. I ask for a concrete test: compute PCR and AvgCosineSim(ΔR_small, ΔR_large) on prefixes sampled from Mbase and from RAST itself, not only from MRL, and report whether the observed alignment persists on those prefixes. This is needed to support the abstract's 'largely model-size invariant' claim.","section":"Sections 2.1 and 4.1 (Eqs. 1-2, 6)"},{"comment":"The central empirical claim of consistent gains is reported without variance or significance statistics. Pass@1 is averaged over 32 runs, yet no confidence intervals, standard deviations, or paired tests are given, so several reported gains are difficult to interpret. For example, Qwen-2.5-32B on GSM8K goes from 93.1 to 93.3 with ΔR1.5B, and Llama-3.1-70B on AIME24 goes from 0.0 to 2.8; both are within the range one would expect from 32-sample sampling noise. The only error bars appear in Figure 6 for a single configuration. Please add bootstrap confidence intervals or paired significance tests for all main tables, or state explicitly which reported differences are not statistically distinguishable.","section":"Tables 1-3 and §3.2"},{"comment":"The 1.5B row is confounded with a prompt mismatch. The footnote says the prompt used for RAST does not match the one used to train the small RL-tuned model, and Appendix A.2 explains that a different template (Prompt b) is used for the 1.5B source. This means the observed smaller gains for ΔR1.5B, and the 'stronger experts yield greater gains' trend in §3.2, may be partly an artifact of prompt mismatch rather than model scale. The claim that RAST transfers across scales would be better supported by using a matched-prompt 1.5B RL model, or by an ablation in which the same mismatch is artificially applied to ΔR7B and ΔR14B.","section":"Table 1 footnote and Fig. 7"}],"minor_comments":[{"comment":"'PRC' is used instead of 'PCR' in the sentence 'The PRC in this experiment is 96.37%'; the acronym is inconsistent with Eq. (2).","section":"Section 4.2"},{"comment":"The figure is difficult to read as printed: the plotted points and axes are not clearly labeled, and the numbers in the caption are not mapped to visible axes. Please redraw with explicit axis labels and a legend.","section":"Figure 4"},{"comment":"'MAHT500' is a typo for MATH500.","section":"Appendix E"},{"comment":"There are several grammatical slips, including 'pass@k for MRL is large than Mbase' and 'the tn+1-th token based on the first n tokens'; these should be corrected.","section":"Section 3.3 and 4.2"},{"comment":"The decoding-configurations table mixes model names and hyperparameter rows; separating base-model and RAST rows would improve readability.","section":"Table 5"},{"comment":"No code release is mentioned in the text beyond the project page; given the revised vLLM implementation and the reproducibility claims, an explicit code link would be valuable.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about limitations, but the central explanatory claim is not yet supported by the measured quantity. The empirical contribution may be sufficient for an applied venue, but for a research paper the alignment analysis needs to be redone on the distribution where RAST actually operates. No concerns about citation practice or novelty; the related-work section is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the empirical result is real and worth building on, but the explanatory hypothesis—that RL-induced probability shifts are largely model-size invariant—is significantly under-supported by the evidence actually presented. The reader's conditional verdict is right, and the stress-test concern lands: the invariance evidence is collected on the RL expert's trajectories, not on the prefixes where RAST actually computes and applies the delta.\n\nWhat's new and worth credit: the specific recipe—taking the logit delta from a zero-RL-from-scratch small model pair and injecting it into a larger base model at decoding time—is, to my knowledge, not in the contrastive-decoding literature. Equation 3 is DExperts/contrastive decoding, and the authors say so, but the delta source and the cross-scale transfer question are new. The experiments are broad: three model families (Qwen-2.5, Llama-3.1, Qwen-1M/Code-R1), six math benchmarks plus three code benchmarks, pass@k and majority@k, plus a robustness sweep over lambda and temperature. The gains over the base models are consistent and sometimes close to or better than the RL-trained ceiling, which is a practically useful result. The paper also reports an honest negative result (ensembling deltas from multiple scales hurts), and the limitations section is candid.\n\nSoft spots:\n- The central hypothesis is not actually tested where the method operates. The PCR and cosine-similarity analyses (Sections 2.1 and 4.1) both use trajectories generated by SimpleRL-32B. RAST, however, decodes on the base model's own prefixes (as modified by the delta). Without measuring delta alignment on those prefixes, the 'size invariance' claim is open to the alternative explanation that the small RL model contributes a fixed output bias—longer, self-verifying text—that helps any base model. The paper's own token-frequency analysis (Figure 10) is consistent with that bias. This is a load-bearing gap because the abstract's conceptual framing leans on the invariance hypothesis.\n- The empirical measurements are otherwise thin in places: 50 trajectories for PCR/cosine-similarity, no error bars in Tables 1–3, and the 1.5B delta case has a prompt-template mismatch (honestly footnoted) that makes the reported 'consistently smaller gains' hard to interpret.\n- No ablation against a non-RL delta, e.g., a supervised or instruction-tuned small-model pair, or a random/length-bias delta. Without that, you can't attribute the gains to the RL training as such.\n\nNone of this is fatal to the main empirical claim, and the paper itself flags the theoretical gap. But the hypothesis is stated more confidently than the evidence supports. This is a solid paper for anyone working on decoding-time reasoning enhancement, cheap reasoning transfer, or RL interpretability. I'd send it to review, but I'd push the authors to add the ablations and a proper invariance test on base-model prefixes before it's fully accepted.","headline":"RAST is a practical empirical contribution with a simple, honest method, but the size-invariance hypothesis is much thinner than the abstract implies.","tokens_in":24427,"tokens_out":4080,"would_cite":true,"duration_ms":44544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the token-level probability shifts reinforcement learning induces in a small model can be transferred at decoding time to a larger base model, recovering most of the RL performance gap without running RL on the…","keywords":["reinforcement learning","LLM reasoning","decoding-time transfer","logit-space correction","model-size invariance","zero-RL","mathematical reasoning","path coverage rate"],"falsifier":"A concrete test: apply a delta computed from a small RL-trained model to a larger base that uses a different tokenizer or a mismatched prompt template, and measure path coverage rate plus recovery on the same benchmarks. The size-invariance hypothesis predicts PCR stays near 95% and recovery stays well above zero; the paper's own AIME24 results, where recovery is roughly 62% and the base gains are small, mark where the claim already starts to weaken, so a dataset or pairing that drives PCR below 90% with recovery near zero would falsify the transfer claim.","tokens_in":23358,"feed_emoji":"🧠","tokens_out":11718,"duration_ms":119519,"temperature":0.7,"pith_summary":"This paper sets out to prove that the reasoning gains from reinforcement learning on a language model are a transferable signal: the small set of token-level probability shifts RL induces in a small model can be added to a larger model's logits at decoding time, giving the larger model most of the reasoning improvement with none of the RL training cost. The claim matters because RL on large models currently means loading several copies of the model and running long training loops; if the signal transfers, the expensive part can be done once on a small model and replayed on any larger model that shares its vocabulary. The authors first measure how much of an RL-trained model's output path a base model already reproduces token by token and find over 95% overlap, supporting their hypothesis that RL adjusts only a sparse set of reasoning-critical tokens. They then define RAST, which decodes with the base logits plus a scaled logit difference from a small RL-trained pair, and show across six math and three code benchmarks that it consistently lifts the base model, recovering between roughly 60% and over 100% of the gap to the fully RL-trained model while using about half the GPU memory. The paper's 1.5B-transfer experiments, where the prompt template did not match, show visibly smaller gains, which marks the practical boundary of the claim.","feed_headline":"RL reasoning transfers from small to large models without retraining","feed_subtitle":"One small model's RL-trained logit correction restores most of the reasoning gap at half the GPU cost.","key_machinery":"The load-bearing object is the delta-logit signal $\\Delta_R(x_{<t}) = S_{\\mathrm{RL}}(x_{<t}) - S_{\\mathrm{base}}(x_{<t})$, the token-level difference in logits between a small RL-trained model and its own base twin, injected additively at every decoding step through $P = \\mathrm{softmax}[M_{\\mathrm{base}} + \\lambda \\Delta_R]$, which amplifies tokens tied to behaviors such as self-verification, branching, and backtracking while leaving ordinary tokens to the base model. Two diagnostics carry the empirical argument: path coverage rate (PCR), the fraction of an RL-produced trajectory that the base model regenerates token for token, which the paper measures above 95% and uses as direct evidence for the sparsity of RL-induced shifts; and cosine similarity between delta signals from different model pairs, which the paper shows correlates with recovery rate so that users can predict transferability before running experiments.","core_discovery":"The central claim is that RL-induced probability shifts are largely model-size invariant, so the logit-space correction $\\Delta_R = S_{\\mathrm{RL}} - S_{\\mathrm{base}}$ learned on a small model pair carries reasoning behavior that can be activated in larger models at inference. The proposed decoding rule is $P(x_t \\mid x_{<t}) = \\mathrm{softmax}[M_{\\mathrm{base}}(x_{<t}) + \\lambda(S_{\\mathrm{RL}}(x_{<t}) - S_{\\mathrm{base}}(x_{<t}))]$, with $\\lambda = 1.0$; path coverage rate, the fraction of the RL trajectory the base regenerates greedily, stays above 95% for 7B, 14B, and 32B models, which the paper reads as evidence that RL steers a small number of reasoning-critical tokens rather than rewriting outputs. On MATH500, AIME24, AMC, Minerva, OlympiadBench, and GSM8K, adding the transferred delta lifts every base model tested, with average recovery rates of the gap to the RL-trained ceiling ranging from 63.4% to 84.8% depending on the pairing, and recovery above 100% in some cells; on Llama-3.1-70B with an 8B delta and on a 14B coding model with a 7B delta the correction transfers across families and domains. The paper also claims the correction widens the sampling search space, so RAST's pass@k reaches or exceeds the RL-trained model's, and that cosine similarity between candidate deltas predicts recovery rate, giving a cheap pre-screening signal.","pith_inferences":["Editorial extension: if RL-induced shifts are size-invariant beyond the two families tested, the economics of reasoning RL change — one small expert's delta becomes a reusable inference-time asset for any larger model sharing its tokenizer, decoupling reasoning quality from per-model training budgets.","Editorial extension: the shared-tokenizer requirement is the hidden boundary of the claim, so a natural test is whether deltas survive cross-vocabulary transfer after a token-alignment or projection step, which the paper does not attempt.","Editorial extension: RAST is a parameter-free distillation of RL behavior into decoding; distilling the delta into a lightweight adapter (as the paper's future-work section suggests) would make the correction persistent and avoid running three models at inference.","Editorial extension: the fact that RAST's pass@k can exceed the RL-trained ceiling hints that the combination of base knowledge and transferred behavior searches more broadly than either alone, which suggests the enhanced sampler could itself generate training data for further RL — a loop the paper does not test."],"forward_implications":["Transplanting a 14B-trained delta into a 32B base yields an average recovery rate of 84.8% across the math benchmarks while needing roughly 160 GB of GPU memory against about 350 GB for full RL training of the 32B model, so most of the reasoning gain arrives at roughly half the memory cost.","Recovery rates vary with the pairing and benchmark, from 63.4% to over 100%; on Minerva the 32B base with the 14B delta exceeds the RL-trained 32B model outright, so the transferred signal can sometimes beat its own ceiling.","Sampling with the transferred delta yields pass@k curves that reach or exceed the RL-trained model's on all six math benchmarks, indicating the correction diversifies the search space instead of narrowing it.","The correction transfers across model families and domains: an 8B Llama-3.1 delta lifts Llama-3.1-70B on all six math benchmarks, and a 7B code delta lifts a 14B coding model by 4.4 points on average, so the mechanism is not specific to one architecture or task type.","Because recovery rate tracks the cosine similarity between candidate delta signals, a user can screen which small RL expert to train by measuring delta alignment first, without running full evaluations."],"supporting_citations":[{"why":"Supplies the small RL-trained models and their base twins whose logit difference defines the transferred delta, and the RL-from-scratch training recipe used throughout.","marker":"[72]"},{"why":"Provides the MATH500 test set used both for the path-coverage alignment study and as a principal math benchmark.","marker":"[18]"},{"why":"The finding that RL reshapes output distributions rather than adding knowledge is the premise the size-invariance hypothesis builds on.","marker":"[70]"},{"why":"Established the zero-RL paradigm that the small expert models are trained under.","marker":"[14]"},{"why":"Supplies the zero-RL code models whose delta is applied to the 14B coding base in the code-reasoning experiments.","marker":"[39]"},{"why":"Provides the GRPO algorithm used to train the RL experts the deltas are derived from.","marker":"[54]"},{"why":"Defines the Qwen-2.5 model family used for the main cross-scale transfer experiments.","marker":"[67]"},{"why":"The contrastive-decoding logit manipulation family that RAST's additive injection formula extends to reasoning.","marker":"[30]"}],"fun_headline_variants":["Small-model RL logit shifts boost large model reasoning","Transfer RL reasoning from small to large LLMs without retraining","RAST: RL probability shifts scale across model sizes","Inject small-model RL deltas into large LLM inference","Logit-space RL transfer: one small model, many sizes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that carries the argument is that the logit-space difference measured on a small model pair transfers unchanged to a larger model's logits, which requires a shared tokenizer, comparable logit scales, and matching prompt templates; the paper's own 1.5B experiments, where the template differed from the training prompt, produced visibly smaller gains, marking where the transfer starts to break.","fun_headline_variants_meta":{"raw":{"variants":["Small-model RL logit shifts boost large model reasoning","Transfer RL reasoning from small to large LLMs without retraining","RAST: RL probability shifts scale across model sizes","Inject small-model RL deltas into large LLM inference","Logit-space RL transfer: one small model, many sizes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1423,"prompt_tokens":1169,"completion_tokens":254,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":785,"completion_tokens_details":{"reasoning_tokens":173}},"tokens_in":785,"tokens_out":254,"duration_ms":3347,"temperature":1.0,"reasoning_tokens":173,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:13:14.706212+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: apply a delta computed from a small RL-trained model to a larger base that uses a different tokenizer or a mismatched prompt template, and measure path coverage rate plus recovery on the same benchmarks. The size-invariance hypothesis predicts PCR stays near 95% and recovery stays well above zero; the paper's own AIME24 results, where recovery is roughly 62% and the base gains are small, mark where the claim already starts to weaken, so a dataset or pairing that drives PCR below 90% with recovery near zero would falsify the transfer claim.","supporting_citations":[],"review_version":1}