{"id":"aee4eff6-2fd8-45f3-98ee-9e57e0a1cfc5","arxiv_id":"2412.16849","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OpenRFT adapts a reasoning foundation model to eight scientific tasks with 100 samples each via data augmentation, self-distilled reasoning SFT, and PPO with a process reward model, achieving 0.447 vs 0.403 average accuracy.","lead":"OpenRFT introduces a reinforcement fine-tuning pipeline that adapts a generalist reasoning model to scientific multiple-choice tasks using only 100 labeled samples per task. It combines question rewriting, self-generated reasoning traces, and process reward supervision, reporting an 11% average accuracy gain over the base model on SciKnowEval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 11% gain is not statistically established: no error bars or significance tests are reported, and the full OpenRFT pipeline (with ICL) scores below its own DA-only ablation.","rationale":"The reader's weakest assumption is the PRM action-space compatibility (footnote 3). That is a genuine limitation, but it is about the internal training signal and does not by itself overturn the observed final-answer accuracy. The more load-bearing issue for the central claim is that the 11% improvement is reported without any estimate of uncertainty. With 100 test items per task, an average absolute gain of 0.044 can easily be noise; the paper's own admission of preliminary results and the fact that the full ICL-inclusive method underperforms an ablation make the headline result fragile. My read reinforces the reader's CONDITIONAL verdict rather than changing it: the empirical claim is plausible but not yet rigorously supported. If a concrete multi-seed rerun with confidence intervals is provided and the pattern persists, I would accept the claim; if the interval excludes zero only for the DA-only variant and not the full OpenRFT, the abstract's general claim should be narrowed.","tokens_in":11210,"tokens_out":6628,"duration_ms":57278,"concrete_test":"Run SFT+RL(PRM)+DA and Vanilla on T1–T8 with at least 5 independent RL seeds and identical train/test splits, then compute per-task mean±std and a paired bootstrap 95% CI over the 100 test items for the average improvement. If the CI includes 0, or if the full OpenRFT run with ICL does not numerically beat the DA-only ablation, the headline claim should be downgraded from a definitive gain to a preliminary observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in §3.4 is that SFT+RL(PRM)+DA improves over Vanilla by an average of 11% (0.447 vs 0.403 in Table 1). The paper says 'we perform three evaluations and report the average accuracy,' but it gives no standard deviations, confidence intervals, or paired significance tests for any method. Each task is evaluated on only 100 test questions (49 for T7), so a single task's standard error is up to about 0.05; several per-task deltas are exactly in that range (e.g., T1: 0.28→0.29, T5: 0.45→0.47). The across-task average improvement (0.044) is therefore possibly within sampling noise, especially without controlling for train/test split or RL seed. Moreover, the full OpenRFT configuration SFT+RL(PRM)+DA+ICL (0.443) is numerically worse than the ablated SFT+RL(PRM)+DA (0.447), so the improvement is not robustly tied to the proposed full method. The paper itself calls the results an 'early exploration' to be updated, which further undercuts treating the 11% as a settled finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OpenRFT, an open-source implementation of reinforcement fine-tuning (RFT) for adapting a generalist reasoning model to domain-specific multiple-choice tasks using only 100 training samples per task. The method combines three uses of the limited domain samples: question rewriting and option shuffling for data augmentation (DA); supervised fine-tuning (SFT) on reasoning traces synthesized by the policy itself (or by a stronger teacher); and PPO reinforcement learning with a process reward model (PRM), optionally augmented with few-shot in-context learning (ICL). The evaluation is on eight SciKnowEval L3 tasks spanning biology, chemistry, physics, and materials science. The main reported result is that SFT+RL(PRM)+DA achieves an average accuracy of 0.447 versus 0.403 for the unmodified Skywork-o1-Open-Llama-3.1-8B policy, an average improvement of about 11%, while the full pipeline with ICL scores 0.443. The paper also reports that using a stronger teacher model (QwQ-32B-Preview) for distillation hurts performance, which it attributes to teacher-student action-space mismatch, and it explicitly labels the work as an early exploration that will be updated.","tokens_in":11488,"tokens_out":4007,"duration_ms":35773,"significance":"If the central empirical claim were statistically robust, the paper would make a useful contribution: it provides a concrete, open-source recipe for RFT-style fine-tuning with very few domain samples, decomposes the pipeline into modules that can be ablated, and is candid about failure modes such as the ICL degradation and the teacher-alignment problem. The release of source code, datasets, and models is a genuine strength, as is the attempt to study the action-space alignment issue empirically rather than assuming it away. However, the headline result is not yet established: no uncertainty quantification is reported, the full proposed pipeline underperforms its own ablated variant, and the teacher-distillation experiment shows a large negative effect that the paper explains only by an untested assumption. The paper is therefore best viewed as a promising preliminary technical report whose main quantitative claim needs additional evidence.","major_comments":[{"comment":"The headline claim that SFT+RL(PRM)+DA achieves an average improvement of 11% over Vanilla is not statistically supported. The text says three evaluations were performed and averaged, but no standard deviations, confidence intervals, or significance tests are reported. With 100 test questions per task (49 for T7), the standard error of a single task accuracy is up to about 0.05, and several contributing per-task deltas are of that magnitude (e.g., T1: 0.28 to 0.29; T5: 0.45 to 0.47). The across-task average gap of 0.044 could therefore be within sampling noise, especially since no information is given about the variability of the train/test split or the RL seed. Moreover, the full OpenRFT configuration SFT+RL(PRM)+DA+ICL (0.443) is numerically worse than the ablated SFT+RL(PRM)+DA (0.447), so the improvement is not robustly tied to the complete proposed method. Please report per-run results, standard deviations or confidence intervals, and paired tests across tasks, and re-state the central claim to clearly distinguish the best ablated variant from the full pipeline.","section":"Section 3.4, Table 1"},{"comment":"The teacher-alignment experiment raises a load-bearing issue for the SFT-based imitation module of Section 2.2. SFT+ (distilling reasoning data from the stronger QwQ-32B-Preview model) performs substantially worse than SFT with self-distilled data and even worse than Vanilla (0.31 vs 0.40 average). The paper attributes this to teacher-student action-space mismatch, but no controlled experiment isolates that cause from other differences, such as reasoning-data quality, prompt format, or the fallback strategy of replacing failed rollouts with the true answer after 64 sampling attempts. Since the deployed OpenRFT pipeline relies on self-distillation rather than a stronger teacher, the paper should provide evidence that self-synthesized traces are reliable, and it should test the action-space hypothesis directly by, for example, reformatting teacher traces into the student's action space before SFT.","section":"Section 3.5, Table 2"},{"comment":"The reward design rests on an unverified assumption about the process reward model. The combined reward R_i = alpha * ori + (1-alpha) * f(pr_i) depends on the PRM's step-level scores, but footnote 3 states that the provider has not confirmed whether the action spaces of the policy model and the PRM are aligned. If the PRM systematically mis-scores reasoning steps, the PPO update can be pushed toward flawed chains even when final answers are correct. The paper does not report any sensitivity analysis for alpha, any evaluation of PRM accuracy on the eight tasks, or any comparison of alternative aggregation functions f(.). Please provide at least a quantitative sanity check (for example, correlation between PRM process scores and outcome correctness, or an ablation with alpha=1) to support the reward design.","section":"Section 2.3.2 and footnote 3"},{"comment":"The experimental setup does not fully specify the data split and the leakage risk of data augmentation. The paper says 100 training and 100 test samples are sampled from each dataset, but it does not state how the split was performed, whether the augmented questions were generated only from the training set, or whether any check was made for overlap or near-duplicates between augmented training questions and test questions. Since the augmentation rewrites question stems and shuffles options, a leakage check is needed to ensure the reported gains are not inflated by memorizing test-like rephrasings. Please clarify the split procedure and add a similarity or containment check.","section":"Section 3.1 and Section 3.3"}],"minor_comments":[{"comment":"There are typos and minor wording issues, including 'vanila' (Section 2.2.1), 'rewerd' (Section 3.2), and 'recomputing function' (Section 2.3.2); these should be corrected in a revision.","section":"Throughout"},{"comment":"The notation is inconsistent: the sampled reasoning data are written as (Qi, ..., S^i_j, ..., A'_i) and the final synthesized data as (Qi, Si, Ai), but the relationship between A'_i and Ai is not made explicit, and the 'at least one data that can infer the correct answer' selection criterion is informal.","section":"Section 2.2.1"},{"comment":"The caption mentions a 'light green dashed line' representing SFT with 100 samples, but the figure is not visible in the manuscript text and the marker is not described in the body; please ensure the figure is legible and the legend is explained.","section":"Section 3.5, Figure 3"},{"comment":"The reference list contains a malformed entry for 'ope, 2024' (Open O1), and the Sentence-BERT model used for ICL retrieval is not cited; please fix the citation format and add the missing reference.","section":"References"},{"comment":"The description of the fallback for failed distillation rollouts says the true answer is 'directly used as the response in the training data' when 64 attempts fail; it would be clearer to state whether this creates a reasoning-less (Q,A) example and how such examples affect the SFT loss.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an openly released technical report rather than a polished journal submission, and the authors themselves describe the results as an 'early exploration' that will be updated. The main risk is that the central quantitative claim, the average 11% improvement, is not statistically established and is not tied to the full proposed pipeline, since the ICL component lowers performance. I do not see the issues as irreparable: adding variance information, significance tests, a leakage check, PRM reliability analysis, and a controlled teacher-alignment experiment would substantially strengthen the paper. If the authors prefer to keep the paper as a short technical report, they should at least soften the strength of the claim to reflect the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is an honest, early technical report on fine-tuning a reasoning foundation model with 100 domain samples, and it does a few things worth knowing about. But don't quote the 11% average gain as a fact; the statistical support isn't there, and the full proposed pipeline actually scores slightly below its own ablated variant on the same table.\n\nWhat's genuinely useful: the paper lays out a clean RFT recipe — question augmentation, self-distilled SFT on reasoning traces, PPO with a process reward model, and few-shot ICL — and reports results on eight SciKnowEval tasks. The components are individually known, but the integration and the empirical comparison with these components turned off is a useful contribution for practitioners. The authors also do the right thing by releasing code, data, and model weights. The teacher-student action-space finding (Table 2) is interesting: using the stronger QwQ-32B as teacher made SFT worse than self-distillation, which is a cautionary data point for anyone doing reasoning-trace distillation. They flag the same concern for their PRM in footnote 3.\n\nThe soft spots are real and mostly statistical. They report three evaluations but give no standard deviations, confidence intervals, or significance tests; with 100 test questions per task (49 for T7), a per-task standard error is up to about 0.05, and several of the per-task deltas are in that range. The across-task average improvement of 0.044 is therefore possibly within sampling noise. The other structural issue: the full OpenRFT configuration (SFT+RL(PRM)+DA+ICL) gets 0.443, numerically worse than the ablated SFT+RL(PRM)+DA at 0.447. So the ICL module doesn't pay off, and the claimed best result comes from a partial configuration. The authors are transparent about this, but it complicates the headline claim.\n\nWho should read this: anyone building a low-data fine-tuning pipeline for reasoning models on scientific multiple-choice tasks. It's a reasonable starting point for a practitioner, not a settled result.\n\nWould I send it to peer review? Yes, as an empirical workshop-style paper, provided the authors add error bars or significance tests and reframe the claim around the partial configuration. As is, it's an arXiv-level early exploration, not a definitive evaluation.","headline":"An honest early RFT recipe with useful components and an interesting teacher-student finding, but the headline 11% gain lacks error bars and the full pipeline underperforms its own ablation.","tokens_in":12000,"tokens_out":2157,"would_cite":false,"duration_ms":17632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With only 100 domain samples per task, a three-part recipe of self-distilled reasoning, process-reward RL, and data augmentation lifts a generalist reasoning model's accuracy by an average of 11%.","keywords":["reinforcement fine-tuning","reasoning foundation models","process reward model","data augmentation","self-distillation","few-shot in-context learning","SciKnowEval","PPO"],"falsifier":"Take the best pipeline and replace each PRM step score with a random permutation of the same scores while keeping the final-answer reward identical; if accuracy stays at 0.447 within noise, then process supervision is not the active ingredient. A complementary check is to run the full pipeline on an unseen ninth SciKnowEval task and see whether the 11% gain reproduces.","tokens_in":11042,"feed_emoji":"🧪","tokens_out":7148,"duration_ms":57097,"temperature":0.7,"pith_summary":"OpenRFT asks whether the recently announced reinforcement fine-tuning paradigm can be reproduced openly with a generalist reasoning model and very few domain examples. The paper's answer is yes: with 100 training samples per task, it builds a pipeline that rewrites questions, self-distills reasoning traces, and then runs PPO with a process reward model that scores each step. On eight scientific multiple-choice tasks from SciKnowEval, the best pipeline, supervised fine-tuning followed by process-reward RL with data augmentation, raises average accuracy from 0.403 to 0.447, an 11 percent relative gain over the unmodified base model. A reader should care because this is a concrete recipe for turning a strong general reasoner into a specialist on tiny datasets, and because the experiments isolate which ingredients matter.","feed_headline":"100 samples per task lift a reasoning model by 11%","feed_subtitle":"Self-distilled reasoning traces plus process-reward RL beat plain fine-tuning on eight scientific tasks.","key_machinery":"The load-bearing object is a language-augmented Markov decision process in which a reasoning step or answer prediction is an action, and the reward for each step is produced by a process reward model. The central identity is the blended reward $R_i = \\alpha \\times or_i + (1-\\alpha) \\times f(pr_i^1,\\ldots,pr_i^m)$, where $or_i$ is 1 if the final answer matches and 0 otherwise, $f$ aggregates step scores (e.g., mean or minimum), and $\\alpha=0.7$. This reward is optimized with PPO after an SFT warm start built from self-synthesized reasoning traces; data augmentation expands the 100 samples sixfold, and a sentence-embedding retriever selects the top-three similar examples as in-context context. The action-space alignment of teacher, policy, and PRM is the mechanism invoked to explain why self-distillation succeeds while distillation from a stronger model fails.","core_discovery":"In the paper's own terms, the discovery is that reinforcement fine-tuning of a reasoning foundation model is feasible with only 100 domain-specific (question, answer) pairs, provided the limited samples are exploited in three coordinated ways: rewriting questions and shuffling options to augment the RL training set, using the policy model itself to synthesize reasoning steps and warm-starting with SFT, and injecting domain knowledge through few-shot in-context learning. The reward is a weighted blend of outcome correctness and PRM step scores, $R_i = \\alpha \\cdot or_i + (1-\\alpha) \\cdot f(pr_i^1,\\ldots,pr_i^m)$ with $\\alpha=0.7$. The strongest result is that SFT+RL(PRM)+DA achieves 0.447 average accuracy versus 0.403 for vanilla, the best among all open Skywork-initialized methods. The paper also reports that distilling reasoning traces from a stronger but action-space-misaligned teacher (QwQ-32B) drops accuracy to 0.31, below the 0.40 vanilla baseline, which the authors read as evidence that teacher-student action-space alignment matters more than raw teacher strength.","pith_inferences":["If action-space alignment is the dominant mechanism, a direct extension is to normalize reasoning-step formats across teacher and student (or fine-tune the PRM on the student's step distribution); the paper's own QwQ result predicts this would turn SFT+ from a 0.31 failure into a gain.","The reported gains could be tested for dependence on process supervision by shuffling the PRM's step scores within each rollout while keeping the final-answer reward; if accuracy stays near 0.447, outcome reward alone is doing the work.","The results are all on multiple-choice tasks with extractable final answers; transferring the recipe to free-form scientific outputs would require new reward definitions, so the 11% claim should not be assumed to generalize.","Self-distillation plus augmentation creates a self-generated data loop, suggesting a self-play variant where the policy's correct rollouts are added back to the training pool could compound the gains beyond a static 100-sample budget."],"forward_implications":["With only 100 samples per task, the best OpenRFT pipeline (SFT + RL with process reward + data augmentation) improves average accuracy by 11% over the vanilla reasoning model, and beats both outcome-only RL (ReFT) and SFT alone.","Data augmentation contributes most when the training set is small; its benefit shrinks as more real samples are added, suggesting that augmented questions carry some noise.","Using a stronger but misaligned teacher model for reasoning distillation can be worse than no distillation, because inconsistent step-level action spaces corrupt the SFT warm start.","Adding few-shot ICL to the RL prompt does not help overall accuracy, but it helps the hardest task (molecule structure prediction), leaving domain-knowledge injection as an open direction.","More domain-specific training data improves all RL-based variants, so RFT remains data-hungry even though it needs far fewer samples than conventional SFT."],"supporting_citations":[{"why":"Defines the RFT setting that OpenRFT reproduces: fine-tuning a reasoning model with dozens of (question, answer) pairs.","marker":"OpenAI, 2024"},{"why":"Supplies both the Skywork-o1 policy model and the Skywork PRM used in all experiments.","marker":"o1 Team, 2024"},{"why":"Provides SciKnowEval, the scientific benchmark and the eight L3 tasks used for evaluation.","marker":"Feng et al., 2024"},{"why":"Supplies the ReFT baseline that OpenRFT compares against and extends with PRM and warm-up.","marker":"Luong et al., 2024"},{"why":"Provides QwQ-32B-Preview, the stronger teacher used in SFT+ to demonstrate the action-space alignment effect.","marker":"Team, 2024b"},{"why":"Supplies the PPO algorithm used for the RL fine-tuning stage.","marker":"Schulman et al., 2017"},{"why":"Supplies the language-augmented MDP formulation that models reasoning steps as actions.","marker":"Wang et al., 2024"},{"why":"Co-supports the MDP grounding of language-model interactions used in the RL formulation.","marker":"Carta et al., 2023"},{"why":"Co-authors the Qwen/QwQ model report for QwQ-32B-Preview used as the stronger distillation teacher.","marker":"Yang et al., 2024"},{"why":"Supplies the RL training framework used to run PPO experiments.","marker":"Hu et al., 2024"}],"fun_headline_variants":["RFT with 100 samples per task beats vanilla fine-tuning","100 samples enough for reinforcement fine-tuning of reasoning models","Self-distilled reasoning traces plus PRM boost RL fine-tuning","RFT with only 100 samples outperforms standard fine-tuning","OpenRFT: RL fine-tuning of reasoning models from 100 samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole recipe depends on the process reward model giving trustworthy step-level scores in the same action space as the policy model; if the PRM mis-scores reasoning steps, RL can be pushed toward flawed chains even when final answers are right, and the authors note this alignment is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["RFT with 100 samples per task beats vanilla fine-tuning","100 samples enough for reinforcement fine-tuning of reasoning models","Self-distilled reasoning traces plus PRM boost RL fine-tuning","RFT with only 100 samples outperforms standard fine-tuning","OpenRFT: RL fine-tuning of reasoning models from 100 samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3080,"prompt_tokens":941,"completion_tokens":2139,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":2053}},"tokens_in":557,"tokens_out":2139,"duration_ms":12170,"temperature":1.0,"reasoning_tokens":2053,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:14:07.851108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the best pipeline and replace each PRM step score with a random permutation of the same scores while keeping the final-answer reward identical; if accuracy stays at 0.447 within noise, then process supervision is not the active ingredient. A complementary check is to run the full pipeline on an unseen ninth SciKnowEval task and see whether the 11% gain reproduces.","supporting_citations":[{"cited_title":"Openai's reinforcement fine-tuning research program","cited_arxiv_id":null,"evidence_quote":"Defines the RFT setting that OpenRFT reproduces: fine-tuning a reasoning model with dozens of (question, answer) pairs."}],"review_version":1}