{"id":"c734e902-e371-4477-a8d7-28d020557e16","arxiv_id":"2504.15777","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Applying LoRA during reinforcement learning to a 1.5B reasoning model yields competitive math and scientific reasoning at roughly $9 of training and evaluation cost.","lead":"Can a tiny language model learn complex reasoning for nine dollars? This paper shows that adding LoRA adapters during reinforcement learning lets a 1.5B model match or beat much more expensive full-model reasoning training on several benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 'surpasses SOTA' rests on selecting Tina's best checkpoint on the same six test benchmarks, while baselines are fixed final checkpoints; a 100-step shift in one run swings the average by 7 points.","rationale":"The reader's weakest assumption matches the load-bearing point, and Appendix D makes the selection effect concrete rather than hypothetical. The Tina-Open-RS2 checkpoint at step 450 is an isolated maximum: adjacent checkpoints are 7 points lower on the same six-benchmark average, so the reported 'best' is not a stable estimate of method performance under a fixed training schedule. Meanwhile, baselines are final released checkpoints, and the one same-dataset comparison (Tina-DeepScaleR) shows a slight deficit. Because of this asymmetry, the abstract's 'sometimes surpasses SOTA' is not yet established, although the weaker claim that LoRA-based RL is competitive and dramatically cheaper is plausible and supported. The paper deserves credit for releasing code, checkpoints, logs, and a transparent cost table, and Appendix A does disclose that the $9 figure is the best-checkpoint path rather than the full-run cost. Still, the headline comparisons should be conditioned on a held-out checkpoint-selection protocol or reported as mean/max over checkpoints, with baseline best-of-run where checkpoints exist. The reader's CONDITIONAL verdict already captures this, so my stress-test does not change the verdict.","tokens_in":27286,"tokens_out":9120,"duration_ms":90603,"concrete_test":"Use the released checkpoints in Appendix D. Split the six benchmarks into two matched halves, e.g., selection = {AIME24, AMC23, GPQA} and reporting = {AIME25, MATH500, Minerva}. For each Tina model, pick the checkpoint maximizing the selection-set average and record its reporting-set average; repeat with the halves swapped. Compare these selection-rule averages against baseline averages on the same reporting sets. If either split yields a Tina reporting average below the corresponding baseline average, the 'surpasses SOTA' claim in the abstract should be downgraded to 'competitive.' Additionally, run a paired bootstrap over the 17 Tina-Open-RS2 checkpoints to estimate the expected maximum under checkpoint-level noise, which would quantify how much of the 50.60 vs 48.74 gap is max-selection inflation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim—that LoRA-based RL 'sometimes surpasses' full-parameter SOTA—requires that Tina scores and baseline scores be produced under comparable model-selection protocols. They are not. In Table 3, each Tina row's 'Steps (% of 1 epoch)' identifies the best checkpoint from Appendix D, chosen by evaluating every saved checkpoint on the same six benchmarks that are then used to report the headline number. This makes the reported score an order statistic over a noisy training trajectory, not the performance of a fixed stopping rule. Section 5 calls these 'held-out evaluations,' but they are the same six reporting benchmarks, not a separate validation set for checkpoint selection. The magnitude is material: for Tina-Open-RS2, the model behind Figure 1 and the 50.60% average, step 450 scores 50.60 while step 400 is 43.58 and step 500 is 43.05; the gap between the best Tina and the top baseline DeepScaleR is only 1.86 average points. Baselines, by contrast, are single released final checkpoints re-evaluated in Table 2. The one controlled same-dataset/same-reward comparison, Tina-DeepScaleR, ends slightly below its baseline (48.38 vs 48.74). It is therefore possible that the superiority headline is largely a selection artifact: maximizing over 17 checkpoints on the evaluation set while comparing against one final checkpoint. This does not undo the paper's useful observation that LoRA-based RL is cheap and competitive, but it does undermine the specific claim of surpassing SOTA.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Tina, a family of 1.5B reasoning models obtained by applying LoRA-based GRPO-style RL to DeepSeek-R1-Distill-Qwen-1.5B. The authors replicate the datasets and reward setups of STILL-3, DeepScaleR, and Open-RS, and report best-checkpoint zero-shot pass@1 scores on AIME24/25, AMC23, MATH500, GPQA, and Minerva. The headline claims are that Tina is competitive with, and sometimes surpasses, full-parameter SOTA baselines built on the same base model, that the best Tina checkpoint reaches 43.33% on AIME24 at an estimated $9 post-training cost, and that LoRA works by rapidly adapting to the rewarded reasoning format. The paper also includes ablations over dataset, learning rate, LoRA rank, and RL algorithm, and releases code, logs, and checkpoints.","tokens_in":27542,"tokens_out":7285,"duration_ms":63170,"significance":"If the comparative claim held, the result would be practically important: it would show that parameter-efficient RL on a 1.5B base can match full-parameter RL at roughly 1/260th of the post-training cost, substantially lowering the barrier to reasoning-model research. The paper has genuine strengths: baselines are re-evaluated under a single framework, hyperparameters are kept fixed by design, the cost accounting in Table 1 is unusually transparent, and all code, logs, and checkpoints are released. The format-adaptation hypothesis is clearly labeled as a hypothesis. However, the central \"surpasses SOTA\" claim is not supported by the evidence as presented, because Tina's best checkpoint is selected on the same six benchmarks used for the reported scores while baselines are fixed final checkpoints, and because key comparisons rest on 30-question splits with large sampling noise. The contribution is therefore a promising empirical study whose headline findings need re-analysis before the claims can be accepted.","major_comments":[{"comment":"The comparison between Tina and the baselines is not protocol-matched. Each Tina row in Table 3 reports the single best checkpoint over training, selected by evaluating every saved checkpoint on the same six benchmarks that are then used as the reporting set; the full trajectories in Appendix D show large swings (e.g., Tina-Open-RS2 scores 50.60 at step 450, 43.58 at step 400, and 43.05 at step 500). The baselines in Table 2, by contrast, are released final checkpoints re-evaluated once. Because the gap between Tina-Open-RS2 and the best baseline (DeepScaleR, 48.74) is only 1.86 average points, a 7-point swing over 50 steps means the claimed superiority can plausibly be a selection artifact. The only same-dataset, same-reward controlled comparison, Tina-DeepScaleR, ends slightly below its baseline (48.38 vs. 48.74). I therefore request that checkpoint selection be done on a held-out validation set, or that final-checkpoint scores and best-of-run baseline scores be reported; without this, the abstract's \"sometimes surpasses\" claim is not supported. Section 5's reference to \"held-out evaluations\" does not resolve the issue because no separate held-out selection set is described.","section":"§4.2, Table 3, and Appendix D"},{"comment":"AIME24 and AIME25 contain only 30 problems each, so a 3.33- or 6.67-point difference is one or two questions. The headline AIME24 figure of 43.33% for Tina-Open-RS2 corresponds to 13/30, versus 11/30 for DeepScaleR and 7/30 for the base model; the paper reports no confidence intervals, standard errors, or sensitivity analyses. Given that the \">20% reasoning performance increase\" and the 43.33% Pass@1 number are central to the abstract, the authors should either report uncertainty estimates (e.g., bootstrap CIs) or avoid presenting single 30-question differences as decisive evidence of superiority.","section":"§4.2, Table 3 (AIME24/AIME25 rows)"},{"comment":"The advertised $9 cost is the retrospective cost of the best checkpoint, not the cost of discovering it. Table 1 shows that reproducing the best checkpoint in the best-performance task costs $9, but the row above it shows that reproducing all checkpoints in that task costs $31; without running all checkpoints and evaluating them on the reporting benchmarks, one cannot know that step 450 is the best. The paper itself states that the $9 is \"the efficient path to the best Tina model,\" and that full-run cost is $31 including evaluation, but the abstract and Figure 1 present the $9 as the cost of the approach. Comparisons with baseline costs (which are full-run costs) should be made on the same basis, for example by reporting both best-of-run and full-run costs and clarifying which one is used in the 260x figure.","section":"Appendix A and Table 1"},{"comment":"The \"less is more\" FLOPs-versus-performance analysis is confounded by the checkpoint-selection protocol. Tina points in Figure 3 are best checkpoints chosen on the evaluation benchmarks, while baseline points are full runs, so the inverse relationship between FLOPs and performance may reflect selection noise rather than a property of LoRA-based RL. Additionally, the phase-transition analysis is qualitative: no operational definition of \"training turning point\" is given, and the placement of the green dashed line appears to be determined by visual inspection of the same curves used to support the hypothesis. The format-adaptation hypothesis is a reasonable conjecture, but the current evidence is not sufficient to present it as the main explanation for the results.","section":"§5 and Figure 3"}],"minor_comments":[{"comment":"The phrase \">20% reasoning performance increase\" does not state the reference point; please specify whether the increase is over the base model's average across six benchmarks, over AIME24 alone, or over the corresponding full-parameter baseline.","section":"Abstract and §4.2"},{"comment":"Model naming is inconsistent: \"Tina-Open-RS\" in the dataset-ablation text and table is the same as \"Tina-Open-RS2,\" and the footnote in Table 4 should be moved into the main text so that readers are not confused when matching rows across tables.","section":"§4.3 and Table 4"},{"comment":"The displayed formula for D_KL in the GRPO objective is not the standard KL divergence and appears to be an estimator formula; please provide a derivation or citation so that the notation is unambiguous.","section":"Appendix B.1"},{"comment":"The appendix tables would be easier to use if the selected best checkpoint per run were highlighted and the selection rule (max average over the six evaluation benchmarks) were stated explicitly; currently the reader must reconstruct this from the Steps column in Table 3.","section":"Appendix D"},{"comment":"Please provide a quantitative criterion for detecting the \"training turning point\" in the reward and length curves; as written, the green dashed lines are not reproducible from the raw logs.","section":"§5 and Appendix E"}],"recommendation":"major_revision","confidential_remarks":"This is a well-organized empirical report with unusually transparent cost accounting and full release of artifacts, but the headline comparative claims need to be re-derived under a fair selection protocol. The issues are fixable within the manuscript's scope, so I would not reject; however, the current version should not be accepted because the \"surpasses SOTA\" and \"$9\" statements are not supported by the evidence as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is worth reading and worth refereeing, but the headline 'surpasses SOTA' should not survive contact with the tables. The core empirical observation—LoRA-based RL on a 1.5B model gets within a couple of points of full-parameter RL at roughly 1/260th of the compute—is plausible, reproducible, and useful. The paper ships code, checkpoints, W&B logs, a full cost breakdown, and careful re-evaluations of baselines under one evaluation harness. That is real work and real value. The ablation set (datasets, learning rate, rank, algorithm) is thorough, and the 'rapid format adaptation' story is explicitly labeled a hypothesis, which is the right level of confidence given the evidence.\n\nThe soft spot is the comparative claim. The stress-test note is correct: in Tables 3 and 4, each Tina row is the single best checkpoint selected by evaluating every saved checkpoint on the same six benchmarks that are then used for reporting. Baselines are single released final checkpoints re-evaluated in Table 2. That is not a like-for-like comparison. The magnitude matters: for Tina-Open-RS2, step 450 scores 50.60, while step 400 is 43.58 and step 500 is 43.05—a 100-step shift swings the average by 7 points. The gap to the strongest baseline, DeepScaleR, is under 2 points. And the one controlled same-dataset/same-reward comparison, Tina-DeepScaleR, lands slightly below its baseline (48.38 vs 48.74). So 'sometimes surpasses SOTA' is mostly a selection artifact. That should be fixed with a held-out validation set for checkpoint selection, or reported as best-of-run for all models, or final-to-final. The paper's own numbers contain the evidence to support this correction.\n\nMinor issues: AIME24/25 are 30-question sets, so single-point differences are noise; no error bars reported. The $9 figure is honest (it includes training and evaluation of the best checkpoint) but is also cherry-picked by definition; the appendix does a good job of showing the full cost picture, so this is a labeling issue, not a hidden cost.\n\nWho it's for: anyone doing RL-based reasoning post-training on small models, and anyone who wants a cheap testbed for RL algorithms. I'd bring it to a reading group and I'd cite the cost numbers. It deserves a serious referee—not a desk reject—but the requested revision should require the checkpoint-selection fix and more careful claim language.","headline":"Useful, honest empirical study of LoRA-based RL for 1.5B reasoning, but the 'surpasses SOTA' headline is an artifact of selecting the best checkpoint on the test benchmarks.","tokens_in":28148,"tokens_out":2114,"would_cite":true,"duration_ms":17857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LoRA-trained 1.5B model matches or beats full-parameter rivals for $9","keywords":["LoRA","reinforcement learning","mathematical reasoning","small language models","GRPO","parameter-efficient fine-tuning","cost-efficient training","AIME24"],"falsifier":"Re-run the Tina recipes with checkpoint selection done on a held-out validation split (for example, a reserved portion of MATH or a separate competition set), evaluate the chosen checkpoint on the six test benchmarks, and compare with baselines selected by the same rule; if the validation-selected Tina no longer matches or beats the full-parameter baselines, the central claim of LoRA-based superiority would be refuted.","tokens_in":27058,"feed_emoji":"🧠","tokens_out":10201,"duration_ms":87632,"temperature":0.7,"pith_summary":"Driven by the question of how cheaply strong reasoning can be trained into a language model, this paper claims that applying parameter-efficient Low-Rank Adaptation (LoRA) during reinforcement learning to a tiny 1.5B-parameter base model yields reasoning performance competitive with, and sometimes better than, state-of-the-art full-parameter RL reasoning models built on the same base. The best Tina model achieves 43.33% Pass@1 on AIME24 and more than a 20% reasoning performance increase, at an estimated post-training and evaluation cost of roughly $9 USD, about a 260x cost reduction. The paper further hypothesizes that LoRA works because it quickly adapts the model to the rewarded reasoning format while preserving the base model's knowledge. If right, this makes RL-driven reasoning research far more accessible: the paper prices the entire experimental program at about $526 to reproduce from scratch.","feed_headline":"LoRA-trained 1.5B model matches or beats full-parameter rivals for $9","feed_subtitle":"Reinforcement learning on a 1.5B model hits 43% on AIME24 at roughly 1/260th the post-training cost.","key_machinery":"The load-bearing mechanism is LoRA-based RL: the base model's weight matrices $W_0$ stay frozen, and a low-rank adapter changes the forward pass to $h(x)=W_0x+ABx$, with $A\\in\\mathbb{R}^{d\\times r}$, $B\\in\\mathbb{R}^{r\\times k}$, and $r\\ll\\min(d,k)$. Training this adapter under a GRPO-style reinforcement-learning objective with verifiable rewards (accuracy, format, length, and related signals) keeps trainable parameters and FLOPs tiny while letting the model learn the structural format of rewarded reasoning chains. The paper also observes a phase transition in format reward and completion length during training, with the best-performing checkpoint appearing just before the transition, which it interprets as support for the format-adaptation hypothesis.","core_discovery":"On its own terms, the paper establishes that LoRA-based RL can extract reasoning gains from the 1.5B DeepSeek-R1-Distill-Qwen-1.5B base model that are comparable to or larger than those obtained by full-parameter RL training on the same base. Across five main Tina configurations that mirror the datasets and reward designs of three public full-parameter RL baselines, average scores on a six-benchmark suite (AIME24/25, AMC23, MATH500, GPQA, Minerva) range from 48.16% to 50.60%, with the best configuration reaching 50.60% against its baseline's 41.60%. All but one configuration beat their corresponding full-parameter baseline average, and the best checkpoint reaches 43.33% Pass@1 on AIME24. The paper prices this at $9 for the best checkpoint (training plus evaluation) and $526 to reproduce every experiment and ablation reported.","pith_inferences":["A testable extension is to train LoRA-RL with format and length rewards only, withholding the accuracy reward; if reasoning accuracy still rises substantially, that would support the format-adaptation mechanism rather than knowledge acquisition.","The reported margin could depend on selecting Tina's best checkpoint from all evaluated checkpoints, whereas baselines are single released checkpoints; a replication using held-out validation for early stopping would give a protocol-fair estimate of the true advantage.","The 260x cost reduction is tied to one base model, current cloud pricing, and the specific reward scaffolding; transferring the recipe to larger bases or to code reasoning could change the ratio, since format adaptation may matter less when knowledge gaps dominate.","Because LoRA adapters are modular and separable, the paper's approach could also be used to toggle reasoning behavior on and off per deployment without storing multiple full model copies."],"forward_implications":["The best Tina checkpoint costs about $9 to train and evaluate, so RL reasoning experiments move within reach of researchers with two L40S-class GPUs and a modest budget.","On the same 1.5B base, LoRA updates can beat several full-parameter RL models, implying that full-parameter post-training is not automatically necessary for math-focused reasoning at this scale.","Because the best checkpoint appears early (19% to 57% of an epoch), training can be stopped well before a full epoch, cutting compute further.","The Dr.GRPO comparison reaches its best checkpoint at 17% of an epoch versus 57% for GRPO, suggesting algorithm choice can improve sample efficiency even within the LoRA setting.","All code, training logs, and checkpoints are released, making the $9 result directly reproducible and auditable."],"supporting_citations":[{"why":"It defines LoRA, the low-rank update mechanism that is the core parameter-efficient training method.","marker":"Hu et al., 2021"},{"why":"It provides the 1.5B base model and the R1-style reinforcement learning framing the paper builds on.","marker":"DeepSeek-AI, 2025"},{"why":"It introduces GRPO, the group-relative policy optimization algorithm used for the RL updates.","marker":"Shao et al., 2024"},{"why":"It supplies a full-parameter baseline, its dataset, and its reward design that one Tina configuration mirrors.","marker":"Luo et al., 2025"},{"why":"It supplies another full-parameter baseline and the dataset and reward pipeline that another Tina configuration mirrors.","marker":"RUCAIBox STILL Team, 2025"},{"why":"It supplies the Open-RS baselines, their curated datasets, reward scaffolding, and the default hyperparameters used.","marker":"Dang and Ngo, 2025"},{"why":"It provides the open training codebase and reward definitions that the Tina pipeline inherits.","marker":"Hugging Face, 2025"},{"why":"It provides the fast inference engine used for the standardized evaluation of all models.","marker":"Kwon et al., 2023"}],"fun_headline_variants":["Tiny LoRA model beats 260x costlier rivals on AIME24","1.5B LoRA model hits 43% on AIME24 for $9","LoRA RL: $9 model outperforms full-parameter training","Tiny model, tiny cost: LoRA RL wins on reasoning","1.5B LoRA RL: $9, 43% AIME24, beats SOTA baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison treats Tina's single best-performing checkpoint, chosen after evaluation, as the representative result, while baselines are represented by a single released checkpoint re-evaluated once; if checkpoint selection had instead been locked in on a held-out set, Tina's claimed advantage over full-parameter baselines could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Tiny LoRA model beats 260x costlier rivals on AIME24","1.5B LoRA model hits 43% on AIME24 for $9","LoRA RL: $9 model outperforms full-parameter training","Tiny model, tiny cost: LoRA RL wins on reasoning","1.5B LoRA RL: $9, 43% AIME24, beats SOTA baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":3021,"prompt_tokens":1019,"completion_tokens":2002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1893}},"tokens_in":635,"tokens_out":2002,"duration_ms":12427,"temperature":1.0,"reasoning_tokens":1893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:18:06.667291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Tina recipes with checkpoint selection done on a held-out validation split (for example, a reserved portion of MATH or a separate competition set), evaluate the chosen checkpoint on the six test benchmarks, and compare with baselines selected by the same rule; if the validation-selected Tina no longer matches or beats the full-parameter baselines, the central claim of LoRA-based superiority would be refuted.","supporting_citations":[],"review_version":1}