REVIEW 4 major objections 5 minor 7 cited by
Tina: Tiny Reasoning Models via LoRA
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read LoRA-trained 1.5B model matches or beats full-parameter rivals for $9
desk verdict Useful, honest empirical study of LoRA-based RL for 1.5B reasoning, but the 'surpasses SOTA' headline is an artifact of selecting the best checkpoint on the test benchmarks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is LoRA-based RL: the base model's weight matrices $W_0$ stay frozen, and a low-rank adapter changes the forward pass to $h(x)=W_0x+ABx$, with $A\in\mathbb{R}^{d\times r}$, $B\in\mathbb{R}^{r\times k}$, and $r\ll\min(d,k)$. Training this adapter under a GRPO-style reinforcement-learning objective with verifiable rewards (accuracy, format, length, and related signals) keeps trainable parameters and FLOPs tiny while letting the model learn the structural format of rewarded reasoning chains. The paper also observes a phase transition in format reward and completion length during training, with the best-performing checkpoint appearing just before the transition, which it interprets as support for the format-adaptation hypothesis.
What would settle it
Re-run the Tina recipes with checkpoint selection done on a held-out validation split (for example, a reserved portion of MATH or a separate competition set), evaluate the chosen checkpoint on the six test benchmarks, and compare with baselines selected by the same rule; if the validation-selected Tina no longer matches or beats the full-parameter baselines, the central claim of LoRA-based superiority would be refuted.
Extended reading notes
Core claim
On its own terms, the paper establishes that LoRA-based RL can extract reasoning gains from the 1.5B DeepSeek-R1-Distill-Qwen-1.5B base model that are comparable to or larger than those obtained by full-parameter RL training on the same base. Across five main Tina configurations that mirror the datasets and reward designs of three public full-parameter RL baselines, average scores on a six-benchmark suite (AIME24/25, AMC23, MATH500, GPQA, Minerva) range from 48.16% to 50.60%, with the best configuration reaching 50.60% against its baseline's 41.60%. All but one configuration beat their corresponding full-parameter baseline average, and the best checkpoint reaches 43.33% Pass@1 on AIME24. The paper prices this at $9 for the best checkpoint (training plus evaluation) and $526 to reproduce every experiment and ablation reported.
Load-bearing premise
The headline comparison treats Tina's single best-performing checkpoint, chosen after evaluation, as the representative result, while baselines are represented by a single released checkpoint re-evaluated once; if checkpoint selection had instead been locked in on a held-out set, Tina's claimed advantage over full-parameter baselines could shrink or disappear.
Editorial extensions
If this is right
- The best Tina checkpoint costs about $9 to train and evaluate, so RL reasoning experiments move within reach of researchers with two L40S-class GPUs and a modest budget.
- On the same 1.5B base, LoRA updates can beat several full-parameter RL models, implying that full-parameter post-training is not automatically necessary for math-focused reasoning at this scale.
- Because the best checkpoint appears early (19% to 57% of an epoch), training can be stopped well before a full epoch, cutting compute further.
- The Dr.GRPO comparison reaches its best checkpoint at 17% of an epoch versus 57% for GRPO, suggesting algorithm choice can improve sample efficiency even within the LoRA setting.
- All code, training logs, and checkpoints are released, making the $9 result directly reproducible and auditable.
Reading between the lines
- A testable extension is to train LoRA-RL with format and length rewards only, withholding the accuracy reward; if reasoning accuracy still rises substantially, that would support the format-adaptation mechanism rather than knowledge acquisition.
- The reported margin could depend on selecting Tina's best checkpoint from all evaluated checkpoints, whereas baselines are single released checkpoints; a replication using held-out validation for early stopping would give a protocol-fair estimate of the true advantage.
- The 260x cost reduction is tied to one base model, current cloud pricing, and the specific reward scaffolding; transferring the recipe to larger bases or to code reasoning could change the ratio, since format adaptation may matter less when knowledge gaps dominate.
- Because LoRA adapters are modular and separable, the paper's approach could also be used to toggle reasoning behavior on and off per deployment without storing multiple full model copies.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Tina, a family of 1.5B reasoning models obtained by applying LoRA-based GRPO-style RL to DeepSeek-R1-Distill-Qwen-1.5B. The authors replicate the datasets and reward setups of STILL-3, DeepScaleR, and Open-RS, and report best-checkpoint zero-shot pass@1 scores on AIME24/25, AMC23, MATH500, GPQA, and Minerva. The headline claims are that Tina is competitive with, and sometimes surpasses, full-parameter SOTA baselines built on the same base model, that the best Tina checkpoint reaches 43.33% on AIME24 at an estimated $9 post-training cost, and that LoRA works by rapidly adapting to the rewarded reasoning format. The paper also includes ablations over dataset, learning rate, LoRA rank, and RL algorithm, and releases code, logs, and checkpoints.
Significance. If the comparative claim held, the result would be practically important: it would show that parameter-efficient RL on a 1.5B base can match full-parameter RL at roughly 1/260th of the post-training cost, substantially lowering the barrier to reasoning-model research. The paper has genuine strengths: baselines are re-evaluated under a single framework, hyperparameters are kept fixed by design, the cost accounting in Table 1 is unusually transparent, and all code, logs, and checkpoints are released. The format-adaptation hypothesis is clearly labeled as a hypothesis. However, the central "surpasses SOTA" claim is not supported by the evidence as presented, because Tina's best checkpoint is selected on the same six benchmarks used for the reported scores while baselines are fixed final checkpoints, and because key comparisons rest on 30-question splits with large sampling noise. The contribution is therefore a promising empirical study whose headline findings need re-analysis before the claims can be accepted.
major comments (4)
- [§4.2, Table 3, and Appendix D] The comparison between Tina and the baselines is not protocol-matched. Each Tina row in Table 3 reports the single best checkpoint over training, selected by evaluating every saved checkpoint on the same six benchmarks that are then used as the reporting set; the full trajectories in Appendix D show large swings (e.g., Tina-Open-RS2 scores 50.60 at step 450, 43.58 at step 400, and 43.05 at step 500). The baselines in Table 2, by contrast, are released final checkpoints re-evaluated once. Because the gap between Tina-Open-RS2 and the best baseline (DeepScaleR, 48.74) is only 1.86 average points, a 7-point swing over 50 steps means the claimed superiority can plausibly be a selection artifact. The only same-dataset, same-reward controlled comparison, Tina-DeepScaleR, ends slightly below its baseline (48.38 vs. 48.74). I therefore request that checkpoint selection be done on a held-out validation set, or that final-checkpoint scores and best-of-run baseline scores be reported; without this, the abstract's "sometimes surpasses" claim is not supported. Section 5's reference to "held-out evaluations" does not resolve the issue because no separate held-out selection set is described.
- [§4.2, Table 3 (AIME24/AIME25 rows)] AIME24 and AIME25 contain only 30 problems each, so a 3.33- or 6.67-point difference is one or two questions. The headline AIME24 figure of 43.33% for Tina-Open-RS2 corresponds to 13/30, versus 11/30 for DeepScaleR and 7/30 for the base model; the paper reports no confidence intervals, standard errors, or sensitivity analyses. Given that the ">20% reasoning performance increase" and the 43.33% Pass@1 number are central to the abstract, the authors should either report uncertainty estimates (e.g., bootstrap CIs) or avoid presenting single 30-question differences as decisive evidence of superiority.
- [Appendix A and Table 1] The advertised $9 cost is the retrospective cost of the best checkpoint, not the cost of discovering it. Table 1 shows that reproducing the best checkpoint in the best-performance task costs $9, but the row above it shows that reproducing all checkpoints in that task costs $31; without running all checkpoints and evaluating them on the reporting benchmarks, one cannot know that step 450 is the best. The paper itself states that the $9 is "the efficient path to the best Tina model," and that full-run cost is $31 including evaluation, but the abstract and Figure 1 present the $9 as the cost of the approach. Comparisons with baseline costs (which are full-run costs) should be made on the same basis, for example by reporting both best-of-run and full-run costs and clarifying which one is used in the 260x figure.
- [§5 and Figure 3] The "less is more" FLOPs-versus-performance analysis is confounded by the checkpoint-selection protocol. Tina points in Figure 3 are best checkpoints chosen on the evaluation benchmarks, while baseline points are full runs, so the inverse relationship between FLOPs and performance may reflect selection noise rather than a property of LoRA-based RL. Additionally, the phase-transition analysis is qualitative: no operational definition of "training turning point" is given, and the placement of the green dashed line appears to be determined by visual inspection of the same curves used to support the hypothesis. The format-adaptation hypothesis is a reasonable conjecture, but the current evidence is not sufficient to present it as the main explanation for the results.
minor comments (5)
- [Abstract and §4.2] The phrase ">20% reasoning performance increase" does not state the reference point; please specify whether the increase is over the base model's average across six benchmarks, over AIME24 alone, or over the corresponding full-parameter baseline.
- [§4.3 and Table 4] Model naming is inconsistent: "Tina-Open-RS" in the dataset-ablation text and table is the same as "Tina-Open-RS2," and the footnote in Table 4 should be moved into the main text so that readers are not confused when matching rows across tables.
- [Appendix B.1] The displayed formula for D_KL in the GRPO objective is not the standard KL divergence and appears to be an estimator formula; please provide a derivation or citation so that the notation is unambiguous.
- [Appendix D] The appendix tables would be easier to use if the selected best checkpoint per run were highlighted and the selection rule (max average over the six evaluation benchmarks) were stated explicitly; currently the reader must reconstruct this from the Steps column in Table 3.
- [§5 and Appendix E] Please provide a quantitative criterion for detecting the "training turning point" in the reward and length curves; as written, the green dashed lines are not reproducible from the raw logs.
Circularity Check
Empirical LoRA-RL report; no circular derivation. The reported best-checkpoint scores are selected on the same six benchmarks, which is a statistical validity concern rather than a definitional reduction.
full rationale
Tina is an empirical systems report: it fixes DeepSeek-R1-Distill-Qwen-1.5B, trains LoRA adapters with GRPO on public datasets and reward pipelines, and evaluates on six benchmarks. LoRA and GRPO are standard external formulations (Appendix B), and the hyperparameters are adopted from OpenR1 and Open-RS rather than fitted to the reported outcomes. The comparative claims rest on re-evaluating released baselines and Tina checkpoints under a single lighteval/vLLM protocol (Section 4.1), which is an externally checkable procedure. The only self-citation by the present authors, Wang and Neiswanger (2025), supports the general statement that multi-step reasoning remains a frontier challenge; it is not load-bearing. One genuine caveat is checkpoint selection: Table 3 reports the best checkpoint from Appendix D, chosen on the same six benchmarks, while baselines are single final checkpoints. This means the headline Tina scores are order statistics over noisy training trajectories, and the 'sometimes surpasses SOTA' claim may be optimistic. But that is a selection-bias or evaluation-protocol issue, not circular reasoning: the reported numbers are observed evaluations, and no fitted parameter is renamed as a prediction. The derivation chain does not reduce to its inputs.
Assumptions & free parameters
free parameters (2)
- Reported best checkpoint step =
varies per run; e.g., step 450 out of 875 for Tina-Open-RS2
- LoRA rank and alpha =
32 and 128 as defaults; ablations vary rank 4 to 64
assumptions (3)
- domain assumption DeepSeek-R1-Distill-Qwen-1.5B provides a strong prior for reasoning, so RL starts from an already-competent base.
- domain assumption The OpenR1 reward functions (accuracy, format, length, etc.) with the listed weights constitute a suitable RL signal.
- standard math GRPO and Dr.GRPO are valid and correctly implemented optimization algorithms.
Cite this review
Pith. "Pith review of Tina: Tiny Reasoning Models via LoRA." pith.science (2026). https://pith.science/paper/RZV2D4YA
@misc{pith2026250415777,
author = {Pith},
title = {Pith review of: Tina: Tiny Reasoning Models via LoRA},
year = {2026},
howpublished = {\url{https://pith.science/paper/RZV2D4YA}},
note = {Machine review of arXiv:2504.15777}
}
abstract
How cost-effectively can strong reasoning abilities be achieved in language models? Driven by this fundamental question, we present Tina, a family of tiny reasoning models achieved with high cost-efficiency. Notably, Tina demonstrates that substantial reasoning performance can be developed using only minimal resources, by applying parameter-efficient updates during reinforcement learning (RL), using low-rank adaptation (LoRA), to an already tiny 1.5B parameter base model. This minimalist approach produces models that achieve reasoning performance which is competitive with, and sometimes surpasses, SOTA RL reasoning models built upon the same base model. Crucially, this is achieved at a tiny fraction of the computational post-training cost employed by existing SOTA models. In fact, the best Tina model achieves a >20\% reasoning performance increase and 43.33\% Pass@1 accuracy on AIME24, at only \$9 USD post-training and evaluation cost (i.e., an estimated 260x cost reduction). Our work reveals the surprising effectiveness of efficient RL reasoning via LoRA. We validate this across multiple open-source reasoning datasets and various ablation settings starting with a single, fixed set of hyperparameters. Furthermore, we hypothesize that this effectiveness and efficiency stem from LoRA rapidly adapting the model to the structural format of reasoning rewarded by RL, while largely preserving the base model's underlying knowledge. In service of accessibility and open research, we fully open-source all code, training logs, and model weights \& checkpoints.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 7 Pith papers
-
LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts
A training method that recovers reinforcement-learning gradient on all-fail math prompts by fitting a temporary LoRA adapter, re-rolling, and splicing successful answers into the RL batch.
-
Resa: Transparent Reasoning Models via SAEs
SAE-Tuning, a sparse-autoencoder-guided SFT procedure, elicits RL-comparable reasoning in 1.5B models from CoT-free QA data at about $1 and 20 minutes of training.
-
RECIPE-TKG: From Sparse History to Structured Reasoning for LLM-based Temporal Knowledge Graph Completion
RECIPE-TKG combines rule-based multi-hop history sampling, contrastive LoRA fine-tuning, and test-time semantic filtering to improve LLM temporal knowledge graph completion, with Hits@10 gains up to 30.6% over prior L...
-
The Hallucination Tax of Reinforcement Finetuning
Standard RFT sharply reduces LLM refusal on unanswerable questions, and adding 10% synthetic unanswerable math during RFT restores refusal with small accuracy losses.
-
Efficient Reasoning on the Edge
LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
-
Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters
The paper proves an upper bound of about sqrt(r/N) on the LoRA generalization gap and claims a matching lower bound, but both proofs contain structural gaps.
-
Get Experience from Practice: LLM Agents with Record & Replay
AgentRR is a proposed paradigm that records agent traces, generalizes them into multi-level experiences, and replays them under safety checks to make LLM agents cheaper, faster, and more reliable.
Reference graph
Works this paper leans on
-
[1]
Note thatϵ and β are parameters controlling the clipping range and KL penalty, respectively. B.2. LoRA Formulation We follow the standard LoRA setup (Hu et al., 2021). Given a frozen pretrained weight matrixW0∈ Rd×k and trainable low-rank matricesA ∈ Rd×r and B ∈ Rr×k with r ≪ min(d, k), the original forward pass h(x)= W0x is modified as ˆh(x)= W0x + ABx ...
work page 2021
-
[3]
URLhttps://arxiv.org/abs/2502.01456. Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small llms: What works and what doesn’t,
-
[4]
URLhttps://arxiv.org/abs/2503.16219. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in llms via reinforcement learning,
-
[5]
Clémentine Fourrier, Nathan Habib, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall
URL https://arxiv.org/abs/2501.12948. Clémentine Fourrier, Nathan Habib, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation,
-
[7]
URLhttps://arxiv.org/abs/2403.14608. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset,
-
[9]
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang
URLhttps://arxiv.org/abs/ 2106.09685. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-Reasoner- Zero: An open source approach to scaling reinforcement learning on the base model,
-
[10]
URLhttps://arxiv.org/abs/2411.16489. Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January
-
[11]
URLhttps://arxiv.org/abs/2411.15124. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. InProceedings of Advances in Neural Informa...
Show all 24 references
-
[12]
URLhttps://arxiv.org/abs/2401.08967. Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, Weihan Cao, Jiangning Liu, Hongwei Liu, Junnan Liu, Songyang Zhang, Dahua Lin, and Kai Chen. Exploring the limit of ...
-
[13]
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan
URL https://arxiv.org/abs/2502.06781. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods,
-
[14]
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto
URLhttps: //arxiv.org/abs/2412.09413. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling,
-
[15]
NovaSky Team
URL https://arxiv.org/abs/2501.19393. NovaSky Team. Sky-T1: Train your own o1 preview model within $450,
-
[16]
OpenThoughts Team
URLhttps://arxiv.org/abs/2412.16720. OpenThoughts Team. Open Thoughts, January
-
[17]
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He
URLhttps://arxiv.org/abs/2408.06195. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimization towards training A trillion parameter models.CoRR, abs/1910.02054,
1910 arXiv
-
[19]
15 Tina: Tiny Reasoning Models via LoRA Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu
URLhttps://arxiv.org/abs/2402.03300. 15 Tina: Tiny Reasoning Models via LoRA Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of European Confer...
-
[20]
URL http://dx.doi.org/10.1145/3689031.3696075
doi: 10.1145/3689031.3696075. URL http://dx.doi.org/10.1145/3689031.3696075. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning,
-
[21]
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He
URLhttps://arxiv.org/abs/2501.09686. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL-Zoo: Investigating and taming zero reinforcement learning for open base models in the wild,
-
[22]
16 Tina: Tiny Reasoning Models via LoRA Appendix A
URL https://arxiv.org/abs/2503.18892. 16 Tina: Tiny Reasoning Models via LoRA Appendix A. Cost Breakdown This section provides further details on how training data amounts, computational cost, time cost, and performance metrics reported in this paper – particularly those prese...
2025 arXiv
-
[24]
lighteval vllm $MODEL_ARGS
20 Tina: Tiny Reasoning Models via LoRA C.2. Evaluation Command Thefollowingistheevaluationcommandweusetocombine lightevaland vLLMforperformanceevaluation on reasoning tasks. TheMODEL_PATH should be replaced with either the local path or huggingface identifier to the model to ...
2000
-
[2019]
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman
URLhttp://arxiv.org/abs/ 1910.02054. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-proof Q&A benchmark. InProceedings of Conference on Language Modeling (COLM),
1910 arXiv
-
[2021]
Edward J
URLhttps: //arxiv.org/abs/2103.03874. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models,
-
[2023]
URL https://github.com/huggingface/ lighteval. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu...
-
[2024]
Cudo Compute
URLhttps://arxiv.org/abs/2412.15287. Cudo Compute. Nvidia L40S pricing. URLhttps://www.cudocompute.com/products/gpu-cloud/ nvidia-l40s. Accessed: 2025-04-21. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Ji...
2025
-
[2025]
Zeyuan Allen-Zhu and Yuanzhi Li
URLhttps://arxiv.org/abs/2503.04697. Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. In Proceedings of International Conference on Learning Representations (ICLR),
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.