REVIEW 4 major objections 5 minor 10 references
Token-Efficient RL for LLM Reasoning
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Under LoRA fine-tuning, a token-level prefix-matching RL method raises Qwen2-1.5B's 3-digit multiplication accuracy from 3.9% to 70.0%, while full-token GRPO barely improves it.
desk verdict Large LoRA-based RL gains on reasoning, but the two core objective equations are wrong as printed, so the empirical claims need correction and replication before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the token-level advantage A(v|p)=E[R|p∘v]−E[R|p], estimated by averaging exact-match rewards over the |G| sampled completions that share each prefix. This is the mechanism that lets T-SPMO assign credit to individual decisions rather than whole trajectories; the policy update maximizes the summed log-probabilities of high-advantage transitions, and a replay mechanism resamples from successful completions to build prefix tries deeper into the output. S-GRPO uses a simpler version of the same idea, a stochastic token mask with an always-on early-token cutoff.
What would settle it
Compare the empirical A(v|p) computed from T-SPMO's 50 completions per prompt against the same quantities computed from a large sample, for example 1000 completions. If the correlation is near zero for long prefixes, the credit-assignment mechanism is not reliably estimating what the objective claims to optimize, and the reported accuracy gains would need another explanation.
Extended reading notes
Core claim
The central discovery is that, under LoRA constraints, updating the policy with a small, carefully chosen set of output tokens outperforms updating on every token. S-GRPO keeps early tokens deterministically and samples later tokens up to a budget k. T-SPMO goes further: it builds a prefix trie over sampled completions, estimates a token-level advantage A(v|p)=E[R|p∘v]−E[R|p] for each unique (prefix, next-token) transition from empirical reward averages, and reinforces transitions whose continuation improves expected reward. On the multiplication task, T-SPMO's fine-grained credit assignment allows correct partial steps to be reinforced even when the final answer is wrong, which the paper identifies as the reason it reaches 70.0% accuracy where full-token GRPO reaches 4.4%.
Load-bearing premise
T-SPMO's token-level advantages are computed by averaging rewards over only 50 sampled completions per prompt, so many prefix (p,v) pairs are observed once or not at all; if those sparse averages misassign credit, the reported gains may not generalize to other prompts or tasks.
Editorial extensions
If this is right
- Reasoning RL becomes feasible on a single partitioned GPU: T-SPMO uses under 5% of the tokens GRPO puts through the loss, and S-GRPO under 50%, with both beating the full-token baseline.
- For exact-match arithmetic tasks, token-level credit assignment is the recommended setting; S-GRPO is preferred when broader output structure matters more than local correctness.
- Full-token GRPO under LoRA should not be assumed to be a safe default: in these experiments it fails to improve over the base model, so selective token optimization may be acting as an implicit regularizer in low-parameter regimes.
- LoRA-compatible, critic-free RL closes a practical gap, letting groups without large compute budgets fine-tune small models on reasoning tasks.
Reading between the lines
- A direct testable extension is to estimate the same token-level advantages from a much larger sample of completions; if accuracy improves with group size, the sparse average is a bottleneck, and if not, the regularizing effect of sparse sampling is the active ingredient.
- The prefix-trie advantage estimator should transfer to other tasks where exact-match reward over structured outputs is available, such as code repair or constrained generation, since it does not rely on arithmetic-specific structure.
- The paper's comparison suggests that gradient interference from full-trajectory losses under LoRA may be the cause of GRPO's failure; one could test this by training GRPO with a much higher KL penalty or smaller learning rate to see if the failure persists.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces two reinforcement learning methods for fine-tuning LLM reasoning under strict memory/compute constraints, specifically with LoRA. S-GRPO is a stochastic variant of GRPO that subsamples tokens contributing to the policy loss. T-SPMO performs token-level credit assignment by building a prefix trie over sampled completions, estimating advantages for each (prefix, next-token) transition from empirical reward averages, and updating the policy with a weighted objective plus L2 regularization on LoRA weights. Experiments on Qwen2-1.5B report strong gains on SVAMP (from 45.0% base accuracy to 70.3% S-GRPO / 71.6% T-SPMO) and on 3-digit multiplication (from 3.9% base to 22.9% S-GRPO / 70.0% T-SPMO), while a full-token GRPO baseline fails to improve over the base model. The paper argues that selective token-level optimization acts as an implicit regularizer in low-parameter LoRA training.
Significance. If the reported results are reproducible and the methods are correctly specified, the paper would make a useful practical contribution: memory-efficient RL for reasoning that works under LoRA constraints, with a surprising and interesting negative result for full-token GRPO. The claimed token-efficiency numbers (Table 3) are also striking. However, the manuscript's formal derivations contain load-bearing errors: Eq. (5) as printed is degenerate, and Eq. (10) is not a policy-gradient objective. Without correction, the reported accuracy gains cannot be attributed to the stated algorithms. The lack of seeds, error bars, and the manual peak-validation checkpoint selection further weaken the empirical claims. The paper ships no code, so the gap between the printed equations and the actual implementation cannot be checked.
major comments (4)
- [Section 3.2, Eq. (5)] The S-GRPO objective is printed with the ratio πθ(y_t|x,y_<t) / πθ(y_t|x,y_<t) annotated 'no grad'. This ratio is identically 1, so the update reduces to a pure KL penalty with no policy-gradient term; as written, Eq. (5) does not implement the described algorithm. The intended ratio presumably involves a frozen old policy πθ_old, as in Eq. (3). This is not a cosmetic typo: it determines whether the reported S-GRPO results can be attributed to the stated method. The manuscript must correct Eq. (5), specify the frozen reference policy, and clarify whether the implementation matches the corrected equation.
- [Section 3.3, Eq. (10)] The T-SPMO objective uses πθ(v|p)·A(v|p) instead of log πθ(v|p)·A(v|p) (or an importance-sampling ratio). Consequently, ∇J_TSPMO = Σ_{(p,v)∈U} A(v|p)∇πθ(v|p), which is not the gradient of expected reward, even if the advantages A(v|p) were exactly the true conditional expectations. The policy gradient theorem requires the log-probability (or an equivalent ratio) precisely so that the expectation over the policy's distribution is recovered. Additionally, U contains only the next tokens observed in the finite set of sampled completions; the sum over this subset does not approximate an expectation over the full vocabulary, and a baseline does not cancel over this partial support. This is a load-bearing error: the central claim that T-SPMO performs token-level RL via baseline-subtracted policy gradient is not supported by Eq. (10). The authors must either provide a correct derivation of an objective that provably optimizes expected reward, or present the actual objective used and empirically justify its optimization target.
- [Section 4.1 and 4.3, Table 6] The multiplication results rely on manually selecting the checkpoint with peak validation accuracy after plateauing, and the hyperparameters (group size, replay budgets, α, k) were chosen via pilot sweeps on the same benchmarks used for the final reported numbers. No seeds, repetitions, or confidence intervals are reported. Given the sensitivity of T-SPMO multiplication accuracy to configuration (Table 6 ranges from 11.1% to 70.0%), the headline 70.0% versus S-GRPO's 22.9% may reflect checkpoint selection and tuning artifacts rather than algorithmic superiority. The paper should report mean±std over multiple seeds, a fixed hold-out validation protocol, and the exact selection procedure; otherwise the central empirical claim of a large T-SPMO advantage on multiplication is not robustly established.
- [Section 3.3, Eqs. (7)-(9)] The token-level advantages are computed by averaging final exact-match rewards over the completions in G (size |G|=50 per prompt). For a typical long reasoning trace, most prefixes appear in only one or a few completions, so many empirical averages are based on a single reward and are zero or extremely high-variance. The paper does not analyze the coverage of the prefix trie, the distribution of G_{p∘v} sizes, or the resulting bias/variance of the advantage estimates. This is a central statistical assumption for T-SPMO's success; the manuscript should provide a quantitative analysis or additional experiments (e.g., varying |G|, comparing against exact value estimates on a small task) to show that the sparse empirical advantages provide a reliable credit-assignment signal.
minor comments (5)
- [Throughout] There are numerous formatting and typographical issues: 'SV AMP' and 'SV AMP' spacing inconsistencies, 'M ULT' in Table 3 header, 'ReST-MCTS*' missing formatting, and the references list is incomplete for the workshop (e.g., no URL for the ICML workshop proceedings). The paper should be carefully copy-edited.
- [Section 3.1, Eq. (4)] The KL estimator in Eq. (4) is printed without a clear statement of which distribution it estimates; the usual unbiased estimator for KL(πθ||πref) involves expectations over πθ. Please clarify the derivation and the direction of the KL term used in the objective.
- [Section 3.3 (Replay-Based Resampling)] The replay procedure is described only in prose. The pseudocode would benefit from a formal specification: how exactly the replay positions are sampled, whether rewards for the restarted generations are used in the same update step, and how the replay budgets C_success and C_failure interact with the number of optimization steps. As written, the method is underspecified and hard to reimplement.
- [Section 4.2, Table 3] The token count for T-SPMO includes replay tokens, but the comparison with S-GRPO and GRPO should also report the wall-clock time and memory usage, since the 'token-efficient' claim is the paper's central motivation. Without time/memory measurements, the practical significance of the token reduction is unclear.
- [Section 4.4] The Discussion speculates that GRPO 'may overfit under LoRA' but provides no evidence (e.g., training vs. validation reward curves, or a simple experiment with a larger LoRA rank). Either provide supporting analysis or soften the speculation to avoid overstating the hypothesis.
Circularity Check
No circularity: headline results are external benchmark accuracies; the cited objective typos and checkpoint-selection practice are correctness/evaluation issues, not reductions to inputs.
full rationale
The paper's central claims are empirical: S-GRPO and T-SPMO are run on Qwen2-1.5B with LoRA and measured against held-out test sets ('We report accuracy on the SVAMP testset and a generated multiplication testset'), with the base model and full-token GRPO as controls. Nothing in the method definitions makes a reported accuracy equal to its training input by construction: the advantage estimates in Eqs. (7)-(9) are empirical averages over sampled completions, and the final numbers are not derived from those averages but from separate test evaluations. There are no load-bearing self-citations; all citations (Williams 1992, Kool et al. 2019, Shao et al. 2024, Hu et al. 2021) are to independent prior methods/models. The printed objective defects flagged in Eqs. (5) and (10) — the ratio πθ/πθ annotated 'no grad' degenerating to 1, and the use of πθ·A instead of log πθ·A in the T-SPMO objective — are correctness/validity problems, not circular equivalences: they concern whether the implemented update matches the claimed policy-gradient derivation, not whether an output is presupposed as an input. Similarly, Section 4.1's 'manual selection of the checkpoint with peak performance for multiplication' is a potential evaluation-protocol/overfitting concern (especially if the selection used test-accuracy information), but it is not a circular reduction of the reported accuracy to the training objective. The sparsity of T-SPMO's advantage estimates with |G|=50 is a variance/credit-assignment concern, not a circularity. Therefore no circular step meeting the quote-and-reduce standard is present; the paper is self-contained against external benchmarks.
Assumptions & free parameters
free parameters (8)
- S-GRPO cutoff alpha =
100 (SVAMP), 50 (multiplication)
- S-GRPO max loss tokens k =
100 for both tasks
- S-GRPO sampling probability P =
not reported
- T-SPMO group size |G| =
50 (T-SPMO), 8 (GRPO/S-GRPO)
- T-SPMO replay budgets C_success, C_failure =
SVAMP: (0,0); multiplication: (1,0)
- Training steps and checkpoint selection =
325 to 450 steps depending on method and task
- LoRA rank and layer fraction =
rank 16, last 1/3 layers (SVAMP); rank 8, last 1/4 layers (multiplication)
- Regularization beta, lambda, and learning rate =
0.01, 0.01, 1e-4
assumptions (7)
- ad hoc to paper The token-level advantage A(v|p)=E[R|p∘v]-E[R|p] estimated by empirical averages over sampled completions is a valid credit assignment signal for optimizing expected reward.
- domain assumption Sampling |G| completions from the current policy and averaging their exact-match rewards gives reliable estimates of E[R|p] and E[R|p∘v].
- ad hoc to paper Restarting generation from a randomly sampled prefix position of a successful or failed completion yields training signal representative of full-length rollouts.
- standard math Group-relative advantage normalization (Eq. 2) is an unbiased variance-reduction baseline.
- standard math The KL estimator in Eq. (4) is unbiased.
- ad hoc to paper Full-token GRPO's failure to improve over the base model is due to overfitting or an implicit regularizer rather than to untuned hyperparameters or insufficient training.
- domain assumption Low-rank adapters on a subset of attention layers are sufficient to learn the targeted arithmetic behaviors.
Cite this review
Pith. "Pith review of Token-Efficient RL for LLM Reasoning." pith.science (2026). https://pith.science/paper/KBUZH7ZM
@misc{pith2026250420834,
author = {Pith},
title = {Pith review of: Token-Efficient RL for LLM Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/KBUZH7ZM}},
note = {Machine review of arXiv:2504.20834}
}
read the original abstract
We propose reinforcement learning (RL) strategies tailored for reasoning in large language models (LLMs) under strict memory and compute limits, with a particular focus on compatibility with LoRA fine-tuning. Building on early policy gradient methods with baseline subtraction, we design critic-free methods that operate on a small, informative subset of output tokens to reduce memory usage and stabilize training. We introduce S-GRPO, a stochastic variant of Group Relative Policy Optimization, and T-SPMO, a token-level prefix matching approach for fine-grained credit assignment. Applied to Qwen2-1.5B, our methods raise accuracy on the SVAMP benchmark from 46% to over 70% and show strong performance on multi-digit multiplication. Surprisingly, full-token GRPO under LoRA fails to improve over the base model, suggesting that selective token-level optimization may act as an implicit regularizer in low-parameter training regimes.
Figures
Reference graph
Works this paper leans on
-
[1]
J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
arXiv 2021
-
[2]
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., Iftimie, A., Karpenko, A., and Passos, A. T. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024
arXiv 2024
-
[3]
Kool, W., van Hoof, H., and Welling, M. Buy 4 reinforce samples, get a baseline for free! In Deep Reinforcement Learning Meets Structured Prediction Workshop at ICLR 2019, 2019. URL https://openreview.net/forum?id=r1lgTGL5DE
work page 2019
-
[4]
Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.\ 2080--2094, Online, 2021. Association for Computational Linguistics. doi:10.18653/v1/2021.naacl-mai...
-
[5]
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
arXiv 2017
-
[6]
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., and Y. Wu, D. G. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
arXiv 2024
-
[7]
Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8 0 (3): 0 229--256, 1992
work page 1992
-
[8]
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024
arXiv 2024
Show all 10 references
-
[9]
Rest-mcts*: Llm self-training via process reward guided tree search
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, J. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024
2024 arXiv
-
[10]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.