REVIEW 5 major objections 5 minor 18 references
L0: Reinforcement Learning to Become General Agents
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that L0's reinforcement-learning pipeline, using a Python code-execution loop and verifiable rewards, turns a 7B instruct model into a more accurate factuality and multi-hop QA agent without supervised agent training.
desk verdict A solid engineering paper with a plausible RLVR result; the numbers are interesting but the unstated HotpotQA split and missing experiment details must be fixed before the claim is trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the agentic policy gradient, which treats a whole action sequence $a_t$ as the policy output and writes the gradient as an expectation over steps of $\nabla \log \pi_\theta(a_t \mid s_t)\hat{A}_t$, with token-level normalization over long sequences and batch-wise advantage normalization for stability. It is paired with a verifiable reward $R_{\text{final}} = 0.9 \cdot \mathbf{1}_{\text{exact match}} + 0.1 \cdot \mathbf{1}_{\text{has answer}}$ plus rewards for format compliance and error-free code execution. The supporting infrastructure is a decoupled worker pool: isolated sandboxed agents handle environment rollouts while a shared inference server serves the latest policy, and a dynamic-sampling heuristic drops zero- and maximum-reward trajectories to keep hard-task training from collapsing.
What would settle it
Compute the overlap between the released 20K training set and each evaluation set, then retrain on the non-overlapping subset and re-run HotpotQA, Musique, and SimpleQA; if the L0-RL minus L0-Scaffold gap largely disappears, the headline gains came from contamination or selection rather than general agent skill.
Extended reading notes
Core claim
The paper's central claim is that a policy-gradient update defined over whole action sequences, where each action is a complete think-and-code step, is enough to convert an instructed LLM into a self-correcting agent when the reward is verifiable. The reward function mixes exact-match answer correctness (weight 0.9), the presence of any answer (weight 0.1), well-formed think/code structure, and successful code execution. The NB-Agent scaffold supplies the structural prior: a Think-Code-Observe loop over a Python kernel in which the model can search the web, store facts in a notepad object, and submit a final answer. After RL, average exact match across HotpotQA, Musique, and Bamboogle rises from 20.52% with the scaffold alone to 38.28%, and the 32B backbone reaches 61.04%.
Load-bearing premise
The load-bearing assumption is that the 20K RL training questions are cleanly separated from the evaluation questions, so the reported accuracy gains reflect a general ability to search, code, and self-correct rather than memorization or filtering that makes questions easy.
Editorial extensions
If this is right
- On the reported benchmarks, most of the gain comes from RL rather than the scaffold: average EM moves from 20.52% to 38.28% after training.
- The 7B L0-RL model outperforms the strongest search-based RL agents on the same suite, indicating that an executable code environment offers more room for RL than a single search-tool loop.
- Models pretrained for reasoning and tool use show the steepest relative gains, with Qwen3-4B-Thinking rising from 14.78% to 44.67% EM.
- Without dynamic sampling, high-difficulty training collapses; with it, format and execution rewards stay stable, so the headline accuracy depends on the sampling mitigation.
Reading between the lines
- The same code-as-action plus verifiable-reward recipe should transfer to other executable domains, such as code repair, database queries, or browser automation, provided a checkable answer exists; the authors leave this extension implicit.
- Because the paper does not document the train/eval split for its 20K filtered RL questions, the size of the reported gains should be re-verified on a strictly non-overlapping holdout before treating them as evidence of general capability.
- The notepad-based external memory suggests context length may not be the limiting factor for long-horizon work; a stress test with increasing step budgets could show whether state management degrades gracefully.
- A matched comparison against supervised fine-tuning on the same scaffold would settle whether RLVR alone is truly sufficient or whether imitation adds value on top.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces L0, an open-source end-to-end RLVR pipeline for training LLM agents, together with the NB-Agent scaffold, which implements a Think-Code-Observe loop in a Jupyter kernel, a verifiable reward combining answer correctness, format compliance, and code execution, and a distributed, sandboxed infrastructure for parallel rollouts. The authors evaluate the approach on HotpotQA, Musique, Bamboogle, and SimpleQA, compare against prompting, RAG, and RL baselines, and ablate task difficulty with a dynamic sampling strategy. The central claim is that RLVR alone, without SFT, enables a base model to develop robust problem-solving skills, with headline gains such as Qwen2.5-7B-Instruct improving from 22% to 41% on HotpotQA and from 30% to 80% on SimpleQA.
Significance. If substantiated, this is a useful contribution to the agentic RL systems literature: the paper ships code, models, a training pipeline, and a controlled within-paper comparison showing large gains from RL over the scaffold alone. The design of a code-as-action scaffold with verifiable rewards and a decoupled worker/inference infrastructure is sensible, and the open-sourcing commitment is a genuine strength. However, the central generalization claim is not yet empirically established because the relationship between the 20K training pool and the evaluation sets is not stated, the reward aggregation and RL hyperparameters are incomplete, and the abstract's headline numbers do not correspond to the primary metric in Table 2.
major comments (5)
- [§4.3 and §4.4] The 20K training pairs are selected from the training sets of 2WikiMultihopQA, TriviaQA, NQ, and HotpotQA, while §4.4 evaluates on HotpotQA without stating which split is used. If the HotpotQA evaluation overlaps the selected training questions, the reported HotpotQA EM gain (22.03 to 40.63 in Table 2) could reflect retrieval of seen questions rather than general agent skill. The authors must state the exact split, release the filtered question IDs, and report results on a demonstrably disjoint evaluation set.
- [§4.3] The filtering criteria of objectivity, temporal stability, and difficulty can silently select questions whose answers are easy to retrieve through web search, and training and evaluation use the same SerpAPI/Jina retrieval environment. To rule out distribution-specific strategy learning, the paper should report the filtering thresholds, per-dataset retention counts, and accuracy before and after filtering, and should add an evaluation on a held-out factuality benchmark drawn from outside the training distribution.
- [§3.2 and §4.3] The total reward R(τ) is described as a combination of three components, but only R_final is formally specified; the stepwise format compliance reward, the code execution reward, and the weights by which they are combined into R(τ) are not defined. Likewise, the dynamic sampling rule that discards zero- or maximum-reward trajectories is described only informally, and RL hyperparameters such as γ, the KL penalty coefficient, learning rate, batch size, and advantage epsilon are omitted. Because the central claim is that RLVR drives the observed gains, the training recipe must be fully specified to be reproducible and to rule out hidden reward shaping.
- [Abstract and §4.4, Table 2] The abstract's statement that accuracy on SimpleQA rises from 30% to 80% matches the LLM-as-judge column (30.45 to 80.40), not the exact-match or F1 columns, conflating a judge-based metric with accuracy. In addition, the phrase 'base model' is inaccurate because the backbones are Qwen2.5-Instruct and Qwen3-Thinking models. Both statements should be corrected so that the headline numbers correspond to a clearly defined metric and to the actual model class.
- [Tables 1 and 2] No error bars, confidence intervals, or number of independent runs are reported for any of the evaluation results. Single-run numbers cannot support precise comparative claims such as the HotpotQA margin between L0-RL (40.63) and ZeroSearch-inst (34.60). The authors should report multiple seeds with variance, or explicitly state the evaluation budget and the deterministic components of the pipeline.
minor comments (5)
- [Abstract and §4.2] There are several typos and grammatical errors, including 'remains significant challenges' in the abstract, 'experoment settings' in §4.2, and 'a more stronger foundation model' in §4.4; a careful proofread is needed.
- [§4.5, Figure 3] Figure 3 is described qualitatively, but the axes, curves, and configurations are not labeled; adding a legend and caption details would make the collapse and the stabilizing effect of dynamic sampling interpretable.
- [Algorithm 1] Algorithm 1 has formatting issues and is not self-contained: the submit_final_answer termination condition is only mentioned in prose, and the state update through context_watcher is not reflected in the pseudocode.
- [§2.2] The Notepad is described as a long-term memory mechanism, but it is implemented as Python variables within an episode and it is unclear whether any state persists across episodes; the scope of the memory should be clarified.
- [§4.1] The LLM-as-a-judge metric is used in Table 2 but the judge model and prompt are not specified; these details are needed for reproducibility.
Circularity Check
No significant circularity: the RLVR derivation is self-contained and supported by held-out evaluations, with no load-bearing self-citation chain.
full rationale
The paper's central derivation is an end-to-end RLVR pipeline with an agentic policy gradient and verifiable rewards. No load-bearing step reduces to its own inputs by construction. The policy-gradient formula is a standard log-probability expansion with token-level normalization adopted from DAPO and advantage estimation from REINFORCE++; these are external, citable techniques rather than self-citations, and the math is an identity rather than an assumption of the result. The reward function uses exact-match plus answer-presence terms, and the evaluation metric is exact match; this objective alignment is normal RLVR practice, not a fitted parameter renamed as a prediction, because the reported numbers come from the evaluation benchmarks rather than from equations that assume the target accuracies. The 20K training subset is drawn from training splits of HotpotQA, TriviaQA, NQ, and 2WikiMultihopQA, while evaluations are reported on HotpotQA, Musique, Bamboogle, and SimpleQA; the paper does not explicitly state the split identifiers or release filtered IDs, which is a legitimate reproducibility and generalization concern, but there is no quoted evidence that the evaluation set equals or overlaps the training subset by construction. No self-citation is used to justify the central claim, no uniqueness theorem is imported from the authors' prior work, and no known result is merely renamed. Accordingly, the appropriate circularity finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- Reward weights in Rfinal =
0.9 exact match, 0.1 has-answer
- Format and code-execution reward scales =
not reported
- Dynamic sampling discard rule =
not specified
- RL hyperparameters (gamma, KL coefficient, learning rate, batch size, advantage epsilon) =
not reported
- Data filtering thresholds =
not specified
assumptions (5)
- standard math The policy gradient estimator with token-level normalization is a valid and stable training objective.
- domain assumption Exact-match final answer plus format and code-execution rewards is a sufficient proxy for task success.
- domain assumption The training queries and evaluation queries are disjoint.
- domain assumption The external APIs (SerpAPI search, Jina Reader) return relevant and stable content during training and evaluation.
- ad hoc to paper Discarding zero/maximum reward trajectories improves stability without biasing the learned policy.
invented entities (2)
-
NB-Agent scaffold
-
Notepad memory object
Cite this review
Pith. "Pith review of L0: Reinforcement Learning to Become General Agents." pith.science (2026). https://pith.science/paper/BNZXF5KU
@misc{pith2026250623667,
author = {Pith},
title = {Pith review of: L0: Reinforcement Learning to Become General Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/BNZXF5KU}},
note = {Machine review of arXiv:2506.23667}
}
read the original abstract
Training large language models (LLMs) to act as autonomous agents for multi-turn, long-horizon tasks remains significant challenges in scalability and training efficiency. To address this, we introduce L-Zero (L0), a scalable, end-to-end training pipeline for general-purpose agents. Featuring a low-cost, extensible, and sandboxed concurrent agent worker pool, L0 lowers the barrier for applying reinforcement learning in complex environments. We also introduce NB-Agent, the agent scaffold within L0, which operates in a "code-as-action" fashion via a Read-Eval-Print-Loop (REPL). We evaluate L0 on factuality question-answering benchmarks. Our experiments demonstrate that a base model can develop robust problem-solving skills using solely Reinforcement Learning with Verifiable Rewards (RLVR). On the Qwen2.5-7B-Instruct model, our method boosts accuracy on SimpleQA from 30 % to 80 % and on HotpotQA from 22 % to 41 %. We have open-sourced the entire L0 system, including our L0 series models, the NB-Agent, a complete training pipeline, and the corresponding training recipes on (https://github.com/cmriat/l0).
Figures
Reference graph
Works this paper leans on
-
[1]
Feng, Lang, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. ``Group-in-Group Policy Optimization for LLM Agent Training.'' arXiv Preprint arXiv:2505.10978
arXiv 2025
-
[2]
Guo, Daya, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, et al. 2025. ``Deepseek-R1: Incentivizing Reasoning Capability in Llms via Reinforcement Learning.'' arXiv Preprint arXiv:2501.12948
arXiv 2025
-
[3]
Ho, Xanh, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. ``Constructing a Multi-Hop QA Dataset for Comprehensive Evaluation of Reasoning Steps.'' In Proceedings of the 28th International Conference on Computational Linguistics, 6609--25
work page 2020
-
[4]
Hu, Jian. 2025. ``REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models.'' arXiv e-Prints, arXiv--2501
work page 2025
-
[5]
Jayasundara, Sakuna Harinda, Nalin Asanka Gamagedara Arachchilage, and Giovanni Russello. 2024. ``RAGent: Retrieval-Based Access Control Policy Generation.'' arXiv Preprint arXiv:2409.07489
arXiv 2024
-
[6]
Jin, Bowen, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. ``Search-R1: Training Llms to Reason and Leverage Search Engines with Reinforcement Learning.'' arXiv Preprint arXiv:2503.09516
arXiv 2025
-
[7]
Joshi, Mandar, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. ``TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.'' In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601--11
work page 2017
-
[8]
Kwiatkowski, Tom, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, et al. 2019. ``Natural Questions: A Benchmark for Question Answering Research.'' Transactions of the Association for Computational Linguistics 7: 452--66
work page 2019
Show all 18 references
-
[9]
Li, Xiaoxi, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. ``Search-O1: Agentic Search-Enhanced Large Reasoning Models.'' arXiv Preprint arXiv:2501.05366
2025 arXiv
-
[10]
Press, Ofir, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. ``Measuring and Narrowing the Compositionality Gap in Language Models.'' In Findings of the Association for Computational Linguistics: EMNLP 2023, 5687--5711
2023
-
[11]
Sun, Hao, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. 2025. ``Zerosearch: Incentivize the Search Capability of Llms Without Searching.'' arXiv Preprint arXiv:2505.04588
2025 arXiv
-
[12]
Trivedi, Harsh, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. ``MuSiQue: Multihop Questions via Single-Hop Question Composition.'' Transactions of the Association for Computational Linguistics 10: 539--54
2022
-
[13]
Wang, Xingyao, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. ``Executable Code Actions Elicit Better LLM Agents.'' In International Conference on Machine Learning, 50208--32. PMLR
2024
-
[14]
Wang, Zihan, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, et al. 2025. ``RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning.'' arXiv e-Prints, arXiv--2504
2025
-
[15]
Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. ``Measuring Short-Form Factuality in Large Language Models.'' arXiv Preprint arXiv:2411.04368
2024 arXiv
-
[16]
Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. 2025. ``Qwen3 Technical Report.'' arXiv Preprint arXiv:2505.09388
2025 arXiv
-
[17]
Yang, Zhilin, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. ``HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering.'' In Proceedings of the 2018 Conference on Empirical Methods in Natural Lang...
2018
-
[18]
Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, et al. 2025. ``DAPO: An Open-Source LLM Reinforcement Learning System at Scale.'' CoRR. CSLReferences document
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.