{"id":"e16a336d-23df-4ed0-94a0-a4b4b3503bbf","arxiv_id":"2506.23667","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"L0 combines a code-as-action agent scaffold with multi-turn RLVR, lifting Qwen2.5-7B HotpotQA EM from 22 to 41 and SimpleQA judge accuracy from 30 to 80.","lead":"This paper introduces L0, a system that trains language models to act as agents by writing and running Python code in a REPL loop, then reinforces correct answers. The authors report large gains on factuality and multi-hop QA benchmarks and say they have open-sourced the code, models, and training recipes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RLVR generalization claim rests on an unreported train/eval separation and a filtered 20K training pool; without evidence of disjointness and transfer to a held-out factuality set, benchmark-specific overfitting remains a live alternative.","rationale":"I read the paper as an engineering contribution: a REPL code-as-action scaffold plus on-policy REINFORCE with verifiable rewards. The scaffold-vs-RL comparison is a credible controlled experiment, and the infrastructure claims are plausible. The load-bearing risk is empirical generalization. The reader's weakest assumption captures the split issue; I agree and sharpen it as a distributional-alignment problem: the filter, the training environment, and the evaluation environment are all aligned toward search-and-copy on QA benchmarks, so the reported gains may not indicate robust problem-solving skills. A clean HotpotQA split would reduce contamination risk but would not eliminate the need for a held-out disjoint benchmark. The RAgent baseline citation mismatch is a real but secondary issue that does not affect the L0-RL vs L0-Scaffold comparison. No machine-checked proof or error bars strengthen the empirical claims, and the open-source release helps reproducibility but is not verifiable from the preprint. Therefore the CONDITIONAL verdict is appropriate: the authors should provide the data split, filtered dataset, and a disjoint transfer evaluation before the general-agent claim is accepted.","tokens_in":7886,"tokens_out":5544,"duration_ms":67586,"concrete_test":"Request the authors release the 20K filtered training question IDs and confirm disjointness from every evaluation question, especially HotpotQA. Then run the identical L0-RL recipe without further changes on a factuality benchmark with no source overlap, such as FreshQA or a freshly curated 500-question set from Wikipedia/Common Crawl, and compare L0-RL vs L0-Scaffold. If the gain shrinks by more than half, the headline result is training-distribution-specific rather than a general 'solely RLVR' capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RLVR alone turns a base model into a general agent is supported mainly by Table 2, and the weakest link is the relationship between the 20K training subset (§4.3) and the evaluation sets (§4.4). The paper does not state whether the HotpotQA evaluation is the standard dev split, does not release the filtered question IDs, and does not report accuracy before and after filtering. More importantly, the filter is described as selecting for objectivity, temporal stability, and difficulty—criteria that can silently select questions whose answers are easy to retrieve via web search. Because training, scaffold, and evaluation all use the same SerpAPI/Jina retrieval environment, the observed gains (e.g., HotpotQA EM 22.03→40.63, SimpleQA EM 21.94→61.68) could reflect the model learning to issue search queries and copy returned snippets rather than general problem-solving. The within-paper L0-Scaffold vs L0-RL comparison is well controlled, but it cannot rule out distribution-specific strategy learning. The abstract's 'base model' phrasing is also imprecise because the backbones are Instruct/Thinking models, but the decisive unresolved issue is generalization beyond the training distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces L0, an open-source end-to-end RLVR pipeline for training LLM agents, together with the NB-Agent scaffold, which implements a Think-Code-Observe loop in a Jupyter kernel, a verifiable reward combining answer correctness, format compliance, and code execution, and a distributed, sandboxed infrastructure for parallel rollouts. The authors evaluate the approach on HotpotQA, Musique, Bamboogle, and SimpleQA, compare against prompting, RAG, and RL baselines, and ablate task difficulty with a dynamic sampling strategy. The central claim is that RLVR alone, without SFT, enables a base model to develop robust problem-solving skills, with headline gains such as Qwen2.5-7B-Instruct improving from 22% to 41% on HotpotQA and from 30% to 80% on SimpleQA.","tokens_in":8161,"tokens_out":4830,"duration_ms":53454,"significance":"If substantiated, this is a useful contribution to the agentic RL systems literature: the paper ships code, models, a training pipeline, and a controlled within-paper comparison showing large gains from RL over the scaffold alone. The design of a code-as-action scaffold with verifiable rewards and a decoupled worker/inference infrastructure is sensible, and the open-sourcing commitment is a genuine strength. However, the central generalization claim is not yet empirically established because the relationship between the 20K training pool and the evaluation sets is not stated, the reward aggregation and RL hyperparameters are incomplete, and the abstract's headline numbers do not correspond to the primary metric in Table 2.","major_comments":[{"comment":"The 20K training pairs are selected from the training sets of 2WikiMultihopQA, TriviaQA, NQ, and HotpotQA, while §4.4 evaluates on HotpotQA without stating which split is used. If the HotpotQA evaluation overlaps the selected training questions, the reported HotpotQA EM gain (22.03 to 40.63 in Table 2) could reflect retrieval of seen questions rather than general agent skill. The authors must state the exact split, release the filtered question IDs, and report results on a demonstrably disjoint evaluation set.","section":"§4.3 and §4.4"},{"comment":"The filtering criteria of objectivity, temporal stability, and difficulty can silently select questions whose answers are easy to retrieve through web search, and training and evaluation use the same SerpAPI/Jina retrieval environment. To rule out distribution-specific strategy learning, the paper should report the filtering thresholds, per-dataset retention counts, and accuracy before and after filtering, and should add an evaluation on a held-out factuality benchmark drawn from outside the training distribution.","section":"§4.3"},{"comment":"The total reward R(τ) is described as a combination of three components, but only R_final is formally specified; the stepwise format compliance reward, the code execution reward, and the weights by which they are combined into R(τ) are not defined. Likewise, the dynamic sampling rule that discards zero- or maximum-reward trajectories is described only informally, and RL hyperparameters such as γ, the KL penalty coefficient, learning rate, batch size, and advantage epsilon are omitted. Because the central claim is that RLVR drives the observed gains, the training recipe must be fully specified to be reproducible and to rule out hidden reward shaping.","section":"§3.2 and §4.3"},{"comment":"The abstract's statement that accuracy on SimpleQA rises from 30% to 80% matches the LLM-as-judge column (30.45 to 80.40), not the exact-match or F1 columns, conflating a judge-based metric with accuracy. In addition, the phrase 'base model' is inaccurate because the backbones are Qwen2.5-Instruct and Qwen3-Thinking models. Both statements should be corrected so that the headline numbers correspond to a clearly defined metric and to the actual model class.","section":"Abstract and §4.4, Table 2"},{"comment":"No error bars, confidence intervals, or number of independent runs are reported for any of the evaluation results. Single-run numbers cannot support precise comparative claims such as the HotpotQA margin between L0-RL (40.63) and ZeroSearch-inst (34.60). The authors should report multiple seeds with variance, or explicitly state the evaluation budget and the deterministic components of the pipeline.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, including 'remains significant challenges' in the abstract, 'experoment settings' in §4.2, and 'a more stronger foundation model' in §4.4; a careful proofread is needed.","section":"Abstract and §4.2"},{"comment":"Figure 3 is described qualitatively, but the axes, curves, and configurations are not labeled; adding a legend and caption details would make the collapse and the stabilizing effect of dynamic sampling interpretable.","section":"§4.5, Figure 3"},{"comment":"Algorithm 1 has formatting issues and is not self-contained: the submit_final_answer termination condition is only mentioned in prose, and the state update through context_watcher is not reflected in the pseudocode.","section":"Algorithm 1"},{"comment":"The Notepad is described as a long-term memory mechanism, but it is implemented as Python variables within an episode and it is unclear whether any state persists across episodes; the scope of the memory should be clarified.","section":"§2.2"},{"comment":"The LLM-as-a-judge metric is used in Table 2 but the judge model and prompt are not specified; these details are needed for reproducibility.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The highest-risk item is the train/eval separation between the 20K HotpotQA training pairs in §4.3 and the HotpotQA evaluation in §4.4. If the evaluation split overlaps the training subset, the headline HotpotQA result would be invalid. I recommend asking the authors to provide the exact split and the filtered question IDs before further consideration, in addition to the missing reward and hyperparameter details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nYou should know two things about this paper. First, the system is real and plausibly useful: NB-Agent is a straightforward REPL code-as-action scaffold with a notepad memory object, and the RL pipeline (agentic gradient, verifiable rewards, dynamic sampling, sandboxed rollouts) is a competent integration of DAPO/REINFORCE++ on top. Second, the headline numbers are interesting but the paper currently does not support them as carefully as it should; the main gap is data hygiene, not the method.\n\nWhat is genuinely new is the integrated package—an open-source agentic RLVR pipeline that works on 7B-scale models and shows large gains over the same scaffold without RL. The within-paper comparison L0-Scaffold vs L0-RL is well controlled and is the strongest evidence that the RL training itself, not the scaffold, drives the gains. The ablation of dynamic sampling on hard data is also a useful, honest demonstration of training collapse and how sampling mitigates it.\n\nWhere it gets soft: the train/eval split issue is real. The 20K training pool is sampled from the training sets of 2Wiki, TriviaQA, NQ, and HotpotQA, and evaluation includes HotpotQA without stating whether that is the standard dev split. The paper should say this explicitly and release filtered question IDs or a split statement. The filtering for objectivity/temporal stability/difficulty could also bias toward easy-to-retrieve questions, though the gains on Bamboogle, Musique, and SimpleQA—none of which are in the training pool—make a pure benchmark-overfitting story less likely. Less central but worth fixing: no error bars, no stated RL hyperparameters (learning rate, KL coefficient, gamma, batch size), the reward aggregation is under-specified, the RAgent baseline is cited as a retrieval-based policy-generation framework but the reference is about access control policy generation, and the abstract's \"base model\" overstates things because the backbones are Instruct/Thinking models. The action-level policy gradient is also mathematically just token-level gradients with the same advantage grouped into actions; that's fine to present as an implementation detail, but presenting it as a core innovation overstates it.\n\nBottom line: this deserves a serious referee. It is an engineering contribution with open source, a controlled scaffold comparison, and a plausible RLVR result that, if cleaned up, would be useful to people building tool-use agents. It should not be desk-rejected; it needs a revision that addresses the data split, adds variance estimates, and fixes the baselines.","headline":"A solid engineering paper with a plausible RLVR result; the numbers are interesting but the unstated HotpotQA split and missing experiment details must be fixed before the claim is trustworthy.","tokens_in":8720,"tokens_out":2461,"would_cite":true,"duration_ms":26876,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that L0's reinforcement-learning pipeline, using a Python code-execution loop and verifiable rewards, turns a 7B instruct model into a more accurate factuality and multi-hop QA agent without supervised agent training.","keywords":["reinforcement learning with verifiable rewards","LLM agents","code-as-action","multi-turn reinforcement learning","question answering","agent scaffold","dynamic sampling","sandboxed rollouts"],"falsifier":"Compute the overlap between the released 20K training set and each evaluation set, then retrain on the non-overlapping subset and re-run HotpotQA, Musique, and SimpleQA; if the L0-RL minus L0-Scaffold gap largely disappears, the headline gains came from contamination or selection rather than general agent skill.","tokens_in":7677,"feed_emoji":"🤖","tokens_out":8965,"duration_ms":86876,"temperature":0.7,"pith_summary":"This paper tries to establish that a language model can become a general tool-using agent through reinforcement learning alone, without supervised fine-tuning on agent behavior. The L0 pipeline pairs a code-as-action agent scaffold, NB-Agent, with a verifiable-reward policy-gradient update and sandboxed rollout infrastructure. On Qwen2.5-7B-Instruct the authors report accuracy rising from 30% to 80% on SimpleQA and from 22% to 41% on HotpotQA. If the claim is right, the expensive ingredient for agent training is a checkable reward plus a scalable execution environment, not collected human demonstrations.","feed_headline":"RL alone lifts 7B model's SimpleQA from 30% to 80%","feed_subtitle":"No supervised fine-tuning: a code-execution loop and verifiable rewards teach a model to search and self-correct.","key_machinery":"The load-bearing mechanism is the agentic policy gradient, which treats a whole action sequence $a_t$ as the policy output and writes the gradient as an expectation over steps of $\\nabla \\log \\pi_\\theta(a_t \\mid s_t)\\hat{A}_t$, with token-level normalization over long sequences and batch-wise advantage normalization for stability. It is paired with a verifiable reward $R_{\\text{final}} = 0.9 \\cdot \\mathbf{1}_{\\text{exact match}} + 0.1 \\cdot \\mathbf{1}_{\\text{has answer}}$ plus rewards for format compliance and error-free code execution. The supporting infrastructure is a decoupled worker pool: isolated sandboxed agents handle environment rollouts while a shared inference server serves the latest policy, and a dynamic-sampling heuristic drops zero- and maximum-reward trajectories to keep hard-task training from collapsing.","core_discovery":"The paper's central claim is that a policy-gradient update defined over whole action sequences, where each action is a complete think-and-code step, is enough to convert an instructed LLM into a self-correcting agent when the reward is verifiable. The reward function mixes exact-match answer correctness (weight 0.9), the presence of any answer (weight 0.1), well-formed think/code structure, and successful code execution. The NB-Agent scaffold supplies the structural prior: a Think-Code-Observe loop over a Python kernel in which the model can search the web, store facts in a notepad object, and submit a final answer. After RL, average exact match across HotpotQA, Musique, and Bamboogle rises from 20.52% with the scaffold alone to 38.28%, and the 32B backbone reaches 61.04%.","pith_inferences":["The same code-as-action plus verifiable-reward recipe should transfer to other executable domains, such as code repair, database queries, or browser automation, provided a checkable answer exists; the authors leave this extension implicit.","Because the paper does not document the train/eval split for its 20K filtered RL questions, the size of the reported gains should be re-verified on a strictly non-overlapping holdout before treating them as evidence of general capability.","The notepad-based external memory suggests context length may not be the limiting factor for long-horizon work; a stress test with increasing step budgets could show whether state management degrades gracefully.","A matched comparison against supervised fine-tuning on the same scaffold would settle whether RLVR alone is truly sufficient or whether imitation adds value on top."],"forward_implications":["On the reported benchmarks, most of the gain comes from RL rather than the scaffold: average EM moves from 20.52% to 38.28% after training.","The 7B L0-RL model outperforms the strongest search-based RL agents on the same suite, indicating that an executable code environment offers more room for RL than a single search-tool loop.","Models pretrained for reasoning and tool use show the steepest relative gains, with Qwen3-4B-Thinking rising from 14.78% to 44.67% EM.","Without dynamic sampling, high-difficulty training collapses; with it, format and execution rewards stay stable, so the headline accuracy depends on the sampling mitigation."],"supporting_citations":[{"why":"Establishes the executable-code-actions premise that the NB-Agent's Think-Code-Observe loop builds on.","marker":"X. Wang et al. 2024"},{"why":"Search-R1 is the closest RL-trained search-agent baseline; L0-RL is directly compared to it.","marker":"Jin et al. 2025"},{"why":"Provides the token-level normalization and dynamic-sampling machinery adapted for long action sequences.","marker":"Yu et al. 2025"},{"why":"Supplies the advantage-estimation and normalization scheme used in the policy gradient.","marker":"Hu 2025"},{"why":"Defines SimpleQA, the factuality benchmark where L0-RL reports 30% to 80% accuracy.","marker":"Wei et al. 2024"},{"why":"Defines HotpotQA, used both as a multi-hop evaluation set and as a source of RL training questions.","marker":"Z. Yang et al. 2018"},{"why":"2WikiMultiHopQA is a source of RL training questions and a multi-hop reasoning evaluation set.","marker":"Ho et al. 2020"},{"why":"ZeroSearch provides the search-engine protocol and baseline configuration for the evaluations.","marker":"Sun et al. 2025"}],"fun_headline_variants":["RL-only agent training: SimpleQA 30% to 80% on 7B model","Code-as-action RL lifts 7B QA: SimpleQA 30→80%, HotpotQA 22→41%","Verifiable rewards alone train 7B agent to 80% on SimpleQA","No SFT needed: RLVR with code REPL makes 7B self-correct","Think-code-observe RL: SimpleQA 80%, HotpotQA 41% on 7B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 20K RL training questions are cleanly separated from the evaluation questions, so the reported accuracy gains reflect a general ability to search, code, and self-correct rather than memorization or filtering that makes questions easy.","fun_headline_variants_meta":{"raw":{"variants":["RL-only agent training: SimpleQA 30% to 80% on 7B model","Code-as-action RL lifts 7B QA: SimpleQA 30→80%, HotpotQA 22→41%","Verifiable rewards alone train 7B agent to 80% on SimpleQA","No SFT needed: RLVR with code REPL makes 7B self-correct","Think-code-observe RL: SimpleQA 80%, HotpotQA 41% on 7B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1636,"prompt_tokens":946,"completion_tokens":690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":562,"tokens_out":690,"duration_ms":6115,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:35:15.778196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between the released 20K training set and each evaluation set, then retrain on the non-overlapping subset and re-run HotpotQA, Musique, and SimpleQA; if the L0-RL minus L0-Scaffold gap largely disappears, the headline gains came from contamination or selection rather than general agent skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the executable-code-actions premise that the NB-Agent's Think-Code-Observe loop builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines HotpotQA, used both as a multi-hop evaluation set and as a source of RL training questions."}],"review_version":1}