{"id":"0761045c-f8c4-4d94-9c36-1ce3640bac48","arxiv_id":"2608.01822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"SearchMaster trains a 9B LLM search agent through self-play with evidence-chain task generation, search-depth rewards, and over-opening penalties, lifting average accuracy on six deep-search benchmarks from 38.19% to 51.52%.","lead":"This paper trains a search agent from tasks it invents, solves, and verifies by itself, using three controls to keep the tasks genuinely multi-hop and the browsing efficient. On six deep-search benchmarks the 9B model's average accuracy rises from 38.19% to 51.52%, including a 30.1-point jump on BrowseComp-Plus.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The frozen Verifier is the sole correctness signal for self-generated tasks; if it accepts plausible but wrong reference answers, the claimed grounded self-play is unproven. A direct Verifier-accuracy audit on retained training data would settle this.","rationale":"The paper is a solid empirical contribution with a plausible mechanism; the strongest claim is that a 9B model improves by +13.3 points average / +30.1 on BrowseComp-Plus from self-play without human labels or demonstrations. For that claim to hold as stated, the self-generated training signal must be informative. The reader's weakest-assumption analysis points to the frozen Verifier, and I agree that this is the most load-bearing uncertainty. Every reward component that selects or scores tasks ultimately depends on V's judgments: Solver correctness z_i, the number of successful rollouts C used by SDR and the seed gate, and the validity/parametric filters. Since V is a frozen copy of the same model family that generates both the tasks and the answers, the loop could converge to self-consistent but factually ungrounded behavior. The final evaluation is external, so the reported numbers are not circular, but the mechanism attribution ('grounded and regulated') is only as strong as V's accuracy. The proposed audit directly measures V's accuracy on retained training data, which is the missing evidence. Secondary issues—lack of error bars, baselines not re-run, no artifact hashes—reinforce but do not replace this concern. The reader's CONDITIONAL verdict remains appropriate; no verdict change is warranted.","tokens_in":14453,"tokens_out":5709,"duration_ms":52006,"concrete_test":"Sample 200 retained seed tasks from the final training iterations (or use the released training data, if available). Have three independent annotators—or a stronger model such as GPT-5 with access to the same local corpus—judge (i) whether each Proposer reference answer y*_j is correct and supported by the evidence chain, and (ii) whether each accepted Solver rollout answer is correct. Compute the frozen Verifier's precision, recall, and agreement on these judgments. If the Verifier's false-acceptance rate on Solver rollouts exceeds ~5%, or its agreement with the independent judges is below ~90%, the training signal is not reliably grounded and the 'grounded self-play' interpretation fails; if agreement is high, the circularity concern is empirically resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the frozen Verifier V (a copy of the initial Qwen3.5-9B) providing trustworthy correctness labels for tasks that the Proposer generates. In 'Solver Rollouts and Search Depth,' the Proposer's reference answer y*_j is generated by the policy and 'the frozen Verifier compares ŷ_i with the reference answer y⋆_j and outputs a correctness score z_i∈{0,1}'. Eq. 6 then sets the Solver reward to bS_i = z_i (when tool use and format are valid), and Eqs. 3-5 use C = Σ z_i and the h_min of successful rollouts for SDR and the seed-level gate. The validity and parametric-knowledge filters also use the same Verifier. Therefore every training signal is filtered through a model from the same family and initialization as the policy being trained. This is not an internal inconsistency, but an unverified empirical precondition: if V is systematically lenient toward Proposer-style reference answers (e.g., it accepts self-consistent but factually wrong evidence chains), the policy can be optimized toward self-consistent errors. The external benchmark judges would catch wrong final answers, so the headline numbers are not themselves circular; however, the causal claim that gains come from 'grounded and regulated' self-play would be unsupported if the Verifier is unreliable. The paper reports no audit of V's judgments on retained training data. Given the 30.1-point gain and the absence of error bars, this is the main load-bearing uncertainty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SearchMaster, a self-play framework for training an LLM-based search agent without human-labeled QA pairs or expert demonstrations. A single Qwen3.5-9B policy acts as both Proposer and Solver inside a local search environment. The Proposer builds explicit evidence chains (ECG) and generates tasks; four filters remove invalid candidates; the Solver produces multiple rollouts; a frozen copy of the initial model serves as Verifier, scoring correctness; a Search-Depth Reward (SDR) assigns task difficulty by the minimum successful search depth; and an Over-Opening Penalty (OOP) regularizes the open-to-search ratio. The policy is updated with GRPO using the resulting rewards. On six deep-search benchmarks, the method improves the backbone from 38.19% to 51.52% average accuracy, including a 30.1-point gain on BrowseComp-Plus, and ablations show monotonic gains from ECG, SDR, and OOP.","tokens_in":14771,"tokens_out":4873,"duration_ms":46774,"significance":"If the results hold, this is a notable advance: it shows that a 9B model can bootstrap competitive deep-search ability from purely self-generated tasks in a local corpus, surpassing much larger open and proprietary baselines on BrowseComp-Plus and transferring to live-web benchmarks. The paper also provides a useful decomposition of the self-play failure modes and an ablation that demonstrates each mechanism contributes. The explicit release commitment for code, checkpoints, and training data is a strength. However, the central causal claim that the training signal is 'grounded and regulated' depends on the reliability of the frozen Verifier, which is a copy of the initial model and is not audited in the paper. The final benchmark numbers are not circular because they use external judges and held-out benchmarks, but the training signal is self-referential. This concern, together with the absence of variance estimates, is the main load-bearing uncertainty.","major_comments":[{"comment":"The frozen Verifier V is the sole correctness signal in training, and V is a frozen copy of the initial Qwen3.5-9B model. The reference answer y*_j is generated by the Proposer, and the validity, parametric-knowledge, and correctness filters all use the same Verifier. Since the Solver base reward is bS_i = z_i and the Proposer reward depends on C = sum_i z_i, a systematically lenient Verifier could train the policy toward self-consistent but factually incorrect answers. The external benchmark evaluation means the headline numbers are not circular, but the causal claim that gains come from 'grounded and regulated' self-play requires direct evidence. Please add a Verifier audit: sample retained training tasks and have human annotators or a strong external judge label whether the reference answers are correct and whether the Verifier's z_i agrees with external correctness, reporting precision, recall, and agreement. A small training comparison using an external verifier would also directly test whether the gains are an artifact of the self-referential signal. This is the main load-bearing issue in the paper.","section":"Solver Rollouts and Search Depth; Eq. (6)"},{"comment":"All reported accuracies are single-run numbers with no error bars, confidence intervals, or significance tests. Several evaluation sets are small (GAIA has 103 instances, SEAL-0 has 111, XBench has 100), so differences of a few points may be within noise. The 30.1-point gain on BrowseComp-Plus is large and unlikely to be noise, but the average online gain of 13.3 points is supported by variable per-benchmark improvements. Please report results over at least three training seeds or provide bootstrapped confidence intervals, and do the same for the ablation in Table 2. This is necessary to assess the magnitude and stability of the claimed improvements.","section":"Table 1 and Table 2"},{"comment":"The ECG diagnostic in Table 3 labels task quality using GLM-5, another LLM judge, rather than human annotation. The paper reports a large increase in 'True Multi-Hop' (from 24.2% to 78.6%) and a drop in 'Invalid' tasks, but without human agreement or error analysis it is unclear how reliable these labels are. Since the central claim is that ECG reduces pseudo multi-hop tasks, please include a human-validated sample (e.g., 200 tasks per condition) or report agreement between GLM-5 and human judges. This would strengthen the behavioral evidence beyond the final accuracy numbers.","section":"Evidence-Chain Task Generation; Table 3"}],"minor_comments":[{"comment":"The caption says 'range' but does not define whether it is min-max, interquartile, or standard deviation. Please specify the exact definition in the caption or text.","section":"Figure 3"},{"comment":"The Avg. column is computed only for the two rows with all six benchmark entries; external baselines with missing entries show dashes. Please state this explicitly in the table caption so the reader does not compare Avg. values across rows with different support.","section":"Table 1"},{"comment":"The notation in Eq. (10) uses gamma_t for the per-token importance ratio and also for the clip argument; the definition appears only in the text after the equation. Please define gamma_t before the equation or restructure the expression for readability.","section":"Eq. (10)"},{"comment":"The parametric-knowledge filter is described only as 'removes questions the Verifier can answer without tools.' Please specify how this is operationalized: for example, whether the Verifier is prompted to answer q_j directly without tool observations and what threshold or decision rule is used.","section":"Method: Quality Filters"},{"comment":"The limitation about operating in a local search environment is acknowledged, but the paper does not discuss the risk that the OpenResearcher corpus may have distributional overlap with current web content or that the offline environment's fixed documents could become stale for live-benchmark transfer. A sentence on corpus diversity and timeliness would be useful.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the novelty claim is credible relative to prior self-play search methods such as Dr. Zero and Search Self-Play. The main risk is the unverified self-referential Verifier: I would want the requested audit before acceptance. I do not see concerns about citation practice or inappropriate overlap with prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper puts real new mechanisms on the table for self-play search-agent training. Evidence-Chain Generator, Search-Depth Reward, and Over-Opening Penalty are concrete, well-motivated fixes for known failure modes, and the ablation shows each one moves BrowseComp-Plus accuracy. The 30.1-point gain on that benchmark and the cross-benchmark average lift from 38.19% to 51.52% are large enough to take seriously, especially with a 9B backbone beating much larger systems.\n\nWhat the paper does well: the experimental section is unusually honest about mechanism-level behavior. Table 3 shows ECG actually raises true multi-hop generation from 24.2% to 78.6%. Figure 3 shows SDR keeps h_min in the 8–10 range instead of decaying to 2–3. Figure 4 shows OOP holds the open/search ratio flat. That is the right kind of evidence for a method paper: not just final numbers but the intended behavioral effect. The writing is clear, the local environment is a substantive reproducibility choice, and the authors explicitly acknowledge the cost of Verifier calls and the local-env coverage limit.\n\nSoft spots, in order of how much they bother me. First, the frozen Verifier is a copy of the initial model, and it is the sole correctness signal for self-generated reference answers. That is a real circularity risk. It is not fatal: final evaluation uses external benchmarks and a different judge, so the headline numbers are not tautological. But the causal claim that gains come from grounded self-play would be stronger with a direct audit of the Verifier on retained training data, e.g., human- or stronger-model-checked agreement rates. Second, all numbers appear to be single-run, no error bars. For a 9B model trained with GRPO at temperature 1.0, variance could be nontrivial, and the ablation gaps are not huge between variants. Third, external baselines are taken from source papers rather than re-run in a shared environment; standard practice, but worth noting when comparing across contexts. Fourth, there are many reward hyperparameters (r0, Hsdr, alpha/beta/lambda_oop) and no sensitivity analysis. Minor, but a reviewer will ask.\n\nThe stress-test concern about the Verifier is legitimate and not overstated. It is the main load-bearing uncertainty. The paper itself does not address it, and the authors' own limitation paragraph stops short of acknowledging the self-verification issue.\n\nWho this is for: anyone working on RL for tool-use agents, synthetic data generation, or self-improving LLMs. It deserves a serious referee: the mechanisms are transferable, the experiments are reproducible in spirit, and the weaknesses are addressable in revision.\n\nRecommendation: send to peer review. Ask the authors for a Verifier reliability audit, repeated runs with variance, and the actual code artifact rather than a forward-looking URL.","headline":"A genuinely novel self-play recipe for search agents with credible ablations; the self-referential Verifier is the main uncertainty, but the results deserve serious referee time.","tokens_in":15334,"tokens_out":1178,"would_cite":true,"duration_ms":12354,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SearchMaster claims that a 9-billion-parameter model can bootstrap deep-search ability by generating, solving, and verifying its own tasks in a local search environment.","keywords":["search agents","self-play","evidence chain","search-depth reward","over-opening penalty","GRPO","deep search","multi-hop question answering"],"falsifier":"Take a sample of SearchMaster-generated tasks with their Verifier scores, have independent annotators or a stronger external judge relabel correctness, and check whether the accepted answers are actually correct; if the Verifier accepts a substantial share of wrong answers, the benchmark gains would not reflect genuine search ability.","tokens_in":14226,"feed_emoji":"🔍","tokens_out":5836,"duration_ms":48270,"temperature":0.7,"pith_summary":"SearchMaster claims that a single language model can train itself into a capable deep-search agent using only tasks it generates, solves, and verifies in a local search environment, with no human-labeled question-answer pairs or expert demonstrations. The paper identifies three ways self-play can go wrong: questions that look multi-hop but are answerable from one document, difficulty estimates based on success rate rather than search depth, and trajectories that open many documents without gathering targeted evidence. It then adds a control for each failure mode. On six deep-search benchmarks, the framework raises a 9-billion-parameter backbone from 38.19% to 51.52% average accuracy, including a 30.1-point jump on BrowseComp-Plus. If the result holds, search-agent skill can be bootstrapped from a model's own exploration, removing one of the main data bottlenecks in deep-research systems.","feed_headline":"Self-play lifts a 9B search agent from 38% to 52%","feed_subtitle":"No human QA pairs or expert demos: the model writes, solves and verifies its own deep-search tasks.","key_machinery":"The machinery is a closed self-play loop in which one shared policy plays two roles. As Proposer, it starts from a seed document, builds an explicit evidence chain using search, open, and find tools, and writes a question plus a reference answer derived from the whole chain. As Solver, it receives only the question and produces tool-use rollouts; a frozen Verifier, an untrained copy of the initial model, scores each answer correct or incorrect. Four filters remove shallow, malformed, invalid, or parametric-knowledge-answerable candidates before Solver rollouts. Rewards combine correctness with the Search-Depth Reward, which uses the minimum search depth among correct rollouts to measure task difficulty, and the Over-Opening Penalty, which clips the open-to-search ratio into a linear penalty. GRPO turns these rewards into token-level advantages, and the same policy is updated on both Proposer and Solver samples.","core_discovery":"The paper reports that the failure modes of search self-play can be corrected by making three quantities explicit. The Evidence-Chain Generator forces task proposals to build a visible chain of evidence across documents before asking a question, which raises the measured fraction of genuinely multi-hop tasks from 24.2% to 78.6% while cutting invalid tasks from 28.7% to 6.4%. The Search-Depth Reward scores a task by the minimum number of unique searches among successful solutions, keeping retained tasks search-intensive as the solver improves. The Over-Opening Penalty subtracts a penalty proportional to the open-to-search ratio, preventing the policy from drifting into repeated document openings. With these controls, verified Proposer and Solver rollouts are optimized jointly with GRPO, and the resulting policy improves the untrained backbone by 30.1 points on BrowseComp-Plus and improves all five live-web benchmarks, despite never training on the live web.","pith_inferences":["Going beyond the paper, the Over-Opening Penalty is stated generically enough that it could be transferred to other tool-use agents, such as coding agents or database-querying agents, where repeated opening of the same resource is a known failure mode.","A natural stress test would be to remove the parametric-knowledge filter and see whether the frozen Verifier's judgments degrade when reference answers are less grounded in retrieved evidence.","The evidence-chain quality could be cheaply monitored during training with an independent lightweight classifier, potentially replacing part of the Verifier's validity-filtering role and reducing compute.","Because the paper's difficulty signal uses only the minimum successful search depth, a task with one lucky shallow solution is treated as easy even if most successful rollouts are deep; an alternative that also accounts for the distribution of depths might change which tasks are retained."],"forward_implications":["Search-agent training can proceed with no human-written questions or expert trajectories, using only an indexed document collection and a base model.","Search behavior learned offline transfers to live, open-web benchmarks, so local self-play may be a viable substitute for expensive web-scale reinforcement learning.","Task difficulty measured by minimum successful search depth prevents the training distribution from collapsing into shallow lookup tasks as the solver improves.","A 9-billion-parameter model trained this way can exceed much larger open and proprietary systems on at least one deep-search benchmark, suggesting model scale is not the only driver of search skill.","The open-to-search penalty offers a reusable control for tool-use efficiency in other agentic reinforcement-learning settings."],"supporting_citations":[{"why":"Search Self-Play is the prior self-play baseline whose task generation is constrained by predefined answers, which this paper extends.","marker":"(Lu et al. 2025a)"},{"why":"Dr. Zero is the prior self-play approach that uses success rate as difficulty signal, the baseline this paper criticizes.","marker":"(Yue et al. 2026)"},{"why":"DeepSeekMath introduces GRPO, the optimization objective used to train both Proposer and Solver.","marker":"(Shao et al. 2024)"},{"why":"OpenResearcher supplies the offline corpus and the local search environment that SearchMaster reuses.","marker":"(Li et al. 2026b)"},{"why":"BrowseComp-Plus is the offline deep-search benchmark where the largest reported gain of 30.1 points is measured.","marker":"(Chen et al. 2026)"},{"why":"Qwen3.5-9B is the backbone and the frozen Verifier; all results are gains over this model.","marker":"(Qwen Team 2026)"},{"why":"BrowseComp provides 1,266 hard-to-find factual web questions used as an online transfer benchmark.","marker":"(Wei et al. 2025a)"},{"why":"GAIA provides 103 text-only tool-use assistant tasks in the online evaluation suite.","marker":"(Mialon et al. 2024)"},{"why":"WebWalkerQA contributes 680 multi-page navigation questions to the online evaluation set.","marker":"(Wu et al. 2025)"},{"why":"XBench-DeepSearch contributes 100 web-search questions used in the reported average.","marker":"(Chen et al. 2025)"}],"fun_headline_variants":["Self-play search agent jumps from 38% to 52%","No human data: self-play search hits 52%","SearchMaster self-play: 30-point gain on BrowseComp-Plus","Grounded self-play guides search agent to 52%","Three controls make self-play work for search agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen Verifier, a copy of the initial model, can reliably tell correct answers from plausible wrong ones on self-generated tasks; if it accepts wrong reference answers, training rewards could push the policy toward self-consistent errors rather than genuine search ability.","fun_headline_variants_meta":{"raw":{"variants":["Self-play search agent jumps from 38% to 52%","No human data: self-play search hits 52%","SearchMaster self-play: 30-point gain on BrowseComp-Plus","Grounded self-play guides search agent to 52%","Three controls make self-play work for search agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000985,"raw_usage":{"total_tokens":4221,"prompt_tokens":1029,"completion_tokens":3192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":3108}},"tokens_in":645,"tokens_out":3192,"duration_ms":21359,"temperature":1.0,"reasoning_tokens":3108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:04:13.448902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of SearchMaster-generated tasks with their Verifier scores, have independent annotators or a stronger external judge relabel correctness, and check whether the accepted answers are actually correct; if the Verifier accepts a substantial share of wrong answers, the benchmark gains would not reflect genuine search ability.","supporting_citations":[],"review_version":1}