{"id":"87a89a60-3da3-4f39-9707-babe9c8d973f","arxiv_id":"2607.10738","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"AWA-RL dynamically sets per-query abstention rewards from model prior capability and on-policy refusal rates, lifting precision up to 10.3% and RA-F1 while sacrificing little accuracy.","lead":"AWA-RL trains LLM search agents to refuse answers when retrieval fails by shaping a dynamic abstention reward from each query's prior success rate and live batch refusal rates. This raises answer precision by up to 10% while keeping accuracy nearly intact, producing more reliable open-domain QA agents.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged offline-pi assumption.","rationale":"The paper's central claim is empirical and modest: a query-aware, dynamically adjusted refusal reward yields a better capability-reliability trade-off than static or data-injection baselines. The only place that claim could fail is if the offline pi became badly mis-calibrated once the policy started abstaining and searching differently. The authors already mitigate this with the on-policy P(Et) term and show stable, γ-responsive dynamics; the concrete re-estimation test above would falsify the residual risk if it existed. Because that risk is already the reader's weakest assumption and the rest of the evidence (tables, figure, released code) is consistent, no verdict adjustment is warranted. CONDITIONAL remains the right call solely for the single-model / modest-gain caveats the reader already listed.","tokens_in":32727,"tokens_out":574,"duration_ms":8990,"concrete_test":"Re-estimate pi on a held-out 500-query slice after the final AWA-RL checkpoint (γ=0.2 cold-start), recompute Rbase, and re-run 20 GRPO steps with the updated priors frozen; if RA-F1 drops >0.02 or refusal rate diverges >10 points from the original plateau, the offline-prior assumption is fragile. Otherwise the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (that a single offline non-abstaining pi remains a faithful enough capability boundary for Rbase=1-(max(pi,ε1))^γ throughout multi-turn RL) is the genuine soft spot, but it is not load-bearing enough to overturn the central claim. Section 2.3 already pairs the static prior with a continuous on-policy excess-refusal penalty P(Et) (EMA-smoothed and step-quantized in Appendix B.1), and Figure 1 shows batch refusal rates converging to distinct, γ-controlled plateaus rather than collapsing. Tables 3–7 further show that at the reported sweet-spot γ=0.2 the cold-start policy improves precision by ~10 points while losing only ~2 points of accuracy, and the same qualitative pattern holds from-scratch. The composite citation reward (Appendix B.2) supplies an independent dense signal that keeps search skill from degrading while the outcome term is free to balance answer/refuse. No internal inconsistency, circular derivation, or reward-hacking failure under the dynamic schedule is visible in the reported evidence. The remaining limitations (single 7B model, modest absolute RA-F1 lift, abstract/body numerical discrepancy) are already correctly noted by the reader and do not undermine the mechanism itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper identifies a vulnerability in outcome-reward RL for search agents: binary rewards encourage correct answers but do not penalize fabrication when retrieval fails, promoting hallucinations. It proposes AWA-RL, which defines a query-specific abstention reward r_ref(q_i) = R_base(q_i) - P(E_t). R_base is derived from an offline estimate of the non-abstaining model's per-query success probability p_i (clipped and transformed by a courage factor γ), while P(E_t) is an on-policy, EMA-smoothed, step-quantized penalty on batch excess refusal. The method is evaluated with GRPO on HotpotQA, 2WikiMultiHopQA and MuSiQue using Qwen2.5-7B (from-scratch and cold-start), against SFT-mixed, unanswerable-data and static-r_ref baselines. A new RA-F1 metric (harmonic mean of precision and accuracy) is introduced; at γ=0.2 the cold-start policy raises precision by ~10 points and RA-F1 by a few points with only a small accuracy drop, and refusal rates converge stably to γ-controlled plateaus.","tokens_in":33127,"tokens_out":1012,"duration_ms":18640,"significance":"If the reported gains hold under broader validation, AWA-RL supplies a practical, lightweight mechanism for jointly optimizing search capability and calibrated abstention inside a single RL loop, addressing a clear gap left by correctness-centric agentic-search methods (Search-R1 and related systems) and by post-hoc or static-refusal approaches. The public release of code, data and weights, the full γ sweeps, training-dynamics plots and per-dataset tables, and the explicit composite citation reward that keeps search skill from collapsing are concrete strengths that make the contribution immediately usable and falsifiable by the community.","major_comments":[{"comment":"Abstract claims an overall RA-F1 lift of 2.9 % while the introduction and Table 3 report 5.2 % (cold-start γ=0.2: 0.585 vs 0.556). The two numbers cannot both be correct; the discrepancy must be resolved and the abstract/body numbers aligned before acceptance.","section":"Abstract / §1 / Table 3"},{"comment":"All primary results use a single 7B model. The Limitations section correctly flags this, yet the central claim that AWA-RL “successfully yields highly capable and reliable search agents” is currently supported only at this scale. At minimum a second model family or a larger checkpoint should be added, or the claim should be explicitly scoped to the 7B regime.","section":"§3 / Limitations"},{"comment":"The offline p_i estimate (Section 2.3) is treated as a fixed capability boundary for the entire subsequent multi-turn RL run. While the on-policy penalty P(E_t) and Figure 1 provide empirical reassurance against collapse, no ablation quantifies how sensitive final RA-F1 is to the quality or staleness of that single offline estimate (e.g., number of samples per query, re-estimation mid-training). A short sensitivity study would strengthen the weakest modelling assumption.","section":"§2.3 / Appendix B.1"}],"minor_comments":[{"comment":"Several table captions and body sentences contain missing spaces (“AWA-RLfrom scratch”, “AWA-RLwith varying”).","section":"Tables 1–3"},{"comment":"The composite advantage weights (2, 3, −1) in Appendix B.2 are presented without ablation; a one-sentence justification or sensitivity note would help readers assess robustness.","section":"Appendix B.2"},{"comment":"RA-F1 is defined as the harmonic mean of Precision and Accuracy; a brief comparison with existing reliability metrics (e.g., those in Xu et al. 2024b) would clarify novelty.","section":"§3 Evaluation Metrics"},{"comment":"Figure 1 caption could state the exact batch size and EMA parameters used so the plateaus are fully reproducible from the plot alone.","section":"Figure 1"}],"recommendation":"minor_revision","confidential_remarks":"The numerical inconsistency between abstract and body is minor but should be fixed before camera-ready; otherwise the paper is a solid, well-executed empirical contribution that fits the journal’s scope on reliable agentic RL. No citation or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a practical paper that fixes a real failure mode. Outcome-reward GRPO for multi-turn search agents rewards correct finals but never teaches the model to refuse when retrieval fails, so it fabricates. AWA-RL is the first clean attempt I have seen to put a query-specific, on-policy refusal reward inside that same loop rather than as a post-hoc patch.\n\nWhat is new is the reward construction itself. They measure a frozen non-abstaining checkpoint’s per-query success rate pi, clip it, raise it to a single “courage” factor γ, set Rbase = 1 − p*, then subtract a batch-level excess-refusal penalty that is EMA-smoothed and step-quantized. The composite advantage also keeps a dense citation term so search skill does not collapse while the outcome term balances answer versus refuse. RA-F1 (harmonic mean of precision and accuracy) is a sensible single number for the trade-off. Tables 1–3 and the full per-dataset appendices show the usual baselines (SFT-mixed refusal trajectories, unanswerable-data RL, static rref) either refuse almost never or collapse into lazy refusal; AWA-RL at γ ≈ 0.2 lifts precision ~10 points for a ~2-point accuracy drop and improves RA-F1, with refusal rates converging to stable γ-controlled plateaus (Figure 1). Code, data, and weights are public.\n\nSoft spots are real but proportional. Everything is on one 7B model and three multi-hop Wikipedia benchmarks; the abstract’s 2.9 % RA-F1 claim does not match the body’s 5.2 % wording; and the offline pi is only an approximation of the capability boundary that later multi-turn RL will actually face. The on-policy penalty and citation reward mitigate the last issue, and the paper is honest about it in the limitations. No circular math, no hidden free parameters that invent the gains, no reward-hacking under the dynamic schedule in the reported runs.\n\nThis is for people who train tool-using agents and care about reliability under incomplete retrieval. It is not a foundational theory paper, but it is a reproducible, tunable recipe that works better than the obvious alternatives. I would send it to peer review; the referees will ask for larger models and a couple of extra domains, but the core idea is already solid enough to discuss.","headline":"Solid engineering fix for over-answering in search-agent RL: query-aware dynamic refusal reward plus RA-F1, modest but clean gains, code released.","tokens_in":33714,"tokens_out":583,"would_cite":true,"duration_ms":8551,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Standard RL for search agents rewards correct answers but never abstention, so they hallucinate when retrieval fails; AWA-RL fixes the incentive by tying refusal rewards to each query’s prior success rate.","keywords":["search agents","reinforcement learning","abstention","hallucination mitigation","GRPO","capability-reliability trade-off","RA-F1","query-specific reward shaping"],"falsifier":"Train the identical cold-start policy with γ = 0.2 on the same 5 000 queries, then measure whether precision still rises ≥ 8 points and RA-F1 still improves while accuracy drops less than 3 points on the three held-out multi-hop benchmarks; if either the precision gain or the RA-F1 gain disappears, the central claim fails.","tokens_in":33659,"feed_emoji":"🛡️","tokens_out":1000,"duration_ms":15258,"temperature":0.7,"pith_summary":"Outcome-reward RL has produced strong open-domain search agents, yet the reward only pays for a correct final answer. When the agent cannot retrieve enough evidence it still fabricates one, because abstaining earns nothing. The paper shows this training bias systematically worsens hallucination. AWA-RL replaces the missing signal with a refusal reward that is first set from the model’s own estimated success probability on that exact query, then continuously adjusted by how much the current batch is over- or under-refusing. A single “courage” factor (γ) controls how aggressively the agent is pushed to try answering rather than refuse. The result is a controllable capability-reliability trade-off: precision rises by as much as 10.3 points and the new RA-F1 metric by 2.9 points, while raw accuracy falls only modestly. Conventional fixes—mixing refusal data or using a fixed refusal bonus—either collapse into lazy refusal or force a rigid zero-sum trade-off; the dynamic, query-aware formulation does neither.","feed_headline":"Search agents learn to refuse when evidence is missing","feed_subtitle":"Dynamic abstention rewards lift precision up to 10 points with only a small accuracy cost","key_machinery":"The query-specific base refusal reward R_base(q_i) = 1 − (max(p_i, ε₁))^γ, where p_i is the non-abstaining model’s estimated success probability on q_i and γ ∈ (0,1] is the courage factor, plus a batch-level dynamic penalty P(E_t) that corrects excess or deficit refusal during GRPO training.","core_discovery":"When search agents are trained only with a binary outcome reward, they learn to over-answer and hallucinate on queries that lie outside their current capability boundary. Dynamically shaping the abstention reward from each query’s pre-estimated success probability and from on-policy batch refusal statistics yields agents that both search effectively and refuse when evidence is insufficient, improving precision and the joint RA-F1 metric with only marginal accuracy loss.","pith_inferences":["The same prior-plus-on-policy shaping could be applied to other sparse-reward agent settings (tool use, code generation, multi-step math) where “I don’t know” is currently unrewarded.","If the offline p_i estimate is replaced by a cheap uncertainty proxy (entropy, self-consistency variance), the method becomes sample-free and could be used online without a preliminary pass.","Because refusal is now an explicit, rewarded action, the resulting agents may expose capability gaps on underrepresented topics more cleanly than pure accuracy metrics, offering a new diagnostic for demographic or cultural bias."],"forward_implications":["Search-agent RL pipelines can stop treating abstention as a post-hoc patch and instead optimise it jointly with retrieval skill under a single outcome-style objective.","A single scalar courage hyper-parameter becomes a practical dial that operators can set to any desired point on the capability-reliability frontier without retraining from scratch.","Static refusal bonuses and synthetic unanswerable-data mixtures become unnecessary once the refusal reward is anchored to the model’s own per-query priors.","The RA-F1 metric supplies a single scalar that future agent papers can report to quantify the joint capability-reliability trade-off."],"fun_headline_variants":["Search agents learn to abstain when retrieval fails","AWA-RL rewards refusal to cut search-agent hallucinations","Dynamic abstention training lifts QA precision by 10 points","RL teaches agents to refuse outside their capability boundary","Abstention-aware rewards yield more reliable search agents"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"An offline estimate of the model’s success probability on each training query, taken before any abstention training, remains a faithful enough capability boundary throughout later multi-turn RL.","fun_headline_variants_meta":{"raw":{"variants":["Search agents learn to abstain when retrieval fails","AWA-RL rewards refusal to cut search-agent hallucinations","Dynamic abstention training lifts QA precision by 10 points","RL teaches agents to refuse outside their capability boundary","Abstention-aware rewards yield more reliable search agents"]},"model":"grok-4.5","effort":"low","cost_usd":0.003514,"raw_usage":{"total_tokens":1127,"prompt_tokens":764,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":35140000,"prompt_tokens_details":{"text_tokens":764,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":284,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":764,"tokens_out":79,"duration_ms":3943,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T09:36:25.924492+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the identical cold-start policy with γ = 0.2 on the same 5 000 queries, then measure whether precision still rises ≥ 8 points and RA-F1 still improves while accuracy drops less than 3 points on the three held-out multi-hop benchmarks; if either the precision gain or the RA-F1 gain disappears, the central claim fails.","supporting_citations":[],"review_version":1}