REVIEW 3 major objections 4 minor 16 references
Standard RL for search agents rewards correct answers but never abstention, so they hallucinate when retrieval fails; AWA-RL fixes the incentive by tying refusal rewards to each query’s prior success rate.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 09:36 UTC pith:6F4AQ4XY
load-bearing objection Solid engineering fix for over-answering in search-agent RL: query-aware dynamic refusal reward plus RA-F1, modest but clean gains, code released. the 3 major comments →
To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
When search agents are trained only with a binary outcome reward, they learn to over-answer and hallucinate on queries that lie outside their current capability boundary. Dynamically shaping the abstention reward from each query’s pre-estimated success probability and from on-policy batch refusal statistics yields agents that both search effectively and refuse when evidence is insufficient, improving precision and the joint RA-F1 metric with only marginal accuracy loss.
What carries the argument
The query-specific base refusal reward R_base(q_i) = 1 − (max(p_i, ε₁))^γ, where p_i is the non-abstaining model’s estimated success probability on q_i and γ ∈ (0,1] is the courage factor, plus a batch-level dynamic penalty P(E_t) that corrects excess or deficit refusal during GRPO training.
Load-bearing premise
An offline estimate of the model’s success probability on each training query, taken before any abstention training, remains a faithful enough capability boundary throughout later multi-turn RL.
What would settle it
Train the identical cold-start policy with γ = 0.2 on the same 5 000 queries, then measure whether precision still rises ≥ 8 points and RA-F1 still improves while accuracy drops less than 3 points on the three held-out multi-hop benchmarks; if either the precision gain or the RA-F1 gain disappears, the central claim fails.
If this is right
- Search-agent RL pipelines can stop treating abstention as a post-hoc patch and instead optimise it jointly with retrieval skill under a single outcome-style objective.
- A single scalar courage hyper-parameter becomes a practical dial that operators can set to any desired point on the capability-reliability frontier without retraining from scratch.
- Static refusal bonuses and synthetic unanswerable-data mixtures become unnecessary once the refusal reward is anchored to the model’s own per-query priors.
- The RA-F1 metric supplies a single scalar that future agent papers can report to quantify the joint capability-reliability trade-off.
Where Pith is reading between the lines
- The same prior-plus-on-policy shaping could be applied to other sparse-reward agent settings (tool use, code generation, multi-step math) where “I don’t know” is currently unrewarded.
- If the offline p_i estimate is replaced by a cheap uncertainty proxy (entropy, self-consistency variance), the method becomes sample-free and could be used online without a preliminary pass.
- Because refusal is now an explicit, rewarded action, the resulting agents may expose capability gaps on underrepresented topics more cleanly than pure accuracy metrics, offering a new diagnostic for demographic or cultural bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper identifies a vulnerability in outcome-reward RL for search agents: binary rewards encourage correct answers but do not penalize fabrication when retrieval fails, promoting hallucinations. It proposes AWA-RL, which defines a query-specific abstention reward r_ref(q_i) = R_base(q_i) - P(E_t). R_base is derived from an offline estimate of the non-abstaining model's per-query success probability p_i (clipped and transformed by a courage factor γ), while P(E_t) is an on-policy, EMA-smoothed, step-quantized penalty on batch excess refusal. The method is evaluated with GRPO on HotpotQA, 2WikiMultiHopQA and MuSiQue using Qwen2.5-7B (from-scratch and cold-start), against SFT-mixed, unanswerable-data and static-r_ref baselines. A new RA-F1 metric (harmonic mean of precision and accuracy) is introduced; at γ=0.2 the cold-start policy raises precision by ~10 points and RA-F1 by a few points with only a small accuracy drop, and refusal rates converge stably to γ-controlled plateaus.
Significance. If the reported gains hold under broader validation, AWA-RL supplies a practical, lightweight mechanism for jointly optimizing search capability and calibrated abstention inside a single RL loop, addressing a clear gap left by correctness-centric agentic-search methods (Search-R1 and related systems) and by post-hoc or static-refusal approaches. The public release of code, data and weights, the full γ sweeps, training-dynamics plots and per-dataset tables, and the explicit composite citation reward that keeps search skill from collapsing are concrete strengths that make the contribution immediately usable and falsifiable by the community.
major comments (3)
- [Abstract / §1 / Table 3] Abstract claims an overall RA-F1 lift of 2.9 % while the introduction and Table 3 report 5.2 % (cold-start γ=0.2: 0.585 vs 0.556). The two numbers cannot both be correct; the discrepancy must be resolved and the abstract/body numbers aligned before acceptance.
- [§3 / Limitations] All primary results use a single 7B model. The Limitations section correctly flags this, yet the central claim that AWA-RL “successfully yields highly capable and reliable search agents” is currently supported only at this scale. At minimum a second model family or a larger checkpoint should be added, or the claim should be explicitly scoped to the 7B regime.
- [§2.3 / Appendix B.1] The offline p_i estimate (Section 2.3) is treated as a fixed capability boundary for the entire subsequent multi-turn RL run. While the on-policy penalty P(E_t) and Figure 1 provide empirical reassurance against collapse, no ablation quantifies how sensitive final RA-F1 is to the quality or staleness of that single offline estimate (e.g., number of samples per query, re-estimation mid-training). A short sensitivity study would strengthen the weakest modelling assumption.
minor comments (4)
- [Tables 1–3] Several table captions and body sentences contain missing spaces (“AWA-RLfrom scratch”, “AWA-RLwith varying”).
- [Appendix B.2] The composite advantage weights (2, 3, −1) in Appendix B.2 are presented without ablation; a one-sentence justification or sensitivity note would help readers assess robustness.
- [§3 Evaluation Metrics] RA-F1 is defined as the harmonic mean of Precision and Accuracy; a brief comparison with existing reliability metrics (e.g., those in Xu et al. 2024b) would clarify novelty.
- [Figure 1] Figure 1 caption could state the exact batch size and EMA parameters used so the plateaus are fully reproducible from the plot alone.
Circularity Check
No circularity: AWA-RL is empirical reward shaping with an offline prior anchor and on-policy batch statistics; reported gains are measured outcomes, not quantities forced by construction.
full rationale
The paper's central construction (Section 2.3) defines a query-specific base refusal reward Rbase(qi)=1-(max(pi,ε1))^γ from a single offline estimate of success probability pi on a frozen non-abstaining checkpoint, then subtracts a dynamic penalty P(Et) computed from the observed batch excess refusal rate (EMA-smoothed and step-quantized in Appendix B.1). This is intentional reward design, not a derivation that reduces a claimed prediction or first-principles result to its own inputs. The composite GRPO advantage further mixes the shaped outcome term with independent dense citation rewards (Appendix B.2). All reported improvements (precision, RA-F1) are post-training measurements on held-out multi-hop QA benchmarks against non-abstaining and static-reward baselines (Tables 1–7, Figure 1); none of the equations equate a fitted parameter to the quantity being claimed. Related-work citations are ordinary background and do not supply a load-bearing uniqueness theorem or ansatz that forces the result. The method is therefore self-contained and free of the enumerated circularity patterns.
Axiom & Free-Parameter Ledger
free parameters (4)
- courage factor γ =
0.2 (sweet spot)
- clip thresholds ε1, ε2 =
0.05
- EMA decay α and quantization step τ =
α=0.5, τ=0.1
- composite advantage weights (2, 3, -1) =
2·Wh(R)+3·Wh(Rcov)-1·Wh(Rred)
axioms (3)
- domain assumption Group Relative Policy Optimization (GRPO) with binary or shaped outcome rewards is a valid and stable RL algorithm for multi-turn search agents.
- ad hoc to paper An offline estimate of per-query success probability on a non-abstaining checkpoint is a sufficiently accurate proxy for the model’s intrinsic capability boundary.
- domain assumption LLM-as-judge scores for answer correctness and citation quality are reliable enough to serve as training rewards.
invented entities (2)
-
RA-F1 (Reliability-Aware F1)
no independent evidence
-
dynamic refusal reward rref(qi)=Rbase(qi)-P(Et)
no independent evidence
read the original abstract
Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.
Figures
Reference graph
Works this paper leans on
-
[1]
The faiss library. Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, and Chao Huang. 2025. Understanding deepresearch via re- ports.arXiv preprint arXiv:2510.07861. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforce- men...
arXiv 2025
-
[2]
Webshaper: Agentically data synthesizing via information-seeking formalization.arXiv preprint arXiv:2507.15061. MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, and 1 oth- ers. 2025. Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, ...
Pith/arXiv arXiv 2025
-
[3]
This EMA acts as a low-pass filter, ensuring that the penalty only responds to sustained trends in over-refusal or under-refusal rather than tran- sient batch noise
Exponential Moving Average (EMA) Smoothing:Instead of using the instanta- neous excess refusal rate Et, we maintain a smoothed running average ¯Et across training steps: ¯Et =α ¯Et−1 + (1−α)E t where α∈[0,1) is the decay factor (set to α= 0.5 in our experiments). This EMA acts as a low-pass filter, ensuring that the penalty only responds to sustained tren...
-
[4]
staircase
Step-wise Quantization:To further prevent microscopic, continuous shifts in the refusal reward, we enforce discrete, step-wise up- dates by quantizing the smoothed excess rate ¯Et. Given a predefined step-tier threshold τ (e.g., τ= 0.1 ), the effective excess rate is discretized as: ˜Et =τ⌊ ¯Et/τ⌉ where ⌊·⌉ denotes rounding to the nearest in- teger. Ultim...
2018
-
[5]
The pred_answer doesn't need to be exactly the same as any of the ground truth answers, but should be semantically same for the question. ,→ ,→
-
[6]
rationale
Each item in the ground truth answer list can be viewed as a ground truth answer for the question, and the pred_answer should be semantically same to at least one of them. ,→ ,→ ,→ question: {question} ground truth answers: {gt_answer} pred_answer: {pred_answer} The output should in the following json format: ```json {{ "rationale": "your rationale for th...
-
[7]
**Question**: The user query
-
[8]
**Ground Truth (GT) Answer**: The correct answer string
-
[9]
Each atomic evidence represents one key piece of information
**Ground Truth (GT) Atomic Evidence**: A list of specific supporting facts that should be cited. Each atomic evidence represents one key piece of information. ,→ ,→ ,→
-
[10]
**Model Citations**: The citations/evidence passages that the model has provided in its answer.,→ ## Goal Evaluate the model's citations on TWO dimensions: A) **Coverage Count**: How many GT Atomic Evidence items are covered by the citations?,→ B) **Redundancy/Irrelevance Severity**: The severity level of redundant or irrelevant citations (0-3 scale).,→ #...
-
[11]
Go through EACH item in the GT Atomic Evidence list one by one.,→
-
[12]
For each GT item, check if the Model Citations contain the **same semantic information**:,→ - Exact wording is NOT required - BUT specific details (names, dates, numbers, entities) MUST match,→ - Vague references do NOT count as coverage
-
[13]
If a GT item is sufficiently covered (even partially across multiple citations), count it as **1**.,→
-
[14]
covered_count
If a GT item is completely missing or only vaguely referenced, count it as **0**.,→ **Output**:`covered_count`= integer, the total number of GT Atomic Evidence items that are covered.,→ **Example**: - GT Atomic Evidence has 5 items: [A, B, C, D, E] - Model Citations cover: A (fully), B (fully), C (missing), D (vaguely), E (fully),→ - covered_count = 3 (on...
1934
-
[15]
,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ }, {
As an important regional hub in the Yangtze River Delta, Nanjing is well-connected by over 60 state and provincial highways to all parts of China. Express highways such as Hu–Ning, Ning–He, Ning–Hang enable commuters to travel to Shanghai, Hefei, Hangzhou, and other important cities quickly and conveniently. Inside the city of Nanjing, there are of highwa...
2011
-
[16]
,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ }, {
As an important regional hub in the Yangtze River Delta, Nanjing is well-connected by over 60 state and provincial highways to all parts of China. Express highways such as Hu–Ning, Ning–He, Ning–Hang enable commuters to travel to Shanghai, Hefei, Hangzhou, and other important cities quickly and conveniently. Inside the city of Nanjing, there are of highwa...
2011
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.