Pith. sign in

REVIEW 3 major objections 4 minor 16 references

Standard RL for search agents rewards correct answers but never abstention, so they hallucinate when retrieval fails; AWA-RL fixes the incentive by tying refusal rewards to each query’s prior success rate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-14 09:36 UTC pith:6F4AQ4XY

load-bearing objection Solid engineering fix for over-answering in search-agent RL: query-aware dynamic refusal reward plus RA-F1, modest but clean gains, code released. the 3 major comments →

arxiv 2607.10738 v1 pith:6F4AQ4XY submitted 2026-07-12 cs.LG cs.AIcs.CL

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

classification cs.LG cs.AIcs.CL
keywords search agentsreinforcement learningabstentionhallucination mitigationGRPOcapability-reliability trade-offRA-F1query-specific reward shaping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Outcome-reward RL has produced strong open-domain search agents, yet the reward only pays for a correct final answer. When the agent cannot retrieve enough evidence it still fabricates one, because abstaining earns nothing. The paper shows this training bias systematically worsens hallucination. AWA-RL replaces the missing signal with a refusal reward that is first set from the model’s own estimated success probability on that exact query, then continuously adjusted by how much the current batch is over- or under-refusing. A single “courage” factor (γ) controls how aggressively the agent is pushed to try answering rather than refuse. The result is a controllable capability-reliability trade-off: precision rises by as much as 10.3 points and the new RA-F1 metric by 2.9 points, while raw accuracy falls only modestly. Conventional fixes—mixing refusal data or using a fixed refusal bonus—either collapse into lazy refusal or force a rigid zero-sum trade-off; the dynamic, query-aware formulation does neither.

Core claim

When search agents are trained only with a binary outcome reward, they learn to over-answer and hallucinate on queries that lie outside their current capability boundary. Dynamically shaping the abstention reward from each query’s pre-estimated success probability and from on-policy batch refusal statistics yields agents that both search effectively and refuse when evidence is insufficient, improving precision and the joint RA-F1 metric with only marginal accuracy loss.

What carries the argument

The query-specific base refusal reward R_base(q_i) = 1 − (max(p_i, ε₁))^γ, where p_i is the non-abstaining model’s estimated success probability on q_i and γ ∈ (0,1] is the courage factor, plus a batch-level dynamic penalty P(E_t) that corrects excess or deficit refusal during GRPO training.

Load-bearing premise

An offline estimate of the model’s success probability on each training query, taken before any abstention training, remains a faithful enough capability boundary throughout later multi-turn RL.

What would settle it

Train the identical cold-start policy with γ = 0.2 on the same 5 000 queries, then measure whether precision still rises ≥ 8 points and RA-F1 still improves while accuracy drops less than 3 points on the three held-out multi-hop benchmarks; if either the precision gain or the RA-F1 gain disappears, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Search-agent RL pipelines can stop treating abstention as a post-hoc patch and instead optimise it jointly with retrieval skill under a single outcome-style objective.
  • A single scalar courage hyper-parameter becomes a practical dial that operators can set to any desired point on the capability-reliability frontier without retraining from scratch.
  • Static refusal bonuses and synthetic unanswerable-data mixtures become unnecessary once the refusal reward is anchored to the model’s own per-query priors.
  • The RA-F1 metric supplies a single scalar that future agent papers can report to quantify the joint capability-reliability trade-off.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same prior-plus-on-policy shaping could be applied to other sparse-reward agent settings (tool use, code generation, multi-step math) where “I don’t know” is currently unrewarded.
  • If the offline p_i estimate is replaced by a cheap uncertainty proxy (entropy, self-consistency variance), the method becomes sample-free and could be used online without a preliminary pass.
  • Because refusal is now an explicit, rewarded action, the resulting agents may expose capability gaps on underrepresented topics more cleanly than pure accuracy metrics, offering a new diagnostic for demographic or cultural bias.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper identifies a vulnerability in outcome-reward RL for search agents: binary rewards encourage correct answers but do not penalize fabrication when retrieval fails, promoting hallucinations. It proposes AWA-RL, which defines a query-specific abstention reward r_ref(q_i) = R_base(q_i) - P(E_t). R_base is derived from an offline estimate of the non-abstaining model's per-query success probability p_i (clipped and transformed by a courage factor γ), while P(E_t) is an on-policy, EMA-smoothed, step-quantized penalty on batch excess refusal. The method is evaluated with GRPO on HotpotQA, 2WikiMultiHopQA and MuSiQue using Qwen2.5-7B (from-scratch and cold-start), against SFT-mixed, unanswerable-data and static-r_ref baselines. A new RA-F1 metric (harmonic mean of precision and accuracy) is introduced; at γ=0.2 the cold-start policy raises precision by ~10 points and RA-F1 by a few points with only a small accuracy drop, and refusal rates converge stably to γ-controlled plateaus.

Significance. If the reported gains hold under broader validation, AWA-RL supplies a practical, lightweight mechanism for jointly optimizing search capability and calibrated abstention inside a single RL loop, addressing a clear gap left by correctness-centric agentic-search methods (Search-R1 and related systems) and by post-hoc or static-refusal approaches. The public release of code, data and weights, the full γ sweeps, training-dynamics plots and per-dataset tables, and the explicit composite citation reward that keeps search skill from collapsing are concrete strengths that make the contribution immediately usable and falsifiable by the community.

major comments (3)
  1. [Abstract / §1 / Table 3] Abstract claims an overall RA-F1 lift of 2.9 % while the introduction and Table 3 report 5.2 % (cold-start γ=0.2: 0.585 vs 0.556). The two numbers cannot both be correct; the discrepancy must be resolved and the abstract/body numbers aligned before acceptance.
  2. [§3 / Limitations] All primary results use a single 7B model. The Limitations section correctly flags this, yet the central claim that AWA-RL “successfully yields highly capable and reliable search agents” is currently supported only at this scale. At minimum a second model family or a larger checkpoint should be added, or the claim should be explicitly scoped to the 7B regime.
  3. [§2.3 / Appendix B.1] The offline p_i estimate (Section 2.3) is treated as a fixed capability boundary for the entire subsequent multi-turn RL run. While the on-policy penalty P(E_t) and Figure 1 provide empirical reassurance against collapse, no ablation quantifies how sensitive final RA-F1 is to the quality or staleness of that single offline estimate (e.g., number of samples per query, re-estimation mid-training). A short sensitivity study would strengthen the weakest modelling assumption.
minor comments (4)
  1. [Tables 1–3] Several table captions and body sentences contain missing spaces (“AWA-RLfrom scratch”, “AWA-RLwith varying”).
  2. [Appendix B.2] The composite advantage weights (2, 3, −1) in Appendix B.2 are presented without ablation; a one-sentence justification or sensitivity note would help readers assess robustness.
  3. [§3 Evaluation Metrics] RA-F1 is defined as the harmonic mean of Precision and Accuracy; a brief comparison with existing reliability metrics (e.g., those in Xu et al. 2024b) would clarify novelty.
  4. [Figure 1] Figure 1 caption could state the exact batch size and EMA parameters used so the plateaus are fully reproducible from the plot alone.

Circularity Check

0 steps flagged

No circularity: AWA-RL is empirical reward shaping with an offline prior anchor and on-policy batch statistics; reported gains are measured outcomes, not quantities forced by construction.

full rationale

The paper's central construction (Section 2.3) defines a query-specific base refusal reward Rbase(qi)=1-(max(pi,ε1))^γ from a single offline estimate of success probability pi on a frozen non-abstaining checkpoint, then subtracts a dynamic penalty P(Et) computed from the observed batch excess refusal rate (EMA-smoothed and step-quantized in Appendix B.1). This is intentional reward design, not a derivation that reduces a claimed prediction or first-principles result to its own inputs. The composite GRPO advantage further mixes the shaped outcome term with independent dense citation rewards (Appendix B.2). All reported improvements (precision, RA-F1) are post-training measurements on held-out multi-hop QA benchmarks against non-abstaining and static-reward baselines (Tables 1–7, Figure 1); none of the equations equate a fitted parameter to the quantity being claimed. Related-work citations are ordinary background and do not supply a load-bearing uniqueness theorem or ansatz that forces the result. The method is therefore self-contained and free of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on a small set of free hyper-parameters that control the refusal boundary and on standard RL and retrieval assumptions; no new physical or mathematical entities are postulated beyond the algorithmic constructs themselves.

free parameters (4)
  • courage factor γ = 0.2 (sweet spot)
    Single scalar that non-linearly maps prior success probability into the base refusal reward; chosen by grid search (optimal reported at 0.2).
  • clip thresholds ε1, ε2 = 0.05
    Lower bounds (default 0.05) that prevent zero or unbounded refusal rewards.
  • EMA decay α and quantization step τ = α=0.5, τ=0.1
    Stabilize the dynamic penalty; set to 0.5 and 0.1 respectively.
  • composite advantage weights (2, 3, -1) = 2·Wh(R)+3·Wh(Rcov)-1·Wh(Rred)
    Relative weights of outcome, citation-coverage and redundancy terms inside the GRPO advantage.
axioms (3)
  • domain assumption Group Relative Policy Optimization (GRPO) with binary or shaped outcome rewards is a valid and stable RL algorithm for multi-turn search agents.
    Adopted without re-derivation from Shao et al. (2024) and Search-R1; used throughout Section 2.
  • ad hoc to paper An offline estimate of per-query success probability on a non-abstaining checkpoint is a sufficiently accurate proxy for the model’s intrinsic capability boundary.
    Core modeling choice that defines Rbase(qi); stated in Section 2.3.
  • domain assumption LLM-as-judge scores for answer correctness and citation quality are reliable enough to serve as training rewards.
    GPT-oss-120b is used as external judge; common practice but unproven for the exact reward scale.
invented entities (2)
  • RA-F1 (Reliability-Aware F1) no independent evidence
    purpose: Single scalar that balances precision and accuracy (recall) for abstaining agents.
    Defined as the harmonic mean of Prec and Acc; introduced to quantify the capability-reliability trade-off.
  • dynamic refusal reward rref(qi)=Rbase(qi)-P(Et) no independent evidence
    purpose: Query- and batch-dependent scalar that replaces the missing abstention term in standard binary RL rewards.
    Central algorithmic construct of AWA-RL; no external validation outside the reported experiments.

pith-pipeline@v1.1.0-grok45 · 36861 in / 2795 out tokens · 37305 ms · 2026-07-14T09:36:25.924492+00:00 · methodology

0 comments
read the original abstract

Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.

Figures

Figures reproduced from arXiv: 2607.10738 by Bei Chen, Chengen Huang, Fengji Zhang, Jacky Keung, Tianyu Fan, Xinyao Niu, Yuxiang Zheng.

Figure 1
Figure 1. Figure 1: Training dynamics of batch refusal ratios over 90 GRPO steps under different [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 linked inside Pith

  1. [1]

    Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, and Chao Huang

    The faiss library. Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, and Chao Huang. 2025. Understanding deepresearch via re- ports.arXiv preprint arXiv:2510.07861. Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforce- men...

  2. [2]

    MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, and 1 oth- ers

    Webshaper: Agentically data synthesizing via information-seeking formalization.arXiv preprint arXiv:2507.15061. MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Jifeng Dai, Xuan Dong, and 1 oth- ers. 2025. Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, ...

  3. [3]

    This EMA acts as a low-pass filter, ensuring that the penalty only responds to sustained trends in over-refusal or under-refusal rather than tran- sient batch noise

    Exponential Moving Average (EMA) Smoothing:Instead of using the instanta- neous excess refusal rate Et, we maintain a smoothed running average ¯Et across training steps: ¯Et =α ¯Et−1 + (1−α)E t where α∈[0,1) is the decay factor (set to α= 0.5 in our experiments). This EMA acts as a low-pass filter, ensuring that the penalty only responds to sustained tren...

  4. [4]

    staircase

    Step-wise Quantization:To further prevent microscopic, continuous shifts in the refusal reward, we enforce discrete, step-wise up- dates by quantizing the smoothed excess rate ¯Et. Given a predefined step-tier threshold τ (e.g., τ= 0.1 ), the effective excess rate is discretized as: ˜Et =τ⌊ ¯Et/τ⌉ where ⌊·⌉ denotes rounding to the nearest in- teger. Ultim...

  5. [5]

    The pred_answer doesn't need to be exactly the same as any of the ground truth answers, but should be semantically same for the question. ,→ ,→

  6. [6]

    rationale

    Each item in the ground truth answer list can be viewed as a ground truth answer for the question, and the pred_answer should be semantically same to at least one of them. ,→ ,→ ,→ question: {question} ground truth answers: {gt_answer} pred_answer: {pred_answer} The output should in the following json format: ```json {{ "rationale": "your rationale for th...

  7. [7]

    **Question**: The user query

  8. [8]

    **Ground Truth (GT) Answer**: The correct answer string

  9. [9]

    Each atomic evidence represents one key piece of information

    **Ground Truth (GT) Atomic Evidence**: A list of specific supporting facts that should be cited. Each atomic evidence represents one key piece of information. ,→ ,→ ,→

  10. [10]

    **Model Citations**: The citations/evidence passages that the model has provided in its answer.,→ ## Goal Evaluate the model's citations on TWO dimensions: A) **Coverage Count**: How many GT Atomic Evidence items are covered by the citations?,→ B) **Redundancy/Irrelevance Severity**: The severity level of redundant or irrelevant citations (0-3 scale).,→ #...

  11. [11]

    Go through EACH item in the GT Atomic Evidence list one by one.,→

  12. [12]

    For each GT item, check if the Model Citations contain the **same semantic information**:,→ - Exact wording is NOT required - BUT specific details (names, dates, numbers, entities) MUST match,→ - Vague references do NOT count as coverage

  13. [13]

    If a GT item is sufficiently covered (even partially across multiple citations), count it as **1**.,→

  14. [14]

    covered_count

    If a GT item is completely missing or only vaguely referenced, count it as **0**.,→ **Output**:`covered_count`= integer, the total number of GT Atomic Evidence items that are covered.,→ **Example**: - GT Atomic Evidence has 5 items: [A, B, C, D, E] - Model Citations cover: A (fully), B (fully), C (missing), D (vaguely), E (fully),→ - covered_count = 3 (on...

  15. [15]

    ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ }, {

    As an important regional hub in the Yangtze River Delta, Nanjing is well-connected by over 60 state and provincial highways to all parts of China. Express highways such as Hu–Ning, Ning–He, Ning–Hang enable commuters to travel to Shanghai, Hefei, Hangzhou, and other important cities quickly and conveniently. Inside the city of Nanjing, there are of highwa...

  16. [16]

    ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ }, {

    As an important regional hub in the Yangtze River Delta, Nanjing is well-connected by over 60 state and provincial highways to all parts of China. Express highways such as Hu–Ning, Ning–He, Ning–Hang enable commuters to travel to Shanghai, Hefei, Hangzhou, and other important cities quickly and conveniently. Inside the city of Nanjing, there are of highwa...