Pith. sign in

REVIEW 3 major objections 5 minor 17 references

Lessons from Training Grounded LLMs with Verifiable Rewards

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a two-stage GRPO reward scheme—first answer correctness and citations, then refusal quality—makes retrieval-augmented models measurably more grounded, especially when the model reasons before answering.

desk verdict A concrete and honest two-stage GRPO recipe for grounded RAG, with one real soft spot: the automatic NLI judge behind both reward and metric is unnamed. read the letter →

arxiv 2506.15522 v1 pith:HTM34XYU submitted 2025-06-18 cs.CL

Shang Hong Sim , Tej Deep Pala , Vernon Toh , Hai Leong Chieu , Amir Zadeh , Chuan Li , Navonil Majumder , Soujanya Poria This is my paper
classification cs.CL
keywords reinforcementlearningretrieval-augmentedgenerationgroundedresponsecitationgroundingrefusalGRPOTrust-Scorereasoningmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ground-GRPO, the method proposed here, trains a retrieval-augmented language model with reinforcement learning using rewards that can be checked automatically: whether each answer statement exactly matches a gold claim, whether its citations are supported by the retrieved documents, and whether refusals to unanswerable queries resemble the gold refusal. The paper's central claim is that this outcome-based reward scheme, applied with the GRPO algorithm in two stages (answer-and-citation first, refusal second), makes models measurably more grounded—more correct, better cited, and more willing to refuse when evidence is missing. Reasoning-augmented models benefit much more than instruction-only models, especially on the refusal component. Combining distillation with GRPO helps on open-ended long-form QA, while on short-form factoid QA the distilled baseline is already strong enough that GRPO adds little or hurts. If the claims hold, RAG systems can be aligned toward verifiable behavior without gold reasoning traces or expensive per-step annotations.

What carries the argument

The central object is a hierarchical outcome-based reward function trained with GRPO, a reinforcement-learning update that compares a group of sampled responses to a question and increases the probability of the higher-scoring ones relative to the group. The reward stack is: exact-match correctness per statement (0.5), citation support judged by an NLI model (+0.5 for correct, −0.5 for incorrect), a hard format reward with a soft tag-count reward, and a refusal reward computed by fuzzy matching against a gold refusal. The two-stage curriculum, answerable-then-mixed, is what carries the argument: it first stabilizes answer extraction and citation behavior, then adds answerability judgments. The NLI-based citation check and the 0.85 fuzzy threshold are the automated proxies that make the rewards 'verifiable' without human annotation.

What would settle it

Recompute Trust-Score and its F1GR and F1GC components on a subset where citation support and refusal equivalence are labeled by humans instead of the NLI model and the 0.85 fuzzy matcher; if the two-stage Ground-GRPO models no longer beat their prompt-only reasoning and instruction baselines on this human-labeled subset, the central claim is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a hierarchical set of verifiable outcome rewards, optimized by GRPO, improves grounded response quality in RAG, and that the improvement is strongest when the policy already reasons. Concretely, the model is rewarded for statements that exact-match gold claims (0.5), for correct citations on those statements (+0.5) and penalized for incorrect citations (−0.5), and for refusing in a way that fuzzy-matches the gold refusal (threshold 0.85). Training first on answerable questions and then on a mixed set stabilizes the learning signal: the two-stage curriculum beats stage-1-only training by 12.5% and stage-2-only by 6.8% in Trust-Score, with the largest component gain in grounded refusals (15.8%). Reasoning variants of Qwen3 and LLaMA-3.1 outperform their instruction-tuned counterparts in 8 of 9 zero-shot configurations, and Ground-GRPO widens the gap; an NLI-based process reward improves reasoning–answer alignment by 8% at a small Trust-Score cost, and removing the bad-citation penalty pushes answer rates toward 100%, collapsing citation quality in larger models.

Load-bearing premise

The load-bearing premise is that the automatic judges used to score citations and refusals—an NLI model for citation support and a 0.85 fuzzy match for refusal similarity—are accurate enough that optimizing rewards against them improves genuine grounding.

Editorial extensions

If this is right

  • Training with the two-stage Ground-GRPO reward should raise Trust-Score on RAG benchmarks, with grounded refusal (F1GR) the most improved component.
  • Reasoning-augmented models are the better substrate for this RL: they show gains across answer correctness, grounded refusal, and citation quality, while instruct models improve mainly in citation quality.
  • Distillation should come before GRPO, not after; the reverse order underperforms by about 15% across models and datasets.
  • On open-ended, long-form QA like ELI5 and ExpertQA, GRPO adds 4–5 Trust-Score points on top of distilled models, but on short-form list-style QA like QAMPARI it hurts answer correctness by up to 10 points.
  • Scaling the Stage-2 training set from 300 to 3000 samples monotonically improves Trust-Score and its subcomponents, so the method benefits from more mixed answerable and unanswerable data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the automatic judges are the weak link, the measured Trust-Score gains overstate genuine grounding improvements; a human-judged subset of citations and refusals would separate real grounding gains from optimization of noisy proxies.
  • The process-reward result suggests a cheap follow-up: alternating or annealing between the outcome rewards and the NLI alignment reward might capture the 8% alignment gain without the 1.2% Trust-Score drop.
  • The finding that removing the bad-citation penalty acts as a mild regularizer on a 4B model but causes citation collapse on an 8B model implies that reward design in this scheme may need per-model-capacity tuning.
  • Because all reported training runs use GRPO, the same staged reward decomposition could be tested with other on-policy optimizers to see whether the benefit comes from the reward structure itself or from the group-relative update.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents Ground-GRPO, a two-stage reinforcement learning procedure that applies GRPO to three verifiable reward components—exact-match answer correctness, NLI-based citation support, and fuzzy-match refusal quality—for RAG models, and evaluates the resulting models on ASQA, QAMPARI, ELI5, and ExpertQA using the Trust-Score metric. The authors report that reasoning-augmented backbones benefit more from GRPO than instruction-tuned backbones, that staged training improves refusal and citation behavior, and that applying GRPO on top of GPT-4 distillation helps on long-form QA. The paper also contributes ablations of training-set size, process supervision, and reward-design choices.

Significance. The paper's strengths are its breadth (three backbone families, four benchmarks, multiple ablations), the transparent reporting of stage-wise and reward-design ablations, and the honest documentation of counterexamples such as Qwen-3-4b Instruct regressing under Ground-GRPO. If the headline effects are real and reproducible, the two-stage, verifiable-reward recipe would be a practically useful addition to the RAG alignment toolbox. However, the core grounding metrics and the reward functions share unvalidated automatic judges, so the central claim that the method improves groundedness rather than proxy scores is not established by the reported experiments. The study also lacks variance estimates or significance tests, making several comparative claims stronger than the evidence supports.

major comments (3)
  1. [Sections 2.1, A.3, and Eq. (6)] The headline F1GR and F1GC improvements are measured with the same automatic components used inside the training rewards. The citation reward in Eq. (2) and Figure 1 is computed by an unspecified NLI model, and F1GC in Section A.3 relies on the same "documents support statement" judgment; the refusal reward in Eq. (6) uses an unspecified fuzzy-matching r_score with threshold 0.85, and F1GR in Section A.1 classifies responses as answers versus refusals. The manuscript never names the NLI model or the fuzzy matcher, and it provides no human validation or alternative-verifier experiment. Without this, the reported gains could reflect optimization of a noisy or gameable proxy rather than genuine grounding; Section 4.7 already shows that an NLI-based signal can improve its target metric (%Align) while decreasing Trust-Score by 1.2%, so the sensitivity is real. Please specify the judges, evaluate on a human-annotated or independently judged subset, and report agreement.
  2. [Section 4.2 and Figure 2] The claim that "Trust-Score improves consistently following GRPO training across all model types and datasets" is contradicted by the paper's own Figure 2: for Qwen-3-4b Instruct, TRUST decreases on ASQA (58.2→55.5), QAMPARI (21.0→14.9), and ELI5 (26.6→25.6). The more general claim that reasoning models "significantly outperform" is also unsupported by any significance testing or variance reporting; with only 300 Stage-1 and 1,000 Stage-2 training examples and no random-seed variation, these differences may be within noise. Please soften the claims to the pattern that actually holds, add a per-configuration breakdown that acknowledges the exceptions, and report confidence intervals or significance tests.
  3. [Section 4.4 and Table 1] The text says that "combining distillation with Ground-GRPO leads to notable performance improvements across all three QA benchmarks," but Table 1 shows SFT→Ground-GRPO decreasing TRUST relative to SFT on QAMPARI for all backbones (e.g., Qwen-3-4b: 52.60→42.79; Qwen-3-8b: 60.82→50.76; LLaMA-3.1: 58.08→47.62) and on ASQA for Qwen-3-8b (70.31→69.52). The subsequent nuanced discussion is more accurate, but the earlier overstatement should be corrected to match the data.
minor comments (5)
  1. [Section 2.1] "hierarchial" in the introduction and method text should be "hierarchical".
  2. [Section 4.8] In the reward-design ablation, the sentence "removing the penalty leads to a drastic drop in F1GC (e.g., from 67.96 to 22.78 on QAMPARI), despite modest gains in F1GC" appears to contain a typo; the second F1GC should presumably refer to F1AC, since Figure 6 shows F1GC dropping while F1AC rises for Qwen-3-8b on QAMPARI.
  3. [Algorithm 1] Algorithm 1 uses is_answerable(Q,D) and r_score(S) without defining them at the point of use; r_score is introduced in Eq. (6), but is_answerable should be defined explicitly as the gold answerability label used in training.
  4. [Sections A.2 and A.3] The metric definitions for P_AC, R_AC, P_cite, and R_cite are described in words but not operationalized; please provide the exact formulas or pseudocode, since these metrics are the basis of the central comparisons.
  5. [General] No code, checkpoints, or configuration files are released; given that the NLI judge and fuzzy matcher are not named, a reproducibility appendix with these artifacts and exact prompts would be needed to allow independent verification.

Circularity Check

1 steps flagged · score 3.0 of 10

Central RL/grounding claims are not circular; one secondary result (process reward improves NLI alignment) is true by construction.

  1. self definitional [Section 4.7, Equations 7 and 8]
    "we introduce a process reward using an NLI model that evaluates whether the model's reasoning supports its final decision to answer or refuse: Rprocess = nli_score (7) ... To measure the alignment between reasoning and final output, we define a simple consistency metric: %Align= 1/N \sum_i 1[NLI(r_i, a_i)→entailment] (8)"

    The training reward Rprocess is literally the NLI entailment score between the reasoning trace and the final decision, and the reported alignment metric %Align is the fraction of examples where the same NLI entailment check succeeds. Training the model to maximize Rprocess therefore directly maximizes %Align, so the observed 8% average improvement in alignment is a restatement of the optimization objective, not an independent empirical discovery. The paper's accompanying Trust-Score drop of 1.2% is independent and more informative, and this circularity is confined to a secondary ablation rather than the central grounding claim.

full rationale

The paper's central claims are not circular. The reasoning-versus-instruct comparison is made on prompt-only baselines before any reward training, so it cannot reduce to the reward design. The two-stage training and distillation-ordering conclusions compare fixed reward configurations on the same external Trust-Score benchmark, with metric formulas provided in Appendix A rather than resting solely on the self-citation to Song et al. (2025b). The rewards and Trust-Score do share automatic components (NLI-based citation support and fuzzy refusal matching), which is a proxy-alignment and reproducibility risk, but the paper never defines the metric as the reward or vice versa, so that overlap does not amount to equation-level circularity. The one genuinely circular passage is the process-supervision ablation in Section 4.7: Rprocess is defined as nli_score, and %Align is the same NLI entailment check, making the reported alignment gain true by construction. Because this is a minor secondary result and the central conclusions about reasoning, stage-wise training, and distillation ordering have independent content, the overall circularity score is 3.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by inherited measurement assumptions. The paper contributes no new physical or mathematical entities; its contribution is a training procedure. The free parameters are hand-set reward weights and a refusal threshold. The axioms are the validity of the automatic judges, NLI entailment, fuzzy refusal matching, and exact-match correctness, plus the self-authored Trust-Score benchmark. GRPO hyperparameters are an additional unquantified domain assumption.

free parameters (2)
  • refusal similarity threshold in R_refusal = 0.85
    Equation 6 treats a response as a refusal only when r_score is greater than 0.85; the paper gives no sensitivity analysis, and this threshold directly controls when Stage 2 rewards refusal.
  • reward coefficients in Stage 1 = EM 0.5, citation 0.5, bad citation -0.5, format 1.0
    Equations 1 to 5 fix these weights by hand; only the bad-citation penalty is ablated, so the other coefficients are unvalidated design choices.
assumptions (5)
  • domain assumption Trust-Score (F1AC, F1GR, F1GC) is a valid measure of response grounding.
    The abstract and all main tables compare models using Trust-Score from Song et al. 2025b, a benchmark authored by overlapping researchers; no human evaluation or external benchmark is used.
  • domain assumption The NLI model used for citation verification and process rewards yields reliable entailment judgments.
    Figure 1 and Equation 7 rely on NLI scores to reward correct citations and reasoning-answer alignment; the specific NLI model and its accuracy are not reported.
  • domain assumption Statement-level exact match is a valid answer-correctness signal.
    Equation 1 rewards 0.5 for any statement containing an exact match, which ignores paraphrastic correctness and helps explain the observed formatting sensitivity on ASQA and QAMPARI.
  • domain assumption Answerability labels and gold claims in the Trust-Align training data are correct.
    Stage 1 selects samples where the answer can be found in documents, and Stage 2 relies on answerable and unanswerable labels; the labeling pipeline is inherited from Trust-Align and not independently verified in this paper.
  • domain assumption GRPO with 8 responses per question over 4 epochs gives a stable optimization signal.
    Section 3.1 fixes these hyperparameters; no repeated seeds or variance are reported, so reward noise and run-to-run variation are unquantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lessons from Training Grounded LLMs with Verifiable Rewards." pith.science (2026). https://pith.science/paper/HTM34XYU

@misc{pith2026250615522,
  author       = {Pith},
  title        = {Pith review of: Lessons from Training Grounded LLMs with Verifiable Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTM34XYU}},
  note         = {Machine review of arXiv:2506.15522}
}
read the original abstract

Generating grounded and trustworthy responses remains a key challenge for large language models (LLMs). While retrieval-augmented generation (RAG) with citation-based grounding holds promise, instruction-tuned models frequently fail even in straightforward scenarios: missing explicitly stated answers, citing incorrectly, or refusing when evidence is available. In this work, we explore how reinforcement learning (RL) and internal reasoning can enhance grounding in LLMs. We use the GRPO (Group Relative Policy Optimization) method to train models using verifiable outcome-based rewards targeting answer correctness, citation sufficiency, and refusal quality, without requiring gold reasoning traces or expensive annotations. Through comprehensive experiments across ASQA, QAMPARI, ELI5, and ExpertQA we show that reasoning-augmented models significantly outperform instruction-only variants, especially in handling unanswerable queries and generating well-cited responses. A two-stage training setup, first optimizing answer and citation behavior and then refusal, further improves grounding by stabilizing the learning signal. Additionally, we revisit instruction tuning via GPT-4 distillation and find that combining it with GRPO enhances performance on long-form, generative QA tasks. Overall, our findings highlight the value of reasoning, stage-wise optimization, and outcome-driven RL for building more verifiable and reliable LLMs.

Figures

Figures reproduced from arXiv: 2506.15522 by the authors.

Figure 1
Figure 1. Overview of the Ground-GRPO framework for reward-guided reinforcement learning in retrieval-augmented genera￾tion (RAG) settings. Given a user query, the Policy Model generates a set of candidate outputs O1, . . . , ON , each of which is evaluated by a Reference Model to compute three distinct reward signals: (1) EM Reward, which checks for exact matches against ground-truth answers and provides a score based on the… view at source ↗
Figure 2
Figure 2. Performance comparison between reasoning and instruct models across three model families (Qwen-3-4b, Qwen-3-8b, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Stage-wise training ablation study across three models (Qwen-3-4b, Qwen-3-8b, LLaMA-3.1-8b) and three datasets [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance trends across different training data sizes for three question-answering datasets. The analysis reveals [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Process supervision ablation study across two language models and three question-answering datasets. The comparison [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Reward Design Ablation Study: Impact of Bad Citation Penalty on Model Performance. We compare Ground￾GRPO training with and without the bad citation penalty component across two model variants (Qwen-3-4b and Qwen-3-8b) on three datasets. The red dashed lines indicate t…
Figure 7
Figure 7. Figure 7: The pipeline of Trust-Align [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 2 canonical work pages

  1. [1]

    I apologize, but I couldn't find an answer to your question in the search results

    1 4 5 6 Questions Documents Seed Set Retrieve top 100 docs using Wikipedia, Sphere GPT-4synthesizer ... Matt Prater at 64 yards[Gold Claim 2], ... OveJohansson in a 1976... [Gold Claim 1]. ... Matt Prater at 64yards [1][3], ... OveJohansson in a 1976...[2][4]. Answerable Questions Unanswerable Questions Positive Answer: “I apologize, but I couldn't find a...

    work page 1976

  2. [3]

    arXiv:2501.12948

    DeepSeek-R1: Incentivizing Rea- soning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Gao, L.; Dai, Z.; Pasupat, P.; Chen, A.; Chaganty, A. T.; Fan, Y .; Zhao, V .; Lao, N.; Lee, H.; Juan, D.-C.; and Guu, K. 2023a. RARR: Researching and Revising What Language Models Say, Using Language Models. In Rogers, A.; Boyd-Graber, J.; and Okazaki,...

    arXiv

  3. [5]

    arXiv:2402.04315

    Training Language Models to Generate Text with Citations via Fine- grained Rewards. arXiv:2402.04315. Ji, B.; Liu, H.; Du, M.; and Ng, S.-K

    arXiv

  4. [6]

    arXiv:2412.12881

    RAG-Star: Enhancing Deliber- ative Reasoning with Retrieval Augmented Verification and Refinement. arXiv:2412.12881. Karpukhin, V .; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t

    arXiv

  5. [10]

    arXiv:2305.18290

    Direct Preference Opti- mization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. Shi, F.; Chen, X.; Misra, K.; Scales, N.; Dohan, D.; Chi, E. H.; Schärli, N.; and Zhou, D

    arXiv

  6. [11]

    arXiv:2403.17104

    Attribute First, then Generate: Locally-attributable Grounded Text Generation. arXiv:2403.17104. Song, H.; Jiang, J.; Min, Y .; Chen, J.; Chen, Z.; Zhao, W. X.; Fang, L.; and Wen, J.-R. 2025a. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. arXiv:2503.05592. Song, M.; Sim, S. H.; Bhardwaj, R.; Chieu, H. L.; Majumder, N...

    arXiv

  7. [12]

    arXiv:2505.09388

    Qwen3 Technical Report. arXiv:2505.09388. Xia, S.; Wang, X.; Liang, J.; Zhang, Y .; Zhou, W.; Deng, J.; Yu, F.; and Xiao, Y

    arXiv

  8. [13]

    arXiv:2407.01796

    Ground Every Sentence: Improv- ing Retrieval-Augmented LLMs with Interleaved Reference- Claim Generation. arXiv:2407.01796. Xu, F.; Shi, W.; and Choi, E

    arXiv

Show all 17 references
  1. [14]

    arXiv:2310.04408

    RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation. arXiv:2310.04408. Ye, X.; Sun, R.; Arik, S. Ö.; and Pfister, T

    arXiv

  2. [15]

    arXiv:2311.09533

    Effective Large Language Model Adaptation for Improved Grounding and Citation Generation. arXiv:2311.09533. Yoran, O.; Wolfson, T.; Ram, O.; and Berant, J

    arXiv

  3. [16]

    arXiv:2310.01558

    Making Retrieval-Augmented Language Models Robust to Irrelevant Context. arXiv:2310.01558. A Metric Details We report performance using Trust-Score, a composite metric that evaluates the trustworthiness of model responses across three dimensions: response truthfulness, factual...

    arXiv

  4. [2020]

    In Webber, B.; Cohn, T.; He, Y .; and Liu, Y ., eds.,Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781

    Dense Passage Retrieval for Open-Domain Question Answering. In Webber, B.; Cohn, T.; He, Y .; and Liu, Y ., eds.,Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781. Online: Association for Computa- tional Linguistics. Le...

    2020

  5. [2021]

    arXiv:2005.11401

    Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. Li, D.; Sun, Z.; Hu, B.; Liu, Z.; Hu, X.; Liu, X.; and Zhang, M. 2024a. Improving Attributed Text Generation of Large Language Models via Preference Learning. In Ku, L.-W.; Martins, A.; and Sr...

    2005 arXiv

  6. [2022]

    arXiv:2203.02155

    Training language models to follow instructions with human feedback. arXiv:2203.02155. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Man- ning, C. D.; and Finn, C

    arXiv

  7. [2023]

    arXiv:2310.11511

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511. Asai, A.; Wu, Z.; Wang, Y .; Sil, A.; and Hajishirzi, H

    arXiv

  8. [2024]

    arXiv preprint arXiv:2407.21783

    The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783. Includes Meta LLaMA-3.1 models (8B, 70B, 405B). Available via Hugging Face: https: //huggingface.co/meta-llama/Llama-3.1-8B-Instruct. Huang, C.; Wu, Z.; Hu, Y .; and Wang, W

    arXiv

  9. [2025]

    arXiv:2503.19470

    ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. arXiv:2503.19470. DeepSeek-AI

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.