Pith. sign in

REVIEW 3 major objections 7 minor 6 references

Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning

T0 review · 3 major / 7 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CRAFT shows that faithfulness-guided alignment improves, not sacrifices, answer accuracy in retrieval-augmented multi-hop QA.

desk verdict CRAFT's accuracy gains are real and worth engaging with, but the faithfulness improvement claim rests on an evaluator that also provides the training reward—so treat the headline synergy as provisional. read the letter →

arxiv 2602.01348 v3 pith:IRPNDOPE submitted 2026-02-01 cs.CL cs.LG

Yu Liu , Wenxiao Zhang , Diandian Guo , Cong Cao , Fangfang Yuan , Qiang Sun , Yanbing Liu , Jin B. Hong
show 1 more author
Zhiyuan Ma
This is my paper
classification cs.CLcs.LG MSC 68T50
keywords retrieval-augmentedgenerationmulti-hopquestionansweringfaithfulreasoningreinforcementlearningLLM-as-a-judgeprocesssupervisiontracesansweraccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether training a retrieval-augmented model to produce evidence-grounded, auditable reasoning hurts its answer accuracy. It answers no: CRAFT, a reinforcement-learning framework, trains models to emit structured traces—plan, cited document set, step-by-step reasoning, and answer—under four rewards covering format, citation validity, answer correctness, and a judge-based faithfulness audit. Across 0.5B to 7B models and three multi-hop benchmarks, CRAFT improves both Exact Match and Faithfulness, and the 7B model matches or exceeds much larger closed-source API models. The central, somewhat counterintuitive result is that the semantic judge reward raises answer accuracy rather than trading it off against faithfulness.

What carries the argument

The load-bearing mechanism is CRAFT's chain of faithfulness: a structured XML-style trace of plan, gold-document indices, reasoning steps, and answer, constrained so each component supervises the next. Training uses group-relative policy optimization with a decomposed reward: deterministic checks for format, citation F1, and answer soft-F1, plus an LLM judge reward that averages four binary faithfulness audits. This design turns open-ended semantic properties—plan adherence, evidence grounding, logical entailment—into differentiable process-level supervision, and makes unfaithful traces detectable at training time.

What would settle it

Take CRAFT-trained models to a held-out multi-hop dataset not in the training mix and have human annotators plus two independent judge models flag claims that are unsupported by the cited documents. If the faithfulness advantage disappears, or if the model's citations do not move when supporting passages are shuffled or replaced with near-duplicate distractors, the model is likely satisfying the judge's checklist rather than learning grounded reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that outcome-level reinforcement learning alone leaves reasoning traces weakly grounded, but adding a semantic faithfulness reward does not degrade accuracy—it improves it. CRAFT combines deterministic rewards (format compliance, citation-set F1 against gold documents, soft-F1 answer match) with a judge-based reward that audits four binary criteria: plan-to-reason consistency, evidence-boundary-to-reason consistency, reason-to-answer entailment, and claim-level grounding in the cited documents. Because the trace fields form a chain, any hallucination breaks at least one verifiable link, making unfaithful reasoning machine-checkable during training. Experiments ac

Load-bearing premise

The load-bearing premise is that the judge model's binary faithfulness audit captures genuine evidence grounding rather than a learnable checklist: the same high-capacity LLM supplies the training reward and the reported evaluation score, and the paper's own validation is limited to 500 human-annotated traces from a single 7B model plus a four-configuration cross-judge ranking check.

Editorial extensions

If this is right

  • Faithfulness can be enforced as a training-time objective, not just measured after generation, without sacrificing accuracy.
  • Structured trace templates can be tuned to desired auditability: full plan-and-citation traces give high auditability, while simpler reason-and-answer traces give high raw faithfulness.
  • Small models (1.5B, 3B) gain substantially from trace structure, and the 0.5B model only learns when some reasoning structure is present.
  • Removing the judge reward costs faithfulness without an accuracy gain, supporting the claim that faithfulness and accuracy are not in tension.
  • A 7B model trained on 20K mixed multi-hop examples can reach competitive accuracy and faithfulness on these benchmarks, comparable to much larger closed-source APIs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: if the judge model is swapped during training, the learned trace behavior should shift toward the new judge's notion of faithfulness; the paper's cross-judge check only re-evaluates, it does not re-train.
  • The accuracy–faithfulness synergy is demonstrated within the training distribution; the paper itself flags that transfer to out-of-distribution multi-hop QA remains open, so a useful stress test is applying CRAFT to a held-out benchmark not in the 20K mixture.
  • The 0.5B model's collapse on the answer-only template suggests structured traces act as a credit-assignment scaffold; this predicts that adding a lightweight intermediate reward to a small model should improve exploration, which could be tested directly.
  • If judge-scored faithful traces are genuinely grounded, they could serve as high-quality training data for downstream auditing or answer verification, producing a measurable downstream benefit beyond benchmark scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes CRAFT, a GRPO-based reinforcement learning framework for the response-generation stage of retrieval-augmented multi-hop QA. The model is trained to emit structured XML-style traces (plan, gold-document citations, reasoning, answer) under a composite reward: deterministic rewards for format compliance, citation validity, and answer correctness, plus a judge-based reward (Rfaith) for semantic faithfulness. The faithfulness reward is computed by Qwen3-Max according to four binary checks (plan-to-reason, citation-boundary, reason-to-answer, and evidence grounding). Experiments on MuSiQue, HotpotQA, and 2WikiMHQA across Qwen2.5 models from 0.5B to 7B report simultaneous gains in EM/F1 and in a Faithfulness metric, with CRAFT 7B reportedly matching or exceeding several larger API models. The authors also ablate trace variants, remove the judge reward, validate the judge against 500 human-annotated MuSiQue traces, and report a cross-judge comparison with Gemini-2.5-Flash.

Significance. If the results hold, the framework is practically valuable for auditable RAG reasoning, and the finding that judge-based process rewards can improve rather than trade off answer accuracy is an interesting counterpoint to outcome-only RL. The paper has real strengths: a thorough ablation over five trace variants and four model scales, deterministic rewards that are directly verifiable, training-dynamics analysis, human validation of the judge, a cross-judge consistency check, and released code. The main weakness is that the Faithfulness metric used to validate the central claim is produced by the same judge model that supplies the Rfaith training reward. The two defenses—human agreement on 500 traces from one 7B model using the same criteria, and aggregate rank preservation over four configurations with a second judge—are not yet sufficient to rule out reward hacking. The accuracy half of the central claim is externally measured and therefore credible; the faithfulness half is currently over-strong.

major comments (3)
  1. [§3.3.2, §4.5, Appendix C (Table 3, Table 12)] Circularity of the faithfulness metric is load-bearing for the title claim. Qwen3-Max supplies Rfaith during training and also produces the reported Faithfulness numbers; the human validation covers only 500 MuSiQue traces from CRAFT 7B and uses the same four binary criteria as the judge. This establishes that the judge applies its checklist consistently with humans, not that satisfying the checklist entails genuine faithfulness or that the trained model did not exploit judge blind spots. The cross-judge check compares aggregate scores over only four configurations; rank preservation does not rule out shared biases. Please add per-trace agreement against an independent judge not used in training, human evaluation across model scales and at least one additional benchmark, or an objective grounding metric (e.g., claim-level precision against gold supporting documents).
  2. [§4.1 and Limitations] All evaluation is in-distribution: the 20K training samples and the 2K test samples come from the same three benchmarks (MuSiQue, HotpotQA, 2WikiMHQA). The Limitations section explicitly says the synergy is established 'within the training distribution first.' The abstract and conclusion, however, state the general result that faithfulness-guided alignment improves answer accuracy. Either temper the claims to the in-distribution setting or add a held-out multi-hop QA benchmark not seen in training (e.g., an additional RAG dataset) to test transfer.
  3. [§4.3, Table 2; §4.2, Table 1] No repeated runs or statistical uncertainty are reported. Several load-bearing comparisons rely on small deltas—for example, adding the judge reward improves CRAFT v1 MuSiQue EM by only +3.0 points (51.37 vs. 54.35), and some cross-template differences are similar in size. Because GRPO is stochastic, single-run numbers are insufficient to establish the 'judge reward improves accuracy' claim. Report at least three seeds with mean and standard deviation (or confidence intervals) for the main tables and for the judge-reward ablation.
minor comments (7)
  1. [§4.1] The retrieval setup is underspecified: how are the candidate document sets D constructed, which retriever is used, and are gold supporting documents derived from the original dataset annotations? This is needed for reproducibility. Also specify the exact prompt templates for base and API models so readers know how unstructured baselines were mapped to the CRAFT trace format.
  2. [Figure 3] The axes labels are unclear: the left y-axis is 'Total Reward' and the right is 'sub-rewards,' but it is not stated whether Total Reward is the normalized GRPO reward or the unnormalized sum, nor what the numerical range represents. Add a legend or caption note.
  3. [Algorithm 1] Line 5 refers to 'Faithfulness Judge Principles (A, B, C, D)' but A–D are not defined in the algorithm; either label the checks A, B, C, D or remove the parenthetical.
  4. [Appendix B.2] The comparison between training-time average Rfaith (0.11, from on-policy high-temperature sampling over all samples) and greedy test-time average (69.33%, computed only on correct-answer samples) is not apples-to-apples. Clarify or provide the test-time average over all samples as well.
  5. [Table 2 and Table 10] The colored small delta numbers in Table 2 may not survive print; use explicit columns such as ΔEM/ΔF1/ΔFaith. Also, Table 10 uses '2WikiMQA' while elsewhere '2WikiMHQA' is used; standardize.
  6. [Eq. (3)] Define w_c in the main text (the weights are only mentioned as equal to 1.0 in §4.1) and state explicitly that all reward components are in [0,1] so the normalization is meaningful.
  7. [§4.5] Cohen's kappa is normally attributed to Cohen (1960); the Artstein (2017) handbook citation is appropriate but should be supplemented with the primary reference.

Circularity Check

2 steps flagged · score 6.0 of 10

Faithfulness gains reflect the training reward itself: Qwen3-Max scores both Rfaith and the reported Faithfulness metric.

  1. fitted input called prediction [§3.3.2 (Eq. 7), §4.5, Appendix A]
    "We employ Qwen3-Max (Team, 2025) as the LLM judge for both the faithfulness reward Rfaith during training and the Faithfulness metric during evaluation."

    The reported 'Faithfulness' outcome is the same Qwen3-Max Audit procedure (Algorithm 1) that defines the training reward Rfaith (Eq. 7). GRPO directly optimizes Rfaith, so Table 1's +22.1% Faithfulness gain is, by construction, the model improving on its own training objective. EM/F1 gains are externally grounded and not affected, but the headline claim that CRAFT improves reasoning faithfulness is an evaluation of the trained objective, not an independent verification of faithful reasoning.

  2. other [§4.5, Table 3]
    "two annotators with NLP research experience independently assessed 500 randomly sampled MuSiQue reasoning traces from CRAFT 7B using the same four-aspect binary criteria as the LLM judge (Algorithm 1)."

    The human validation compares the judge to human raters applying the judge's own four criteria; it does not test whether satisfying those criteria entails genuine grounding. A policy exploiting judge leniency or checklist blind spots could pass all four checks while remaining unfaithful. The cross-judge check (Appendix C) also only compares aggregate scores over four configurations with another LLM judge, so it does not break the construction; two judges can share the same biases.

full rationale

The central circularity is that the Faithfulness metric is identical to the training reward Rfaith: both are Qwen3-Max's Algorithm 1 Audit. Reporting that CRAFT 'improves Faithfulness' is therefore reporting improvement on the training objective, not an external property. The paper's two defenses are partial: human annotation on 500 MuSiQue traces from a single 7B model uses the same criteria as the judge, and the cross-judge evaluation with Gemini-2.5-Flash compares only four aggregate configurations (Spearman ρ=1.0), which does not rule out shared judge bias or per-instance reward hacking. The accuracy half of the claim remains credible because EM and F1 are computed against gold labels and are not optimized by the judge. Self-citations (OPERA, HopRAG) appear only in related work and are not load-bearing. Overall, the faithfulness-improvement claim is partially circular, hence score 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on benchmark-specific supervision (gold support sets), a hand-defined judge-based notion of faithfulness, standard GRPO assumptions, and in-distribution evaluation. The judge's validity is only partially established, and no new physical or conceptual entities are introduced.

free parameters (3)
  • Reward weights w_c (Rfmt, Rgold, Rfaith, Rans) = 1.0 each
    Eq. (3) / Appendix A: all weights set to 1.0 as a 'natural default'; the balance of deterministic vs judge rewards is a hand-chosen design choice that the reported trade-off depends on.
  • Training data size and composition = 20,000 (10k MuSiQue, 5k HotpotQA, 5k 2WikiMHQA)
    §4.1: the amount and mix of training data are chosen, not derived; all results are in-distribution relative to this data.
  • Judge temperature = 0.0
    Appendix A: judge sampled deterministically; this removes stochasticity but leaves the judge's outputs as a fixed function.
assumptions (5)
  • domain assumption Gold supporting document sets S* are available for training and evaluation.
    §3.1, Eq. (5): Rgold uses S*; this is a benchmark-specific annotation not generally available in deployed RAG.
  • ad hoc to paper The four binary judge checks in Algorithm 1 operationalize faithfulness.
    Algorithm 1 defines faithfulness; human validation is limited to 500 MuSiQue CRAFT7B traces (Table 3) and cross-judge ranking to four configurations (Appendix C).
  • standard math GRPO update (Eqs. 1-2) optimizes the policy as intended.
    Shao et al. 2024; standard single-step RL update.
  • domain assumption Test sets (2,000/benchmark) are disjoint from the 20,000 training entries despite being sampled from the same benchmarks.
    §4.1: no explicit split or leakage control is described; the evaluation is in-distribution by construction.
  • ad hoc to paper Chain-of-faithfulness: any unfaithful trace violates at least one of the four checks.
    §3.2: this is the design principle motivating CRAFT; it is plausible but unproven, and the 0.5B v5 collapse shows limits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning." pith.science (2026). https://pith.science/paper/IRPNDOPE

@misc{pith2026260201348,
  author       = {Pith},
  title        = {Pith review of: Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRPNDOPE}},
  note         = {Machine review of arXiv:2602.01348}
}
read the original abstract

Retrieval-augmented generation (RAG) can achieve strong answer accuracy on multi-hop questions, but outcome-level rewards often leave reasoning traces weakly grounded and difficult to audit. Under noisy retrieval, models may exhibit right-answer-wrong-reason failures, where the final answer is correct but the supporting rationale exploits shortcuts or unsupported evidence. We therefore ask whether faithfulness-guided alignment hurts answer accuracy in post-retrieval reasoning. To study this question, we propose CRAFT (Calibrated Reasoning with Answer-Faithful Traces), a reinforcement learning framework for the response-generation stage of retrieval-augmented multi-hop question answering. CRAFT trains models to produce structured reasoning traces with configurable auditability, while combining deterministic rewards for format compliance, answer correctness, and citation validity with a judge-based reward for semantic faithfulness. Experiments across model scales and benchmarks show that CRAFT unlocks task-specific reasoning capacity from 1.5B upward, improving both answer accuracy and Faithfulness; at 0.5B, performance remains sharply template-dependent. At 7B, CRAFT improves Faithfulness over the Base model in all evaluated settings and remains competitive with strong closed-source models. Code is available at https://github.com/Ameame1/CRAFT.

Figures

Figures reproduced from arXiv: 2602.01348 by the authors.

Figure 1
Figure 1. Comparison of existing approaches (top) and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of CRAFT. The framework comprises three components: CRAFT Trace Variants (top-left) defines structured trace templates (CRAFTv1–CRAFTv5); Decomposed Reward Auditor (bottom-left) provides four reward signals (Rfmt, Rgold, Rfaith, Rans); Faithfulness Audit Principle (bottom-right) enables machine-checkable reasoning verification. Top-right: a concrete CRAFTv1 trace with all four judge checks passing (Rfaith =… view at source ↗
Figure 3
Figure 3. Training dynamics of GRPO across scales (0.5B, 1.5B, 3B, 7B) using [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Training data distribution (t-SNE). 20,000 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Training dynamics using CRAFTv2 template [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Training dynamics using CRAFTv3 template. Note: Rgold not applicable [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Training dynamics using CRAFTv4 template. Note: Rgold not applicable [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Training dynamics using CRAFTv5 template. Note: Rfaith and Rgold not applicable. (§B.4), format error patterns (§B.5), and model scale sensitivity (§B.6). B.1 Training Dynamics for Additional Variants Figures 5–8 show the training dynamics for CRAFTv2–CRAFTv5. Consiste…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 linked inside Pith

  1. [3]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    Direct Preference Optimization: Your Lan- guage Model Is Secretly A Reward Model.Advances in Neural Information Processing Systems, 36:53728– 53741. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms.arXiv preprint arXiv:1707.06347. Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang ...

    arXiv 2017

  2. [4]

    arXiv preprint arXiv:2502.10835

    Back Attention: Understanding And Enhanc- ing Multi-Hop Reasoning In Large Language Models. arXiv preprint arXiv:2502.10835. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank Responses To Align Language Models with Human Feedback without Tears.Advances in Neural Infor- mation Processing Systems, 36:10935–10950...

    arXiv 2023

  3. [5]

    w/o judge reward

    as the LLM judge for both the faithful- ness reward Rfaith during training and the Faith- fulness metric during evaluation. The judge is prompted with the query, retrieved documents, and the model’s trace, and returns binary scores for each applicable audit dimension. We use tempera- ture 0.0 for deterministic judge outputs. Because Qwen3-Max serves dual ...

    1958

  4. [6]

    Ryan Hunter-Reay won

    as an independent judge with identical eval- uation prompts. Gemini-2.5-Flash is not involved in any stage of CRAFT training. Table 12 reports the faithfulness scores from both judges across all model configurations un- der CRAFT v1. Gemini-2.5-Flash assigns system- atically lower absolute scores than Qwen3-Max (mean |∆| = 1.92), which is expected given k...

  5. [2023]

    InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488

    Enabling Large Language Models To Generate Text with Citations. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 6465–6488. Google DeepMind. 2025. Gemini 2.5 Flash. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024....

    arXiv 2023

  6. [2025]

    In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 4405–4424, Suzhou, China

    GRADE: Generating Multi-Hop QA And Fine- GRAined Difficulty Matrix for RAG Evaluation. In Findings of the Association for Computational Lin- guistics: EMNLP 2025, pages 4405–4424, Suzhou, China. Association for Computational Linguistics. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike...

    arXiv 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.