{"id":"7d9d4390-43f2-4d35-9d85-3588819560e5","arxiv_id":"2412.14510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A staged DPO-based training method improves RAG generators' answer completeness, robustness to noise, and citation quality on four QA datasets.","lead":"The paper proposes PA-RAG, a training method that aligns retrieval-augmented generation (RAG) models with three goals: giving complete answers, resisting noisy documents, and citing sources properly. It uses supervised fine-tuning followed by staged preference optimization, and reports large gains on question answering benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Citation-quality gains may be inflated because TRUE NLI is both the training-label oracle (§3.1, §3.2.3) and the evaluation metric (§4.1.2); an independent NLI/human check is required.","rationale":"The central claim is that PA-RAG significantly improves RAG generators, with large gains in EM and citation quality. The EM component is credible because it uses external gold answers. The citation component, however, is only as valid as the TRUE NLI scorer. The reader flagged this as the weakest assumption, and I agree it is the most load-bearing: if TRUE is gameable, the reported 49.77%/39.58% improvements are not evidence of better citations, and the paper's distinct contribution over SFT baselines remains only an EM improvement. The paper's appeal to ALCE's human correlation is insufficient because correlation for base models does not guarantee robustness under optimization against the same model; this is a standard reward-hacking failure mode. The proposed test breaks the circularity by using an independent evaluator and human annotation. If the improvements persist, the concern is resolved and the CONDITIONAL verdict can be upgraded. If not, the citation claims should be retracted and the paper's contribution narrowed to EM/correctness. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":23328,"tokens_out":5333,"duration_ms":47169,"concrete_test":"Score the released generations from the base generators and PA-RAG models (code/data at the provided repo) with an independent NLI evaluator (e.g., a DeBERTa-v3-large MNLI model or a GPT-4o judge) and with human annotators on 200 randomly sampled outputs per dataset. Recompute citation recall/precision using these labels. If the PA-RAG over base delta is not consistently large (e.g., less than half the reported margin) under the independent evaluator or human annotators, the citation improvements are largely artifacts of TRUE-circularity. Also report variance across 3 random seeds to ensure the point estimates are stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PA-RAG's headline citation improvements (avg +49.77% recall, +39.58% precision in Table 1) are computed with the TRUE NLI model (T5-11B), as stated in §4.1.2. The same model serves as the oracle in the citation rewrite mechanism (§3.1): it verifies citations, traverses the powerset of golden docs to find a supporting subset, and removes 'irrelevant' citations; it also constructs the chosen/rejected pairs for the citation-quality DPO stage (§3.2.3). Thus the training pipeline optimizes directly against TRUE, and the evaluation then scores with TRUE. This closed loop can reward artifacts (e.g., lexical overlap, sentence-level entailment heuristics) rather than genuine grounding. ALCE's human-correlation result applies to off-the-shelf generators, not to models explicitly optimized against TRUE; reward hacking can inflate the metric without improving human-judged support. The EM gains (+13.97%) rest on exact-match against gold answers and are not affected by this concern, but the citation-specific claim, which is a major part of the paper's contribution, is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PA-RAG, a multi-stage fine-tuning framework for retrieval-augmented generation (RAG) generators. After an instruction fine-tuning stage that uses ChatGPT-3.5 and a citation-rewrite mechanism to build high-quality responses, the generator is optimized with three sequential DPO stages targeting response informativeness, response robustness, and citation quality. The method is evaluated on ASQA, WebQuestions, Natural Questions, and TriviaQA using LLAMA 2-7B, LLAMA 2-13B, and LLAMA 3-8B as base generators. The authors report average absolute improvements of 13.97% in EM, 49.77% in citation recall, and 39.58% in citation precision, and they release their training data and code.","tokens_in":23550,"tokens_out":3845,"duration_ms":33322,"significance":"If the reported gains hold up, PA-RAG is a valuable contribution to RAG alignment: it shows that multi-perspective DPO training can improve an end-to-end generator along several dimensions, and the released 58.9k instruction instances and 48.7k preference instances are useful resources. The ablation study (Table 2) and the preference-order experiments (Table 3) provide informative evidence for the staged design. The EM improvements are substantial and are not affected by the evaluation circularity noted below. However, the citation-quality results, which form a major part of the contribution, are currently unverified because the same NLI model is used both as the label oracle for training and as the evaluation metric. The lack of variance estimates and significance tests also weakens the quantitative claims. With independent citation evaluation and more rigorous experimental statistics, this would be a solid paper.","major_comments":[{"comment":"The TRUE T5-11B NLI model is used both to construct the citation labels (citation verification, powerset citation construction, simplification, and the chosen/rejected pairs for the citation-quality DPO stage) and to compute the citation recall and precision numbers in Table 1. The reported citation gains are therefore at least partly self-referential: the generator is explicitly optimized to satisfy TRUE, and the same model is then used as the scoring oracle. The ALCE human-correlation result cited in §4.1.2 applies to off-the-shelf generators, not to a model that has been trained to game TRUE, so reward hacking remains a real risk. Please add an independent citation-quality evaluation, for example with a different NLI model or with human annotation on a sample, and report the agreement.","section":"§3.1, §3.2.3, §4.1.2"},{"comment":"All headline numbers (13.97% EM, 49.77% citation recall, 39.58% citation precision) are single-run results with no error bars, no multiple seeds, and no significance tests. DPO training is known to be sensitive to random seed and hyperparameters, so the improvements may not be robust. Please report at least three seeds with mean and standard deviation, and add paired significance tests (e.g., bootstrap or approximate randomization) for the main comparisons, especially PA-RAG versus the SFT baseline.","section":"§4.2, Table 1"},{"comment":"The 'SFT on chosen' baseline is only run on LLAMA2-7B-CHAT and only for the response-informativeness and response-robustness stages. There is no SFT baseline for the citation-quality stage, nor for the other two backbones. Since one of the paper's central claims is that preference optimization outperforms SFT, this claim currently rests on a single backbone and a single data configuration. Please extend the SFT baseline to all backbones and to the citation-quality stage, or substantially qualify the claim.","section":"§4.1.3, Table 4"},{"comment":"The comparison with Self-RAG and RetRobust uses different backbone sizes (13B) and different training procedures, and the claim that PA-RAG 'significantly outperforms the baselines' is not backed by significance testing. Please restrict the claim to same-backbone comparisons or provide significance tests across the full table; otherwise the superiority over pipeline baselines is not established.","section":"§4.1.3, Table 1"}],"minor_comments":[{"comment":"The phrase 'lea to a significant performance decline' should be corrected to 'leads to a significant performance decline.'","section":"§4.5"},{"comment":"Some values in Table 2 are run together (e.g., '77.6868.32' in the LLAMA2-7B ASQA row), making the table hard to read; please add explicit separators between columns.","section":"Table 2"},{"comment":"The figure labels contain typos: 'Constract' should be 'Construct' and 'Constuct' should be 'Construct.'","section":"Figure 3"},{"comment":"The ChatGPT prompt has a typo: 'Qustion' should be 'Question.'","section":"Appendix D"},{"comment":"The paper says 'All generators were trained on the same dataset,' but the preference data for later stages is generated by LLAMA2-7B-CHAT at different stages of fine-tuning; please clarify whether the same generated preference data is reused for the 13B and 8B backbones, since this affects the interpretation of the cross-backbone generalization claim.","section":"§4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the released data/code is a genuine asset. The main risk is the self-referential citation evaluation; I would encourage the editor to require an independent NLI or human evaluation before acceptance. The lack of statistical rigor is also a concern for a paper that makes 'significant' claims. The EM results are plausible and provide a useful signal that the method works, but the citation-specific claims need stronger evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real, reproducible method for aligning RAG generators, and the EM gains are believable. But the citation-quality headline is inflated by a closed loop: TRUE NLI creates the training labels in §3.1 and §3.2.3 and then scores the outputs in §4.1.2. You cannot report +49.77% citation recall against the same oracle you optimized for. The method may still help, but the citation claim is unverified.\n\nWhat's genuinely new: staged multi-perspective preference optimization for RAG, with separate DPO stages for informativeness, robustness, and citation quality. The data construction is careful, including the citation rewrite mechanism and the noisy-document sampling. The released data and code are a plus. The ablation in Table 2 shows each stage adds something, and the order study in Table 3 supports the curriculum intuition. The comparison of DPO vs SFT in Table 4 is also useful, showing catastrophic forgetting under sequential SFT.\n\nSoft spots: the circularity is real. TRUE is both oracle and evaluator. The fact that ALCE found TRUE correlates with human judgment does not transfer to a model explicitly optimized against TRUE; reward hacking could inflate citation recall/precision without improving actual support. An independent NLI model or human evaluation is needed before I'd believe the citation numbers. That's the main issue. Smaller issues: SFT-on-chosen only runs on Llama-2-7B, no error bars or significance tests, and the DPO beta hyperparameter is not reported. These are fixable with a revision. The EM gains (+13.97% absolute on average) are not affected by the circularity and are plausible given the staged training. Also, the claim that PA-RAG \"far surpasses\" SFT is only tested on one backbone; the comparison is incomplete.\n\nOverall, this deserves serious peer review. The method is well-engineered, the writing is clear, and the released artifacts let others reproduce. I would insist on an independent evaluation of citation quality before accepting the headline claim.","headline":"Staged DPO for RAG is a genuinely useful recipe, but the citation-quality gains are unverified because the same NLI model both labels and grades.","tokens_in":24127,"tokens_out":1663,"would_cite":true,"duration_ms":15285,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a RAG generator with three staged preference objectives—informativeness, noise robustness, and citation quality—can lift answer correctness and citation quality together while keeping the generator end-to-end.","keywords":["retrieval-augmented generation","preference optimization","direct preference optimization","citation quality","response robustness","response informativeness","RAG alignment","LLM fine-tuning"],"falsifier":"Take a random sample of PA-RAG and baseline outputs, have human annotators mark each claim as supported, unsupported, or irrelevant-cited, and compute citation recall and precision from those human labels; if PA-RAG's margin over SFT-on-chosen shrinks to near zero, the central citation-quality claim fails. A second check is to rerun the pipeline with a different NLI model or with human-supplied citation labels and see whether the 49.77% average citation-recall gain survives.","tokens_in":1493,"feed_emoji":"🎯","tokens_out":1604,"duration_ms":56320,"temperature":0.7,"pith_summary":"The paper is trying to establish that the weaknesses of retrieval-augmented generation—incomplete answers when context is noisy, and poor citation support—can be fixed by aligning the generator itself, not by adding retrieval or verification steps around it. It proposes PA-RAG, a training scheme that first teaches the model via instruction fine-tuning to use and cite documents, then runs three Direct Preference Optimization stages that reward making answers complete, ignoring irrelevant documents, and citing accurately. Across four question-answering datasets and three LLMs, the trained generators improve exact match by about 14 absolute points on average, with large gains in citation recall and precision. If the claim holds, RAG systems can get substantially more reliable answers and citations while keeping the generator a single end-to-end language model.","feed_headline":"Preference training lifts RAG answer accuracy by 14 points","feed_subtitle":"Three-stage DPO also nearly doubles citation recall while keeping the generator end-to-end.","key_machinery":"The load-bearing mechanism is the pairing of a citation rewrite mechanism with staged Direct Preference Optimization (DPO). The rewrite mechanism uses an NLI model to verify whether each cited document entails the claim, searches over the powerset of prompt documents to find a supporting citation, and removes irrelevant citations; it produces both the chosen outputs for SFT and the chosen/rejected pairs for the citation-quality DPO stage. The staged DPO schedule first optimizes response informativeness (use golden documents), then response robustness (ignore noisy documents), then citation quality, an ordering the paper shows beats the reversed or mixed order.","core_discovery":"The paper claims that a general-purpose LLM used as a RAG generator can be aligned with the three central RAG requirements—informativeness, robustness, and citation quality—in a single end-to-end model, without adding reranking or verification components to the pipeline. The recipe is supervised fine-tuning on rewritten high-quality responses, followed by three sequential DPO stages, each targeting one requirement. On four QA datasets and three LLMs, this produces average absolute gains of 13.97% in exact match, 49.77% in citation recall, and 39.58% in citation precision over the base generators, and it outperforms both SFT-only training and pipeline baselines.","pith_inferences":["Editorial extension: because the same NLI oracle is used to build citation labels and to compute citation metrics, the reported citation gains should be read as upper bounds until confirmed by human or independent automatic evaluation.","Editorial extension: the preference-data construction relies on datasets that provide short golden answers; applying PA-RAG to open-ended generation without such answer supervision is untested.","Editorial extension: a natural next step is to replace the NLI oracle with a reward model trained on human citation judgments, or to reuse the staged DPO recipe for other multi-objective text-generation tasks.","Editorial extension: training uses a single retriever and document corpus, so the robustness improvements may not transfer unchanged to other retrievers or document collections."],"forward_implications":["RAG generators can be aligned to several partly conflicting requirements in one end-to-end model, so deployment needs no extra reranking or verification passes.","Direct Preference Optimization is a better training signal than SFT for these trade-offs: the paper shows SFT on the same chosen outputs degrades when asked to optimize robustness after informativeness, while DPO keeps improving.","Staged optimization order matters: informativeness before robustness beats the reverse or a single mixed objective, suggesting a curriculum over RAG skills.","The released 58.9k instruction-tuning instances and 48.7k preference-optimization instances are reusable for training other backbone LLMs.","Citation recall and precision can be improved simultaneously in practice, with the citation-quality stage mainly raising the balanced F1 score."],"supporting_citations":[{"why":"Supplies the Direct Preference Optimization algorithm used in all three preference-alignment stages.","marker":"Rafailov et al., 2023"},{"why":"Defines the ALCE evaluation metrics and the citation recall/precision protocol that the paper follows.","marker":"Gao et al., 2023a"},{"why":"Provides the TRUE NLI model that verifies citation support in both data construction and evaluation.","marker":"Honovich et al., 2022"},{"why":"Provides the Self-RAG pipeline baseline that PA-RAG is compared against.","marker":"Asai et al., 2024"},{"why":"Provides the RetRobust SFT-based robustness baseline that PA-RAG is compared against.","marker":"Yoran et al., 2024b"},{"why":"Provides the GTR dense retriever used to retrieve top documents for building training and evaluation inputs.","marker":"Ni et al., 2022"}],"fun_headline_variants":["PA-RAG: Multi-perspective preference optimization aligns RAG","Three-stage DPO aligns RAG generators to key requirements","Preference optimization lifts RAG EM by 14 points, doubles citation recall","End-to-end RAG alignment via multi-perspective preference optimization","Single-model RAG alignment: SFT and three DPO stages"],"cache_read_input_tokens":26240,"weakest_assumption_plain":"The load-bearing premise is that the NLI judge used to check citations agrees with human judgment; that same judge both creates the training labels and measures the reported citation scores.","fun_headline_variants_meta":{"raw":{"variants":["PA-RAG: Multi-perspective preference optimization aligns RAG","Three-stage DPO aligns RAG generators to key requirements","Preference optimization lifts RAG EM by 14 points, doubles citation recall","End-to-end RAG alignment via multi-perspective preference optimization","Single-model RAG alignment: SFT and three DPO stages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2496,"prompt_tokens":919,"completion_tokens":1577,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1486}},"tokens_in":535,"tokens_out":1577,"duration_ms":10688,"temperature":1.0,"reasoning_tokens":1486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:09:52.347547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of PA-RAG and baseline outputs, have human annotators mark each claim as supported, unsupported, or irrelevant-cited, and compute citation recall and precision from those human labels; if PA-RAG's margin over SFT-on-chosen shrinks to near zero, the central citation-quality claim fails. A second check is to rerun the pipeline with a different NLI model or with human-supplied citation labels and see whether the 49.77% average citation-recall gain survives.","supporting_citations":[],"review_version":1}