{"id":"14381474-67fb-4f0f-bb20-bd054921cbe2","arxiv_id":"2412.14905","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DePaC combines context-aware negative training with information-calibrated aggregation to reduce fact fabrication and fact omission in parallel-context RAG systems.","lead":"This paper proposes DePaC, a method that reduces hallucinations in retrieval-augmented language models by training the model to refuse irrelevant contexts and by weighting context windows that add the most new information. It reports consistent gains across nine question-answering and retrieval tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 15's rejection-token filter is inverted as written, selecting the very windows DePaC is supposed to discard; the central mechanism is internally inconsistent.","rationale":"The reader's conditional verdict identified the binary-relevance assumption of NegTrain as the weakest point, but the most load-bearing issue is more basic: the equations that define DePaC's selection rule contradict the prose. Equation 11 and the simplified Eq. 15 both treat the rejection token td as a positive selection signal rather than a filter. This is an internal inconsistency, not a disagreement with external consensus, and it directly threatens the central claim that DePaC reduces fact fabrication by refusing irrelevant windows. The flaw is likely a sign/negation typo: replacing '= td' with '!= td' in the indicator, or changing the argmin penalty from '-' to '+', would make the formulas match the described behavior. Because the preprint does not release code, the reader cannot tell whether the experiments used the corrected rule. If the corrected rule is what produced the reported numbers, the paper needs a formula correction but its empirical conclusion may stand; if the as-printed rule was used, the results are unexplained. This additional condition does not change the overall verdict: the paper was already CONDITIONAL pending clarification and independent replication, and it should remain so with the added requirement that Eqs. 11 and 15-16 be fixed and validated against the reported results.","tokens_in":14725,"tokens_out":4014,"duration_ms":35379,"concrete_test":"Obtain or reimplement DePaC's selection rule from Eqs. 15-16 exactly as printed (with I(top token = td)) and compare against the intended rule I(top token != td) on FuncNR and EntLR. If the as-printed rule performs at chance or emits UNK while the corrected rule reproduces Table 3, the paper's formulas must be corrected and the empirical claim re-verified under the corrected mechanism; if the authors' released code uses the corrected rule, the equations should be amended.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, after defining NegTrain, the paper says td is used 'to explicitly judge the usefulness of each context window' and that selected windows 'should not be aligned to the rejection token.' The formulas say the opposite. Equation 11 sets pi = arg min_{pi,j} [H(pi,j) - β·DKL(pi,j||pi,c) - γ·I(argmax_k p_{i,jk}=td)] with γ>0; a window whose top token is td receives a smaller objective value and is therefore more likely to be selected. The simplified form in Appendix B, Eq. 15, is pi = arg max_{pi,j} [max_k p_{i,jk} + β·DKL(pi,j||pi,c)] · I(argmax_k p_{i,jk}=td). Since the indicator is 1 only for windows whose top token is td and 0 for all other windows, the argmax is forced onto a rejection-token window; Eq. 16's positive score is multiplied by zero for normal windows. This is not a subtle sign preference: if implemented literally, DePaC would aggregate 'UNK' contexts, and the reported accuracy improvements in Tables 1 and 3 could not arise from the described algorithm. The likely intended condition is I(argmax_k p_{i,jk} != td), or a +γ penalty in the argmin form. As printed, the central derivation contradicts its own design, independent of the reader's concern about binary relevance splits in NegTrain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DePaC, a method for retrieval-augmented generation with parallel context extension, combining two components: context-aware negative training (NegTrain) that fine-tunes a backbone LLM to emit a rejection token for irrelevant documents, and information-calibrated aggregation (ICA) that selects context windows based on a combination of output certainty and KL divergence from the non-document distribution. The method is evaluated on six information-seeking tasks and three document QA tasks, with claims that it reduces fact fabrication and fact omission hallucinations relative to vanilla inference, YaRN, AVP, and NBCE baselines. The paper also contributes a complexity analysis showing linear rather than quadratic scaling with the number of documents.","tokens_in":14975,"tokens_out":3972,"duration_ms":33666,"significance":"If the proposed mechanism were correctly specified and the empirical results held up under fair comparison, DePaC would be a useful contribution to RAG and parallel context extension, especially because it combines negative supervision with an information-theoretic selection criterion and demonstrates gains on nine tasks with two backbone models. The complexity analysis is a strength. However, the central aggregation equation contains an inverted rejection-token filter that contradicts the stated design, and the experimental protocol compares a fine-tuned model against non-fine-tuned baselines, so the significance of the reported improvements cannot currently be assessed.","major_comments":[{"comment":"The rejection-token filter in the aggregation objective is inverted. In Eq. (11), the minimization objective subtracts γ·I(argmax_k p_{i,jk}=td) with γ>0, so a window whose top token is the rejection token receives a smaller objective value and is therefore more likely to be selected. In the simplified form of Appendix B, Eq. (15) maximizes \\hat{C}(p_{i,j},p_{i,c})·I(argmax_k p_{i,jk}=td); the indicator is nonzero only for rejection-token windows, forcing the argmax onto exactly the windows the text says should be excluded. This directly contradicts the sentence in Section 3 that selected windows 'should not be aligned to the rejection token.' If implemented literally, DePaC would aggregate 'UNK' contexts, which cannot produce the accuracy gains reported in Tables 1 and 3. The condition should be I(argmax_k p_{i,jk} ≠ td) (or equivalently a +γ penalty in the argmin form), and the corresponding simplification must be corrected. As printed, the central mechanism is internally inconsistent.","section":"Section 3, Eq. (11) and Appendix B, Eq. (15)"},{"comment":"The experimental comparison is unfair because DePaC fine-tunes the backbone on 19K constructed examples, while all baselines (Vanilla, YaRN, AVP, NBCE) are used without any fine-tuning. The reported gains could therefore be attributable to the additional training rather than to NegTrain and ICA. To support the claim that DePaC's aggregation and negative training are responsible for the improvements, the paper needs a control baseline fine-tuned on a comparable positive-only or instruction-following dataset, or an evaluation where all methods use the same fine-tuned backbone. This applies to Tables 1, 2, 3, and 5.","section":"Section 3 (Implementation Details) and Section 5 (Tables 1-3)"},{"comment":"The hallucination metric is defined post hoc and is not a principled measure. Incorrect outputs containing any of 27 manually selected omission phrases are counted as fact omission, and all other errors are counted as fact fabrication. This rule can over-attribute errors to omission (e.g., a fabricated answer that happens to include 'not mentioned') and under-attribute errors that express omission without a trigger phrase. There is no inter-annotator agreement or human validation. Consequently, the strong claims in Figure 5, such as 'DePaC completely avoids fact omission on EntLR and fact fabrication on MVIH,' rest on an unvalidated automatic classifier. Please provide a human-annotated sample or a more rigorous evaluation protocol.","section":"Appendix G (Hallucination Definition and Evaluation Setup)"},{"comment":"NegTrain assumes a binary relevance split in the training data: a document is either fully relevant (paired with the gold answer) or fully irrelevant (paired with the rejection token). Real RAG retrievers often return partially relevant documents or hard negatives that contain a mixture of useful and misleading information. The experiments in Tables 4 and 5 include multi-hop QA, where the required information is spread across documents, but the training data construction described in the Implementation Details samples only 'unrelated fragment' as distractors. The paper does not test whether the rejection token reliably transfers to partially overlapping or hard-negative documents, which is central to the claim that DePaC works in RAG scenarios.","section":"Section 3 (Context-aware Negative Training) and Figure 2"}],"minor_comments":[{"comment":"There are numerous typos and inconsistent notations, e.g., 'independ' in the Introduction, 'conducte' in Section 5.2, 'implicate' in Appendix B, 'A VP' instead of 'AVP' in Tables 1 and 2, and 'MVH' instead of 'MVIH' in Figure 6. A careful proofreading is needed.","section":"Throughout"},{"comment":"All results are reported as single point estimates without error bars, multiple seeds, or significance tests. For a paper that makes claims of 'significant alleviation' and 'consistent' improvements, the absence of variance estimates is a substantial presentation gap.","section":"Tables 1-5"},{"comment":"The complexity comparison assumes all documents have the same length n, but in practice documents vary; the O(k·n²·d·m) bound should be stated with this assumption made explicit, and the 'linear in k' claim is only linear for fixed n.","section":"Section 4 (Complexity Analysis)"},{"comment":"The training-loss curves in Figure 8 are described as showing that NegTrain increases the probability difference between refusing unrelated and related documents, but the figure only displays loss values and no quantitative comparison is given; please add a numerical metric or a clearer analysis.","section":"Appendix D (Figure 8)"}],"recommendation":"major_revision","confidential_remarks":"The sign inversion in Eq. (15) is serious because it is the exact mechanism the paper claims to implement. It suggests the reported experiments were likely run with a corrected condition (or a different code path) that is not documented. I would ask the authors to provide the exact implementation code or a precise pseudocode listing and to re-run the main experiments after fixing the equations. The unfair fine-tuning comparison is the second major obstacle; if not addressed, the empirical contribution cannot be evaluated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DePaC is a sensible combination of negative training and an information-aware selection rule for parallel-context RAG, and the experiments are broad and consistently positive. But there's a load-bearing sign error in the main equations that inverts the rejection-token filter, so the method as printed cannot produce the reported results. The paper needs a fix and a stronger experimental setup before I'd trust it.\n\nWhat's new: The two mechanisms are known individually, but packaging them for PCE in RAG, plus the linear-vs-quadratic complexity argument, is a useful contribution. The ablation study (w/o NegTrain, w/o ICA) isolates both components, and the gains are large on the info-seeking tasks. The DocQA results with two backbones are also solid. Credit where due: the paper ships a clear description of the training data construction and the complexity analysis is correct.\n\nSoft spots: The central issue is in Section 3 and Appendix B. Equation 11 is an argmin with a term -γ·I(argmax=td); since γ>0, a window whose top token is td gets a smaller objective and is thus preferred. The simplified form, Eq. 15, multiplies by I(argmax=td), so the argmax is forced onto a rejection-token window. The text says the selected window 'should not be aligned to the rejection token,' so the equations are the opposite of the design. This looks like a typo (probably I(argmax≠td) or a +γ penalty), but it's not optional: as written, the method would aggregate UNK contexts and the table results could not follow.\n\nBeyond that: DePaC is fine-tuned while baselines are not, so some gains may be fine-tuning effects; the PosTrain ablation only partially controls for this. No error bars or significance tests, and no code/data release. The 'first to apply PCE to RAG' claim is shaky given An et al. (2024) from the same group.\n\nBottom line: The idea is worth exploring, but the paper as written is not reproducible. A referee could catch the sign issue and ask for a fix, plus better controls. I'd send it out, but with a clear note about the equations.","headline":"Sign error in the main equations inverts DePaC's rejection-token filter; the empirical story is strong but the method as written cannot work.","tokens_in":15546,"tokens_out":3323,"would_cite":false,"duration_ms":27404,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DePaC cuts fact fabrication and fact omission in RAG with refusal training and information-calibrated aggregation.","keywords":["retrieval-augmented generation","hallucination","parallel context extension","negative training","information-calibrated aggregation","fact fabrication","fact omission","large language models"],"falsifier":"Build a test set in which every retrieved document is topically related to the question but none supports an answer, and compare how often DePaC produces a confident fabricated answer versus emitting its refusal token; if the refusal rate is no higher than vanilla's, the negative-training component has not generalized to partially relevant distractors and the paper's hallucination claim would not stand for realistic retrieval noise.","tokens_in":14495,"feed_emoji":"🧠","tokens_out":11479,"duration_ms":90535,"temperature":0.7,"pith_summary":"Retrieval-augmented generation still lets large language models hallucinate, and parallel context extension (PCE) makes this worse in two specific ways: a model can fabricate a fact that no window supports, or an uninformative window can dominate the aggregation and cause the model to omit a fact another window supports. DePaC targets both. It fine-tunes the backbone with a context-aware negative training loss: when a document is unrelated to the question, the model is trained to output a rejection token (UNK) instead of an answer; when the document is relevant, it is trained on the ground-truth answer. It then aggregates windows with an information-calibrated rule that prefers a window with both high output certainty and high KL divergence from the no-document distribution, making it robust to both fabrication and omission. On six information-seeking tasks and three document-question-answering tasks, the paper reports that DePaC outperforms vanilla inference, the YaRN context-extension method, average-aggregation PCE, and NBCE, and that its inference cost grows linearly with the number of documents rather than quadratically. The paper claims both components are necessary, since ablations removing either one lower accuracy.","feed_headline":"DePaC curbs RAG hallucinations while keeping inference time linear","feed_subtitle":"Refusal training plus information-aware window selection cuts fabricated and omitted answers in grounded QA.","key_machinery":"Two mechanisms carry DePaC. First, context-aware negative training (NegTrain) builds a dataset in which some question-document pairs are positive (the document is the oracle that supports the answer) and others are negative (the document is an unrelated distractor), and the fine-tuning loss is cross-entropy against the gold answer for positives and against a rejection token $t_d=$UNK for negatives; this gives the model a built-in abstain signal. Second, information-calibrated aggregation (ICA) scores every parallel window at each generation step by combining the model's certainty in its top token with the KL divergence from the context-free distribution $p_{i,c}$, and the selected window is the one that is both confident and informative while windows that have learned to emit the rejection token are skipped. The complexity analysis rests on the observation that DePaC processes all documents in parallel with effective sequence length $n$ per window, so attention cost is $O(k n^2 d m)$ rather than vanilla's $O(k^2 n^2 d m)$.","core_discovery":"DePaC's central claim is that the two common failure modes of PCE for RAG are separable and fixable. Fact fabrication is treated as a training problem: DePaC fine-tunes the model so that an irrelevant context forces the first output token to be a rejection token $t_d$ (set to UNK), while relevant contexts continue with the gold answer; this teaches the model to abstain rather than invent. Fact omission is treated as an aggregation problem: rather than trusting the window with lowest entropy or the majority vote, DePaC scores each window by the certainty of its most likely token plus $\\beta$ times the KL divergence between the window's distribution and the context-free distribution $\\Delta(p_{i,j}, p_{i,c}) = D_{\\mathrm{KL}}(p_{i,j}\\,\\|\\,p_{i,c})$, and it discards windows whose top token is the rejection token. The paper argues this selection rule lets the model switch windows token by token, so it can assemble answers from multiple documents, and the reported results support that DePaC reduces the measured hallucination percentages on FuncNR, EntLR, and MVIH, in one case to zero, while improving F1 and accuracy across the nine tasks.","pith_inferences":["The paper's binary relevant/irrelevant training suggests an untested boundary: real retrievers produce partially relevant documents, and one straightforward extension would be to train a graded refusal output, so the rejection token's strength correlates with answerability rather than being a hard switch.","Because ICA only uses the top-token probability plus a scalar KL divergence, it may underweight rare but correct tokens; a natural follow-up would be to add a query-specific beam over the union of top-$k$ tokens across windows, which could further cut fact omission.","The rejection-token idea could transfer to general abstention in QA: the same UNK-based fine-tuning could give LLMs an explicit 'I do not know' behavior when they are prompted with context that is relevant-looking but uninformative.","DePaC's evaluation measures hallucination on exact-match tasks with clear answers; extending the same two-type taxonomy to open-ended generation would require a stricter entailment-based detector, which the paper leaves for future work."],"forward_implications":["RAG pipelines can run on models with modest context limits: instead of truncating a concatenated mega-document, the system can score many candidate documents independently and carry the best window forward at every generated token.","Fine-tuning with a rejection token gives a calibrated 'not answerable from context' behavior, so downstream applications can route unanswered queries to a retriever refresh or a human rather than trusting a fabricated response.","Because ICA explicitly favors windows whose distribution shifts most relative to the context-free prior, it is naturally suited to multi-hop questions that need evidence from different documents assembled token by token.","The linear-in-documents inference cost makes PCE-style RAG practical for large candidate sets, where vanilla concatenation becomes quadratic.","Ablations imply that negative training is not a regularization aside: removing it, or reverting aggregation to lowest-uncertainty, measurably reduces accuracy, so both components should be present in a deployment."],"supporting_citations":[{"why":"Supplies the NBCE lowest-uncertainty aggregation baseline and the output-distribution formalization that DePaC modifies with KL-based scoring and rejection-token filtering.","marker":"Su et al., 2024"},{"why":"One of the average-aggregation PCE baselines that DePaC is directly compared against in the DocQA and information-seeking experiments.","marker":"Hao et al., 2022"},{"why":"The other average-aggregation PCE baseline, providing the parallel-context-window method that DePaC must beat.","marker":"Ratner et al., 2023"},{"why":"Provides the C4-fragment negative/positive construction recipe for NegTrain and the FuncNR and EntLR information-seeking tasks that DePaC evaluates on.","marker":"An et al., 2024"},{"why":"Defines the KL divergence used in information-calibrated aggregation to measure a window's information increment over the context-free distribution.","marker":"Kullback and Leibler, 1951"},{"why":"Supplies the C4 corpus from which NegTrain's oracle and distractor fragments are sampled.","marker":"Raffel et al., 2020"},{"why":"The Mistral-7B backbone model that is fine-tuned by NegTrain and serves as the default evaluation model.","marker":"Jiang et al., 2023a"},{"why":"Contributes the Qasper long-document QA dataset used in the DocQA evaluation.","marker":"Dasigi et al., 2021"},{"why":"Supplies the MultifieldQA DocQA task and the long-context evaluation setup for document understanding.","marker":"Bai et al., 2023"},{"why":"Provides the NarrativeQA long-book QA dataset used to test DePaC on narrative understanding.","marker":"Kočiský et al., 2018"}],"fun_headline_variants":["DePaC slashes RAG hallucinations, keeps linear speed","Context-aware training stops RAG fabrication and omission","DePaC: refuse if irrelevant, pick info-rich windows","RAG hallucination fix: DePaC with negative training","DePaC curbs RAG lies and omissions in one shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on a clean binary split between fully relevant and fully irrelevant documents; real retrievers often return partially relevant or hard-negative documents, and the learned refuse-to-answer signal was never trained to handle that middle ground.","fun_headline_variants_meta":{"raw":{"variants":["DePaC slashes RAG hallucinations, keeps linear speed","Context-aware training stops RAG fabrication and omission","DePaC: refuse if irrelevant, pick info-rich windows","RAG hallucination fix: DePaC with negative training","DePaC curbs RAG lies and omissions in one shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000445,"raw_usage":{"total_tokens":2287,"prompt_tokens":1021,"completion_tokens":1266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1182}},"tokens_in":637,"tokens_out":1266,"duration_ms":8256,"temperature":1.0,"reasoning_tokens":1182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:48:17.361135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a test set in which every retrieved document is topically related to the question but none supports an answer, and compare how often DePaC produces a confident fabricated answer versus emitting its refusal token; if the refusal rate is no higher than vanilla's, the negative-training component has not generalized to partially relevant distractors and the paper's hallucination claim would not stand for realistic retrieval noise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The other average-aggregation PCE baseline, providing the parallel-context-window method that DePaC must beat."}],"review_version":1}