{"id":"c0acdfea-730e-48b9-850b-07d5535a3149","arxiv_id":"2504.15629","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Post-processing citation correction using keyword, semantic, BERTScore, fine-tuned, and LLM-based matching improves RAG citation accuracy by up to 15.46% relative in the authors' evaluation.","lead":"This paper describes CiteFix, a set of post-processing methods that repair incorrect citations in answers from retrieval-augmented generation (RAG) systems. The authors report up to 15.46% relative accuracy gains across several language models, and argue this could let products switch to cheaper models without losing quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 15.46% improvement rests on 50 questions and post-hoc method selection; no significance tests or absolute baseline values are reported.","rationale":"The reader's weakest assumption identifies a real and important limitation: CiteFix can only re-rank citations among documents already retrieved by the RAG retriever, so it cannot fix hallucinations or facts grounded in documents the retriever missed. This bounds the method's ceiling and is worth stating explicitly as a limitation. However, the more load-bearing concern for the paper's central claim is the statistical reliability of the measured improvement. The headline number is derived from 50 questions with a binary metric, no error bars, and post-hoc selection of the best method per model; the reader flagged similar issues in the rationale but did not elevate them to the weakest assumption. Both concerns justify a conditional verdict, so the reader's CONDITIONAL is appropriate. The paper's contributions are real: a clear problem, several plausible and cheap post-processing methods, latency measurements, and honest discussion of limitations. The route to acceptance is empirical strengthening, not a rewrite of the method. I therefore recommend no change to the reader's verdict.","tokens_in":9070,"tokens_out":4709,"duration_ms":44616,"concrete_test":"Run a larger, pre-registered evaluation on a new set of at least 300 questions, fixing the correction method per model before observing results (e.g., for Model C, Keyword+Semantic Context as the primary comparison). Compute the per-question binary MQLA difference between baseline and CiteFix, then report the 95% bootstrap confidence interval for the mean difference and a McNemar or sign test on the paired binary outcomes. Also report inter-annotator agreement (Cohen's kappa) on the MQLA components. If the lower confidence bound includes zero or the p-value exceeds 0.05, the headline improvement is not statistically reliable. In addition, report absolute baseline and post-correction MQLA so the magnitude of the effect can be interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, repeated in the Abstract and Section 5, is that CiteFix yields a relative improvement of up to 15.46% in MQLA. This number comes from Table 2, which is based on only 50 representative questions (Section 4.2). MQLA, defined in Algorithm 1, is a strict binary per-question pass/fail: a question scores 1 only if all five component metrics are >= 0.8 and there is at most one hallucinated fact. With n=50, the 95% Wilson interval for an accuracy near 50% spans roughly +/-14 percentage points; a 15.5% relative improvement (perhaps 7-8 percentage points absolute) can be produced by flipping only a few of the 50 questions and is well within sampling noise. The paper reports no confidence intervals, no paired significance test, and no inter-annotator agreement for the human MQLA judgments. Furthermore, Table 3 evaluates two methods on each of four LLMs and appears to select the better method per model after seeing the results; this post-hoc selection over eight method-model pairs inflates the reported 'up to 15.46%' figure. The paper also never states the absolute baseline MQLA, only relative changes, making it impossible to judge the practical importance of the improvement. Because the cost-saving model-swap recommendation depends on this quantitative demonstration, the evidence is not yet sufficient to establish the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses citation errors in RAG systems. The authors propose post-processing methods that, after the LLM generates an answer, segment the response into factual points and re-rank the already-retrieved documents by similarity to each factual point, selecting the top C_i as corrected citations. Several scoring functions are proposed: lexical overlap, lexical+query-semantic hybrid, BERTScore with Longformer, a ColBERT-style fine-tuned BERTScore trained on LLM-generated positive/negative pairs, and a lightweight LLM-based citation selection; an attention-map approach is sketched as future work. The evaluation is a human audit of 50 questions on an unnamed RAG system, using a composite MQLA metric defined in Algorithm 1. The paper reports relative MQLA improvements of up to 15.46% and suggests this enables replacing a large LLM with a 12x cheaper and 3x faster model at comparable accuracy.","tokens_in":9340,"tokens_out":7151,"duration_ms":61209,"significance":"If validated, the contribution is practically useful: post-hoc citation fixing is cheaper than retraining or retrieval augmentation changes, and the paper is one of the few to address citation correction rather than detection. The latency figures (e.g., 0.015 s per factual point for keyword+semantic) are valuable. However, the evidence is currently too weak to support the headline quantitative claim: n=50, a binary metric, no confidence intervals, no significance tests, post-hoc method selection, and an unexplained 15.46% figure. The methods are plausible, and the manuscript is transparent about being early-stage, but the evaluation must be strengthened before the central claim can be accepted.","major_comments":[{"comment":"The evaluation is based on 50 questions with no confidence intervals, significance tests, or absolute baseline MQLA values. Since Algorithm 1 defines MQLA as a binary pass/fail per question, the reported relative improvements (e.g., +15.5% for Keyword+Semantic on Model C) may correspond to flipping only a few questions. Please report absolute MQLA for the baseline and each method, compute exact 95% confidence intervals (e.g., Wilson intervals) for the differences, and run a paired test such as McNemar's test on the 50 question-level outcomes. This is necessary to support the central quantitative claim in the Abstract and Section 5.","section":"Section 4.2, Table 2"},{"comment":"The headline claim of a relative improvement of 15.46% does not match the numbers in Table 2, which reports +15.5% for Keyword+Semantic Context and +15.8% for Fine-tuned BERTScore on Model C, while Table 1 reports +15.4% in parentheses. Please state precisely which configuration yields the 15.46% figure, whether it is a single model result or an average, and if an average, specify the averaging procedure. The inconsistency undermines the paper's central quantitative claim.","section":"Abstract and Section 5"},{"comment":"For each LLM the better of the two methods appears to be selected after seeing the results (e.g., KSC for Model A, FBS for Model C), and the 'up to 15.46%' claim appears to use the best possible result. This post-hoc selection over eight method-model pairs inflates the reported improvement. Report all method-model combinations in a single table, and either pre-register the selection rule or use a holdout set for method selection before evaluating on the test questions.","section":"Section 4.3, Table 3"},{"comment":"The correction methods only re-rank the retrieved documents {x̂_i}, so they cannot fix a factual point whose true source is absent from the retrieved set. The paper's 80% 'incorrectly cited' figure is based on a definition that counts a fact as incorrectly cited only if the fact is present in some retrieved document, meaning retrieval recall is assumed for those facts. Please report the retrieval set size and the recall of true sources for incorrectly cited facts, and discuss how the method's ceiling depends on retriever quality. This is central to the claim that 80% of unverifiable facts are fixable by re-ranking.","section":"Sections 3 and 4.1"},{"comment":"The evaluation uses two human auditors over 2.5 days per row, but no inter-annotator agreement is reported (e.g., Cohen's kappa) for the MQLA components. Since MQLA involves subjective judgments (relevancy of keywords, completeness, correctness), the absence of reliability data weakens confidence in the measured improvements. Please report agreement statistics and, if disagreements occurred, describe how they were resolved.","section":"Section 4.2, Algorithm 1"}],"minor_comments":[{"comment":"The notation is confusing: x_i is used for factual points in Eq. (1), but \\hat{x}_i is used for retrieved documents earlier in the same section. Consider using distinct symbols (e.g., s_i for factual points and d_j for retrieved documents) to avoid ambiguity.","section":"Section 3, Eq. (1)"},{"comment":"State clearly that the positive/negative pairs for fine-tuning are LLM-generated and describe the steps taken to prevent overlap between the training pairs and the 50 evaluation questions. Otherwise, the improvement from the fine-tuned BERTScore could partly reflect the evaluator being tuned to the test distribution.","section":"Section 3.4"},{"comment":"Table 1 is very hard to parse: columns are combined (e.g., 'Model Cents per 1K O/P tokens' with 'Relative Mean Question Level Accuracy'), and the parenthetical notation is not immediately clear. Consider splitting the table into separate panels for cost, MQLA, and error decomposition, and add a clearer legend for the numbers inside/outside parentheses.","section":"Table 1"},{"comment":"The paper says it proposes 'six computationally light weight methods,' but only four are evaluated (Keyword, Keyword+Semantic, BERTScore, Fine-tuned BERTScore), with LLM-based matching tested in a limited way and attention maps only as a proof of concept. Please align the method count with what is actually evaluated or explicitly mark the unimplemented methods as future work.","section":"Abstract and Introduction"},{"comment":"The cost-saving model-swap recommendation is based on comparing Model C+FBS against Model A baseline in Table 2, but no statistical test is reported for this head-to-head comparison. Please report the absolute MQLA values for both conditions and a significance test to substantiate the claim that CiteFix enables a 12x cheaper model with comparable performance.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a product engineering report from an industry lab. The core idea is sensible and the latency measurements are useful, but the evaluation is far too thin for the strength of the claims made in the Abstract and Section 5. The authors should either add statistical rigor (absolute values, confidence intervals, paired tests) or substantially soften the claims and present the work as a case study. The inconsistency in the headline number (15.46%) should be corrected before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CiteFix is a useful industrial case study, not yet a demonstrated result. The paper's real contribution is framing citation accuracy as a post-processing reranking problem and showing that simple similarity-based correctors can shift which citations a RAG answer points to. That framing is new; prior work trains or prompts the generator. The methods themselves are off-the-shelf (keyword overlap, BERTScore, fine-tuned late interaction, LLM prompting), but the paper is honest about that and the engineering details are clear. The latency numbers are useful, and the attention-map appendix is a nice toy proof of concept.\n\nThe soft spot is exactly where the abstract makes its biggest claim. The headline 15.46% relative improvement comes from 50 questions, a binary pass/fail metric, and two human auditors with no inter-annotator agreement reported. With n=50, the Wilson interval around a ~50% MQLA is roughly +/-14 points, so a relative jump of 15% could be a handful of flipped questions. Table 3 selects the better method per model after seeing results, which inflates the 'up to' number. The paper never reports absolute MQLA values, only relative changes, so the reader cannot tell whether Model C with CiteFix is at 45% or 80%. The cost-swap recommendation (12x cheaper, 3x faster) hinges entirely on those thin numbers. The assumption that the true source is in the retrieved set is acknowledged and is fine for a post-processor, but it means the method cannot fix hallucinated or missing-source facts, which the authors' own 80%-incorrectly-cited figure suggests is the intended target.\n\nThe citation pattern is fine: the related work covers the ingredients and the gap. There are no invented baselines. The missing piece is comparison to existing attribution evaluation or correction baselines beyond their own methods, and there are no public artifacts.\n\nWho this is for: engineers building commercial RAG products who want a lightweight, low-latency citation corrector, and researchers working on attribution evaluation. It is a credible industrial report, and the problem is real. But the evidence as written does not establish the headline improvement.\n\nMy advice: send to peer review, but with a clear message that the evaluation needs to be substantially strengthened before publication: larger question set, pre-registered method selection, significance testing, absolute MQLA numbers, and at least one public benchmark baseline. The core idea is worth referee time.","headline":"A genuinely useful framing of citation correction as post-processing reranking, but the 15.46% headline rests on 50 questions and post-hoc method selection, so treat the quantitative claim as unproven.","tokens_in":9896,"tokens_out":1829,"would_cite":false,"duration_ms":17269,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a post-processing step re-ranking citations against already retrieved documents improves RAG accuracy by 15.46%, enough for a 12x cheaper model to match a larger baseline.","keywords":["retrieval-augmented generation","citation accuracy","source attribution","hallucination","BERTScore","keyword matching","semantic similarity","inference cost"],"falsifier":"Audit a question set where, for each factual point, the true source document has been removed from the retrieved set and a similar non-supporting document left in. If the correctors re-cite the non-supporting document and MQLA remains high, the methods are optimizing similarity rather than support; if MQLA collapses, the re-ranking is recovering genuine sources.","tokens_in":8867,"feed_emoji":"🔗","tokens_out":11655,"duration_ms":96553,"temperature":0.7,"pith_summary":"This paper sets out to show that much of what users experience as hallucination in retrieval-augmented generation is actually citation error: the model writes a fact that came from one retrieved document but attaches a citation to a different one. It proposes six post-processing algorithms that split a generated answer into factual points and re-rank the retrieved documents against each point, choosing as corrected citations the top documents by a similarity score. On human-audited questions across four LLMs, the best two correctors raise Mean Question Level Accuracy by up to 15.46% relative, with latency overhead on the order of 0.015 seconds per factual point for the keyword-plus-semantic method. The payoff the authors aim at is economic: a small model with the corrector matches the baseline accuracy of a larger model while costing about 12x less and running about 3x faster.","feed_headline":"RAG citation fix lifts accuracy 15.5% and cuts model cost 12x","feed_subtitle":"Most bad citations are re-ranking errors among retrieved documents; smaller LLMs then match larger ones.","key_machinery":"The load-bearing mechanism is the scoring rule in Eq. 1, $s_{ij}=f(x_i,\\hat{x}_j)$, which compares each segmented factual point $x_i$ against every retrieved document $\\hat{x}_j$; the corrected citation list for a factual point is the top $C_i$ documents under this score. The paper instantiates $f$ in five ways: token-intersection keyword matching; a weighted combination of keyword match and the retriever score, $f=\\lambda f_{\\text{keyword}}+(1-\\lambda)r(q,\\hat{x}_j)$ with $\\lambda=0.8$; BERTScore over Longformer embeddings, taking the mean over factual-point tokens of their maximum token similarity in the document; the same BERTScore objective after fine-tuning on in-domain positive/negative reference pairs trained with cross-entropy loss, following the ColBERT late-interaction idea; and a lightweight second LLM prompted to output only reference numbers. A sixth idea, reading the generating model's attention maps to identify which document contributed to each factual point, is reported as a toy proof of concept and left to future work.","core_discovery":"The central claim is that citation accuracy in RAG should be treated as a post-generation re-ranking problem. After the answer is split into factual points $x_i$, each with $C_i$ original citations, the corrected citations are defined as the top $C_i$ retrieved documents $\\hat{x}_j$ under $s_{ij}=f(x_i,\\hat{x}_j)$. The paper's audit data show that for Model C roughly 80% of unverifiable facts were present in the retrieved set but attached to the wrong reference, so this re-ranking can recover them without altering the answer text. With the keyword+semantic corrector or the fine-tuned BERTScore corrector, Model C's MQLA exceeds Model A's uncorrected baseline, which the authors take as evidence that a 12x cheaper and 3x faster model can replace a larger one at comparable quality.","pith_inferences":["The paper does not develop a refusal mechanism, but Eq. 1 gives one for free: a factual point whose best retrieved-document score falls below a threshold is probably unsupported, so the same pipeline could flag hallucinations rather than merely re-cite them.","Because the corrector can only choose among retrieved documents, its ceiling is set by retrieval recall; a reader should expect the 15.46% gain to shrink in corpora where the true sources are often missed by the retriever.","The attention-map proof of concept hints at a near-zero-cost alternative: if attention over prompt tokens reliably localizes the source document for each fact, a future corrector could avoid the second LLM and the embedding pass entirely."],"forward_implications":["RAG products can add a citation corrector without waiting for a better generator; the keyword+semantic method costs about 0.015 seconds per factual point at p90, while the fine-tuned BERTScore method costs about 0.389 seconds.","Swapping a larger model for a 12x cheaper and 3x faster one becomes viable: Model C plus the corrector matches or exceeds Model A's baseline MQLA.","Different LLMs pair best with different correctors, so the corrector should be chosen per model; for Model A, keyword+semantic matching gives the largest lift, while Model C gains most from fine-tuned BERTScore.","The error mix shifts from a heavy majority of incorrect citations toward a balance between incorrect citations and hallucinations, and the absolute number of incorrect citations drops.","The correction step is streaming-compatible and operates as answers are generated, so the added latency stays away from the time-to-first-token path."],"supporting_citations":[{"why":"Provides the roughly 74% citation accuracy figure for generative search engines that motivates the correction problem.","marker":"(Liu et al., 2023)"},{"why":"Surveys hallucination in natural language generation, giving the paper the category of unverifiable facts it uses to distinguish incorrect citations from pure hallucination.","marker":"(Ji et al., 2023)"},{"why":"Documents gaps in LLM citation generation that the paper positions itself against and extends by adding post-processing correction.","marker":"(Gao et al., 2023b)"},{"why":"Supplies BERTScore, the semantic similarity metric used by the BERTScore and fine-tuned correctors.","marker":"(Zhang et al., 2020)"},{"why":"Provides the Longformer embeddings used to compute the BERTScore token similarities.","marker":"(Beltagy et al., 2020)"},{"why":"Motivates the late-interaction fine-tuning design behind the fine-tuned BERTScore corrector.","marker":"(Khattab, 2020)"},{"why":"Provides the document embeddings used to construct positive and negative training pairs for fine-tuning.","marker":"(Amazon-Titan-V2, 2024)"}],"fun_headline_variants":["RAG citation post-fix lifts accuracy 15.5%, enables 12x cheaper model","Fixing RAG citations 15.5% better, then upgrade to a 12x cheaper model","15.5% more accurate RAG citations via post-processing, 12x cost cut","CiteFix: post-process citations, boost accuracy 15.5%, cut cost 12x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true source for every factual point is already present in the retrieved document set, so an error can be fixed by changing which document is cited rather than by retrieving new material.","fun_headline_variants_meta":{"raw":{"variants":["RAG citation post-fix lifts accuracy 15.5%, enables 12x cheaper model","Fixing RAG citations 15.5% better, then upgrade to a 12x cheaper model","15.5% more accurate RAG citations via post-processing, 12x cost cut","CiteFix: post-process citations, boost accuracy 15.5%, cut cost 12x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1470,"prompt_tokens":950,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":420}},"tokens_in":566,"tokens_out":520,"duration_ms":4494,"temperature":1.0,"reasoning_tokens":420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:20:58.459949+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a question set where, for each factual point, the true source document has been removed from the retrieved set and a similar non-supporting document left in. If the correctors re-cite the non-supporting document and MQLA remains high, the methods are optimizing similarity rather than support; if MQLA collapses, the re-ranking is recovering genuine sources.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the document embeddings used to construct positive and negative training pairs for fine-tuning."}],"review_version":1}