Pith. sign in

REVIEW 5 major objections 5 minor 28 references

CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a post-processing step re-ranking citations against already retrieved documents improves RAG accuracy by 15.46%, enough for a 12x cheaper model to match a larger baseline.

desk verdict A genuinely useful framing of citation correction as post-processing reranking, but the 15.46% headline rests on 50 questions and post-hoc method selection, so treat the quantitative claim as unproven. read the letter →

arxiv 2504.15629 v2 pith:PQKKOM4R submitted 2025-04-22 cs.IR cs.CL

Harsh Maheshwari , Srikanth Tenneti , Alwarappan Nakkiran This is my paper · ORCID
classification cs.IRcs.CL
keywords retrieval-augmentedgenerationcitationaccuracysourceattributionhallucinationBERTScorekeywordmatchingsemanticsimilarityinferencecost
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that much of what users experience as hallucination in retrieval-augmented generation is actually citation error: the model writes a fact that came from one retrieved document but attaches a citation to a different one. It proposes six post-processing algorithms that split a generated answer into factual points and re-rank the retrieved documents against each point, choosing as corrected citations the top documents by a similarity score. On human-audited questions across four LLMs, the best two correctors raise Mean Question Level Accuracy by up to 15.46% relative, with latency overhead on the order of 0.015 seconds per factual point for the keyword-plus-semantic method. The payoff the authors aim at is economic: a small model with the corrector matches the baseline accuracy of a larger model while costing about 12x less and running about 3x faster.

What carries the argument

The load-bearing mechanism is the scoring rule in Eq. 1, $s_{ij}=f(x_i,\hat{x}_j)$, which compares each segmented factual point $x_i$ against every retrieved document $\hat{x}_j$; the corrected citation list for a factual point is the top $C_i$ documents under this score. The paper instantiates $f$ in five ways: token-intersection keyword matching; a weighted combination of keyword match and the retriever score, $f=\lambda f_{\text{keyword}}+(1-\lambda)r(q,\hat{x}_j)$ with $\lambda=0.8$; BERTScore over Longformer embeddings, taking the mean over factual-point tokens of their maximum token similarity in the document; the same BERTScore objective after fine-tuning on in-domain positive/negative reference pairs trained with cross-entropy loss, following the ColBERT late-interaction idea; and a lightweight second LLM prompted to output only reference numbers. A sixth idea, reading the generating model's attention maps to identify which document contributed to each factual point, is reported as a toy proof of concept and left to future work.

What would settle it

Audit a question set where, for each factual point, the true source document has been removed from the retrieved set and a similar non-supporting document left in. If the correctors re-cite the non-supporting document and MQLA remains high, the methods are optimizing similarity rather than support; if MQLA collapses, the re-ranking is recovering genuine sources.

Watch

Extended reading notes

Core claim

The central claim is that citation accuracy in RAG should be treated as a post-generation re-ranking problem. After the answer is split into factual points $x_i$, each with $C_i$ original citations, the corrected citations are defined as the top $C_i$ retrieved documents $\hat{x}_j$ under $s_{ij}=f(x_i,\hat{x}_j)$. The paper's audit data show that for Model C roughly 80% of unverifiable facts were present in the retrieved set but attached to the wrong reference, so this re-ranking can recover them without altering the answer text. With the keyword+semantic corrector or the fine-tuned BERTScore corrector, Model C's MQLA exceeds Model A's uncorrected baseline, which the authors take as evidence that a 12x cheaper and 3x faster model can replace a larger one at comparable quality.

Load-bearing premise

The load-bearing premise is that the true source for every factual point is already present in the retrieved document set, so an error can be fixed by changing which document is cited rather than by retrieving new material.

Editorial extensions

If this is right

  • RAG products can add a citation corrector without waiting for a better generator; the keyword+semantic method costs about 0.015 seconds per factual point at p90, while the fine-tuned BERTScore method costs about 0.389 seconds.
  • Swapping a larger model for a 12x cheaper and 3x faster one becomes viable: Model C plus the corrector matches or exceeds Model A's baseline MQLA.
  • Different LLMs pair best with different correctors, so the corrector should be chosen per model; for Model A, keyword+semantic matching gives the largest lift, while Model C gains most from fine-tuned BERTScore.
  • The error mix shifts from a heavy majority of incorrect citations toward a balance between incorrect citations and hallucinations, and the absolute number of incorrect citations drops.
  • The correction step is streaming-compatible and operates as answers are generated, so the added latency stays away from the time-to-first-token path.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not develop a refusal mechanism, but Eq. 1 gives one for free: a factual point whose best retrieved-document score falls below a threshold is probably unsupported, so the same pipeline could flag hallucinations rather than merely re-cite them.
  • Because the corrector can only choose among retrieved documents, its ceiling is set by retrieval recall; a reader should expect the 15.46% gain to shrink in corpora where the true sources are often missed by the retriever.
  • The attention-map proof of concept hints at a near-zero-cost alternative: if attention over prompt tokens reliably localizes the source document for each fact, a future corrector could avoid the second LLM and the embedding pass entirely.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper addresses citation errors in RAG systems. The authors propose post-processing methods that, after the LLM generates an answer, segment the response into factual points and re-rank the already-retrieved documents by similarity to each factual point, selecting the top C_i as corrected citations. Several scoring functions are proposed: lexical overlap, lexical+query-semantic hybrid, BERTScore with Longformer, a ColBERT-style fine-tuned BERTScore trained on LLM-generated positive/negative pairs, and a lightweight LLM-based citation selection; an attention-map approach is sketched as future work. The evaluation is a human audit of 50 questions on an unnamed RAG system, using a composite MQLA metric defined in Algorithm 1. The paper reports relative MQLA improvements of up to 15.46% and suggests this enables replacing a large LLM with a 12x cheaper and 3x faster model at comparable accuracy.

Significance. If validated, the contribution is practically useful: post-hoc citation fixing is cheaper than retraining or retrieval augmentation changes, and the paper is one of the few to address citation correction rather than detection. The latency figures (e.g., 0.015 s per factual point for keyword+semantic) are valuable. However, the evidence is currently too weak to support the headline quantitative claim: n=50, a binary metric, no confidence intervals, no significance tests, post-hoc method selection, and an unexplained 15.46% figure. The methods are plausible, and the manuscript is transparent about being early-stage, but the evaluation must be strengthened before the central claim can be accepted.

major comments (5)
  1. [Section 4.2, Table 2] The evaluation is based on 50 questions with no confidence intervals, significance tests, or absolute baseline MQLA values. Since Algorithm 1 defines MQLA as a binary pass/fail per question, the reported relative improvements (e.g., +15.5% for Keyword+Semantic on Model C) may correspond to flipping only a few questions. Please report absolute MQLA for the baseline and each method, compute exact 95% confidence intervals (e.g., Wilson intervals) for the differences, and run a paired test such as McNemar's test on the 50 question-level outcomes. This is necessary to support the central quantitative claim in the Abstract and Section 5.
  2. [Abstract and Section 5] The headline claim of a relative improvement of 15.46% does not match the numbers in Table 2, which reports +15.5% for Keyword+Semantic Context and +15.8% for Fine-tuned BERTScore on Model C, while Table 1 reports +15.4% in parentheses. Please state precisely which configuration yields the 15.46% figure, whether it is a single model result or an average, and if an average, specify the averaging procedure. The inconsistency undermines the paper's central quantitative claim.
  3. [Section 4.3, Table 3] For each LLM the better of the two methods appears to be selected after seeing the results (e.g., KSC for Model A, FBS for Model C), and the 'up to 15.46%' claim appears to use the best possible result. This post-hoc selection over eight method-model pairs inflates the reported improvement. Report all method-model combinations in a single table, and either pre-register the selection rule or use a holdout set for method selection before evaluating on the test questions.
  4. [Sections 3 and 4.1] The correction methods only re-rank the retrieved documents {x̂_i}, so they cannot fix a factual point whose true source is absent from the retrieved set. The paper's 80% 'incorrectly cited' figure is based on a definition that counts a fact as incorrectly cited only if the fact is present in some retrieved document, meaning retrieval recall is assumed for those facts. Please report the retrieval set size and the recall of true sources for incorrectly cited facts, and discuss how the method's ceiling depends on retriever quality. This is central to the claim that 80% of unverifiable facts are fixable by re-ranking.
  5. [Section 4.2, Algorithm 1] The evaluation uses two human auditors over 2.5 days per row, but no inter-annotator agreement is reported (e.g., Cohen's kappa) for the MQLA components. Since MQLA involves subjective judgments (relevancy of keywords, completeness, correctness), the absence of reliability data weakens confidence in the measured improvements. Please report agreement statistics and, if disagreements occurred, describe how they were resolved.
minor comments (5)
  1. [Section 3, Eq. (1)] The notation is confusing: x_i is used for factual points in Eq. (1), but \hat{x}_i is used for retrieved documents earlier in the same section. Consider using distinct symbols (e.g., s_i for factual points and d_j for retrieved documents) to avoid ambiguity.
  2. [Section 3.4] State clearly that the positive/negative pairs for fine-tuning are LLM-generated and describe the steps taken to prevent overlap between the training pairs and the 50 evaluation questions. Otherwise, the improvement from the fine-tuned BERTScore could partly reflect the evaluator being tuned to the test distribution.
  3. [Table 1] Table 1 is very hard to parse: columns are combined (e.g., 'Model Cents per 1K O/P tokens' with 'Relative Mean Question Level Accuracy'), and the parenthetical notation is not immediately clear. Consider splitting the table into separate panels for cost, MQLA, and error decomposition, and add a clearer legend for the numbers inside/outside parentheses.
  4. [Abstract and Introduction] The paper says it proposes 'six computationally light weight methods,' but only four are evaluated (Keyword, Keyword+Semantic, BERTScore, Fine-tuned BERTScore), with LLM-based matching tested in a limited way and attention maps only as a proof of concept. Please align the method count with what is actually evaluated or explicitly mark the unimplemented methods as future work.
  5. [Section 5] The cost-saving model-swap recommendation is based on comparing Model C+FBS against Model A baseline in Table 2, but no statistical test is reported for this head-to-head comparison. Please report the absolute MQLA values for both conditions and a significance test to substantiate the claim that CiteFix enables a 12x cheaper model with comparable performance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CiteFix's citation corrections are evaluated against human judgments, not the similarity scores used to generate them.

full rationale

The paper proposes post-processing algorithms that reassign citations by selecting the top retrieved documents under a similarity metric (Eq. 1). This is a reranking procedure and does not assume the target conclusion. The headline MQLA metric is defined from independently assessed components (relevancy of URLs and keywords, correctness, completeness) and is scored by human auditors, while the reported improvements are relative to a no-correction baseline. The fine-tuned BERTScore model is trained on LLM-generated positive/negative pairs, but its evaluation is based on external human judgments, so the model is not judged on its own training labels. Hyperparameters such as λ=0.8 are empirically chosen, but the correctness improvement is not definitionally forced by that choice. The paper contains no load-bearing self-citation chain; cited works (BERTScore, ColBERT, Longformer, etc.) are independent external methods. Any concerns about the small 50-question sample, lack of significance testing, or post-hoc method selection are statistical-validity issues, not circularity of the derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a small number of fitted choices (lambda=0.8, number of citations preserved, best-method-per-model selection) and on a set of domain assumptions about retrieval coverage and similarity-as-support. No invented entities are introduced. The paper does not provide independent evidence for the key assumption that the true source is always in the retrieved set.

free parameters (4)
  • lambda = 0.8
    Weight in Eq. 2 balancing keyword and semantic context scores; chosen empirically (Section 3.2).
  • Number of citations preserved per factual point = original count Ci
    The correction keeps the same number of citations as the LLM generated; this count is not optimized or analyzed.
  • Best-method-per-model selection = chosen per LLM from Table 3
    The reported headline improvements select the best performing method for each model after seeing the results, which is a form of post-hoc fitting.
  • Hard-negative difficulty levels = n in {14,11,8,5,4,3}
    Chosen by hand for training data generation (Section 3.4); no sensitivity analysis.
assumptions (5)
  • domain assumption The retrieved document set contains the correct source for each factual point.
    Section 3 states the correction selects from retrieved documents only; if the true source is not retrieved, the citation cannot be fixed.
  • domain assumption Lexical or semantic similarity between a factual point and a document is a valid proxy for whether the document supports the fact.
    All methods define f as a similarity metric and assume the top-Ci most similar documents are the correct citations (Eq. 1 and following).
  • domain assumption Human auditor judgments on 50 questions are stable and representative of real-world performance.
    Section 4 uses 50 questions and two humans with no inter-annotator agreement reported; the composite MQLA metric uses hard thresholds and a hallucination cap.
  • domain assumption The classification of unverifiable facts into 'incorrectly cited' versus 'hallucinated' is accurate.
    Section 1 relies on the 80% incorrect-citation figure to motivate the approach; the division is based on whether the fact appears in any retrieved document.
  • standard math Standard behavior of BERTScore, Longformer, and LLM prompting as described.
    The methods assume these models work as documented and that fine-tuning on generated pairs transfers to the evaluation distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction." pith.science (2026). https://pith.science/paper/PQKKOM4R

@misc{pith2026250415629,
  author       = {Pith},
  title        = {Pith review of: CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQKKOM4R}},
  note         = {Machine review of arXiv:2504.15629}
}
read the original abstract

Retrieval Augmented Generation (RAG) has emerged as a powerful application of Large Language Models (LLMs), revolutionizing information search and consumption. RAG systems combine traditional search capabilities with LLMs to generate comprehensive answers to user queries, ideally with accurate citations. However, in our experience of developing a RAG product, LLMs often struggle with source attribution, aligning with other industry studies reporting citation accuracy rates of only about 74% for popular generative search engines. To address this, we present efficient post-processing algorithms to improve citation accuracy in LLM-generated responses, with minimal impact on latency and cost. Our approaches cross-check generated citations against retrieved articles using methods including keyword + semantic matching, fine tuned model with BERTScore, and a lightweight LLM-based technique. Our experimental results demonstrate a relative improvement of 15.46% in the overall accuracy metrics of our RAG system. This significant enhancement potentially enables a shift from our current larger language model to a relatively smaller model that is approximately 12x more cost-effective and 3x faster in inference time, while maintaining comparable performance. This research contributes to enhancing the reliability and trustworthiness of AI-generated content in information retrieval and summarization tasks which is critical to gain customer trust especially in commercial products.

Figures

Figures reproduced from arXiv: 2504.15629 by the authors.

Figure 1
Figure 1. Improvements in RAG accuracy for various [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the workflow of the proposed [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualisation of Attention Score. See Ap [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 4 canonical work pages

  1. [1]

    Amazon-Titan-V2. 2024. https://aws.amazon.com/bedrock/amazon-models/titan/ Amazon titan foundation models . Accessed: 2025-01-15

    work page 2024

  2. [2]

    Patrice B \'e chard and Orlando Marquez Ayala. 2024. Reducing hallucination in structured outputs via retrieval-augmented generation. arXiv preprint arXiv:2404.08189

    arXiv 2024

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. http://arxiv.org/abs/2004.05150 Longformer: The long-document transformer

    arXiv 2020

  4. [4]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. http://arxiv.org/abs/1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding

    arXiv 2019

  5. [6]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.398 Enabling large language models to generate text with citations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--6488, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.398 2023

  6. [7]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023 c . http://arxiv.org/abs/2305.14627 Enabling large language models to generate text with citations

    arXiv 2023

  7. [8]

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. http://arxiv.org/abs/2204.04991 True: Re-evaluating factual consistency evaluation

    arXiv 2022

  8. [9]

    I Hsu, Zifeng Wang, Long T Le, Lesly Miculicich, Nanyun Peng, Chen-Yu Lee, Tomas Pfister, et al. 2024. Calm: Contrasting large and small language models to verify grounded generation. arXiv preprint arXiv:2406.05365

    arXiv 2024

Show all 28 references
  1. [10]

    Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024. http://arxiv.org/abs/2402.04315 Training language models to generate text with citations via fine-grained rewards

    2024 arXiv

  2. [11]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  3. [12]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172

    2019 arXiv

  4. [13]

    Omar Khattab. 2020. http://arxiv.org/abs/2004.12832 Colbert: Efficient and effective passage search via contextualized late interaction over bert

    2020 arXiv

  5. [14]

    Xinze Li, Yixin Cao, Liangming Pan, Yubo Ma, and Aixin Sun. 2024. http://arxiv.org/abs/2310.05634 Towards verifiable generation: A benchmark for knowledge-aware language model attribution

    2024 arXiv

  6. [15]

    Liu, Tianyi Zhang, and Percy Liang

    Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. http://arxiv.org/abs/2304.09848 Evaluating verifiability in generative search engines

    2023 arXiv

  7. [16]

    Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. http://arxiv.org/abs/2309.07852 Expertqa: Expert-curated questions and attributed answers

    2024 arXiv

  8. [17]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022...

    2022 arXiv

  9. [18]

    Perplexity AI . 2024. https://www.perplexity.ai Perplexity AI : AI -powered search engine . Accessed: November 21, 2024

    2024

  10. [19]

    M Towhidul Islam Tonmoy, Krishnav Rajbangshi, Shravani Nag, Aman Chadha, Amit P

    Vipula Rawte, S. M Towhidul Islam Tonmoy, Krishnav Rajbangshi, Shravani Nag, Aman Chadha, Amit P. Sheth, and Amitava Das. 2024. http://arxiv.org/abs/2403.19113 Factoid: Factual entailment for hallucination detection

    2024 arXiv

  11. [20]

    Fran c ois Rousseau and Michalis Vazirgiannis. 2013. Composition of tf normalizations: new insights on scoring functions for ad hoc ir. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 917--920

    2013

  12. [21]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567

    2021 arXiv

  13. [22]

    Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, and Soujanya Poria. 2024. http://arxiv.org/abs/2409.11242 Measuring and enhancing trustworthiness of llms in rag through grounded attributions and learning to refuse

    2024 arXiv

  14. [23]

    Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. 2024. http://arxiv.org/abs/2312.09075 Towards verifiable text generation with evolving memory and self-reflection

    2024 arXiv

  15. [24]

    Andrew Trotman, Antti Puurula, and Blake Burgess. 2014. Improvements to bm25 and language models examined. In Proceedings of the 19th Australasian Document Computing Symposium, pages 58--65

    2014

  16. [25]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. http://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2023 arXiv

  17. [26]

    Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. http://arxiv.org/abs/2305.06311 Automatic evaluation of attribution by large language models

    2023 arXiv

  18. [27]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. http://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert

    2020 arXiv

  19. [28]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  20. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.