Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that prepending page context to every retrieved evidence improves RAG retrieval and answer quality, and that counterfactual evidence-removal attribution explains answers more accurately than similarity-based attribution.

desk verdict ConfQuestions is the real contribution; contextualization adds a modest, credible finding; the counterfactual attribution claim is not yet supported by the evaluation. read the letter →

arxiv 2412.10571 v3 pith:NKDUKDBY submitted 2024-12-13 cs.CL cs.IR

Rishiraj Saha Roy , Joel Schlotthauer , Chris Hinze , Andreas Foltyn , Luzian Hahn , Fabian Kuech This is my paper · ORCID
classification cs.CLcs.IR
keywords retrievalaugmentedgenerationconversationalquestionansweringevidencecontextualizationcounterfactualattributionexplainabilityheterogeneousdocumentstablesbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAGonite is a RAG system for conversational question answering over enterprise-style wikis that mix passages, lists, and tables. The paper tries to show that two simple interventions fix two common RAG weaknesses: prepend document context to each evidence at indexing time, and explain answers by removing evidence clusters and measuring how much the generated answer changes. On a new benchmark called ConfQuestions, with 300 conversational questions in English and German tied to 215 public wiki pages, the contextualization intervention improves retrieval Precision@1 from 0.398 to 0.528 and answer relevance from 0.388 to 0.529, and the counterfactual attribution beats naive cosine attribution on the paper's URL-match accuracy. The paper also releases the benchmark and the full pipeline. A reader should care because these are cheap, model-agnostic changes that could make enterprise RAG answers more accurate and their explanations causal rather than merely plausible.

What carries the argument

The central mechanism is evidence contextualization at indexing time: each evidence (passage, list, verbalized table row, or whole table) is stored as its raw content plus the page title, previous heading, preceding evidence, and following evidence, so localizing words that live outside the chunk are visible to both the retriever and the LLM. The second central mechanism is counterfactual attribution (Algorithm 1): the retrieved top-ten evidences are first clustered by semantic redundancy; each cluster is removed one at a time, the LLM regenerates the answer with that cluster absent, and the cluster's contribution is one minus the mean cosine similarity between the original and counterfactual answers, normalized by masked softmax. Clustering is essential because removing one of two redundant chunks would otherwise wrongly count the other as the sole cause of the answer.

What would settle it

Run the same counterfactual attribution on a set of questions with hand-labeled evidence-level gold, including questions where the gold page falls outside the top ten; if the top-scoring evidence then matches at no better than the naive cosine attribution baseline, the central claim that counterfactual attribution outperforms standard attribution would be undercut.

Watch

Extended reading notes

Core claim

The paper's central claim has two halves. First, indexing each retrieved unit together with its document context—page title, the heading immediately above it, the preceding evidence, and the following evidence—makes both retrieval and answer generation better than indexing the raw chunk alone, with the page title being the single most useful addition and no context component hurting performance. Second, attribution of an answer to evidence is better done counterfactually than by cosine similarity: remove a cluster of similar evidence from the prompt, regenerate the answer, and take one minus the average similarity between the original and regenerated answers as that evidence's contribution. This reaches roughly 0.80 accuracy on the paper's URL-match metric, beating the naive similarity baseline.

Load-bearing premise

The attribution evaluation assumes that matching the source URL of the highest-scoring evidence to the gold answer URL is a faithful proxy for having found the correct evidence, and it only measures questions for which the gold page was already in the retrieved top ten.

Editorial extensions

If this is right

  • If contextualization generalizes, any RAG system that currently indexes bare chunks can gain retrieval and answer quality simply by storing each chunk with its title, heading, and neighbors, with no model changes.
  • Because the page title alone gives the largest single gain and no context component hurts, content-management pipelines could adopt context enrichment as a default preprocessing step.
  • Counterfactual attribution gives users explanations tied to how the answer would change, not to how similar the answer is to the evidence, so explanations should be more reliable when redundant evidence is present.
  • Verbalizing tables and indexing both whole tables and individual rows is a viable way to bring heterogeneous documents into RAG.
  • The ConfQuestions benchmark provides a reusable testbed for comparing RAG systems on mixed passage, list, and table evidence in two languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains may be largest on corpora like enterprise wikis, where pages often reuse terms from titles and headings; on flatter corpora the benefit of added context could shrink, but this is an empirical question the benchmark does not settle.
  • The attribution distribution is only as causal as the LLM's regenerations are faithful to the evidence pool; if the model ignores the removed evidence or falls back on parametric knowledge, the estimated contributions are biased, and the paper's out-of-scope checks were not directly tied to attribution steps.
  • The URL-match evaluation could be sharpened by evidence-level gold labels; until then, the attribution accuracy numbers are an upper bound on locating the exact supporting chunk, and a finer-grained answer-nugget deletion variant could yield more actionable explanations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents RAGONITE, a RAG demo for conversational QA over heterogeneous enterprise wiki documents. It contributes (i) evidence contextualization at indexing time by concatenating page title, previous heading, and neighboring evidence to each evidence, with table rows verbalized; (ii) counterfactual attribution that removes each evidence cluster, regenerates the answer with an LLM, and uses the average similarity between the original and counterfactual answers as a negative contribution; and (iii) ConfQuestions, a new benchmark of 300 conversational questions in English and German over 215 public Confluence pages. Experiments compare retrieval P@1 and GPT-4o-judged answer relevance with and without various context components, and compare naive cosine attribution with counterfactual attribution on a filtered subset of questions. The central claims are that contextualization improves RAG performance and that counterfactual explanations outperform standard attribution.

Significance. RAGONITE addresses a practical gap: enterprise RAG pipelines often lack document context for heterogeneous evidence and rely on similarity-based attribution. The ConfQuestions benchmark is a useful public resource with balanced passage/list/table answer types and bilingual questions, and the public code release supports reproducibility. The contextualization result is credible: the effect sizes are large and consistent across many slices, for example P@1 of 0.398 to 0.528 and answer relevance of 0.388 to 0.529 for ALL versus NONE in Table 2. The counterfactual attribution idea is interesting and applicable to closed LLMs, unlike ContextCite, but the current evidence does not yet establish that counterfactual attribution outperforms naive attribution because of the evaluation weaknesses detailed below.

major comments (4)
  1. [§7.1 and §7.3] Attribution accuracy is defined as a URL-level match and is measured only over the 364/600 question instances for which the gold URL is already in the top-10 retrieved evidences (§7.3). The paper itself notes in §7.1 that gold answers are often composed of nuggets spread over the whole document and that evidence-level labels would be preferable. A top-attributed evidence on the gold page can therefore be a page-level hit but an evidence-level miss, and the excluded 236 questions make the numbers conditional on successful retrieval rather than end-to-end explanation quality. The authors should report the excluded questions as failures for all methods and, if feasible, add an evidence-level or human-judged evaluation on a sample; at minimum the conditional nature of Table 3 must be stated wherever the attribution claim is made.
  2. [Table 3] The claim that counterfactual attribution outperforms standard attribution rests on a 0.799 versus 0.772 accuracy for the best CFA variant on 364 binary outcomes. No confidence intervals, standard errors, or paired significance tests are reported, and the difference is 2.7 percentage points; on the English slice the naive baseline wins (0.807 versus 0.786), and on complex questions the no-cluster CFA is tied with naive at 0.772. The authors should report paired tests (for example, McNemar or bootstrap over questions) and confidence intervals for each slice before claiming superiority.
  3. [§5, §7.2, §7.4] The default configuration was selected from the same ablation study on the same test set: §7.4 states that "the study leading to this table was also used to select our default configuration," and the same test data are then used for the main results. This selection-on-test-set practice makes the reported advantages of the chosen configuration optimistic. In addition, the temperature parameter t=0.05 in §5 is explicitly chosen because it skews degenerate distributions to 1–3 evidences, reflecting the average number of gold evidences in ConfQuestions, which is supervision from test labels. Although softmax temperature preserves the top-1 argmax and may not affect Table 3's accuracy directly, it is still a benchmark-informed choice; it should be fixed a priori or tuned on a validation split.
  4. [§7.3, Table 2] The text says contextualization "significantly improved" retrieval and answering, but no significance tests accompany Table 2. The effect sizes are large and consistent (for example, P@1 0.398 to 0.528 and answer relevance 0.388 to 0.529 for ALL versus NONE), so I do not doubt the direction of the effect, but the word "significantly" should be replaced with a descriptive statement or supported by paired tests over the 600 question instances.
minor comments (5)
  1. [Algorithm 1, §5] The number of Monte Carlo iterations m in Algorithm 1 is never specified; report the value used, the LLM used for counterfactual answers, and the computational cost.
  2. [§7.3] The enumeration that with-clusters is best in 5/8 cases and without-clusters in 2/8 omits the slice where the naive baseline is best (English questions, 0.807); the counts should cover all three methods or be phrased as 5/2/1.
  3. [Abstract, §6] The abstract says "300 hand-created conversational questions, each in English and German," while §7.2 refers to 600 English and German questions; clarify that there are 300 questions in two languages, yielding 600 test instances.
  4. [§7.1] Answer relevance is judged by GPT-4o, the same model family used for generation; a small human-agreement study would strengthen the answer-quality results.
  5. [§4, §7.4] The contextualization experiments compare against no context only; adding common alternatives such as coreference resolution and sliding-window chunking as baselines would better support the claim that the proposed contextualization scheme is effective relative to existing practice.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's empirical comparisons are self-contained; acknowledged URL-level proxy and in-sample tuning are correctness concerns, not definitional reductions.

full rationale

The central claims of RAGonite are empirical: evidence contextualization improves retrieval and answer quality, and counterfactual attribution outperforms naive similarity-based attribution. Neither claim is derived from an equation that assumes its own conclusion. Contextualization is implemented by concatenating page title, preceding heading, and surrounding evidence at indexing time; the improvements in Table 2 are ablations over the same benchmark, not fitted parameters renamed as predictions. The counterfactual attribution algorithm (Algorithm 1) defines contribution as one minus the average similarity between the original answer and counterfactual answers, and the evaluation in Sec. 7.1 uses URL match as a proxy for evidence correctness; the paper explicitly acknowledges this proxy and the consequent difficulty of evidence-level annotation. This weakens the attribution claim but does not make it circular. The removal of 236 questions whose gold URL was not in the top-10 (Sec. 7.3) conditions the reported accuracy on successful retrieval, which limits external validity but is not a definitional identity. The temperature parameter t is set using the statement that it 'reflects the average number of gold evidences in ConfQuestions' (Sec. 5); this is an in-sample use of label statistics and a correctness/overfitting concern, but a softmax temperature preserves the argmax ranking, so it cannot determine the top-1 URL accuracy reported in Table 3. Likewise, Sec. 7.4 states that the ablation study was used to select the default configuration on the same 600 questions; this risks optimistically in-sample numbers but is model selection rather than a circular derivation. Self-citations in the reference list ([1], [13], [15]) provide background or prior art and are not load-bearing: no uniqueness theorem or core ansatz is imported from the authors' own prior work to force the present choices. Overall, the paper's results are empirical demonstrations with acknowledged proxy metrics and test-set tuning concerns, not predictions that reduce to their inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the benchmark being representative, the LLM judge being reliable, the similarity-drop metric being a valid causal proxy, and several algorithm parameters being appropriate. The most fragile pieces are the URL-level attribution scoring and the test-set-informed temperature choice.

free parameters (6)
  • Temperature t = 0.05
    Set so attribution distributions skew to 1 to 3 top evidences because this 'reflects the average number of gold evidences in ConfQuestions' (Sec. 5, Parameter selection). This is test-set-informed tuning.
  • DBSCAN epsilon = 0.005
    Chosen so that clusters of 2 to 3 evidences form among the 10 retrieved evidences (Sec. 5).
  • DBSCAN minPts = 2
    Set to 2 because only ten evidences are considered (Sec. 5).
  • Top-k retrieved evidences = 10
    Default number of evidence elements inserted into the LLM prompt (Sec. 7.2); selected through the same ablation study used to pick the default configuration.
  • Default RAGONITE configuration = ALL context, verbalization, row and table indexing, GPT-4o, BGE embeddings, hybrid search, BGE plus RRF reranking
    The paper states the ablation study in Table 4 'was also used to select our default configuration,' meaning the reported configuration was selected on the same benchmark used for evaluation (Sec. 7.4).
  • Monte Carlo iterations m = not specified
    Algorithm 1 takes m as input, and the paper mentions repeating generation m times, but never states the value actually used (Sec. 5).
assumptions (5)
  • domain assumption ConfQuestions is a representative enterprise-wiki benchmark with correct gold URLs and answers.
    The benchmark was written by the same group, and no inter-annotator agreement or external validation is provided (Sec. 6).
  • domain assumption GPT-4o answer relevance judgments are a valid substitute for human relevance judgments.
    Answer relevance is scored by GPT-4o as 0, 0.5, or 1 against gold answers without human calibration (Sec. 7.1).
  • domain assumption The drop in answer similarity after removing an evidence cluster measures the causal contribution of that evidence.
    Algorithm 1 defines contribution as 1 minus average cosine similarity; the paper does not justify this as a causal quantity beyond intuition (Sec. 5).
  • domain assumption DBSCAN clustering with the tuned epsilon removes all redundant evidence, so each cluster removal is a faithful counterfactual.
    The method assumes clustering resolves duplicate evidence, but epsilon and minPts are set by hand for a single ten-evidence setting (Sec. 5).
  • domain assumption A URL match between the top-attributed evidence and the gold answer URL is sufficient to score an explanation as correct.
    Attribution accuracy uses source URL, not the exact evidence passage, as the target (Sec. 7.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems." pith.science (2026). https://pith.science/paper/NKDUKDBY

@misc{pith2026241210571,
  author       = {Pith},
  title        = {Pith review of: Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKDUKDBY}},
  note         = {Machine review of arXiv:2412.10571}
}
read the original abstract

Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation typically rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions: it has 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages. These documents are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual explanations outperform standard attribution.

Figures

Figures reproduced from arXiv: 2412.10571 by the authors.

Figure 1
Figure 1. The RAGonite workflow enhances RAG pipelines at both ends, preprocessing evidence and explaining answers. of the answer to the retrieved units of evidence, are solely based on statistical similarity between the answer and the evidence [6]: these are not causal [10], but rather only plausible explanations. Moreover, current pipelines support raw text and rarely mention how tabular elements could be handled. Contribut… view at source ↗
Figure 2
Figure 2. An annotated walkthrough the of RAGonite demo. Blue boxes guide the reader and are not part of the UI (Sec. 3) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Contextualized evidence as in answer prompt. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: LLM prompt for rephrasing a conversational ques [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: LLM prompt for generating an answer from the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: LLM prompt for evaluating a generated answer [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions

    cs.IR 2026-08 conditional novelty 6.0 of 10

    Adding missing support repairs one third of joint-eligible RAG failures and deleting verified nonsupport repairs one seventh, but the exact counterfactual response is not recoverable from the original failure and vari...

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. 2019. Look before you hop: Conversational question answering over knowledge graphs using judicious context expansion. In CIKM

    work page 2019

  2. [2]

    Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry

    work page

  3. [3]

    Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In SIGIR

    work page 2009

  4. [4]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv (2023)

    work page 2023

  5. [5]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, et al. 2020. Retrieval- augmented generation for knowledge-intensive NLP tasks. In NeurIPS

    work page 2020

  6. [6]

    Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. 2023. A survey of large language models attribution. arXiv (2023)

    work page 2023

  7. [7]

    Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. 2023. Table- GPT: Table-tuned gpt for diverse table tasks. arXiv (2023)

    work page 2023

  8. [8]

    Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. UniK- QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering. In NAACL

    work page 2022

Show all 20 references
  1. [9]

    Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and James Hendler

  2. [10]

    Judea Pearl. 1997. Causation, action, and counterfactuals. In SYLI

    1997

  3. [11]

    Judea Pearl and Dana Mackenzie. 2020. The Book of Why: The New Science of Cause and Effect. Hachette Book Group USA

    2020

  4. [12]

    Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversa- tional question answering challenge. TACL 7 (2019)

    2019

  5. [13]

    Rishiraj Saha Roy and Avishek Anand. 2022. Question Answering for the Curated Web Tasks and Methods in QA over Knowledge Bases and Text Collections. Springer

    2022

  6. [14]

    Viju Sudhi, Sinchana Bhat, Max Rudat, and Roman Teucher. 2024. RAG-Ex: A Generic Framework for Explaining Retrieval Augmented Generation. In SIGIR

    2024

  7. [15]

    Khanh Hiep Tran, Azin Ghazimatin, and Rishiraj Saha Roy. 2021. Counterfactual explanations for neural recommenders. In SIGIR

    2021

  8. [16]

    Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv (2024)

    2024

  9. [17]

    Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. In ICLR

    2024

  10. [18]

    Wenhao Yu. 2022. Retrieval-augmented generation across heterogeneous knowl- edge. In NAACL. 8

    2022

  11. [2022]

    arXiv (2022)

    T-RAG: End-to-End Table Question Answering via Retrieval-Augmented Generation. arXiv (2022)

    2022

  12. [2024]

    In NeurIPS

    ContextCite: Attributing Model Generation to Context. In NeurIPS

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.