{"id":"1588526d-0714-43d0-87a2-5c663cc69be8","arxiv_id":"2412.10571","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RAGONITE adds document context to retrieved evidence and explains answers by removing evidence, evaluated on a new bilingual 300-question benchmark over heterogeneous Confluence pages.","lead":"This paper presents RAGONITE, a RAG system that adds document context to retrieved passages and explains answers by removing each evidence and measuring how much the answer changes. It also releases ConfQuestions, a 300-question bilingual benchmark over 215 public Confluence pages, and reports that both contextualization and counterfactual attribution improve performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution claim rests on URL-level accuracy on a retrieval-filtered subset with no significance test; the small CFA-vs-naive margin may vanish under evidence-level evaluation.","rationale":"I read the paper in good faith: the contextualization results in Table 2 are consistent, monotone, and plausible; the benchmark release is useful; and the system walkthrough is a legitimate demo contribution. The point of stress is the attribution evaluation. The reader's weakest assumption identified the URL-match proxy and the top-10 filter, and I agree. I would go further: the reported advantage of CFA over naive is numerically small (0.799 vs 0.772 on 364 questions) and is reported without any uncertainty quantification, so even if the metric were accepted, the superiority claim is not statistically established. The proposed test—evidence-level annotation on a sample and a full-600 inclusion rule—directly targets the proxy and selection issues. If the test shows the advantage persists, the conditional verdict can stand or be upgraded; if it does not, the paper remains a useful systems/benchmark contribution but its second central claim should be downgraded. No concerns about author conduct or internal consistency were identified.","tokens_in":11856,"tokens_out":5433,"duration_ms":54435,"concrete_test":"Sample 100 of the 364 questions from Table 3, and have two annotators independently mark which of the top-10 retrieved evidences (not URLs) actually contain the gold answer content, with adjudication for disagreements. Recompute all three Table 3 columns at evidence level on this sample, scoring a method as correct only when the highest-attributed evidence is one of the marked gold evidences. Also recompute on the full 600 by counting questions with no retrieved gold URL as incorrect for both methods. If the evidence-level accuracy of CFA-with-clusters does not exceed naive accuracy by a margin whose 95% bootstrap confidence interval excludes zero, the headline attribution claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The second half of the central claim—that counterfactual explanations outperform standard attribution—is supported only by Table 3, and the evaluation there has two properties that together make the claim fragile. First, Sec. 7.1 defines accuracy as whether the highest-attributed evidence's source URL equals the gold answer URL; this is a page-level proxy, not evidence-level correctness, and the authors explicitly note that gold answers are often nuggets spread over a document. Thus an evidence that is on the gold page but did not actually support the answer counts as a success. Second, Sec. 7.3 drops the 236/600 questions whose gold URL was not in the retrieved top-10, so the reported numbers are conditional on a successful retrieval and are not end-to-end explanation quality. On the remaining 364 questions, the best counterfactual variant reaches 0.799 versus 0.772 for naive similarity; a 0.027 difference over 364 binary outcomes without confidence intervals or a paired significance test is not enough to establish superiority. If evidence-level labels were used, or if the 236 excluded questions were scored as failures for both methods, the apparent advantage could shrink or disappear. Because this is the only evidence for the causal-attribution claim, the overall central claim is not yet secured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RAGONITE, a RAG demo for conversational QA over heterogeneous enterprise wiki documents. It contributes (i) evidence contextualization at indexing time by concatenating page title, previous heading, and neighboring evidence to each evidence, with table rows verbalized; (ii) counterfactual attribution that removes each evidence cluster, regenerates the answer with an LLM, and uses the average similarity between the original and counterfactual answers as a negative contribution; and (iii) ConfQuestions, a new benchmark of 300 conversational questions in English and German over 215 public Confluence pages. Experiments compare retrieval P@1 and GPT-4o-judged answer relevance with and without various context components, and compare naive cosine attribution with counterfactual attribution on a filtered subset of questions. The central claims are that contextualization improves RAG performance and that counterfactual explanations outperform standard attribution.","tokens_in":12120,"tokens_out":9005,"duration_ms":81500,"significance":"RAGONITE addresses a practical gap: enterprise RAG pipelines often lack document context for heterogeneous evidence and rely on similarity-based attribution. The ConfQuestions benchmark is a useful public resource with balanced passage/list/table answer types and bilingual questions, and the public code release supports reproducibility. The contextualization result is credible: the effect sizes are large and consistent across many slices, for example P@1 of 0.398 to 0.528 and answer relevance of 0.388 to 0.529 for ALL versus NONE in Table 2. The counterfactual attribution idea is interesting and applicable to closed LLMs, unlike ContextCite, but the current evidence does not yet establish that counterfactual attribution outperforms naive attribution because of the evaluation weaknesses detailed below.","major_comments":[{"comment":"Attribution accuracy is defined as a URL-level match and is measured only over the 364/600 question instances for which the gold URL is already in the top-10 retrieved evidences (§7.3). The paper itself notes in §7.1 that gold answers are often composed of nuggets spread over the whole document and that evidence-level labels would be preferable. A top-attributed evidence on the gold page can therefore be a page-level hit but an evidence-level miss, and the excluded 236 questions make the numbers conditional on successful retrieval rather than end-to-end explanation quality. The authors should report the excluded questions as failures for all methods and, if feasible, add an evidence-level or human-judged evaluation on a sample; at minimum the conditional nature of Table 3 must be stated wherever the attribution claim is made.","section":"§7.1 and §7.3"},{"comment":"The claim that counterfactual attribution outperforms standard attribution rests on a 0.799 versus 0.772 accuracy for the best CFA variant on 364 binary outcomes. No confidence intervals, standard errors, or paired significance tests are reported, and the difference is 2.7 percentage points; on the English slice the naive baseline wins (0.807 versus 0.786), and on complex questions the no-cluster CFA is tied with naive at 0.772. The authors should report paired tests (for example, McNemar or bootstrap over questions) and confidence intervals for each slice before claiming superiority.","section":"Table 3"},{"comment":"The default configuration was selected from the same ablation study on the same test set: §7.4 states that \"the study leading to this table was also used to select our default configuration,\" and the same test data are then used for the main results. This selection-on-test-set practice makes the reported advantages of the chosen configuration optimistic. In addition, the temperature parameter t=0.05 in §5 is explicitly chosen because it skews degenerate distributions to 1–3 evidences, reflecting the average number of gold evidences in ConfQuestions, which is supervision from test labels. Although softmax temperature preserves the top-1 argmax and may not affect Table 3's accuracy directly, it is still a benchmark-informed choice; it should be fixed a priori or tuned on a validation split.","section":"§5, §7.2, §7.4"},{"comment":"The text says contextualization \"significantly improved\" retrieval and answering, but no significance tests accompany Table 2. The effect sizes are large and consistent (for example, P@1 0.398 to 0.528 and answer relevance 0.388 to 0.529 for ALL versus NONE), so I do not doubt the direction of the effect, but the word \"significantly\" should be replaced with a descriptive statement or supported by paired tests over the 600 question instances.","section":"§7.3, Table 2"}],"minor_comments":[{"comment":"The number of Monte Carlo iterations m in Algorithm 1 is never specified; report the value used, the LLM used for counterfactual answers, and the computational cost.","section":"Algorithm 1, §5"},{"comment":"The enumeration that with-clusters is best in 5/8 cases and without-clusters in 2/8 omits the slice where the naive baseline is best (English questions, 0.807); the counts should cover all three methods or be phrased as 5/2/1.","section":"§7.3"},{"comment":"The abstract says \"300 hand-created conversational questions, each in English and German,\" while §7.2 refers to 600 English and German questions; clarify that there are 300 questions in two languages, yielding 600 test instances.","section":"Abstract, §6"},{"comment":"Answer relevance is judged by GPT-4o, the same model family used for generation; a small human-agreement study would strengthen the answer-quality results.","section":"§7.1"},{"comment":"The contextualization experiments compare against no context only; adding common alternatives such as coreference resolution and sliding-window chunking as baselines would better support the claim that the proposed contextualization scheme is effective relative to existing practice.","section":"§4, §7.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a demo/system contribution, and the benchmark and code are genuine assets. My recommendation is driven by the attribution evaluation and the selection-on-test-set issue, not by the contextualization result, which I find convincing in direction. I would be satisfied with a revision that adds paired significance tests, reports Table 3 on all 600 questions with excluded cases scored as failures, and moves configuration and parameter choices off the test set; the \"causal\" language may need softening unless evidence-level evaluation is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the ConfQuestions benchmark is the real contribution, and the contextualization results are solid. The counterfactual attribution claim is not yet supported by the evidence as presented.\n\nThe benchmark is genuinely useful. 300 conversational questions in EN and DE over 215 Confluence pages, balanced across passages, lists, and tables, simple and complex, with hand translations and author annotations. That is a lot of careful work, and it fills a real gap for RAG evaluation on heterogeneous enterprise wikis. The system is open source, so others can reproduce and build on it.\n\nThe contextualization finding is credible. Table 2 shows consistent gains from adding page title, heading, and neighbor text across every slice—complexity, answer type, language. No significance tests are reported, and gains are modest for some slices, but the pattern is coherent and matches the motivation. Adding context at indexing time is a simple trick that clearly helps here.\n\nThe soft spot is the attribution evaluation. Accuracy is defined as URL match between the top-attributed evidence and the gold answer URL—a page-level proxy, not evidence-level correctness, and the authors admit gold answers are often nuggets spread over a document. They also drop the 236/600 questions where the gold URL was not retrieved in the top-10, so the numbers are conditional on retrieval success. On the remaining 364 questions, the best counterfactual variant gets 0.799 vs 0.772 for naive similarity. That difference is within noise without confidence intervals or a paired test. The temperature parameter is explicitly set to skew distributions toward the average number of gold evidences in ConfQuestions, which further inflates the apparent advantage. Because this is the only evidence for the causal attribution claim, that half of the abstract is not secured.\n\nThat said, the paper is honest about these limits—the URL proxy is acknowledged, and the filtering is explained. The authors simply stretch the conclusion further than the evaluation supports.\n\nWho benefits: RAG practitioners and anyone building enterprise QA will get ideas and a benchmark worth trying. A serious referee should engage, but the attribution claim needs strengthening: report significance, handle excluded questions as failures, and if possible annotate evidence-level labels. As is, I'd treat the attribution results as a promising pilot, not a demonstrated win.\n\nRecommendation: send to peer review with a request for a rewritten attribution evaluation.","headline":"ConfQuestions is the real contribution; contextualization adds a modest, credible finding; the counterfactual attribution claim is not yet supported by the evaluation.","tokens_in":12638,"tokens_out":2881,"would_cite":true,"duration_ms":25009,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prepending page context to every retrieved evidence improves RAG retrieval and answer quality, and that counterfactual evidence-removal attribution explains answers more accurately than similarity-based attribution.","keywords":["retrieval augmented generation","conversational question answering","evidence contextualization","counterfactual attribution","explainability","heterogeneous documents","tables","benchmark"],"falsifier":"Run the same counterfactual attribution on a set of questions with hand-labeled evidence-level gold, including questions where the gold page falls outside the top ten; if the top-scoring evidence then matches at no better than the naive cosine attribution baseline, the central claim that counterfactual attribution outperforms standard attribution would be undercut.","tokens_in":11693,"feed_emoji":"🔎","tokens_out":5500,"duration_ms":49214,"temperature":0.7,"pith_summary":"RAGonite is a RAG system for conversational question answering over enterprise-style wikis that mix passages, lists, and tables. The paper tries to show that two simple interventions fix two common RAG weaknesses: prepend document context to each evidence at indexing time, and explain answers by removing evidence clusters and measuring how much the generated answer changes. On a new benchmark called ConfQuestions, with 300 conversational questions in English and German tied to 215 public wiki pages, the contextualization intervention improves retrieval Precision@1 from 0.398 to 0.528 and answer relevance from 0.388 to 0.529, and the counterfactual attribution beats naive cosine attribution on the paper's URL-match accuracy. The paper also releases the benchmark and the full pipeline. A reader should care because these are cheap, model-agnostic changes that could make enterprise RAG answers more accurate and their explanations causal rather than merely plausible.","feed_headline":"Adding page context improves RAG; removing evidence explains answers","feed_subtitle":"A RAG system with contextualized evidence outperforms plain chunks, and counterfactual attribution beats cosine similarity for answer…","key_machinery":"The central mechanism is evidence contextualization at indexing time: each evidence (passage, list, verbalized table row, or whole table) is stored as its raw content plus the page title, previous heading, preceding evidence, and following evidence, so localizing words that live outside the chunk are visible to both the retriever and the LLM. The second central mechanism is counterfactual attribution (Algorithm 1): the retrieved top-ten evidences are first clustered by semantic redundancy; each cluster is removed one at a time, the LLM regenerates the answer with that cluster absent, and the cluster's contribution is one minus the mean cosine similarity between the original and counterfactual answers, normalized by masked softmax. Clustering is essential because removing one of two redundant chunks would otherwise wrongly count the other as the sole cause of the answer.","core_discovery":"The paper's central claim has two halves. First, indexing each retrieved unit together with its document context—page title, the heading immediately above it, the preceding evidence, and the following evidence—makes both retrieval and answer generation better than indexing the raw chunk alone, with the page title being the single most useful addition and no context component hurting performance. Second, attribution of an answer to evidence is better done counterfactually than by cosine similarity: remove a cluster of similar evidence from the prompt, regenerate the answer, and take one minus the average similarity between the original and regenerated answers as that evidence's contribution. This reaches roughly 0.80 accuracy on the paper's URL-match metric, beating the naive similarity baseline.","pith_inferences":["The reported gains may be largest on corpora like enterprise wikis, where pages often reuse terms from titles and headings; on flatter corpora the benefit of added context could shrink, but this is an empirical question the benchmark does not settle.","The attribution distribution is only as causal as the LLM's regenerations are faithful to the evidence pool; if the model ignores the removed evidence or falls back on parametric knowledge, the estimated contributions are biased, and the paper's out-of-scope checks were not directly tied to attribution steps.","The URL-match evaluation could be sharpened by evidence-level gold labels; until then, the attribution accuracy numbers are an upper bound on locating the exact supporting chunk, and a finer-grained answer-nugget deletion variant could yield more actionable explanations."],"forward_implications":["If contextualization generalizes, any RAG system that currently indexes bare chunks can gain retrieval and answer quality simply by storing each chunk with its title, heading, and neighbors, with no model changes.","Because the page title alone gives the largest single gain and no context component hurts, content-management pipelines could adopt context enrichment as a default preprocessing step.","Counterfactual attribution gives users explanations tied to how the answer would change, not to how similar the answer is to the evidence, so explanations should be more reliable when redundant evidence is present.","Verbalizing tables and indexing both whole tables and individual rows is a viable way to bring heterogeneous documents into RAG.","The ConfQuestions benchmark provides a reusable testbed for comparing RAG systems on mixed passage, list, and table evidence in two languages."],"supporting_citations":[{"why":"Provides the contemporary counterfactual-attribution baseline whose evaluation style the paper contrasts with its own URL-match accuracy.","marker":"[2]"},{"why":"Supplies reciprocal rank fusion, the ranking-combination method used in RAGonite's hybrid retrieval.","marker":"[3]"},{"why":"Defines retrieval-augmented generation, the paradigm the whole system is built on.","marker":"[5]"},{"why":"Supplies the verbalization scheme used to linearize table rows into LLM-friendly evidence.","marker":"[8]"},{"why":"Introduces counterfactual explanations for model outputs, the conceptual source of the evidence-removal attribution idea.","marker":"[15]"}],"fun_headline_variants":["Page context lifts RAG; removing evidence explains answers","Counterfactual attribution outperforms similarity in RAG","RAGONITE: contextualize evidence, then counterfactually attribute","For better RAG, add page context; for better explanations, remove evidence","Contextualized evidence and counterfactual attribution improve RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attribution evaluation assumes that matching the source URL of the highest-scoring evidence to the gold answer URL is a faithful proxy for having found the correct evidence, and it only measures questions for which the gold page was already in the retrieved top ten.","fun_headline_variants_meta":{"raw":{"variants":["Page context lifts RAG; removing evidence explains answers","Counterfactual attribution outperforms similarity in RAG","RAGONITE: contextualize evidence, then counterfactually attribute","For better RAG, add page context; for better explanations, remove evidence","Contextualized evidence and counterfactual attribution improve RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1602,"prompt_tokens":981,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":534}},"tokens_in":597,"tokens_out":621,"duration_ms":5553,"temperature":1.0,"reasoning_tokens":534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:50:38.739884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same counterfactual attribution on a set of questions with hand-labeled evidence-level gold, including questions where the gold page falls outside the top ten; if the top-scoring evidence then matches at no better than the naive cosine attribution baseline, the central claim that counterfactual attribution outperforms standard attribution would be undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the contemporary counterfactual-attribution baseline whose evaluation style the paper contrasts with its own URL-match accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies reciprocal rank fusion, the ranking-combination method used in RAGonite's hybrid retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the paradigm the whole system is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the verbalization scheme used to linearize table rows into LLM-friendly evidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces counterfactual explanations for model outputs, the conceptual source of the evidence-removal attribution idea."}],"review_version":1}