{"id":"e016ea2e-9e59-4bf6-a65b-3f4229672a54","arxiv_id":"2504.12560","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CDF-RAG combines RL-based query refinement, causal graph retrieval, and hallucination checking to improve accuracy and causal consistency in retrieval-augmented question answering.","lead":"This paper introduces CDF-RAG, a retrieval-augmented generation system that rewrites questions with reinforcement learning, retrieves from text and a causal graph, and verifies answers against causal pathways. The authors report accuracy gains over prior RAG methods on four question-answering datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test split is reported: the causal graph, instruction-tuning, and RL reward training are built from the four evaluation benchmarks, so the reported accuracy and CRC gains may reflect benchmark leakage rather than causal reasoning.","rationale":"The paper's central claim is empirical state-of-the-art accuracy and causal retrieval coverage. The reader's weakest assumption identifies the same core issue: the evaluation may not be independent of the training and graph-construction data. My reading of the appendices confirms that the causal graph, instruction-tuning data, and RL training curriculum are all derived from the four evaluation datasets with no stated split, and that GPT-4 is used both to construct the system's causal graph and to define the gold standard for CRC. This is not a disagreement about causal methodology conventions; it is an internal validity problem that directly threatens the reported gains. The absence of error bars or significance tests further weakens confidence, but the leakage risk is the more load-bearing concern because it would invalidate the headline accuracy numbers even if the architecture works as described. The public code link is a positive sign, and a concrete split-based re-evaluation could settle the issue. Since the current manuscript provides no evidence that the evaluation queries are disjoint from training/graph data, I agree with the reader's rejection and would not change the verdict until the authors provide a clean held-out evaluation or demonstrate that the released implementation already respects such a split.","tokens_in":21636,"tokens_out":3342,"duration_ms":33610,"concrete_test":"Inspect the public repository (github.com/elakhatibi/CDF-RAG) for dataset split definitions. If no held-out split exists, re-run the full pipeline with a strict split: use one half of each dataset for graph construction and query-refinement training and the other half for evaluation; report accuracy, CRC, and F1 on the held-out half with standard errors. If the held-out numbers are substantially lower than Table 1 (e.g., the MedMCQA GPT-4 accuracy drops from 0.94 toward the 0.78 Gym-RAG baseline), leakage is confirmed. Also verify that the GPT-4-verified gold-standard causal graph used for CRC is built from held-out evidence only, not from the same graph used by the system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—state-of-the-art accuracy across all four benchmarks—rests on an evaluation that is not separated from training. Appendix A.1 states that the causal graph and query-refinement prompts are constructed from CosmosQA, MedQA, MedMCQA, and AdversarialQA; Appendix A.3 states the PPO query-refinement agent is trained by 'sampling queries from MedQA, CosmosQA, and AdversarialQA'; Appendix B.1 fine-tunes all non-GPT-4 backbones on the same 'multi-task dataset.' No train/test split is defined anywhere, and Appendix B.4 reports evaluation on 'approximately 2,200 queries spanning four benchmark datasets' without stating that these queries are disjoint from the training or graph-construction data. Consequently, the RL agent and fine-tuned generators may have been optimized on the exact queries and answers used for evaluation, and the Neo4j causal graph may encode test-set passages. The same GPT-4 verification builds both the system's causal graph (Section 2.1) and the gold-standard graph used to compute CRC (Section 4.2), making the causal metric circular. Any of these alone would inflate the reported numbers; together they undercut the generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CDF-RAG, a retrieval-augmented generation framework that adds a causal knowledge graph, reinforcement-learning-based query refinement, dual-path semantic/causal retrieval, response verification against causal structures, and hallucination correction. The authors evaluate CDF-RAG on CosmosQA, MedQA, MedMCQA, and AdversarialQA with GPT-4, LLaMA 3-8B, Mistral, and Flan-T5, reporting state-of-the-art accuracy, causal retrieval coverage, and hallucination metrics.","tokens_in":21893,"tokens_out":5680,"duration_ms":54406,"significance":"If its empirical claims were supported, CDF-RAG would be a useful contribution to causality-aware RAG: the integration of RL-driven query rewriting, structured causal graph traversal, and verifier-based correction is a coherent architectural idea, and the public code release is a positive step. However, the current evidence does not support the stated conclusions because the evaluation protocol appears to train and construct knowledge resources on the same benchmarks used for testing, and the causal metrics are circular. The significance of the claimed improvements therefore cannot be assessed from the reported results.","major_comments":[{"comment":"No train/test split is reported anywhere. Appendix A.1 states that the causal graph and query-refinement prompts are constructed from CosmosQA, MedQA, MedMCQA, and AdversarialQA; Appendix A.3 trains the PPO query-refinement agent by sampling queries from MedQA, CosmosQA, and AdversarialQA; Appendix B.1 fine-tunes every non-GPT-4 backbone on the same multi-task dataset; and Appendix B.4 reports evaluation on approximately 2,200 queries from these four benchmarks without asserting that they are disjoint from the training or graph-construction data. As a result, the graph, the RL policy, and the fine-tuned generators may all encode information from the evaluation sets, making the accuracy and causal-coverage gains in Tables 1 and 2 uninterpretable as generalization results.","section":"Appendix A.1, A.3, B.1, B.4"},{"comment":"The Causal Retrieval Coverage metric is circular. Section 4.2 defines CRC by checking retrieved items against a 'gold-standard causal graph constructed using GPT-4 verification,' while Section 2.1 builds the system's own causal graph using the same GPT-4 verification procedure. There is no independent reference, so CRC measures agreement with GPT-4 rather than causal coverage with respect to an external ground truth. The reported perfect CRC score of 1.00 on MedMCQA therefore does not constitute evidence of causal retrieval quality unless an independently constructed graph is provided.","section":"Section 4.2 vs. Section 2.1"},{"comment":"The formal definitions of the reported metrics are underspecified. The causal consistency score Scausal uses an indicator I(C→E ⊨ ŷ) with no operational definition of entailment between a causal triple and a generated response; the hallucination score Shallucination depends on an unexplained claim extraction function Y and a set intersection K∩Y; and the groundedness metric in Section 4.2 is described only as 'span-level alignment.' Without these definitions, the metrics cannot be reproduced or verified, and the claims based on them are not falsifiable as stated.","section":"Sections 2.4 and 2.5"},{"comment":"The headline claim that 'across all datasets and models, CDF-RAG achieves the highest accuracy scores' is supported only by point estimates. No standard errors, confidence intervals, significance tests, or per-cell evaluation sizes are reported for Table 1 or Tables 4 and 5, and the total of roughly 2,200 queries in Appendix B.4 is not tied to the per-method and per-dataset results. The observed margins, such as the 16% accuracy gain over Gym-RAG on MedMCQA, need statistical support before they can be interpreted as robust improvements.","section":"Table 1 and Section 4.1"},{"comment":"Hallucination Rate is defined as the percentage of responses flagged by the LLM verifier, which is the same mechanism used for hallucination correction within CDF-RAG. This makes HR a measure of consistency with the system's own verifier rather than an objective measure of hallucination. The 'lowest HR' claims in Tables 4 and 5 require an external or human-annotated hallucination benchmark to be meaningful.","section":"Appendix B.4 and Section 4.3"}],"minor_comments":[{"comment":"Several references use placeholder citations such as '(et al., 2020)', '(et al., 2025)', and '(et al., 2024)' instead of proper author names; these need to be completed.","section":"References"},{"comment":"Table 2 omits the MedQA rows for both CRC and Context Relevance, even though the text and Figure 3 report MedQA-specific results; the table should be completed or the omission should be explained.","section":"Table 2"},{"comment":"The sentence 'The improvement is especially notable with GPT-4 and GPT-4 Mistral LLaMA 3-8B Flan-T5' is garbled and should be rewritten to list the model names correctly.","section":"Figure 3"},{"comment":"The bullet point 'We introduceCDF-RAG' is missing a space, and the abstract's statement that existing RAG frameworks 'primarily rely on semantic similarity and correlation-driven retrieval' is an overgeneralization that should be qualified in light of existing structured and causality-aware RAG work.","section":"Introduction and Contributions"},{"comment":"The prompt templates appear more than once, with the Simplification Prompt Template repeated in slightly different wording; please consolidate these into a single canonical version.","section":"Appendix A.4"}],"recommendation":"reject","confidential_remarks":"The manuscript's central empirical claim cannot be accepted without a complete redo of the experimental protocol: the training and evaluation sets must be disjoint, the causal gold standard must be independent of the system's construction procedure, and the metrics need clear operational definitions and statistical support. The publicly available code is a useful asset for such a revision, but as submitted, the reported numbers do not provide reliable evidence for the claimed state-of-the-art performance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the architecture is a legitimate integration of known pieces: RL-based query refinement, dual-path retrieval over a causal graph and a vector store, and a hallucination-correction loop. The authors ablate each component and release code, which is more than many papers in this area do. Second, the empirical claims as presented are not trustworthy. The evaluation leaks information from the test benchmarks into training and graph construction, and the main causal metric measures agreement with GPT-4 rather than with independent ground truth.\n\nThe novelty is modest but real. The closed feedback loop—query refinement influencing retrieval, retrieval influencing verification, verification triggering regeneration—is a sensible engineering contribution over single-shot causal RAG or static refinement methods. If the numbers were solid, this would be a useful framework for medical QA and other domains where causal coherence matters. The ablations show monotonic gains as modules are added, which is a good sign that the pieces do something.\n\nThe soft spots are load-bearing. Appendix A.1 builds the causal graph from the four evaluation datasets, Appendix A.3 trains the RL agent by sampling queries from three of them, and Appendix B.1 fine-tunes all non-GPT-4 backbones on a multi-task dataset derived from the same sources. Appendix B.4 evaluates on roughly 2,200 queries across those four datasets without ever stating that the evaluation queries are disjoint from the training data. That is a textbook leakage risk, and it directly undermines the 'state-of-the-art accuracy across all datasets and models' claim. The causal metric is also circular: the gold-standard graph for CRC is constructed using GPT-4 verification, and the system's own graph is verified by GPT-4, so the metric mostly measures agreement with GPT-4's judgments. Hallucination rate is defined as answers the LLM verifier flags, which is again the system evaluating itself. There are no error bars or significance tests anywhere, and key pieces like the entailment indicator and the claim set Y are never operationalized.\n\nThese are not minor presentation issues. They go to the validity of the central claim. The framework itself is coherent and could work, but the current evidence cannot distinguish real gains from benchmark overfitting and circular scoring.\n\nWho should read this? Someone working on causal RAG or query refinement might read it for architectural ideas, with the numbers treated as entirely unverified. I would not cite the results in my own work yet.\n\nMy recommendation: send it to peer review, but with a clear expectation of major revision. The reviewers should demand a proper train/test split, an independent causal gold standard, and significance testing. If the authors can supply those, the paper has a chance. If not, it should not be accepted.","headline":"A plausible RAG integration whose central accuracy claim is undermined by a missing train/test split and a circular causal metric.","tokens_in":22418,"tokens_out":1865,"would_cite":false,"duration_ms":20972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dynamic feedback loop that refines queries with reinforcement learning, retrieves over a causal graph as well as text passages, and verifies generated answers against causal paths yields the highest accuracy across four…","keywords":["retrieval-augmented generation","causal reasoning","query refinement","reinforcement learning","causal graph retrieval","hallucination detection","question answering"],"falsifier":"Rebuild the causal graph and the query-refinement training data from sources that exclude every evaluation question and its answer, then rerun the four-benchmark comparison; if the accuracy and causal-coverage margins collapse, the gains came from the graph encoding answer-related information rather than from the feedback loop.","tokens_in":21426,"feed_emoji":"🧠","tokens_out":7315,"duration_ms":67774,"temperature":0.7,"pith_summary":"Retrieval-augmented generation often retrieves topically relevant but causally shallow evidence. CDF-RAG claims to fix this by turning retrieval into a closed loop: a reinforcement-learned policy rewrites the user's question, a dual-path retriever pulls both vector-similar passages and directed causal paths from a prebuilt graph, and a causal check verifies the answer before it is delivered. The paper reports that this loop outperforms standard and query-refining RAG methods on four question-answering benchmarks spanning commonsense, adversarial, and biomedical reasoning, across several language-model backbones. If the results hold, causal structure could become a standard component of retrieval rather than an optional add-on.","feed_headline":"RL query rewriting plus causal graphs tops RAG on four QA benchmarks","feed_subtitle":"A reinforcement-learned query refiner plus causal-graph checks cuts hallucinations and lifts accuracy across models.","key_machinery":"The load-bearing mechanism is the causal dynamic feedback loop. A PPO-trained policy maps the query embedding to one of three refinement actions—expand, simplify, or decompose—then retrieval runs over two paths at once: dense semantic search over passage embeddings and traversal of a directed causal graph built from verified cause-effect triples. The retrieved union is rewritten and fed to the generator, and a causal consistency score checks whether the generated claims are entailed by the retrieved causal paths; if the score falls below a threshold, generation is retried under stricter grounding constraints. A separate hallucination score triggers a similar correction. This loop is what the paper claims converts correlation-based retrieval into causality-aware generation.","core_discovery":"The paper's central discovery claim is that injecting causal structure into every stage of a RAG pipeline—query rewriting, retrieval, and answer verification—produces consistently higher accuracy than retrieval based on semantic similarity alone. Concretely, CDF-RAG reports the highest accuracy on all four evaluated benchmarks with four different LLM backbones, with gains such as 16 percent over the strongest baseline on the biomedical dataset when using GPT-4. It also reports higher causal retrieval coverage, higher groundedness, and lower hallucination rates, attributing the gains to the feedback loop rather than to any single module.","pith_inferences":["If the causal graph is built from the same benchmarks used for evaluation, part of the measured gain may be graph exposure rather than new causal reasoning; a transfer test with an externally built graph would separate the two.","The framework amplifies whatever causal coverage the graph has; it does not itself discover missing causal links, so its guarantees are upper-bounded by graph quality.","A natural extension is to apply the same loop to open-domain RAG where no domain graph exists, using the vector retriever alone as a fallback; the comparison would show when graph construction pays for itself."],"forward_implications":["Across all four datasets and all four tested backbones, CDF-RAG reports the highest accuracy among the compared RAG systems, including on biomedical and adversarial questions.","Causal retrieval coverage reaches near-perfect levels on the biomedical benchmark (1.00 with GPT-4), suggesting the graph traversal routinely surfaces query-aligned cause-effect paths.","The stepwise ablation shows each added component—RL refinement, causal graph, rewriter, hallucination correction—improves F1 and groundedness while lowering hallucination rate from 0.18 to 0.07.","Because the final answer is checked against retrieved causal paths before delivery, generation is steered toward evidence-backed causal claims rather than plausible co-occurrence text."],"supporting_citations":[{"why":"Supplies the causal-pair classifier used to extract cause-effect triples for graph construction.","marker":"Tan et al., 2023"},{"why":"Supplies the commonsense reasoning dataset used as one of the four evaluation benchmarks.","marker":"Huang et al., 2019"},{"why":"Supplies the medical question-answering dataset used as a second evaluation benchmark.","marker":"Jin et al., 2020"},{"why":"Supplies the biomedical multi-subject dataset used as the fourth evaluation benchmark.","marker":"Pal et al., 2022"},{"why":"Supplies the adversarial reading-comprehension dataset used as the third evaluation benchmark.","marker":"Bartolo et al., 2020"},{"why":"Supplies the PPO algorithm used to train the query-refinement policy.","marker":"Schulman et al., 2017"},{"why":"Supplies the dense passage retrieval approach that anchors the semantic retrieval pathway.","marker":"Karpukhin et al., 2020"},{"why":"Supplies RQ-RAG, a query-refinement baseline that CDF-RAG is compared against.","marker":"Chan et al., 2024"},{"why":"Supplies RAG-Gym, the reward-guided refinement baseline that CDF-RAG is compared against.","marker":"et al., 2025"}],"fun_headline_variants":["Causal feedback loop sharpens RAG accuracy across four benchmarks","Causal checks in RAG cut hallucinations and lift accuracy","Iterative causal refinement improves RAG on multiple QA sets","Causal graphs strengthen RAG reasoning and reduce errors","CDF-RAG: dynamic causal feedback yields more accurate answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal metrics assume that the causal graph built with help from the same LLM that later checks the answers is an independent source of causal truth.","fun_headline_variants_meta":{"raw":{"variants":["Causal feedback loop sharpens RAG accuracy across four benchmarks","Causal checks in RAG cut hallucinations and lift accuracy","Iterative causal refinement improves RAG on multiple QA sets","Causal graphs strengthen RAG reasoning and reduce errors","CDF-RAG: dynamic causal feedback yields more accurate answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2592,"prompt_tokens":854,"completion_tokens":1738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1656}},"tokens_in":470,"tokens_out":1738,"duration_ms":14849,"temperature":1.0,"reasoning_tokens":1656,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:28:39.760187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the causal graph and the query-refinement training data from sources that exclude every evaluation question and its answer, then rerun the four-benchmark comparison; if the accuracy and causal-coverage margins collapse, the gains came from the graph encoding answer-related information rather than from the feedback loop.","supporting_citations":[],"review_version":1}