{"id":"49101c81-129e-49fa-a9ec-9caeb9917b00","arxiv_id":"2506.16988","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent RAG framework with hybrid retrieval and a citation-generating reviser improves faithfulness by 10.7% over standard RAG on a synthetic 500-question QA dataset.","lead":"RAGentA is a four-agent retrieval-augmented generation system that adds in-line citations and a revision step, reporting 10.7% higher faithfulness than a standard RAG baseline. The paper, entered in the 2025 SIGIR LiveRAG challenge, evaluates its claims on a single 500-question synthetic benchmark built from the FineWeb index.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Faithfulness gain may be an artifact of asymmetric judge input: RAGentA's answers are scored against their self-selected cited passages, while uncited Standard RAG is scored against the full retrieved set.","rationale":"The core claim is that RAGentA's four-agent architecture with in-line citations improves faithfulness by 10.7% over standard RAG. What would have to be true is that the faithfulness measurement compares the two systems fairly. The manuscript's own evaluation description leaves a gap: only RAGentA has citations, so 'cited passages' is well-defined for it, while the baseline has no citations and no stated judge-input protocol. The most plausible reading—that the judge receives all retrieved documents for Standard RAG but only the cited subset for RAGentA—makes the comparison asymmetric and could mechanically produce the reported gap. This is a correctness-risk issue, not an outside-consensus disagreement: it is about whether the metric measures the same quantity across systems. I therefore agree with the CONDITIONAL verdict, and the proposed symmetric-protocol experiment is a cheap, decisive check. Credit where due: the paper reports its own negative finding (second-stage retrieval limited value), discloses the computational overhead, and makes code public; nothing here suggests selective reporting, only an underspecified evaluation protocol.","tokens_in":8946,"tokens_out":5466,"duration_ms":54941,"concrete_test":"For each of the 500 test questions, use the shared top-20 retrieved document set (the same set supplied to both systems) as the sole passage context for the faithfulness judge for both RAGentA and Standard RAG, ignoring RAGentA's in-line citations when constructing the judge input. Recompute the mean faithfulness scores under this symmetric protocol. If the 0.7044 vs 0.6362 gap collapses or shrinks by more than half, the reported +10.7% faithfulness advantage is an artifact of giving RAGentA a self-selected passage subset rather than evidence that the multi-agent pipeline produces better-grounded answers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 states that the LLM judge 'receives the predicted answer, the ground-truth answer, and the cited passages.' For RAGentA the cited passages are the documents picked out by its in-line citations. Standard RAG 'does not incorporate agent-based reasoning' and produces no citations, so the paper never specifies what passage context the judge was given for the baseline; the only controlled element is the initial retrieved document set. If Standard RAG was scored against the full top-20 set while RAGentA was scored against only the passages it chose to cite, the faithfulness metric is not measuring the same object for the two systems: selective citation can raise 'grounded in the provided passages' even if the answer is no better grounded in the shared corpus. The +10.7% faithfulness gap (0.7044 vs 0.6362) could then be an evaluation-protocol artifact rather than a property of the multi-agent architecture, and this confound would persist even if the Llama-3.3-70B judge agreed perfectly with the official Claude-based autoevaluator.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes RAGentA, a training-free multi-agent RAG framework for attributed question answering. Retrieval combines BM25 and E5 via a weighted interpolation (Eq. 1, alpha=0.35) to select top-20 documents. Four agents then work in sequence: Agent-1 generates per-document answers, Agent-2 filters documents using a log-probability relevance score with a dynamic threshold (n=0.5), Agent-3 produces a final answer with in-line citations, and Agent-4 checks completeness, optionally reformulates the query, retrieves additional documents, and merges revised answers. The system is evaluated on 500 synthetic QA pairs generated with DataMorgana from the FineWeb index. Reported results are Recall@20 of 0.5650 for the hybrid retriever versus 0.5020 for BM25 (+12.5%), correctness of 0.8346 versus 0.8256 for Standard RAG (+1.1%), and faithfulness of 0.7044 versus 0.6362 (+10.7%). The paper concludes that the multi-agent architecture and hybrid retrieval improve grounding and answer quality.","tokens_in":9146,"tokens_out":3585,"duration_ms":35287,"significance":"If the evaluation is sound, the paper offers a useful, reproducible, training-free recipe for attributed QA. The hybrid retrieval result is clearly positive and the architectural extension of MAIN-RAG with citation generation and a reviser agent is a plausible mechanism for improving faithfulness. Strengths include public code, adoption of previously published hyperparameters rather than test-set tuning, and an honest discussion of the small correctness gain and computational overhead. The central caveat is that the headline faithfulness and correctness comparison rests entirely on a local LLM judge whose input context appears asymmetric between the two systems and which has not been validated against the official LiveRAG autoevaluator; until that is addressed, the +10.7% faithfulness advantage is not yet demonstrated.","major_comments":[{"comment":"The paper does not specify which document context the LLM judge receives for the Standard RAG baseline. For RAGentA the 'cited passages' are the documents selected by its in-line citations, while Standard RAG produces no citations; if the baseline was scored against the full top-20 retrieved set, the faithfulness metric measures different objects for the two systems. Selective citation can raise 'grounded in the provided passages' even if the answer is no better grounded in the shared corpus. Please state the exact judge input for both systems and re-run the comparison with identical passage sets (e.g., full top-20 for both, or the baseline's own extracted citations), and report both settings.","section":"§5.2, Table 3"},{"comment":"The Llama-3.3-70B-Instruct judge is substituted for the closed-source Claude-3.5 Sonnet autoevaluator without any agreement study, calibration check, or human audit. The entire reported correctness and faithfulness comparison depends on this judge agreeing with the official evaluator, and citation-dense outputs could be rated more leniently by the local judge. Please report an agreement metric (e.g., Cohen's kappa or Spearman correlation) on a sample, or a human-audited subset of 50-100 items, and show that the relative ranking of RAGentA versus Standard RAG is stable under the official or human scoring.","section":"§5.2"},{"comment":"All conclusions are based on a single synthetic dataset of 500 questions with no variance estimates or significance tests. The MRR@20 gain (+2.0%) and correctness gain (+1.1%) are small and could easily arise from sampling noise. Please report bootstrap confidence intervals or paired significance tests (e.g., Wilcoxon signed-rank) for the retrieval, correctness, and faithfulness comparisons, and ideally validate on an established benchmark such as ALCE or the official LiveRAG evaluation set.","section":"§5.1, §5.2"}],"minor_comments":[{"comment":"Table 3 reports RAGentA correctness as 0.8346, while the text in §5.2 states 0.8348; please reconcile the discrepancy.","section":"Table 3, §5.2"},{"comment":"The abstract reports +1.09% and +10.72%, while §5.2 reports +1.1% and +10.7%; unify the numerical precision.","section":"Abstract, §5.2"},{"comment":"Figure 1 is dense and the paths from Agent-4 to reformulation, retrieval, and answer merging are difficult to follow; a step-numbered legend or a simplified sequence diagram would improve readability.","section":"Figure 1"},{"comment":"The alpha=0.35 value is taken from a regulatory-text study [13]; briefly justify why this value transfers to open-domain QA over FineWeb, or report sensitivity to alpha in the retrieval results.","section":"§4.1, Eq. (1)"},{"comment":"The 'completeness check' in Agent-4 and the 'coverage' component of the correctness metric are conceptually related but not formally connected; a sentence defining how completeness is operationalized would help the reader.","section":"§4.2.4"}],"recommendation":"major_revision","confidential_remarks":"This is a LiveRAG challenge system description, and the evaluation scale is typical for such contributions; nevertheless, the asymmetric judge-input issue in §5.2 is a correctness problem, not a presentation problem, and must be fixed before the faithfulness claim can be accepted. The related work is adequate and the framing as an extension of MAIN-RAG is appropriate. I would not require a new benchmark, but I would require an explicit protocol statement and a judge-agreement or human-audit check."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: RAGentA is a clearly written LiveRAG challenge report. It does one useful thing—it takes MAIN-RAG and adds a fine-grained citation agent and a completeness-checking reviser—but its central faithfulness claim is not yet established because the evaluation may have measured the two systems against different passage sets.\n\nWhat's new: the four-agent setup, with Agent-3 generating in-line citations and Agent-4 decomposing the question, checking coverage, and launching targeted second-stage retrieval, is a concrete extension of MAIN-RAG. Hybrid retrieval (BM25+E5, alpha=0.35 from prior work) gives a +12.5% Recall@20 gain that looks solid. The authors also state plainly that second-stage retrieval currently adds limited value and that the pipeline is computationally heavy. That honesty matters.\n\nSoft spots, in order of size. First, the faithfulness comparison is asymmetric. The judge receives \"the predicted answer, the ground-truth answer, and the cited passages.\" For RAGentA, those are the passages its own citations select. Standard RAG produces no citations, and the paper never says what passage set the baseline was scored against. If the baseline saw the full top-20 and RAGentA saw only its self-selected subset, the +10.7% faithfulness gap is an artifact. The stress-test note lands; the paper doesn't rule this out. Second, the Llama-3.3-70B judge is a proxy for the official Claude autoevaluator with no agreement study. Third, the benchmark is one synthetic 500-question set without error bars, and MAIN-RAG itself is not a reported baseline. The architecture's individual contributions are therefore hard to isolate.\n\nWho gets value: someone wanting a concrete, training-free multi-agent RAG recipe, or someone teaching evaluation design in RAG. The related work is well covered and the code is public.\n\nRecommendation: I'd send it to peer review—the architecture is real and the evaluation problem is fixable—but the faithfulness experiment needs to be redone with matched judged-context for both systems. If the gap survives that, it's a decent workshop contribution; if not, the paper still has a useful negative lesson.","headline":"Four-agent RAG pipeline with fine-grained citations is clearly described, but the headline faithfulness gain may be an evaluation artifact because the judge likely saw different passage sets for baseline vs. system.","tokens_in":9693,"tokens_out":2787,"would_cite":false,"duration_ms":26874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-agent RAG pipeline reports +10.7% faithfulness over standard RAG, with correctness nearly unchanged.","keywords":["retrieval-augmented generation","multi-agent systems","attributed question answering","in-line citations","faithfulness","hybrid retrieval","answer correctness","large language models"],"falsifier":"Mechanically append [X] citation markers to Standard RAG answers without changing their text and re-run the same LLM judge; if faithfulness jumps toward RAGentA's 0.70, the judge is rewarding formatting, not grounding. Alternatively, have human raters score a random sample of the 500 answers and compare the faithfulness gap.","tokens_in":8726,"feed_emoji":"🤖","tokens_out":8229,"duration_ms":78662,"temperature":0.7,"pith_summary":"This paper tries to show that a retrieval-augmented generation system can produce answers that are both correct and faithfully grounded in sources by splitting the work among four specialized LLM agents. The main claim is that adding an agent that writes in-line citations and a reviser agent that checks whether every part of the question has been answered materially improves faithfulness, the degree to which answers are supported by retrieved documents. On the authors' 500-question synthetic benchmark, the framework scores 0.7044 for faithfulness against 0.6362 for a standard single-prompt RAG pipeline (+10.7%), while correctness rises only slightly from 0.8256 to 0.8348 (+1.1%). The framework also combines sparse and dense retrieval, improving Recall@20 by 12.5% over the best single retriever. If the gains hold under independent evaluation, the framework offers a training-free recipe for making LLM answers verifiable in settings where users need to check sources.","feed_headline":"Four-agent RAG lifts faithfulness 10.7% over standard RAG","feed_subtitle":"A reviser agent adds in-line citations and second-pass retrieval so answers are better grounded in source documents.","key_machinery":"The load-bearing mechanism is the interplay between Agent-3's claim-level citation generation and Agent-4's completeness check. Agent-3 emits each fact as a claim with markers in the standardized [X] format, converting grounding from an implicit property into an output that can be inspected and scored. Agent-4 parses the query into sub-components, tags each one as fully, partially, or not answered, and for unanswered gaps issues follow-up queries to the retrieval system, excluding documents already seen, then merges the supplemental answers. The other load-bearing piece is the hybrid retrieval score $S_{\\text{hybrid}}(d) = \\alpha S_{\\text{sparse}}(d) + (1-\\alpha)S_{\\text{dense}}(d)$ with $\\alpha=0.35$, which fuses exact-term matching with dense embeddings so the agents start from a broader relevant pool. Agent-2's relevance filter, scored as $\\log p(\\text{Yes}) - \\log p(\\text{No})$ with a dynamic threshold, then trims that pool before generation.","core_discovery":"RAGentA's central claim is that reliability in RAG is improved less by better retrieval alone than by making attribution an explicit, inspectable step of generation. Four agents cooperate: an initial predictor generates a candidate answer for each retrieved document; a judge scores each document's usefulness and filters out noisy ones using a dynamic threshold; a final predictor writes the answer as claims, each with inline citation markers; and a reviser agent decomposes the original question, marks each portion as fully, partially, or not answered, and when gaps remain reformulates queries, retrieves new documents, and merges the new material into a unified answer. The authors report that this pipeline yields a +10.7% faithfulness gain over standard RAG with only +1.1% correctness gain, and that the hybrid sparse-dense retriever recovers 12.5% more relevant documents at rank 20 than the best single retriever. The faithfulness figure is the paper's central assertion: claim-level attribution plus second-pass completion creates answers that are better grounded in the retrieved evidence.","pith_inferences":["Implicit in the results but not tested: the faithfulness gap may be inflated by the judge rewarding the mere presence of citation markers; a citation-precision audit, checking whether each [X] marker actually supports its claim, would separate formatting from true grounding.","The framework's advantage likely concentrates on the benchmark's multi-aspect and comparison questions, where Agent-4 can actually find missing pieces; a per-category breakdown would show where the gain lives and where correctness stays flat.","A practical extension suggested by the architecture is to make the number of revision rounds budget-aware, since Agent-4's second-stage retrieval is the main source of the noted computational overhead, and to early-stop when all components are tagged fully answered.","A testable variant would remove Agent-2's relevance filtering and keep only citation generation; if faithfulness stays near 0.70, filtering is not the driver, and simpler pipelines could be used."],"forward_implications":["If the reported scores hold, any RAG deployment that already retrieves top-k documents can adopt the four-agent loop without retraining and obtain more source-grounded answers, at the cost of extra LLM calls.","In-line citations in the standardized [X] format make individual claims auditable, so users can verify each statement against its cited passage rather than trusting the whole answer.","The hybrid retrieval result implies that fusing sparse and dense signals is a cheap way to improve recall before any agent reasoning begins, since it outperforms both single retrievers on the benchmark.","Agent-4's dynamic refinement should matter most for multi-aspect questions, where a single retrieval pass often misses one component; the paper's fully/partially/not-answered decomposition directly targets that failure mode.","Because correctness gains are small and the authors note the second-stage retrieval currently adds limited value, the framework's practical advantage is in trustworthiness rather than raw accuracy."],"supporting_citations":[{"why":"Supplies the underlying three-agent scoring-and-filtering architecture and the log-probability relevance score that this paper extends.","marker":"[2]"},{"why":"Supplies the faithfulness scoring rubric (full, partial, or no support) that the LLM judge applies.","marker":"[3]"},{"why":"Generates the synthetic 500-question benchmark on which all headline numbers are measured.","marker":"[5]"},{"why":"Establishes the approach of generating text with in-line citations that Agent-3's claim-level attribution follows.","marker":"[7]"},{"why":"Defines the standard RAG pipeline that serves as the baseline in the controlled comparison.","marker":"[12]"},{"why":"Provides the hybrid fusion weighting of $\\alpha=0.35$ used in the sparse-dense retriever.","marker":"[13]"},{"why":"Inspires the coverage component of the correctness metric, based on recall of vital information.","marker":"[15]"},{"why":"Supplies the dense embedding model used alongside BM25 in the hybrid retriever.","marker":"[23]"}],"fun_headline_variants":["Multi-agent RAG boosts faithfulness 10.7% over standard RAG","Agentic RAG with inline citations improves faithfulness by 10.7%","Four-agent RAG improves answer grounding, +10.7% faithfulness","RAGentA: multi-agent RAG lifts faithfulness 10.7%, adds citations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire faithfulness advantage rests on the automatic judge used for scoring agreeing with the official challenge evaluator; no agreement study is reported, so a judge that rewards the presence of citation markers could explain the +10.7% gap.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent RAG boosts faithfulness 10.7% over standard RAG","Agentic RAG with inline citations improves faithfulness by 10.7%","Four-agent RAG improves answer grounding, +10.7% faithfulness","RAGentA: multi-agent RAG lifts faithfulness 10.7%, adds citations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3830,"prompt_tokens":957,"completion_tokens":2873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2788}},"tokens_in":573,"tokens_out":2873,"duration_ms":20940,"temperature":1.0,"reasoning_tokens":2788,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:14:19.809533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Mechanically append [X] citation markers to Standard RAG answers without changing their text and re-run the same LLM judge; if faithfulness jumps toward RAGentA's 0.70, the judge is rewarding formatting, not grounding. Alternatively, have human raters score a random sample of the 500 answers and compare the faithfulness gap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the faithfulness scoring rubric (full, partial, or no support) that the LLM judge applies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hybrid fusion weighting of $\\alpha=0.35$ used in the sparse-dense retriever."}],"review_version":1}