{"id":"9bc7ffd4-59f1-41aa-8935-ac6ca7aeee18","arxiv_id":"2505.10143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GE-Chat combines graph-based RAG, chain-of-thought entity matching, and NLI sentence selection to produce fine-grained evidence for LLM answers.","lead":"This paper presents GE-Chat, a framework that builds a knowledge graph from an uploaded document and uses it to retrieve sentence-level evidence for an LLM's answers. If it works as described, it could help users check whether an AI answer is grounded in the document by highlighting the exact supporting sentences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4) conflates relevance with conciseness; since GE-Chat explicitly minimizes length while baselines are verbose, the claimed evidence-retrieval improvement may be a length artifact rather than better evidence identification.","rationale":"The reader's weakest assumption focused on noise in LLM-based KG construction and NLI sentence selection. That is a legitimate component-level risk, but it is not the most load-bearing concern because even a perfectly constructed KG and perfect NLI would not validate the central claim if the evaluation metric rewards the very behavior GE-Chat is designed to optimize. The metric confound in Eq. (4) directly undermines the empirical support for 'consistently improve' and 'sentence-level fine-grained identification.' The reader already issued a CONDITIONAL verdict due to missing empirical details; my concern provides a specific, actionable reason why those details may not support the claim even when supplied. I therefore do not change the verdict, but I sharpen the condition: the authors must show that the improvement is not simply a length effect. This is a correctness risk, not an ad hominem or an external-consensus disagreement; it is testable from the released dataset and metric definitions.","tokens_in":7023,"tokens_out":3401,"duration_ms":31820,"concrete_test":"Recompute the headline comparison in Fig. 4 (or on the released 1000-case dataset) using Eq. (4) with the conciseness factor set to 1 (i.e., cosine similarity only), and also report sentence-level overlap metrics such as token-level F1 or exact sentence match between predicted evidence and human-annotated groundtruth evidence. If GE-Chat's advantage over direct LLM reflection disappears, shrinks materially, or reverses under length-controlled or overlap-based evaluation, the central claim is a conciseness artifact rather than improved evidence identification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GE-Chat consistently improves each model's evidential responses by providing sentence-level fine-grained identification of supporting evidence (Sec. 1, Sec. 3.2). The only quantitative basis is the Evidencescore metric in Eq. (4): Evidencescore = (1/N) sum [ cos(E_i, E_gt_i) * min(1, L_gt_i / L_i) ]. The min(1, L_gt_i / L_i) factor penalizes generated evidence that is longer than the groundtruth but never penalizes shorter output. GE-Chat's Sec. 2.4 explicitly optimizes conciseness through Eq. (2), F(s_n) = alpha * prob(s_n entails S') - beta * len(s_n), with alpha = beta = 0.5. The paper itself reports that the direct LLM source-reflection baselines 'generate too many words' (Sec. 3.2). Therefore the comparison is structurally biased: the proposed method is tuned to minimize length, the baselines are not, and length is a rewarded component of the evaluation metric. The observed 'consistent improvement' in Fig. 4 could be driven primarily by shorter outputs rather than by more accurate identification of the exact evidence sentences. The metric also uses cosine similarity over the whole generated evidence block, not sentence-level alignment with the groundtruth evidence, so it does not directly measure the claimed sentence-level fine-grained capability. Because no numeric results are reported in the text, the magnitude of the improvement and its statistical reliability cannot be assessed independently of this confound. This is the most load-bearing concern because it threatens the validity of the empirical evidence for the paper's central claim, regardless of whether the KG construction or NLI selection is noisy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GE-Chat, a retrieval-augmented generation framework that constructs a knowledge graph from an uploaded document, elicits chain-of-thought reasoning steps from an LLM, searches an n-hop sub-graph by entity matching to retrieve source chunks, and then applies an NLI-based, length-penalized objective to select concise evidence sentences. The authors release a 1000-case dataset covering ten domains and report, in a single figure, that applying GE-Chat to several LLMs improves an evidence-quality score relative to the models' direct source reflection.","tokens_in":7332,"tokens_out":4208,"duration_ms":42374,"significance":"If the claimed improvement is real, GE-Chat offers a practical and model-agnostic way to produce sentence-level evidence citations, which would be useful for trustworthiness assessment. The paper also contributes a public dataset and a demo, and its pipeline combines established components (GraphRAG, CoT, sub-graph search, NLI) in a coherent way. However, the current manuscript does not provide enough quantitative evidence to support the central claim: the experimental section reports only a figure with no numeric values, the evaluation metric conflates relevance with conciseness, and no comparison with prior evidence-highlighting baselines is made. The significance is therefore conditional on a substantially strengthened evaluation.","major_comments":[{"comment":"The evaluation metric in Eq. (4), Evidencescore = (1/N) sum [cos(E_i, E_gt_i) * min(1, L_gt_i / L_i)], rewards shorter outputs by the min(1, L_gt_i / L_i) factor and never penalizes outputs shorter than the ground truth, while the proposed method's selection objective in Eq. (2), F(s_n) = 0.5 * prob(s_n entails S') - 0.5 * len(s_n), explicitly optimizes for conciseness. Since the manuscript itself reports that the direct LLM baselines 'generate too many words' (Sec. 3.2), the observed 'consistent improvement' in Fig. 4 may be driven primarily by a length effect rather than by more accurate identification of supporting evidence. To address this confound, the authors should report cosine similarity and length (or conciseness) as separate metrics, add a length-controlled baseline, and include an ablation that removes the length term from F.","section":"Sec. 3.2, Eq. (4) and Sec. 2.4, Eq. (2)"},{"comment":"The metric computes cosine similarity over the entire generated evidence block E_i against the ground-truth block E_gt_i, which does not directly measure the sentence-level fine-grained evidence identification claimed in Sec. 1 and Sec. 2.4. A block-level cosine score can be high even if the selected sentences are not the exact ground-truth sentences. Please supplement the evaluation with sentence-level or token-level alignment metrics (for example, token-level F1, ROUGE, or overlap of selected sentence indices with the human-annotated evidence sentences).","section":"Sec. 3.2, Eq. (4)"},{"comment":"All quantitative results are reported only in Fig. 4, with no per-model numeric values, no error bars, no standard deviations, and no significance tests. The central claim that GE-Chat 'consistently improves the performance of each model's evidential-based responses' cannot be verified from a single figure. Please provide a table with per-model Evidence scores for both the baseline and GE-Chat, along with the number of runs and a paired statistical test (e.g., Wilcoxon signed-rank test over the 1000 cases).","section":"Sec. 3.2"},{"comment":"The ground-truth answers and evidence sentences are constructed with GPT-4o plus 'human-correction,' but no details are given about the annotation protocol, the number of annotators, their expertise, or inter-annotator agreement. Since all reported evidence scores measure agreement with this ground truth, its reliability is load-bearing. Please describe the annotation process and report agreement statistics (e.g., Cohen's kappa or a second-annotation overlap measure).","section":"Sec. 3.1"},{"comment":"The only baseline is the direct LLM source reflection of the evidence. The manuscript does not compare GE-Chat with prior chunk-level evidence-retrieval or evidence-highlighting methods, such as the systems cited in refs. [7] and [13], or with simpler post-hoc sentence-selection baselines. Without such comparisons, it is unclear whether the reported improvement comes from the specific graph-based and CoT components or from the final NLI sentence selector. At minimum, please add a strong non-graph baseline (e.g., chunk retrieval followed by the same NLI-based sentence selection) to isolate the contribution of the knowledge-graph and CoT stages.","section":"Sec. 3.2"}],"minor_comments":[{"comment":"The sentence 'The datasets are released at 1' refers to a bare placeholder number; the actual footnote contains a Google Drive URL. Please provide a stable, citable dataset identifier (e.g., Zenodo or a DOI) and replace the placeholder.","section":"Abstract and footnote 1"},{"comment":"The notation 'prob(s_n|− S′)' contains an unexplained minus sign before S'; the prose defines the intended quantity as the entailment probability prob(s_n entails S'). Please fix the notation for clarity.","section":"Sec. 2.4, Eq. (2)"},{"comment":"The claim that 'the direct evidence retrieval ability of GPT4o is the best' is made without quantitative support in the text. If GPT-4o is used as a reference upper bound, please include its score in the figure or a table so the reader can calibrate the other results.","section":"Sec. 3.2"},{"comment":"Sec. 4 states that the evaluation covers 'ten diverse LLMs,' but Sec. 3.2 says GE-Chat was applied to models except GPT-4o; please clarify the exact set of models tested and report their names.","section":"Sec. 4 vs. Sec. 3.2"},{"comment":"The ACM reference-format line lists the year as 2018 for a WWW '25 paper; please correct the placeholder year. Also, reference [3] (EvidenceChat) appears to be an unpublished manuscript; if it is the authors' own prior work, please provide a stable link or a published version.","section":"References"},{"comment":"Figure 4 is the only quantitative result but the axes, units, and the set of models are not described in the caption or text. Please make the figure self-contained and accessible.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a 5-page short-paper or demo submission rather than a full archival journal paper; the experimental evaluation, as reported, is too thin to support the broad claim. The technical pipeline is coherent and the dataset is a useful contribution, but a major revision with a complete experimental table, proper baseline comparisons, and a confound-controlled analysis is needed. I would also ask the authors to clarify their contribution relative to their own prior EvidenceChat work (ref. [3])."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GE-Chat is a sensible pipeline—GraphRAG construction, CoT elicitation, sub-graph search, then NLI-based sentence selection—for giving sentence-level evidence for LLM answers. The released 1,000-case dataset with human-annotated evidence is a real asset for anyone working on evidential QA. The complexity analysis is straightforward and the writing is clear about what the components do. That part is solid.\n\nThe soft spot is the evaluation, and it is load-bearing. Equation (4) defines Evidence score as cos(E_i, E_gt_i) * min(1, L_gt_i / L_i). That penalty rewards shorter outputs and never punishes overly short ones. Meanwhile the method's own optimization objective in Eq. (2) explicitly minimizes sentence length with a fixed 0.5 weight, and the paper admits the direct-reflection baselines “generate too many words.” So the comparison is structurally tilted: the proposed method is tuned to be concise, the baselines are not, and conciseness is rewarded in the metric. The “consistent improvement” in Fig. 4 could easily be mostly a length artifact rather than better evidence identification. On top of that, the text of the paper gives no numeric scores, no per-model breakdown, no error bars, and no significance tests. I can’t tell from the submission how large the gains are or whether they are reliable. The decision to exclude GPT-4o from the comparison, while explained, also means the strongest baseline is missing.\n\nThe stress-test note you forwarded hits the right spot. I don't think the KG noise concern is the main issue—that is a standard engineering risk. The metric confound is the thing that needs to be addressed before the central claim can be accepted.\n\nThe citation pattern is mostly fine. The self-citations to EvidenceChat are appropriate because that is the direct predecessor. The reliance on LightRAG and the RAGAS conciseness metric is also legitimate.\n\nWho is this for? People building RAG systems that need to show users where answers came from, and anyone who wants a sentence-level evidence dataset. As a peer reviewer, I would engage with it—the dataset and framework are worth a serious look—but I would insist on seeing numeric results, a length-controlled comparison (e.g., truncating baseline outputs to the same length as groundtruth), and ideally an ablation that removes the length penalty from the metric to separate relevance from conciseness. Right now the empirical claim is not verified, but the artifact and the problem formulation justify referee time.","headline":"Plausible engineering contribution with a useful released dataset, but the central empirical claim is undermined by a length confound in the evaluation metric and the absence of numeric results.","tokens_in":7902,"tokens_out":1020,"would_cite":false,"duration_ms":11555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GE-Chat shows that a knowledge graph built from a user's uploaded document can lift an LLM's evidence from whole chunks to exact supporting sentences, and applying it consistently improves evidential responses across models.","keywords":["LLMs","evidential answering","retrieval-augmented generation","knowledge graph","chain-of-thought","natural language inference","sentence-level evidence","hallucination"],"falsifier":"Take a document with a known answer and ground-truth evidence sentence, and run the pipeline twice: once with true entity names and once with entity names replaced by random tokens before relation probing and sub-graph search. If the final selected evidence sentence is identical and still correct, the knowledge graph is not doing the claimed work; if it changes or becomes wrong, the graph is load-bearing. A second check is to inspect a case where the selected sentence does not entail the answer by human judgment and see whether the NLI ranking placed that sentence first.","tokens_in":6835,"feed_emoji":"🔍","tokens_out":8785,"duration_ms":78848,"temperature":0.7,"pith_summary":"GE-Chat is a retrieval-augmented generation framework that tries to show that an LLM's answers to questions about an uploaded document can be paired with exact evidence sentences from that document, rather than whole paragraphs or the model's own possibly unreliable source reflection. The paper argues that building a knowledge graph from the document, using chain-of-thought reasoning to identify relevant entities, searching an n-hop sub-graph to retrieve source chunks, and then selecting sentences by an entailment-based score consistently improves evidence quality across a range of open and closed LLMs. If this is right, users could quickly see which parts of an answer are grounded in the uploaded material and which are not, without having to trust the model's own claims about its sources. The paper also releases a 1000-case dataset with human-annotated evidence sentences so the claim can be checked.","feed_headline":"Graph-grounded RAG lifts evidence quality for ten LLMs","feed_subtitle":"GE-Chat builds a knowledge graph from the uploaded document, then highlights the exact sentences behind each answer.","key_machinery":"The load-bearing object is the derived knowledge graph $\\mathcal{G}$, built by an LLM extracting entities from document chunks and probing their relations. Around it, three mechanisms carry the argument: a chain-of-thought template that produces answer sentences plus reasoning steps; a sub-graph search that matches each reasoning step to entities in $\\mathcal{G}$ and expands to second-hop neighbours to retrieve source chunks; and an optimization step that scores each candidate sentence $s_n$ by $F(s_n) = \\alpha\\,\\mathrm{prob}(s_n \\vdash S') - \\beta\\, \\mathrm{len}(s_n)$, with $\\alpha=\\beta=0.5$, where the entailment probability comes from an off-the-shelf NLI model applied to the sentence pair. The graph anchors generated content to the document's original entities, the n-hop search converts that anchor into a small set of source chunks, and the entailment objective selects short sentences that logically support the answer.","core_discovery":"The paper's central claim is that evidence for an LLM's conclusion can be identified at the sentence level by constraining the search to the original document: first parse the document into a knowledge graph, elicit the model's chain-of-thought, map those reasoning steps to graph entities, expand through two-hop relations to retrieve source chunks, and then choose the sentence in each chunk that best entails the answer while staying concise. On a new 1000-case dataset spanning ten domains, the authors report that wrapping existing LLMs in GE-Chat consistently improves their evidential-response scores compared with the models' direct source reflection. They treat this as evidence that the framework generalizes across models, including smaller ones with weak instruction-following, and that sentence-level evidence can be produced in real time from the document graph.","pith_inferences":["This suggests the framework's evidence quality is partly bottlenecked by the LLM's graph-building ability; swapping in a deterministic entity and relation extractor in the same pipeline would test whether graph fidelity, not generation quality, drives the reported gains.","The k-hop search could be extended to dynamic expansion for multi-hop questions, so the framework may generalize from evidence for single answers to evidence supporting a full reasoning chain.","The entailment-based sentence optimizer is independent of the graph search, so it could be bolted onto other retrieval pipelines as a general post-processor that turns chunk-level hits into sentence-level citations.","Because the released dataset includes human-annotated evidence, it could support a follow-up calibration study that compares the cosine-and-conciseness score against human judgments of which evidence actually persuades a reader."],"forward_implications":["Users will see exact supporting sentences from the uploaded document beside each answer, instead of a whole highlighted chunk or the model's own guess at its sources.","Models that answer well but struggle to reflect on their own sources will still produce reliable evidence, because the pipeline does not depend on the model's source-reflection ability.","Every selected evidence sentence is anchored to entities and relations from the original document, so responses remain traceable to the uploaded material by construction.","The pipeline is fast enough for deployment: entity extraction is $O(n\\cdot l)$, relation probing is $O(m^2)$, and the two-hop search is $O(2e)$."],"supporting_citations":[{"why":"supplies the graph-based retrieval-augmented generation foundation the framework builds on.","marker":"[11]"},{"why":"supplies the lightweight graph-based RAG implementation used as the work basis.","marker":"[6]"},{"why":"supplies the chain-of-thought reasoning principle that the CoT template elicits from the LLM.","marker":"[20]"},{"why":"inspires the specific CoT template used to produce answers and logic steps.","marker":"[24]"},{"why":"defines the direct LLM source-reflection baseline that GE-Chat compares against.","marker":"[3]"},{"why":"represents the chunk-level source highlighting that GE-Chat refines to sentence-level evidence.","marker":"[17]"},{"why":"supplies the cosine-similarity relevance term in the evidence score.","marker":"[12]"},{"why":"supplies the conciseness term in the evidence score.","marker":"[16]"}],"fun_headline_variants":["Knowledge graph pinpoints sentence-level evidence","Trace LLM answers to exact source sentences","Graph-grounded retrieval finds evidence sentences","Ten-domain test: graph RAG boosts evidence quality","Pinpoint the evidence behind every LLM answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the knowledge graph built by the LLM accurately reflects the document's entities and relations and that the entailment model reliably ranks candidate sentences; if either is noisy, the retrieved evidence can point to the wrong sentences.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graph pinpoints sentence-level evidence","Trace LLM answers to exact source sentences","Graph-grounded retrieval finds evidence sentences","Ten-domain test: graph RAG boosts evidence quality","Pinpoint the evidence behind every LLM answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1818,"prompt_tokens":911,"completion_tokens":907,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":840}},"tokens_in":527,"tokens_out":907,"duration_ms":8885,"temperature":1.0,"reasoning_tokens":840,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:15:14.495850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a document with a known answer and ground-truth evidence sentence, and run the pipeline twice: once with true entity names and once with entity names replaced by random tokens before relation probing and sub-graph search. If the final selected evidence sentence is identical and still correct, the knowledge graph is not doing the claimed work; if it changes or becomes wrong, the graph is load-bearing. A second check is to inspect a case where the selected sentence does not entail the answer by human judgment and see whether the NLI ranking placed that sentence first.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the graph-based retrieval-augmented generation foundation the framework builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the direct LLM source-reflection baseline that GE-Chat compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the cosine-similarity relevance term in the evidence score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the conciseness term in the evidence score."}],"review_version":1}