{"id":"340d519b-adbc-4ca7-90f9-a4deeff82e6c","arxiv_id":"2501.02020","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A semantic-graph-enhanced uncertainty model, combining AMR-based entity relations with NLI contradiction scores, improves sentence- and passage-level hallucination detection in LLMs.","lead":"This paper presents a method that builds a semantic graph from parsed text and uses it to improve uncertainty-based hallucination detection in large language models. The approach propagates uncertainty along entity relations and calibrates passage-level scores with sentence contradiction probabilities, reporting gains over baselines on two datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual review of extracted triples on test data means Table 2 does not evaluate the automatic method; the reported 19.78% gain may be an artifact of human-curated graphs.","rationale":"The reader's weakest assumption focuses on the reliability of automatically constructed graphs and explicitly cites Appendix A.1's manual review; my concern is the same, sharpened: manual review on test data is not merely a source of noise but a direct threat to the validity of every reported number, because the core novel component (semantic graph) is not what is being evaluated. The other flaws noted by the reader—test-set hyperparameter tuning and mismatched Chinese backbones—are also serious, but they are secondary in the sense that they concern the fairness of comparisons; the manual review problem concerns whether the evaluated system is the proposed system at all. Even the ablation studies and the graph-calibration analysis use the human-corrected graphs, so they cannot isolate the automatic graph's contribution. A single experiment removing the manual-review step would settle whether the method itself, rather than human curation, drives the reported gains. Since this concern reinforces the reader's REJECT verdict, the verdict should remain unchanged.","tokens_in":16389,"tokens_out":5246,"duration_ms":53614,"concrete_test":"Re-run the complete pipeline on WikiBio and NoteSum with the automatic graph construction—AMR parsing, spaCy coreference and entity linking, rule-based triple extraction—explicitly omitting the manual review step described in Appendix A.1. Use the same hyperparameters, projection functions, and backbones as in Table 2. Compare the resulting AUC, Pearson, and Spearman scores against FOCUS and against the paper's reported numbers. If the improvements over FOCUS disappear or shrink substantially, the headline claim depends on human curation and the method as an automatic detector is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed semantic-graph uncertainty method automatically improves hallucination detection. For this to hold, the semantic graph must be produced by the pipeline described in Section 'Semantic Graph Construction' (AMR parsing, coreference resolution, entity linking). However, Appendix A.1 states: 'Finally, we manually review all the extracted triples to obtain more accurate results.' This manual review is applied to the test passages themselves, so every result in Table 2—including the 19.78% passage-level improvement and all ablation/substitution experiments—is produced with human-corrected graphs, not the automatic method. This is a form of test-set leakage: the human reviewer sees the passage to be detected and may use world knowledge or even label information when repairing triples. Even if the review is label-blind, the method as evaluated is not reproducible without human effort and is not the reference-free, efficient method claimed in the abstract. The paper does not report results with fully automatic graph construction, so the headline improvement cannot be attributed to the proposed algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UncerSema4HalluDetec, an uncertainty-based hallucination detection method that constructs a passage-level semantic graph from AMR parses, coreference resolution, and entity linking; propagates entity-level uncertainty along graph edges via relation-based propagation (Eq. 2); and calibrates passage-level uncertainty with NLI contradiction probabilities over neighboring sentences (Eq. 7). The method is evaluated on the WikiBio dataset and on a new Chinese NoteSum dataset, reporting improvements over GPT-3 uncertainty baselines, SelfCheckGPT, and FOCUS, with a headline 19.78% improvement in passage-level Spearman correlation.","tokens_in":16650,"tokens_out":6041,"duration_ms":58958,"significance":"If the evaluation were valid, the idea of using a semantic graph to propagate and calibrate token-level uncertainty is a plausible and potentially useful contribution to single-pass hallucination detection. The paper is also transparent in reporting full hyperparameter sweeps and substitution experiments. However, the current experimental protocol does not support the claimed gains: the semantic graph is manually corrected on the test passages, and the hyperparameters and projection functions are selected on the WikiBio test set. The reported improvements are therefore not attributable to the automatic algorithm as described in the abstract.","major_comments":[{"comment":"The evaluation uses manually corrected semantic graphs on the test passages. Appendix A.1 states: 'Finally, we manually review all the extracted triples to obtain more accurate results.' Since the graph edges directly enter the propagated uncertainty (Eq. 2) and the neighbor set used for calibration (Eq. 7), every result in Table 2—including the 19.78% passage-level gain—is produced by a human-assisted pipeline, not by the automatic AMR/spaCy pipeline described in the main text. The paper neither reports results with fully automatic graph construction nor quantifies the effect of the manual review. Consequently, the abstract's characterization of the method as reference-free and efficient is not supported, and the headline numbers cannot be reproduced without human labor on each test passage.","section":"Appendix A.1; Section 'Semantic Graph Construction'; Table 2"},{"comment":"The free parameters α, β, λ, k and the sentence- and passage-level projection functions are chosen by maximizing the reported evaluation metrics on the WikiBio dataset with LLaMA-30B. Appendix A.5 says 'we adjust the following four parameters ... on the WikiBio dataset' and 'we test on three projection functions ... on the WikiBio dataset.' Because no separate validation set is described, the WikiBio numbers in Table 2 are selection results rather than an unbiased estimate of performance. The reported improvements and the ablation conclusions in Table 3 therefore do not establish generalization; the comparisons with baselines are also affected because the baseline scores are not tuned in the same way.","section":"Appendix A.5; Tables 4-8; Table 2"},{"comment":"The NoteSum comparison is not controlled for backbone. Appendix A.4 states that for the Chinese dataset the FOCUS baseline uses Alpaca-Chinese-13B and Alpaca-Chinese-33B, while Table 2 labels the proposed method's rows as LLaMA-13B and LLaMA-30B. If the proposed method was run with the original LLaMA models on Chinese text while the baseline used a Chinese-tuned model, the comparison is confounded; if the proposed method also used the Chinese-adapted models, this is not stated. The paper should specify the exact backbone used for the proposed method on NoteSum and run baselines under identical conditions.","section":"Appendix A.4; Table 2"},{"comment":"The contribution of the graph-based calibration is not isolated from the manual graph correction. The '- graph' ablation in Table 3 compares the full model against averaging sentence uncertainties, but both conditions use the manually reviewed triples from Appendix A.1. Therefore the drop of about 2 Pearson points cannot be attributed to the automatic semantic graph; it may reflect the human-corrected neighbor structure. A proper comparison would use the same automatic graph construction in both conditions and report the effect of manual correction separately.","section":"Section 'Passage-level Uncertainty', Eq. (7); Table 3"}],"minor_comments":[{"comment":"The attention score att(s', o) in Eq. (2) does not specify which layer or head of the LLaMA attention matrix is used; since attention scores are not probabilities over semantic relations, this choice should be stated for reproducibility.","section":"Eq. (2)"},{"comment":"The notation C_j^i is introduced as a sorted list of top-k probabilities, but the formula applies max() and variance to it; defining C_j^i as the set of top-k probability values would remove the ambiguity.","section":"Eq. (1)"},{"comment":"The annotation agreement of 0.76 for NoteSum is reported without defining the metric; please specify whether this is Cohen's kappa, Krippendorff's alpha, or simple percentage agreement.","section":"Appendix A.2"},{"comment":"The baseline description contains an incomplete citation with a placeholder '( ?)' after 'ChatGPT3.5'; this should be corrected.","section":"Appendix A.4"},{"comment":"The λ sweep is non-monotonic (e.g., NonFact* is 65.34 at λ=0.6 and 61.16 at λ=0.7), yet the chosen value is justified only by the tabulated point estimates; reporting variance or confidence intervals across runs would strengthen the sensitivity analysis.","section":"Table 6"}],"recommendation":"reject","confidential_remarks":"I am recommending rejection because the two evaluation-protocol issues—manual triple correction on test data and test-set hyperparameter selection—directly undermine the paper's central empirical claim. The framework is interesting, but as submitted the reported numbers do not constitute valid evidence for the automatic method. A future submission that reruns the experiments with fully automatic graph construction, a proper validation split, and controlled baselines could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nBottom line: the semantic-graph idea is worth knowing, but the evaluation as reported doesn't support the headline 19.78% gain. The paper proposes to build an AMR-based semantic graph over a passage, propagate token-level uncertainty along entity relations, and use NLI contradiction scores between neighboring sentences to calibrate passage-level scores. The combination is genuinely new relative to FOCUS and SelfCheckGPT, and the ablations are thorough: they test each component, substitution alternatives, and visualize the uncertainty separations. Code is released.\n\nThe soft spots are load-bearing. First, the hyperparameters α, β, λ, k and the projection functions for sentence and passage scores are all selected by maximizing the evaluation metrics on the WikiBio dataset (Appendix A.5, Tables 4–8). That is tuning on the test set. Every reported number for the proposed method on WikiBio—including the 19.78% passage-level improvement—is the result of fitting those degrees of freedom to the very labels being predicted. Second, Appendix A.1 states the extracted AMR triples are manually reviewed on the test passages. So Table 2 evaluates a semi-automatic pipeline with human-curated graphs, not the automatic method described in the abstract. The paper does not report results with fully automatic graph construction, so the main claim as stated is unsupported. Third, the NoteSum comparison uses Alpaca-Chinese backbones for FOCUS instead of the LLaMA models used for the proposed method; that confounds method and backbone.\n\nI want to be fair: the paper is not sloppy in its disclosure. The appendices document exactly what was done, which is unusual and helpful. But the documentation reveals why the central claim doesn't hold. With a held-out tuning split, matched baselines, and automatic graph construction, the method might show real gains. As submitted, the evidence is not reliable. The paper deserves a serious referee to push for those fixes, but my recommendation would be reject in current form.","headline":"Semantic graph idea is fresh but the evaluation is compromised by test-set tuning and manual graph curation; reject as submitted.","tokens_in":17157,"tokens_out":2776,"would_cite":false,"duration_ms":23591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that propagating uncertainty over a semantic graph of entities and sentences detects hallucinations better than scoring tokens independently, with a 19.78-point gain in passage-level detection on the Chinese dataset.","keywords":["hallucination detection","uncertainty estimation","semantic graph","Abstract Meaning Representation","passage-level detection","natural language inference","large language models","factuality"],"falsifier":"Run the published pipeline on a third dataset using only automatic graph construction, with no manual triple review; if the passage-level gain over FOCUS collapses to noise, the reported advantage depends on human cleanup rather than on the semantic-graph mechanism. A second check: build passages in which two sentences contradict each other but share no linked entity; if the graph-based calibration does not raise the passage score above simple sentence averaging, the method is not actually capturing long-range contradictions.","tokens_in":16211,"feed_emoji":"📈","tokens_out":11490,"duration_ms":103268,"temperature":0.7,"pith_summary":"The paper tries to show that uncertainty-based hallucination detection fails when it treats each token as independent, and that a semantic graph of entity and sentence relations can fix that. The authors construct a passage-level graph by parsing each sentence into an Abstract Meaning Representation, linking entities across sentences, and then use the graph twice: to propagate uncertainty between related entities and to calibrate each sentence's score against contradicting neighbors. They report consistent gains over state-of-the-art baselines at both levels, with a 19.78-point improvement in passage-level Spearman correlation on a Chinese note-summarization dataset. If the claim holds, hallucination detection becomes more accurate without needing external knowledge bases or repeated sampling, which is what makes uncertainty-based detection attractive in the first place.","feed_headline":"19.78% gain: semantic graph catches passage-level hallucinations","feed_subtitle":"Entity and sentence relations catch contradictions that token-averaging methods miss.","key_machinery":"The load-bearing object is the passage-level semantic graph, a graph built from AMR parses of each sentence, with Abstract Meaning Representation encoding who did what to whom and sentence graphs stitched together by coreference resolution and entity linking. Two mechanisms use it: relation-based uncertainty propagation, which routes uncertainty from subject to object entities along the dependency path, weighted by attention and penalized by relation intensity, and graph-based uncertainty calibration, which scales each sentence's uncertainty by the NLI contradiction probability with each neighboring sentence. The graph's role is to decide which tokens and sentences are actually related, so that uncertainty flows along true semantic connections rather than the work of scoring everything independently.","core_discovery":"The central claim is that hallucination is a relational phenomenon, not a token-level one, and that uncertainty scores should therefore be computed over a semantic graph rather than averaged over independent tokens. The paper's pipeline parses each sentence into an AMR graph, links the sentence graphs into a passage graph via coreference resolution and entity linking, and identifies the entity tokens and relation paths inside that graph. On top of this graph, relation-based uncertainty propagation sends an object entity's uncertainty from its subject entities along dependency edges, weighting by attention and discounting by relation intensity to avoid the overestimation that plagues propagation over all preceding tokens. For passage-level scoring, graph-based uncertainty calibration multiplies each sentence's uncertainty by the contradiction probability a natural-language-inference model assigns between that sentence and each of its graph neighbors. The authors' claim is that these two graph mechanisms, propagation and calibration, are what let the method beat token-averaging baselines, especially on moderate or no-hallucination sentences and on passages where separate sentences contradict each other.","pith_inferences":["Editorial inference: because triple extraction required manual review in the paper's own setup, a robustness check on fully automatic graphs, without the manual cleanup documented in Appendix A.1, would show whether the approach scales beyond these two datasets.","Editorial inference: the contradiction-calibration step could be turned into an explanation tool; the specific neighbor sentence and relation path that drive a high calibrated score are readable evidence for why a passage was flagged, which the paper does not develop.","Editorial inference: a natural extension is to replace or augment the NLI model with external fact sources, since the graph already identifies the entities and relations whose factual accuracy would need checking; the paper only mentions this possibility as future work.","Editorial inference: the method's reliance on AMR parsing suggests it may be weaker on informal or heavily elliptical text, where AMR parses are noisier; testing on dialogue or social-media data would bound the method's domain of validity."],"forward_implications":["Sentence-level detection improves because uncertainty propagates only along real dependency relations, preventing the overestimation that propagation across all preceding tokens suffers on moderate or no-hallucination sentences.","Passage-level detection improves because a sentence that contradicts a non-adjacent neighbor in the semantic graph receives a boosted uncertainty score, catching hallucinations that sentence averaging would miss.","The method is reference-free and single-pass: it needs one forward pass of the LLM for token probabilities plus graph construction and NLI scoring, with no external retrieval or repeated sampling.","The same pipeline transfers to a new domain and language: the authors report gains on both English WikiBio and Chinese NoteSum, and ablations attribute the improvement to the graph components rather than to token-level statistics alone."],"supporting_citations":[{"why":"supplies the WikiBio dataset, the GPT-3 token-probability uncertainty baselines, and the SelfCheckGPT sampling baseline that the method must outperform.","marker":"Manakul, Liusie, and Gales 2023"},{"why":"FOCUS is the strongest uncertainty-based baseline; its overestimation from propagating across all preceding keywords is the specific failure the paper's relation-based propagation targets.","marker":"Zhang et al. 2023c"},{"why":"provides the transition-based AMR parser used to build the sentence-level semantic graphs.","marker":"Xu, Lee, and Huang 2023"},{"why":"provides the DeBERTa-v3-Large NLI model that produces the contradiction probabilities in the passage-level calibration.","marker":"He, Gao, and Chen 2023"},{"why":"supports the sequence-decay term by documenting that hallucination accumulates as generated text gets longer.","marker":"Varshney et al. 2023"},{"why":"motivates the quantile-based global uncertainty measure used to summarize all token uncertainties in a sentence.","marker":"Gupta et al. 2024"},{"why":"documents that a large share of hallucination is entity-related, motivating the entity-centered uncertainty computation.","marker":"Pagnoni, Balachandran, and Tsvetkov 2021"}],"fun_headline_variants":["Semantic graph uncertainty improves hallucination detection by 19.78%","Graph-based uncertainty modeling catches passage-level hallucinations","Relational uncertainty: 19.78% gain in hallucination detection","Semantic graph: 19.78% better passage-level hallucination detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the automatically built semantic graph is accurate enough that uncertainty flows along true relations and not spurious ones; the authors' manual review of extracted triples in Appendix A.1 shows the graph construction is not fully automatic, so noisy graphs could misdirect propagation and calibration.","fun_headline_variants_meta":{"raw":{"variants":["Semantic graph uncertainty improves hallucination detection by 19.78%","Graph-based uncertainty modeling catches passage-level hallucinations","Relational uncertainty: 19.78% gain in hallucination detection","Semantic graph: 19.78% better passage-level hallucination detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2585,"prompt_tokens":958,"completion_tokens":1627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1553}},"tokens_in":574,"tokens_out":1627,"duration_ms":12361,"temperature":1.0,"reasoning_tokens":1553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:28:50.201411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published pipeline on a third dataset using only automatic graph construction, with no manual triple review; if the passage-level gain over FOCUS collapses to noise, the reported advantage depends on human cleanup rather than on the semantic-graph mechanism. A second check: build passages in which two sentences contradict each other but share no linked entity; if the graph-based calibration does not raise the passage score above simple sentence averaging, the method is not actually capturing long-range contradictions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the WikiBio dataset, the GPT-3 token-probability uncertainty baselines, and the SelfCheckGPT sampling baseline that the method must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"documents that a large share of hallucination is entity-related, motivating the entity-centered uncertainty computation."}],"review_version":1}