{"id":"1dc7b366-a1db-435d-bb93-9c5d345ecc7b","arxiv_id":"2412.11404","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new attention-based attribution method, with set union aggregation and dependency parsing augmentation, reports the best fine-grained attribution accuracy on QuoteSum and VERI-GRAN.","lead":"Researchers built a method that points to the exact sentence in a source document that supports each part of an AI-generated answer. It combines the model's attention patterns with grammar-based clues and reports the best accuracy yet on fine-grained attribution benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is conditional on an unmeasured lexical-overlap confound: fine-grained benchmarks use near-verbatim spans and DEP expands spans to full clauses, so attention-based attribution may be riding on surface matching; an overlap baseline is required.","rationale":"The paper is a careful empirical study: the set-union aggregation is clearly specified (Alg. 1), the DEP expansion is rule-based and ablated, the faithfulness metric follows Cohen-Wang et al., and the latency comparison is informative. The core vulnerability is not internal logic but the benchmark setup: fine-grained attribution is evaluated on spans that are near-verbatim passages, and DEP's clause-level expansion maximizes the chance that some token in the expanded span attends to the correct source passage. Prior baselines (HSSAVG, CCI) also operate on the same spans, so the relative ordering may survive; however, the paper's causal claim—that attention weights provide semantically faithful evidence—is not established unless a surface-overlap control is added. The reader's conditional verdict already identifies this risk; my pass sharpens it into a falsifiable test. If the lexical baseline fails to match ATTN UNION DEP, the concern is resolved and the SOTA claim stands. If it matches, the claim must be downgraded to 'competitive on verbatim benchmarks' and the method should be re-evaluated on paraphrased or abstractive answers. No formal proof or released code is available to replace this empirical control. Verdict remains conditional, unchanged from the reader: the SOTA claim should not be fully accepted until the overlap baseline is reported.","tokens_in":18681,"tokens_out":6850,"duration_ms":62927,"concrete_test":"On QuoteSum and VERI-GRAN, re-run the Table 1 protocol with a pure lexical baseline: for each target span, score every passage by the fraction of dependency-expanded span tokens contained in the passage (or by BM25/ROUGE-L) and select the highest-scoring passage. If this overlap-only baseline reaches or exceeds ATTN UNION DEP's 93.3/84.6 and 94.0/78.2, the SOTA result is explained by near-verbatim span overlap rather than by attention semantics. As a follow-up, paraphrase the target spans while keeping the same human-validated evidence labels and rerun the comparison; a collapse in ATTN UNION DEP's advantage would confirm that the method does not generalize beyond verbatim attribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's central claim is not yet secured because the two fine-grained benchmarks are built around spans that are near-verbatim extracts of a source passage—a fact the authors concede in Limitations—and no lexical-overlap baseline is reported. ATTN UNION DEP deliberately grows a target token into an entire dependency clause before accumulating evidence (Sec. 3.3), so the passage choice is effectively scored by aggregate token overlap between the expanded span and each document. Given known positional and syntactic biases in attention maps, the large DEP gains (e.g., 80.4 to 93.3 on Qwen2 QuoteSum) could therefore come from surface matching rather than from attention weights encoding semantic support. The faithfulness experiment (Sec. 4.3) is run on freshly generated answers, not on the benchmark spans used for Table 1, so it does not certify those spans. The comparison to HSSAVG and CCI is internal and may remain valid, but the advertised mechanism and the breadth of the SOTA claim are unproven without controlling for verbatim overlap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ATTN UNION DEP, a model-internals method for fine-grained attribution in RAG. For each response token, the top-k prompt tokens by averaged attention from a selected middle layer are collected as evidence; evidence sets are combined by union across target-span tokens, isolated tokens are removed, and dependency parsing is used to expand each target token to the clause containing its governing verb and related constituents. The method is evaluated on QuoteSum and VERI-GRAN against CCI and HSSAVG, on sentence-level attribution against self-citation and ATTR FIRST, on a faithfulness test using log-probability drops, and in a latency comparison. The main empirical claim is that ATTN UNION DEP sets a new state of the art in fine-grained attribution (Table 1).","tokens_in":18850,"tokens_out":7163,"duration_ms":68080,"significance":"The two proposed techniques are simple and generally applicable: set-union aggregation avoids recomputation over target spans, and dependency parsing supplies a linguistically motivated way to give decoder representations access to later context. The paper includes careful ablations, hyperparameter sensitivity plots, a cross-lingual transfer experiment, and an efficiency study, all of which are strengths. If the fine-grained results survive a control for lexical overlap, the method would be a practically useful and substantially faster attributor than existing gradient- or hidden-state based approaches.","major_comments":[{"comment":"The paper's central SOTA claim is not yet secured because the fine-grained benchmarks use near-verbatim target spans, as the authors concede in §6, and no lexical-overlap baseline is reported. ATTN UNION DEP expands each target token into a dependency clause (§3.3), so the passage-level score is accumulated over many tokens of the expanded span; for spans that are copied from a source passage, this score may largely reflect token overlap between the expanded span and each candidate passage. The reported gains over HSSAVG (e.g., 80.4→93.3 on Qwen2/QuoteSum, 77.1→94.0 on Llama2/QuoteSum) could therefore be explained by surface matching rather than by attention weights encoding semantic support. Please add baselines such as BM25, TF-IDF cosine, or exact n-gram overlap between the DEP-expanded span and each passage, and an ablation that replaces attention scores with uniform or random weights while keeping the DEP expansion and union aggregation identical.","section":"§4.1, Table 1; §6 (Limitations)"},{"comment":"The DEP ablation does not isolate the contribution of the dependency expansion from the contribution of the underlying similarity metric. The comparisons against SENT COMP and against JinaBERT encoder features all use representational or lexical similarity signals; none controls for the expanded span itself. A direct control is to apply the same expansion and evidence aggregation while replacing the attention-based similarity S with a bag-of-words overlap score. Without such a control, the conclusion that 'recognizing atomic facts via DEP is more effective' (§4.2.1) conflates the dependency expansion with the chosen similarity metric and does not establish that attention weights provide the semantic support claimed for the method.","section":"§3.3, §4.2.1"},{"comment":"The faithfulness experiment does not evaluate the benchmark spans used in Table 1. The protocol generates new answers with greedy decoding, applies CTI to identify context-sensitive tokens, and then measures log-probability drops after deleting the attributed passage. This is a reasonable causal check, but it does not show that the high Table 1 accuracy on QuoteSum/VERI-GRAN spans reflects causal support for those particular spans. Please either run the deletion test on the benchmark spans with the original generator or explicitly restrict the faithfulness claim to newly generated answers.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The text states that UNION underperforms AVG without DEP, but Table 2 shows the opposite in three of the four comparisons: HSSUNION vs. HSSAVG are 80.0 vs. 80.4 (Qwen2/QuoteSum), 80.1 vs. 77.1 (Llama2/QuoteSum), 73.1 vs. 67.1 (Qwen2/VERI-GRAN), and 65.3 vs. 64.5 (Llama2/VERI-GRAN). Only the first comparison favors AVG, and by only 0.4 points. This sentence should be corrected and the surrounding interpretation revised.","section":"§4.2.3, Table 2"},{"comment":"The fine-grained results are reported without error bars or significance tests. Because VERI-GRAN contains only 197 instances, a bootstrap confidence interval would help the reader judge whether differences of a few points (e.g., ATTN UNION vs. HSSAVG on Llama2/VERI-GRAN, 66.7 vs. 64.5) are meaningful.","section":"§4.1, Table 1"},{"comment":"The claim that attention-based similarity is 'superior' to hidden-state similarity is supported only by selected comparisons in Fig. 3; the adjacent-bar comparisons are not quantified or tested. Please report the relevant numbers and, if possible, a paired comparison.","section":"§3.4"},{"comment":"The GPT-4 baseline is cited from Phukan et al. (2024) and may use a different prompt template or evaluation protocol; this should be stated explicitly so that the reader does not treat the comparison as controlled.","section":"§4.1, Baselines"},{"comment":"The Chinese synthetic dataset construction translates only the answers while keeping questions and passages in English. This is a reasonable first test, but the paper should note that it does not evaluate full Chinese RAG pipelines with Chinese documents.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid and clearly written engineering contribution with useful ablations and an efficiency story. The main risk is the missing lexical-overlap baseline: given that the fine-grained benchmarks use verbatim spans and DEP expands spans to full clauses, the SOTA claim could be an artifact of surface matching. I would support acceptance if the authors add the proposed control and the gains persist; otherwise the claim should be substantially narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a solid, clearly written empirical paper with a genuinely new combination—token-wise set-union aggregation and dependency-parsing augmentation—applied to fine-grained attribution in RAG. The results are consistent across two backbones and two datasets, and the ablation showing that DEP helps with union but not with hidden-state averaging is a good piece of evidence that the mechanism is more than just adding context tokens. The latency numbers also give practitioners a reason to try it over gradient-based CCI.\n\nThe soft spot is real but not fatal. The fine-grained benchmarks use near-verbatim spans, and the paper provides no lexical-overlap baseline. That is a genuine gap, as the stress-test note says. If a simple Jaccard-overlap scorer gets 93% on QuoteSum, then the attention weights may be riding on surface matching, and the DEP gain could be just an artifact of expanding spans to include more words that appear verbatim in the passage. The paper concedes the verbatim nature of the benchmarks in Limitations, which is honest, but it does not resolve the confound. The faithfulness experiment is run on freshly generated answers, not the benchmark spans, so it does not certify the Table 1 results. The absence of error bars or significance tests on the main table is also a problem: differences of 5–10 points are large, but we have no sense of variance.\n\nThat said, the paper is not hiding anything. The limitations section is straightforward, the hyperparameter choices are validated, and the comparison to HSSAVG and CCI is fair and internally consistent. The concern that attention weights are biased by syntax or position is real, but the paper's own ablation—where DEP hurts HSSAVG—suggests the method is not simply adding overlapping tokens; if it were, hidden-state averaging should also benefit from an expanded span.\n\nBottom line: this paper deserves a serious referee. The method is new, the experiments are well-designed, and the claim is within reason, but the SOTA claim is conditional. I would ask a referee to require a lexical-overlap baseline (e.g., Jaccard, TF-IDF, or BM25 on the expanded span against passages) and uncertainty estimates before publication. The paper is a meaningful contribution to fine-grained attribution, and with that additional control, it would be a solid accept.","headline":"Attention + dependency-parsing set-union attribution is a plausible advance, but the SOTA claim needs a lexical-overlap baseline and error bars before it fully lands.","tokens_in":19405,"tokens_out":2932,"would_cite":true,"duration_ms":26854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-grained attribution for RAG answers can be made state-of-the-art by unioning token-level attention evidence and expanding target spans via dependency parsing.","keywords":["fine-grained attribution","retrieval-augmented generation","attention weights","dependency parsing","model internals","evidence citation","attribution faithfulness"],"falsifier":"Rerun ATTN UNION DEP on the same benchmarks after paraphrasing the retrieved documents so that no target span appears verbatim; if accuracy collapses, the reported gains are explained by near-verbatim span overlap instead of by the attention-plus-dependency mechanism.","tokens_in":18447,"feed_emoji":"🧩","tokens_out":9848,"duration_ms":77648,"temperature":0.7,"pith_summary":"This paper proposes that fine-grained attribution—pinpointing, for each span of a retrieval-augmented answer, the supporting text in the retrieved documents—can be made substantially more accurate by two model-internal techniques. The first attributes each response token separately and then takes the union of its evidence sets, preserving token-level granularity instead of averaging it away. The second uses dependency parsing to expand the target span with the tokens that form its atomic fact, such as the subject, verb, and object, so the attributor can use context that a causal decoder cannot see from the target position. With averaged attention weights from a selected middle layer as the similarity score, the method reports a new state of the art on the QuoteSum and VERI-GRAN benchmarks.","feed_headline":"Token-union attention outdoes all prior RAG attribution","feed_subtitle":"Unioning per-token evidence and dependency-parsing span expansion set new accuracy records on two QA benchmarks.","key_machinery":"The machinery rests on a token-wise attribution set e(ri)={dj | w(ri,dj)>0}, built from a similarity matrix S in which each response token's score against each prompt token is thresholded at the top-k value (k=2). For a target span t, the evidence is the union ⋃_{ri∈t} e(ri), with scores summed and isolated evidence tokens removed if no other evidence token lies within τ=2 positions. The second mechanism DEP applies a dependency parse (via LAL-Parser) to the answer sentence, finds the closest verb ancestor of a target token, collects that verb's successors except punctuation and irrelevant coordinating constituents, and then augments the token's attribution with the union of those atomic-fact tokens' attributions. The similarity score S is the mean over heads of the attention weights at the layer ⌊L/2⌋+1 of the decoder, which the paper argues outperforms hidden-state cosine similarity and is much cheaper than gradient-based saliency.","core_discovery":"The central claim is that the two mechanisms are complementary and each produces a large gain. Token-wise union alone performs on par with the existing average-hidden-state method (HSSAVG); dependency augmentation alone also helps; but the combination lifts fine-grained attribution accuracy to 93.3 and 84.6 for Qwen2, and 94.0 and 78.2 for Llama2, on QuoteSum and VERI-GRAN respectively, outperforming the two published fine-grained attributors (CCI and HSSAVG) and a reported 2-shot GPT-4 baseline. The paper additionally claims the dependency-augmented union is faithful to the generator as measured by log-probability drops when the attributed evidence is removed, and that union aggregation makes the method faster because token-wise evidence is computed once and reused for every target span.","pith_inferences":["The dependency-expansion idea might be portable to other model-internal probes, such as hallucination detection, where a false claim usually hinges on one atomic fact rather than a whole sentence.","The paper's rule-based coordinating-constituent pruning is English-specific, but the attention-union core is language-neutral, so a learned or multilingual dependency parser could extend the same recipe to other languages with less hand-tuning.","Because evidence sets are computed once per response, the same machinery could enable interactive post hoc attribution, letting a user click any span of an already-generated answer without re-running the model.","The benchmarks contain mostly verbatim spans; if the method is applied to abstractive answers, the dependency enrichment may need to be paired with a paraphrase-aware similarity metric to keep accuracy."],"forward_implications":["The method establishes a new accuracy benchmark for fine-grained attribution on QuoteSum and VERI-GRAN, beating the previous best fine-grained attributors by roughly 13–17 percentage points.","Because token-wise evidence is computed once and reused across all spans of a response, fine-grained attribution becomes fast enough for interactive use, with per-span latency of 22.7 ms on QuoteSum versus 84.5 ms for HSSAVG and 2921.8 ms for CCI.","Dependency augmentation improves faithfulness of attribution to the generation process, approaching the oracle upper bound in the paper's log-probability-drop measurement.","The two techniques transfer to sentence-level attribution and to other similarity metrics, improving the citation quality of self-citation and attribute-then-generate pipelines.","Approximating attention with an open-source LLM preserves accuracy, so the method can attribute answers from black-box generators."],"supporting_citations":[{"why":"Supplies the HSSAVG baseline, the VERI-GRAN dataset, and the evaluation protocol the proposed method must beat.","marker":"Phukan et al., 2024"},{"why":"Introduces contrastive feature attribution, the saliency-based method that underlies the CCI baseline.","marker":"Yin and Neubig, 2022"},{"why":"Adapts CCI to model-internals attribution and provides the CTI method used to select context-sensitive target spans in the faithfulness experiments.","marker":"Qi et al., 2024"},{"why":"Provides the Inseq toolkit through which the CCI baseline is implemented.","marker":"Sarti et al., 2023"},{"why":"Provides SEMQA/QuoteSum, one of the two fine-grained attribution benchmarks.","marker":"Schuster et al., 2024"},{"why":"Provides the LAL-Parser used to construct the dependency trees that DEP relies on.","marker":"Mrini et al., 2020"},{"why":"Supplies the ContextCite removal-based log-probability-drop measurement used to test faithfulness.","marker":"Cohen-Wang et al., 2024"},{"why":"Provides the ALCE sentence-level attribution benchmarks and the self-citation baseline.","marker":"Gao et al., 2023b"}],"fun_headline_variants":["Attention union plus dependency parsing tops RAG attribution","Token union and dependency parsing boost attribution accuracy","Dependency-augmented attention sets new attribution records","Faster, more accurate fine-grained attribution via attention union"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a response token's mean attention to prompt tokens in a chosen middle layer encodes genuine semantic support, so that the top-k attended tokens are the right evidence; if attention is determined more by syntax, position, or attention sinks, the reported gains could be coincidental.","fun_headline_variants_meta":{"raw":{"variants":["Attention union plus dependency parsing tops RAG attribution","Token union and dependency parsing boost attribution accuracy","Dependency-augmented attention sets new attribution records","Faster, more accurate fine-grained attribution via attention union"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1150,"prompt_tokens":871,"completion_tokens":279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":487,"tokens_out":279,"duration_ms":2972,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:58:25.769725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun ATTN UNION DEP on the same benchmarks after paraphrasing the retrieved documents so that no target span appears verbatim; if accuracy collapses, the reported gains are explained by near-verbatim span overlap instead of by the attention-plus-dependency mechanism.","supporting_citations":[],"review_version":1}