{"id":"ca4e78b3-f7e1-4a1f-bf48-50e5635e1fd0","arxiv_id":"2506.02138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PA-LRP extends Layer-wise Relevance Propagation to attribute relevance to positional encodings in Transformers, improving faithfulness of explanations.","lead":"A new explainability technique, PA-LRP, adds positional encoding to the set of components attributed by Layer-wise Relevance Propagation in Transformers. It claims to fix a blind spot in state-of-the-art LRP and to improve explanation faithfulness in language and vision models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theoretical grounding of PA-LRP is not established: Lemma 3.2's conservation proof is circular and Eq. 10's RoPE attribution rule has a dimensional mismatch, so the central 'conservation violation' motivation needs correction or softening.","rationale":"The reader's weakest assumption correctly identifies that Lemma 3.1's conservation violation depends on redefining the input space. My pass confirms that and goes further: Lemma 3.2's proof is circular, and Eq. 10 has a concrete dimensional mismatch. These are internal to the paper's formal claims and affect the central claim because the abstract and contributions rest on 'theoretically grounded' and 'violation of conservation.' If the proofs are restatements or the main RoPE rule is not well-defined, the paper's scientific contribution reduces to an empirical recipe. The experiments are plausible, the code is public, and the gains, while modest and lacking error bars, could still be real; I would not reject on this basis. The paper should be accepted only with revisions that correct the conservation accounting, specify the RoPE attribution rule so dimensions match, and substantiate 'significantly outperforms' with error bars or statistical tests. This matches the reader's CONDITIONAL verdict, so I recommend no change in the overall decision.","tokens_in":17148,"tokens_out":8591,"duration_ms":81907,"concrete_test":"Run the public PA-LRP code on a single RoPE transformer layer with D=4 and L=2; print the shapes of R(tilde Q_i), R(tilde K_i), and R(R_i,k) before the flattening in Eq. 9. If R(R_i,k) cannot be assembled as a DxD (or D^2) tensor from the two D-vectors, Eq. 10 is undefined as published. Independently, compute total relevance before and after the RoPE layer with and without sinks: if R(l)=R(l-1)+R_P(l) is used, the per-layer total grows, contradicting the conservation definition R(output)=R(input) given in Section 2.2; a corrected rule should show R(l-1)=R(l)+R_P(l).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that existing LRP methods violate conservation and that PA-LRP restores it. This is not established. Lemma 3.1 (Appendix E) is definitional: it redefines the input space as separate semantic and positional tensors (Eq. 4) and then observes that attributing only to E omits P. For the summed tensor Z=P+E that AttnLRP actually propagates, conservation is not violated. Lemma 3.2's proof is circular: it asserts the recursion R(l)=R(l-1)+R_P(l) (Eq. 22), which is exactly the equality R(M)=R_E+sum_l R_P(l) that it claims to prove. Under the paper's own conservation definition (R(output)=R(input) per layer), diverting R_P(l) to a sink should give R(l)=R(l-1)-R_P(l), with R_P(l) conserved in the sink, not added to the running total. Moreover, Eq. 10 defines R(R_i,k) as the average of two D-dimensional relevance vectors R(tilde Q_i) and R(tilde K_i), while R_i,k is a DxD matrix, later flattened to a D^2 vector in Eq. 8. The rule as written does not specify how a D^2-dimensional positional attribution is obtained from two D-vectors. These are internal inconsistencies, not disagreements with external consensus. The empirical results may survive, but the 'theoretically grounded' support for the headline claim is currently missing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PA-LRP, a Layer-wise Relevance Propagation (LRP) extension for Transformer explainability that attributes relevance to positional encodings (PE). The authors reformulate the explainability input space as a set of position-token pairs (Eq. 4), introduce LRP rules for input-level PEs (learnable and sinusoidal) and attention-level PEs (RoPE and ALiBi), and claim that existing LRP methods violate the conservation property because they ignore PE. They provide three lemmas intended to prove conservation violation, conservation satisfaction, and low faithfulness of existing methods, and they evaluate PA-LRP on DeiT vision models and on LLaMA-based NLP models using perturbation and segmentation tests, reporting improvements over the AttnLRP baseline. The paper also releases code and includes ablations isolating a 'PE Only' attribution map.","tokens_in":17428,"tokens_out":3680,"duration_ms":30791,"significance":"If the theoretical claims and empirical results held, PA-LRP would be a useful contribution to Transformer explainability: it identifies a genuinely overlooked component (positional encoding) and provides an open-source implementation with experiments across vision and NLP models. The 'PE Only' ablation is a valuable probe of how much positional signal LRP-based methods currently discard, and the multi-sink design is an interesting engineering choice. However, the paper's central theoretical grounding is not established: the conservation lemmas are largely definitional or circular, and the RoPE attribution rule contains a dimensional mismatch. The empirical gains are also small and inconsistent, with no error bars or significance tests, so the headline claim of 'significantly outperforming' the state of the art is not quantitatively supported. The empirical contribution may survive a revision, but the theoretical claims need substantial reframing or removal.","major_comments":[{"comment":"The claimed conservation violation is a consequence of the chosen bookkeeping rather than a property of the network's computation. The model consumes the summed input Z = P + E; for that tensor, standard LRP conservation holds. Attributing relevance only to E in the final explanation omits P by construction of the output space, not because the Transformer's computation violates conservation. Lemma 3.1 should be reframed as a design choice about which input components to explain, not as a violation of conservation.","section":"Section 3.1 / Appendix E, Lemma 3.1"},{"comment":"The proof of Lemma 3.2 is circular. Equation (22) asserts the recursion R(l) = R(l-1) + R_P(l), which is exactly the equality R(M) = R_E + sum_l R_P(l) that the lemma claims to prove. Under the paper's own conservation definition (output relevance equals input relevance per layer), diverting R_P(l) to a separate sink would imply R(l) = R(l-1) - R_P(l) if the sink is external, or R(l) = R(l-1) if the sink is included in the conserved quantity. The recursion must be derived from the propagation rules, not assumed.","section":"Section 3.4 / Appendix E, Lemma 3.2"},{"comment":"There is a dimensional mismatch in the RoPE attribution rule. Equation (10) defines R(R_i,k) as the average of two D-dimensional relevance vectors R(\\tilde Q_i) and R(\\tilde K_i), but R_i,k is a D×D rotation matrix per Eq. (7), later flattened to a D^2-dimensional vector in Eqs. (8)-(9). The rule does not specify how a D^2-dimensional positional attribution is constructed from two D-dimensional vectors; as written, the RoPE attribution step is undefined.","section":"Section 3.3, Eq. (10) and Eqs. (7)-(9)"},{"comment":"The claim that the method 'significantly outperforms' the baseline is not supported by the reported numbers. In Table 4, PA-LRP is worse than AttnLRP in three of six positive-perturbation settings (Base Target, Tiny Predicted, Tiny Target), and in Table 1 the Tiny-LLaMA generation AU-MSE of the full method (4.915) is worse than the PE-Only ablation (3.918). No error bars, confidence intervals, or significance tests are reported anywhere in the paper, so the aggregate 'significant' claim is not established.","section":"Table 4 and Section 4.1, Tables 1 and 3"},{"comment":"Lemma 3.3 is an illustrative existence proof for a deliberately degenerate model in which all semantic weights are zero and the model uses only positional information. It does not quantify faithfulness, and it does not show that existing LRP methods have low faithfulness in general. The lemma should be labeled as a motivating example rather than a general faithfulness theorem, or it should be replaced with a more rigorous statement.","section":"Appendix E, Lemma 3.3"}],"minor_comments":[{"comment":"The row label 'Tiny-LLaMa-2 7B' is inconsistent with the model name TinyLLaMA used in the text and with the paper's own description in Section 4.1; this is likely a copy-paste error and should be corrected.","section":"Table 1"},{"comment":"The notation R(\\tilde Q_i) and R(\\tilde K_i) is used before \\tilde Q_i and \\tilde K_i are defined; the definitions in Eq. (21) appear only in the appendix proof. Please define them in Section 3.3.","section":"Section 3.3, Eq. (10)"},{"comment":"The LRP rule for subtraction is written with the epsilon term in the denominator as i + (-j) + epsilon, which is confusing for an operation i - j; the sign convention should be stated explicitly.","section":"Appendix B, Eq. (16)"},{"comment":"The caption says the figure visualizes 'the violation of the conservation rule,' but the figure plots the ratio of positional to total relevance. The measured relevance shares do not by themselves demonstrate a conservation violation; the connection should be explained.","section":"Figure 8"},{"comment":"The abbreviation 'PA-LRP' is used in the abstract but introduced only in Section 1; please define it at first use.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The theoretical section needs substantive rework rather than cosmetic changes: Lemmas 3.1 and 3.2 should be reframed as design choices or derived properly, and Eq. (10) must be fixed to define the RoPE attribution map. The empirical evaluation would also need error bars or significance tests to support the 'significant' claim. If the authors can address these points, the paper's contribution — identifying PE as a missing attribution target and providing an open-source implementation — could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the PE-specific LRP rules. Nobody had explicitly propagated relevance through positional encodings before, and covering learnable, sinusoidal, RoPE, and ALiBi with public code is a genuine contribution. The ablation finding that PE-only attribution is competitive with AttnLRP on several metrics is the most striking result in the paper, and it suggests positional relevance carries signal that prior methods threw away.\n\nThat said, the theoretical framing needs a serious rewrite. Lemma 3.1 does not show that ignoring PE violates conservation; it shows that if you redefine the input space as separate semantic and positional tensors and then drop the positional term, the sum no longer matches. Standard LRP on the summed input Z=P+E conserves fine, so the \"violation\" is a consequence of the chosen bookkeeping, not the network's computation. Lemma 3.2's proof is circular: the recursion in Eq. 22 is exactly the equality it claims to prove, and the sign convention (adding R_P(l) to R(l-1)) is asserted rather than derived. The dimensional mismatch in Eq. 10 is more concrete: R(R_i,k) is a D×D matrix, but the RHS averages two D-vectors, with no explanation of how that yields a D^2-dimensional positional attribution. That is an internal inconsistency, not a disagreement about taste.\n\nThe empirical claims also overshoot. Gains over AttnLRP are small and mixed: in DeiT positive perturbation, the method loses in half the cases, and no error bars are given anywhere. The text's \"significant outperformance\" phrasing is not supported. The 51.41% AU-MSE improvement claimed for Tiny-LLaMa in Section 4.1 appears to belong to the PE-Only ablation, not to the full method, based on Table 1. That should be corrected.\n\nWho is this for? Researchers actively working on LRP for Transformers will find the rules implementable and the PE-Only result worth investigating. The theory should be trimmed to a remark about conservation under the chosen input-space reformulation, or removed entirely.\n\nRecommendation: send to peer review, but expect major revision. The method is worth engaging with; the claims need to be tempered and the internal inconsistencies fixed.","headline":"PE-aware LRP rules are a real gap worth filling, but the conservation theory is mostly bookkeeping and the proof of Lemma 3.2 is circular; the empirical gains are real but overstated.","tokens_in":18000,"tokens_out":3000,"would_cite":false,"duration_ms":28575,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prior LRP explanations for Transformers omit positional encoding, and that PE-aware LRP rules restore conservation and improve attribution in vision and NLP.","keywords":["Layer-wise Relevance Propagation","Transformer explainability","positional encoding","PA-LRP","RoPE","conservation property","attention attribution","vision transformer segmentation"],"falsifier":"On a synthetic task where the label is determined only by token position, compare AttnLRP's attribution map with the true position rule: if AttnLRP assigns zero or near-zero relevance to the decisive position while the model is accurate, the paper's claim that positional relevance is missing is directly confirmed, whereas if AttnLRP already highlights the position through the summed input $E+P$, the conservation-violation argument is exposed as bookkeeping rather than a computational gap.","tokens_in":16880,"feed_emoji":"📍","tokens_out":7629,"duration_ms":72942,"temperature":0.7,"pith_summary":"The paper tries to establish that every existing LRP-based explanation method for Transformers has been ignoring a real contributor to predictions: the positional encoding. It argues that this omission breaks LRP's conservation property once the input is viewed as position-token pairs, and it throws away a distinct, structurally meaningful kind of relevance. The proposed method, PA-LRP, adds specialized LRP rules for learnable, sinusoidal, Rotary, and ALiBi positional encodings, with per-layer sinks that absorb positional relevance. If the paper is right, Transformer explanations become more faithful and more complete at no extra forward-pass cost, and the positional relevance maps alone are already competitive with the previous state of the art.","feed_headline":"PE-aware LRP rules beat standard Transformer explanations","feed_subtitle":"New propagation rules route relevance through positional encodings, which restores conservation and produces sharper attribution maps in…","key_machinery":"The central object is the reformulated input space of position-token pairs, with each layer given its own positional sink so that positional relevance is not aggregated away. The rules that carry the argument are: for input-level PE, the LRP-$\\epsilon$ addition rule $R(P_i') = P_i' R(z_i)/(P_i' + E_i + \\epsilon)$; for RoPE, flattening the rotation matrix and setting $R(R_{i,k}) = \\tfrac12 R(\\tilde Q_i) + \\tfrac12 R(\\tilde K_i)$; and a multi-sink aggregation that sums only positive semantic and positional contributions across layers. Sinusoidal PE is reparameterized as a learnable-PE-style matrix so the same rule applies, and ALiBi gets an analogous distance-based rule.","core_discovery":"On the paper's own terms, the central discovery is that positional encodings are not a neutral preprocessing detail for Layer-wise Relevance Propagation; they carry a substantial, unique share of the relevance that standard LRP-based methods discard. By reformulating the input space as $\\{(E_i, (P_{i,1}, \\dots, P_{i,K}))\\}$ rather than $\\{E_i\\}$ and by defining LRP rules that propagate relevance through the PE computation itself, the paper obtains explanations that satisfy conservation and, in its experiments, outperform the AttnLRP baseline on perturbation and segmentation tests for both language models and vision Transformers.","pith_inferences":["The conservation violation in Lemma 3.1 is a bookkeeping result: standard LRP conserves relevance for the summed tensor $E+P$, and the paper's claim requires accepting that the input space should be split into separate semantic and positional components; if a reader rejects that split, the method's theoretical motivation shifts to its empirical gains.","The same position-token separation could be ported to other propagation-based attribution methods, where positional terms are currently folded into the input; a testable extension would be whether those methods also gain from explicit PE sinks.","PE-only maps might serve as a cheap unsupervised segmentation or object-localization signal in vision Transformers, since the paper's qualitative results show the positional channel highlights whole objects rather than sparse discriminative patches.","One could test the positional-concepts hypothesis directly by comparing PE-only relevance on images of objects in typical versus atypical contexts, since the paper predicts that typical-context objects receive more positional relevance."],"forward_implications":["PA-LRP can be layered on top of any LRP method that propagates through attention, not only the AttnLRP baseline, so the PE-aware rules are a modular addition.","Transformer explanations now include a separate positional channel that captures whole-object, structural, and ordering information; combining it with semantic relevance produces less fragmented maps.","Because the method uses only an equivalent forward and backward pass and no model modification, it applies to large zero-shot models such as LLaMA 3 at the cost of a single backward pass.","The PE-only ablation shows that positional relevance alone is a competitive explanation signal, meaning previous methods were discarding information that can by itself rival state-of-the-art attributions.","Conservation is restored for both input-level and attention-level PE, giving PA-LRP the same theoretical grounding as standard LRP."],"supporting_citations":[{"why":"Primary baseline and the attention-propagation framework that PA-LRP extends with positional rules.","marker":"[2]"},{"why":"Prior conservative LRP rules for LayerNorm and attention that established the propagation style PA-LRP builds on.","marker":"[3]"},{"why":"Introduces Layer-wise Relevance Propagation and the conservation principle central to the paper's lemmas.","marker":"[9]"},{"why":"Hybrid attention-LRP method whose positive-relevance aggregation PA-LRP adapts for multi-sink summation.","marker":"[16]"},{"why":"Deep Taylor Decomposition, used to justify the LRP-epsilon rules applied to PE terms.","marker":"[31]"},{"why":"Defines ALiBi, the attention-level positional bias for which the paper gives a PA-LRP rule.","marker":"[34]"},{"why":"Defines RoPE, the rotary positional encoding for which the paper derives rotation-matrix relevance propagation.","marker":"[41]"},{"why":"Introduces the Transformer architecture and positional encoding, the object the paper makes attributable.","marker":"[45]"}],"fun_headline_variants":["Positional encoding is the missing piece in Transformer LRP","New LRP rules route relevance through positional encodings","Fixing LRP's conservation violation with positional attribution","Why LRP fails without positional relevance—and how to fix it","Position-aware LRP yields sharper Transformer explanations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the input space for explanation should be split into separate semantic embeddings and positional embeddings, so that relevance assigned to $E+P$ must be decomposed into two parts; if one treats $E+P$ as a single tensor, standard LRP already conserves relevance and the claimed violation disappears.","fun_headline_variants_meta":{"raw":{"variants":["Positional encoding is the missing piece in Transformer LRP","New LRP rules route relevance through positional encodings","Fixing LRP's conservation violation with positional attribution","Why LRP fails without positional relevance—and how to fix it","Position-aware LRP yields sharper Transformer explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000743,"raw_usage":{"total_tokens":3277,"prompt_tokens":869,"completion_tokens":2408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":2339}},"tokens_in":485,"tokens_out":2408,"duration_ms":16964,"temperature":1.0,"reasoning_tokens":2339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:29:27.362931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic task where the label is determined only by token position, compare AttnLRP's attribution map with the true position rule: if AttnLRP assigns zero or near-zero relevance to the decisive position while the model is accurate, the paper's claim that positional relevance is missing is directly confirmed, whereas if AttnLRP already highlights the position through the summed input $E+P$, the conservation-violation argument is exposed as bookkeeping rather than a computational gap.","supporting_citations":[{"cited_title":"Attnlrp: attention-aware layer-wise relevance propagation for transformers","cited_arxiv_id":null,"evidence_quote":"Primary baseline and the attention-propagation framework that PA-LRP extends with positional rules."},{"cited_title":"Xai for transformers: Better explanations through conservative propagation","cited_arxiv_id":null,"evidence_quote":"Prior conservative LRP rules for LayerNorm and attention that established the propagation style PA-LRP builds on."},{"cited_title":"Transformer interpretability beyond attention visualization","cited_arxiv_id":null,"evidence_quote":"Hybrid attention-LRP method whose positive-relevance aggregation PA-LRP adapts for multi-sink summation."}],"review_version":1}