{"id":"e502dfb9-eebc-472a-a3f5-b0902463aecb","arxiv_id":"2501.15061","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A polarity-aware linear attention with a learnable channel-wise power function improves vision transformer accuracy by up to 4.6% while keeping linear complexity.","lead":"PolaFormer is a linear attention mechanism for vision transformers that separates positive and negative query-key interactions and rescales them with a learnable power function, reporting accuracy gains of up to 4.6% over prior efficient attention baselines. It matters because it targets the speed-versus-expressiveness gap in attention, relevant for high-resolution images and long sequences.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's entropy-reduction guarantee is false: the proof's reduction to a univariate f is invalid, and even the power function used fails on a two-key example.","rationale":"The reader's weakest assumption correctly identified the entropy-reduction theory as the risky component, pointing to Lemma 2's failure for f(0)>0 and Lemma 1's reversed Jensen inequality. My stress test agrees that the theory is the load-bearing weakness, but finds an even more fundamental problem: the theorem is false for the exact power function the paper deploys, because the vector-level inner product is not reducible to a univariate f of the scalar inner product. The counterexample with x=(1,1,1), k1=(10,10,10), k2=(15,1,1) and g(t)=t² directly refutes Eq. (8). This matters because the abstract and Sec. 4.2 cite Theorem 1 as the justification for the learnable power function. The empirical evaluation, however, is broad and consistent: ImageNet-1K, COCO, ADE20K, and LRA results are reported with multiple backbones and ablations, and the complexity analysis in Eq. (10) is plausible. A false supporting theorem does not necessarily invalidate the architecture's observed performance, but the paper must either remove or replace the theorem, or reframe the power function as a purely empirical design choice. Since the reader already conditioned acceptance on fixing this theory, the verdict should remain unchanged: conditional acceptance pending correction of the theoretical claim.","tokens_in":18371,"tokens_out":10927,"duration_ms":101489,"concrete_test":"Compute PSE for the explicit counterexample with x=(1,1,1), k1=(10,10,10), k2=(15,1,1), g(t)=t²: original scores (30,17) give PSE ≈ 0.654 nats, transformed scores (300,227) give PSE ≈ 0.683 nats, violating Eq. (8). Also test Lemma 2 with f(t)=t²+1 and (a,b)=(10,9); the ratio f(a)/f(b) < a/b, and PSE increases. Either check settles whether the theorem as stated can hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is Theorem 1 in Sec. 4.2 and its proof in Appendix A.1. The proof assumes that applying g element-wise and taking inner products yields a univariate function f(⟨x,y⟩)=⟨g(x),g(y)⟩, so that Lemma 2 can be invoked on each scalar ⟨x,y_n⟩. No such well-defined f exists: for fixed x, keys y and y′ with the same inner product can give different transformed inner products (e.g., x=(1,1), y=(1,0), y′=(0.5,0.5) both give ⟨x,y⟩=1, but with g(t)=t² the transformed values are 1 and 0.5). Lemma 2 itself is also false for f with f(0)>0, e.g., f(t)=t²+1 and (a,b)=(10,9). More seriously, Theorem 1 is false even for the paper's own power function g(t)=t^p, p>1. Take x=(1,1,1), y1=(10,10,10), y2=(15,1,1), g(t)=t². Original scores are ⟨x,y1⟩=30, ⟨x,y2⟩=17, PSE≈0.654 nats; transformed scores are ⟨g(x),g(y1)⟩=300, ⟨g(x),g(y2)⟩=227, PSE≈0.683 nats, contradicting Eq. (8). Lemma 1's Jensen step is also reversed: for convex g, E[g(q_i)g(k_i)] ≥ g(E q_i)g(E k_i), not ≤, so the claimed concavity is wrong. Since the learnable power function is explicitly justified by Theorem 1, the paper's stated theoretical guarantee is unproven and false as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PolaFormer, a linear-attention mechanism that separately models same-signed and opposite-signed query-key interactions through a polarity-aware decomposition, mixes these streams with learnable sign-aware coefficient matrices, and applies a channel-wise learnable power function to restore the 'spiky' low-entropy behavior of softmax attention. The authors provide a theoretical claim (Theorem 1) that element-wise functions with positive first and second derivatives reduce the entropy of the attention distribution, and they support this with experiments on ImageNet-1K, COCO, ADE20K, and Long Range Arena, reporting gains up to 4.6% while preserving linear complexity in sequence length.","tokens_in":18788,"tokens_out":13934,"duration_ms":115246,"significance":"If the empirical results are reproducible, PolaFormer is a potentially useful linear-attention variant with broad, consistent improvements across classification, detection, segmentation, and LRA, and the released code is a concrete strength. However, the paper's headline theoretical guarantee, Theorem 1, is false as stated, and the proof in Appendix A.1 contains several invalid steps. Since the learnable power function is explicitly motivated by this theorem, the theoretical foundation for the 'spikiness' component is not established. The significance of the work therefore currently rests on the empirical evaluation, which is extensive but does not by itself support the theoretical claims made in the abstract and Section 4.2.","major_comments":[{"comment":"Theorem 1 is false as stated, and the proof is invalid. For x=(1,1,1), y1=(10,10,10), y2=(15,1,1), and g(t)=t^2 (which satisfies g'>0 and g''>0), the original scores are ⟨x,y1⟩=30 and ⟨x,y2⟩=17, giving PSE≈0.654 nats; the transformed scores are ⟨g(x),g(y1)⟩=300 and ⟨g(x),g(y2)⟩=227, giving PSE≈0.683 nats, which violates Eq. (8) for the paper's own power function. The proof breaks down because (i) Lemma 1 defines f(⟨x,y⟩):=⟨g(x),g(y)⟩, but no such univariate function exists for d>1: with x=(1,1), y=(1,0), and y'=(0.5,0.5), both have ⟨x,y⟩=1 but ⟨g(x),g(y)⟩=1 and ⟨g(x),g(y')⟩=0.5 for g(t)=t²; (ii) the Jensen step in Eq. (15) has the inequality reversed for convex g (since E[g(X)]≥g(E[X])), and the conclusion that f is 'concave ... having a positive second derivative' is self-contradictory; and (iii) Lemma 2's key claim that d:=f(a)/f(b)>c:=a/b is not implied by f'>0 and f''>0 when f(0)>0, e.g., f(t)=t²+1000, a=10, b=9 gives d≈1.018<c≈1.111. The main-text theorem also omits the 'dimensions are independently distributed' condition that is inserted in the appendix statement, and that probabilistic assumption does not justify a deterministic inequality on fixed vectors.","section":"§4.2, Appendix A.1, Eq. (8)"},{"comment":"Because the entropy-reduction claim is false for the very power function adopted in Eq. (9), the statement that Theorem 1 'provides insights into why commonly used feature maps such as ReLU or ELU+1 fail to reduce entropy' and the design of learnable exponents p are not supported by the stated theory. The counterexample in the previous comment uses exactly g(t)=t² with p=2, showing that the power function can increase PSE. The empirical success of the power function may still be valid, but the theoretical justification advanced in the abstract and in Section 4.2 for the 'loss of attention spikeness' motivation is not established. This is load-bearing because the choice of g(x;p) rests entirely on Theorem 1.","section":"§4.2, Eq. (9)"},{"comment":"The text says the polarity-aware mixing handles 'both types of interactions without introducing additional learnable parameters' (Introduction and Section 4.1), but immediately defines G_s ∈ R^{N×d/2} and G_o ∈ R^{N×d/2} as 'two learnable polarity-aware coefficients matrices applied with element-wise multiplication'. This is an internal contradiction that must be resolved. Additionally, the R^{N×d/2} notation suggests parameters that depend on the sequence length N, which would be atypical and inconsistent with the complexity analysis in Eq. (10); if these are intended as per-channel scalars, the notation should be corrected to R^{d/2} or an equivalent broadcast shape.","section":"§1 and §4.1, Eq. (6)"}],"minor_comments":[{"comment":"In the proof of Lemma 2, the line 'H1 = h(c) > H2 = h(c)' appears to be a typo for 'H2 = h(d)' with d>c; as written, the inequality is between identical arguments. Also, the claim that the argument 'extendable to N > 2' is asserted without proof.","section":"Appendix A.1, Lemma 2"},{"comment":"The checkmark pattern in Table 3 is ambiguous; the text refers to 'row 1 and row 4' for DCN versus DWC, but the columns are not clearly aligned with the headers. Please reformat with explicit row labels and unambiguous column markers.","section":"§5.4, Table 3"},{"comment":"The sentence 'For instance, in Table 1, our DeiT-T-PolaFormer surpasses other DeiT variants from 0.5% to 6.3%' is followed by a discussion of PVT-T results; clarify which model is being compared and to which baselines.","section":"§5.1, third paragraph"},{"comment":"The phrase 'the complexity complexity of PolaFormer' should be 'the computational complexity of PolaFormer'.","section":"Eq. (10)"},{"comment":"The phrase 'The training interation is set to 40000' should be 'The training iteration is set to 40000'.","section":"Appendix A.2"},{"comment":"The kernel notation SM(q,k)=exp(q_i k_j^⊤) mixes vector and scalar notation; please use ⟨q,k⟩ or q^⊤k consistently.","section":"§3.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The empirical evaluation is broad and could support publication once the theoretical claim is corrected or removed. The false Theorem 1 is a serious problem: it is stated prominently in the abstract, and the power function that is a core design element is claimed to be justified by it. I would not accept the paper with the theorem as currently stated, but the empirical core is defensible and the architecture is a reasonable contribution if the theory is retracted or replaced with a correct statement. If the authors choose to keep Theorem 1 unchanged, I would recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PolaFormer is worth reading for the mechanism, not for the theory. The polarity split plus per-channel learnable power function gives consistent gains across ImageNet, COCO, ADE20K, and LRA, and the ablations are honest enough that I believe the effect is real. The entropy-reduction theorem, on the other hand, is false as stated.\n\nWhat's new and good: decomposing q and k into positive and negative parts and treating same-signed and opposite-signed interactions separately is a clean idea that prior linear-attention work didn't do. The learnable exponent per channel is a natural extension of FLatten's fixed norm. The empirical work is broad: three vision tasks plus LRA, multiple backbones, and the gains over FLatten and the base models are consistent, in the 0.3–4.6% range. The efficiency numbers (Table 6) also look reasonable. Code is public. I'd trust the experiments.\n\nWhere it gets soft: Theorem 1. The proof in Appendix A.1 defines f(⟨x,y⟩)=⟨g(x),g(y)⟩, but no such univariate f exists: two different y's with the same inner product against x can give different transformed inner products. Lemma 1's Jensen step is reversed—for convex g you get E[g(q)g(k)] ≥ g(E q)g(E k), not ≤. Lemma 2 needs f(0)=0 to make d>c, and even the paper's own power function fails it: take x=(1,1,1), y1=(10,10,10), y2=(15,1,1), g(t)=t². The PSE goes from about 0.654 to 0.683, contradicting Eq. (8). So the theorem is not just missing rigor; it's false. The method may still work—and apparently does—but the theoretical motivation needs to be rewritten as an observation or conjecture, not a proven guarantee.\n\nAlso minor: Section 4.1 says the split introduces 'no additional learnable parameters,' then immediately introduces learnable Gs and Go. And Gs, Go are sized R^{N×d/2}, meaning their parameter count grows with sequence length; that deserves a clarification, even if it's a per-token scale.\n\nWho this is for: anyone working on efficient attention who wants a working engineering recipe. It deserves a serious referee, but with a hard ask: fix or demote the theory, and clarify the parameter dependence. I'd cite it for the mechanism and the empirical comparison, not for the proof.","headline":"Solid empirical linear-attention paper with a real mechanism, but the stated entropy-reduction theorem is false as written and needs to be fixed or demoted.","tokens_in":19202,"tokens_out":2202,"would_cite":true,"duration_ms":19664,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PolaFormer shows that linear attention can recover the information lost when softmax is replaced: explicitly modeling same-signed and opposite-signed query-key interactions and rescaling each channel with a learnable power function…","keywords":["polarity-aware attention","linear attention","vision transformers","entropy reduction","learnable power function","kernelized attention","Long Range Arena","ImageNet classification"],"falsifier":"Search numerically over pairs of positive sequences $(a, b)$ and increasing convex functions $g$ with $g'(x)>0$ and $g''(x)>0$—for example $g(x) = x^2 + 1$ applied to $(1, 0.1)$—and check whether $\\mathrm{PSE}(g(a), g(b))$ is always strictly less than $\\mathrm{PSE}(a, b)$; a single counterexample pair where the transformed entropy is not lower would falsify Theorem 1 and the paper's stated theoretical basis for the power function.","tokens_in":18216,"feed_emoji":"⚡","tokens_out":9632,"duration_ms":76017,"temperature":0.7,"pith_summary":"The paper argues that linear attention falls short of softmax attention for two correctable reasons: it discards all interactions involving negative query or key coordinates, and it produces flatter, higher-entropy attention distributions. PolaFormer answers both by decomposing queries and keys into positive and negative parts, computing same-signed and opposite-signed similarities in separate streams with learnable mixing weights, and rescaling each channel with a learnable power function. The paper claims a theorem that any element-wise function with positive first and second derivatives strictly reduces the entropy of the attention distribution, which justifies the power-function sharpening. On ImageNet-1K, COCO, ADE20K, and the Long Range Arena, PolaFormer improves over linear-attention baselines by up to 4.6% while keeping complexity linear in sequence length.","feed_headline":"Linear attention recovers negative query-key interactions, up 4.6%","feed_subtitle":"A sign-aware two-stream attention plus learnable power scaling restores softmax-like focus without quadratic compute.","key_machinery":"The machinery is a polarity-aware attention that runs two parallel linear-attention streams: one for same-signed pairs $(q^+, k^+)$ and $(q^-, k^-)$, and one for opposite-signed pairs $(q^+, k^-)$ and $(q^-, k^+)$, with value vectors split along the channel dimension and the two output halves gated by learnable sign-aware coefficient matrices $G_s$ and $G_o$. The second component is the positive-sequence entropy (PSE) measure, which defines the entropy of a non-negative sequence and is used to prove the theorem; the chosen entropy-reducing function is a per-channel power function $x^p$ with learnable exponent $p = 1 + \\alpha\\sigma(w)$, sharpening strong similarities relative to weak ones. A depthwise convolution is added to counteract the low-rank tendency of the softmax kernel, which the paper argues causes degenerate solutions when the value space is compact.","core_discovery":"The central claim is that the information loss in linear attention is not inevitable: writing $q = q^+ - q^-$ and $k = k^+ - k^-$ shows that the full dot product consists of same-signed minus opposite-signed terms, and a learnable sign-aware mixing of the two streams—rather than a direct subtraction—can preserve negative-value interactions while keeping attention weights non-negative and training stable. To restore spikiness, the paper introduces a channel-wise learnable power function $g(x_i) = x_i^{p_i}$ with $p_i = 1 + \\alpha\\sigma(w_i)$ applied to the ReLU-transformed positive and negative parts, and proves that element-wise functions with $g'>0$ and $g''>0$ reduce the positive-sequence entropy of the resulting attention distribution. Inserted into DeiT, Swin, PVT, and PVTv2 backbones, the module reports consistent gains: up to +3.7% top-1 accuracy on ImageNet-1K, up to +4.6% on COCO detection metrics, and up to +2.6% mIoU on ADE20K, while its LRA variants average up to 60.72% with substantially lower peak memory than softmax attention.","pith_inferences":["This suggests the sign decomposition could transfer to linear attention in language and audio modeling, where negative query-key correlations may carry different semantics than in vision.","If the entropy-reduction theorem turns out to hold only for power functions rather than the full claimed class, the learnable exponent still stands on its own as a sharpening layer for any kernel attention; the empirical gains would not require the general theorem.","The learned exponents p per channel offer a diagnostic: channels that converge near p=1 are already sufficiently sharp, while channels with large p are where linear attention over-smooths most, which could guide architectural capacity allocation.","The use of a convolution specifically to raise attention rank suggests rank deficiency, not just spikiness, is a bottleneck for linear attention, and the same rank-repair idea could improve other sub-quadratic attention designs."],"forward_implications":["Linear attention can encode negative-value interactions without sacrificing the O(N) per-head complexity, because both the same-signed and opposite-signed streams use the same kernelized associative trick.","A learnable per-channel power exponent gives the model a differentiable, data-dependent way to sharpen attention maps, separating strong from weak query-key responses.","Dropping the module into existing backbones (DeiT, Swin, PVT, PVTv2) improves ImageNet-1K top-1 accuracy by +0.3% to +3.7% at comparable FLOPs.","On dense prediction, fine-tuning PolaFormer backbones improves COCO detection by up to +4.6% AP and ADE20K segmentation by up to +2.6% mIoU, consistent with the claim that fine-grained attention benefits from lower entropy.","On the Long Range Arena, the best PolaFormer variant averages 60.72% with roughly 0.22x the peak memory of softmax attention, indicating the benefits carry over to long-sequence tasks outside vision."],"supporting_citations":[{"why":"Supplies the kernelized linear-attention formulation and the 1+ELU feature map that PolaFormer builds on and aims to improve.","marker":"Katharopoulos et al., 2020"},{"why":"Provides the FLatten Transformer power-function rescaling that PolaFormer extends with per-channel learnable exponents, and serves as a direct comparison baseline.","marker":"Han et al., 2023a"},{"why":"Skyformer contributes the Gaussian-kernel/Nystrom attention background and the LRA training setup that PolaFormer adopts for long-sequence evaluation.","marker":"Chen et al., 2021"},{"why":"Cosformer is the representative ReLU-based non-negative linear attention whose dropped negative interactions the paper identifies as the information loss to fix.","marker":"Qin et al., 2022"},{"why":"Defines the Long Range Arena benchmark used to compare PolaFormer against other linear attention models on long-sequence tasks.","marker":"Tay et al., 2021"},{"why":"Supports the claim that the softmax kernel is low-rank, which motivates the convolution added to counter degenerate solutions in PolaFormer.","marker":"Verma, 2021"}],"fun_headline_variants":["Polarity split restores negative query-key signal in attention","Sign-aware linear attention recovers softmax-like focus, up 4.6%","Learnable power scaling sharpens linear attention, gains 4.6%","Two-stream polarity attention: sign matters for ViTs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that any increasing strictly convex element-wise function maps every positive ratio $a/b$ to a strictly larger ratio $f(a)/f(b)$, which is what the entropy-reduction proof needs; if the ratio-widening property fails for some functions in the class, the theorem's guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["Polarity split restores negative query-key signal in attention","Sign-aware linear attention recovers softmax-like focus, up 4.6%","Learnable power scaling sharpens linear attention, gains 4.6%","Two-stream polarity attention: sign matters for ViTs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000289,"raw_usage":{"total_tokens":1723,"prompt_tokens":1004,"completion_tokens":719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":642}},"tokens_in":620,"tokens_out":719,"duration_ms":6097,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:40:22.939422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search numerically over pairs of positive sequences $(a, b)$ and increasing convex functions $g$ with $g'(x)>0$ and $g''(x)>0$—for example $g(x) = x^2 + 1$ applied to $(1, 0.1)$—and check whether $\\mathrm{PSE}(g(a), g(b))$ is always strictly less than $\\mathrm{PSE}(a, b)$; a single counterexample pair where the transformed entropy is not lower would falsify Theorem 1 and the paper's stated theoretical basis for the power function.","supporting_citations":[],"review_version":1}