{"id":"211f3daf-e5bf-4fb8-b5da-f658ac67e2c9","arxiv_id":"2507.00698","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MALA adds a scale-sensitive offset to linear attention so attention sharpens with query magnitude, improving accuracy across several tasks.","lead":"This paper proposes MALA, a tweak to linear attention that brings back the query-size sensitivity softmax attention has. In tests across image, text, and speech tasks, the modified attention often beats both standard softmax and earlier linear-attention designs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof of Eq. (12)'s concentration claim needs every attention score to stay positive, but the paper's zero-negative-score assertion is unmeasured and mathematically not guaranteed for unbounded magnitude scaling.","rationale":"The reader identified the positivity assumption as the weakest point, and this stress-test agrees: the derivation of Eq. (12) and the Appendix proof of p_m>p depend on all attention scores remaining positive under magnitude scaling, yet the paper only asserts that non-positive scores have zero empirical frequency without any measurement. The concern is sharpened here by noting that, because the raw similarities r_j cannot all exceed their mean, there will always be at least one key whose denominator crosses zero at some finite scaling factor a; the only open question is whether trained models ever reach such a. This is an internal correctness risk rather than a disagreement with consensus, and it is independent of whether the reported task numbers are reproducible. If the concrete test shows no negative scores throughout the tested magnitude range and p_m>p for all pairs, the central mechanism claim would be supported. If it does not, the paper's attribution of its gains to magnitude-aware concentration would be weakened even if the empirical results stand. The reader's CONDITIONAL verdict remains the right assessment: the method may well work, but the load-bearing proof and the empirical support for its key premise are not yet fully established, and the missing code and architecture details reinforce the conditionality rather than overturning it.","tokens_in":16228,"tokens_out":11767,"duration_ms":157594,"concrete_test":"Instrument the released implementation (or re-train MALA DeiT-T/Swin-T per the ablation): for each layer/head/token, compute r_j, S, β, γ and record min(Attn_j) and the count of Attn_j≤0 at initialization and after training. Then, with the direction of ϕ(Q_i) fixed, scale its magnitude by a∈{1,1.5,2,4,8}, recompute the ratio in Eq. (12) for key pairs with p>1, and record the largest a for which all N scores remain positive and the fraction of pairs with p_m>p. If non-positive scores occur in any of these conditions, or if p_m≤p at any tested a, the proof premise fails and the paper's mechanism claim is unsupported; if the full tested range stays positive and p_m>p, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the Appendix proof of p_m>p (Eqs. 20-22). It requires A_n - γx to remain positive for x up to c=aβ/(a+β-1), i.e., the lowest attention score must remain positive when ϕ(Q_i) is scaled by a. The paper asserts this holds because the observed frequency of non-positive scores is zero, but gives no measurement protocol and no bound on the admissible range of a. The condition is not structural: with r_j=ϕ(Q_i)ϕ(K_j)^T, S=Σ r_j, β=1+1/S, and γ=S/N, the score is βr_j-S/N. At a=1 a score is negative whenever r_j<S/(βN), which is possible. Moreover, for any key with r_n<S/N, the pole A_n/γ=βNr_n/S is below β, while c(a) increases from 1 to β as a goes from 1 to infinity; hence there is a finite a at which the denominator in Eq. (22) changes sign. Since not all r_j can exceed their mean, unbounded magnitude scaling inevitably produces negative scores. Once negative scores appear, the derivative argument is not valid on [1,c], and p_m can drop below p or become negative. Therefore Eq. (12)'s monotone concentration claim, and the attributed advantage over Softmax Attention, is unproved for the magnitude range claimed unless positivity is verified empirically for every layer, head, token, and encountered a.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that linear attention ignores query magnitude, which makes its attention distribution too smooth compared with softmax attention, and proposes Magnitude-Aware Linear Attention (MALA). MALA adds a scale-dependent boost β and an offset γ so that attention scores still sum to one while the ratio between two keys' scores grows when the query is scaled, but grows fractionally rather than exponentially. The authors prove this ratio growth in the appendix under a positivity condition, assert that negative/zero scores never occur in their experiments, and report strong empirical results across image classification, detection, segmentation, NLP, speech recognition, and image generation. The central theoretical claim is the monotone-concentration property in Eq. (12), whose proof depends on an unverified and, for unbounded scaling, unsustainable positivity assumption.","tokens_in":16497,"tokens_out":11192,"duration_ms":134055,"significance":"If the positivity gap is closed, this is a useful and simple contribution: a parameter-free modification of linear attention that preserves O(N) complexity and shows consistent gains over softmax attention in direct ablations (Table 9: 75.1 vs 72.2 on DeiT-T; 85.3 vs 83.0 on Swin-S). The paper also includes extensive benchmarks across seven task families, a kernel-function ablation, and an ablation isolating the β and γ terms. However, the theoretical explanation for MALA's behavior currently rests on a positivity condition that is asserted empirically but not measured, and that cannot hold for unbounded magnitude scaling; therefore the mechanism claimed in Eqs. (10)-(12) is not yet established. The significance is conditional on repairing this proof or adding a nonnegative variant.","major_comments":[{"comment":"The proof that p_m > p requires A_n/γ > aβ/(a+β−1) for the scaled query, i.e., all attention scores must remain positive along the scaling path. The manuscript justifies this with the statement in Section 3.3 that negative/zero scores have \"actual frequency of occurrence equal zero,\" but no measurement protocol is given, no bound on the scale factor a is provided, and the assertion is made only for image classification, object detection, and segmentation, not for the NLP, speech, or generation experiments. Moreover, the condition cannot hold for all a in a nonuniform score vector: writing r_j = ϕ(Q_i)ϕ(K_j)^T and S = Σ r_j, the score after scaling ϕ(Q_i) by a is s_j(a) = a(r_j − S/N) + r_j/S. For any key with r_j < S/N, s_j(a) becomes negative for all sufficiently large a, and since not all r_j can exceed their mean unless they are all equal, unbounded magnitude scaling inevitably produces negative scores; the denominator in Eq. (22) then crosses zero. A concrete check is that for r_j < S/N, positivity requires a < (r_j/S)/(S/N − r_j). Consequently, the monotone-concentration claim of Eq. (12) is unproved over the magnitude range claimed, and the \"fractional growth\" advantage over Softmax Attention is not established. Please either (i) provide a formal or empirical bound on the range of a encountered at every layer, head, and token, with a specified verification protocol and results covering all reported task families, or (ii) modify Eq. (8) to guarantee nonnegative attention scores by construction. This gap is load-bearing because the paper's theoretical explanation of MALA's behavior is exactly this concentration property.","section":"Appendix A, Eq. (22); Section 3.3, Eq. (12)"},{"comment":"Even if p_m > p were proved, the paper's conclusion that \"attention becomes more concentrated on keys that originally received higher attention\" does not follow from the ratio of two keys alone; one must show that all scores above the mean increase while all scores below the mean decrease. In this specific construction that stronger property is in fact true, since s_j(a) = a(r_j − S/N) + r_j/S has derivative r_j − S/N with respect to a, independent of positivity. I recommend replacing the ratio-based argument with this direct per-score monotonicity argument, which is simpler and does not require positivity for the monotonicity itself. However, positivity is still needed to ensure that the resulting scores form a valid nonnegative attention distribution, so this repair does not remove the need to address the first major comment.","section":"Section 3.3, Eq. (12) and Fig. 2"},{"comment":"The direct comparisons to softmax attention are only reported at DeiT-T and Swin-T/Swin-S scales. The main tables compare MA ViT to published backbones with different architectures and training schedules, so the broad statement in Section 5 that MALA \"achieves superior performance\" over softmax attention is not fully supported by a same-architecture softmax baseline at the larger scales. A same-architecture softmax counterpart for at least the base or large setting would make the claim more rigorous. This is secondary to the positivity issue but affects the strength of the conclusion.","section":"Section 4.8, Table 9"}],"minor_comments":[{"comment":"The expression \"= p s\" appears to be a typographical error for p^a; please correct the notation and use a consistent variable for the scaling factor throughout Section 3.2.","section":"Eq. (6)"},{"comment":"The arrow notation \"Softmax−→Linear\" in Table 1 is unexplained; please specify that replacing Q by Q/||Q|| in softmax attention removes magnitude information and makes the distribution resemble linear attention.","section":"Table 1"},{"comment":"Reference [16] is malformed (the title and author list are merged into one line), and references [39] and [40] appear to be the same paper; please consolidate and fix the bibliography.","section":"References"},{"comment":"The benchmark name \"LMB\" is likely a typo for \"LAMBADA\"; if so, please spell out the benchmark name for clarity.","section":"Table 6"},{"comment":"The visualizations of attention distributions are qualitative; adding a quantitative measure such as entropy or a Gini coefficient would strengthen the \"more balanced\" claim.","section":"Section 3.3, Fig. 2 and Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the experimental effort is substantial, but the central proof is conditional on an unverified positivity assumption that is mathematically unsustainable for unbounded magnitude scaling. I believe this is fixable by adding a rigorous verification protocol or by modifying the normalization to be nonnegative. The related-work discussion is mostly fair, though there is a cluster of self-citations; given the topic, that is not unusual. Scope is appropriate for a computer vision venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the mechanism — offset linear attention scores by a β/γ term derived from query/key sums — is genuinely new and the controlled ablation is the right way to show it: 75.1 vs 72.2 on DeiT-T, 83.7 vs 81.3 on Swin-T, 85.3 vs 83.0 on Swin-S. Second, the proof that query-magnitude scaling concentrates attention has a real gap. The stress-test is right: for any query with a below-average key, there is a finite scaling a that flips the denominator in Eq. (22) negative, so the p_m > p argument only holds within an unquantified range of magnitudes. The authors assert that negative scores don't occur in practice, but that is an unmeasured empirical claim, not a bound. This narrows the theoretical contribution but doesn't sink the empirical one.\n\nWhat is new is the β/γ offset normalization and the fractional-growth sharpening property. The observation that linear attention ignores query magnitude is not fully original — InLine Attention and the group's prior RALA touch it — but the specific fix is. The breadth of experiments, across classification, detection, segmentation, NLP, speech, and generation, is impressive, though MA ViT's architecture is under-specified and no error bars are reported. The efficiency plots show a real throughput advantage over softmax.\n\nSoft spots: the positivity condition on the proof; no code; the claim about being \"more balanced\" than softmax is qualitative. The ablation on DeiT/Swin is the most trustworthy part of the empirical story.\n\nThis paper deserves a serious referee. I would accept it for review and push the authors to release code, report error bars, and state the admissible magnitude range or replace the uniform-positivity condition with something verifiable. I'd also try the MALA mechanism in my own work — it's simple and the controlled results are strong. Bring it to reading group: the proof gap is instructive.","headline":"New offset-normalized linear attention that works well empirically, but the concentration proof is only valid under an unquantified positivity condition that cannot hold for unbounded scaling.","tokens_in":17047,"tokens_out":4364,"would_cite":true,"duration_ms":49010,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that restoring the query magnitude to linear attention recovers softmax-like focus at linear complexity, improving accuracy across vision, language, speech, and generation.","keywords":["linear attention","query magnitude","softmax attention","attention score distribution","vision transformer","magnitude-aware linear attention","normalization","efficient attention"],"falsifier":"Scan a trained MALA model’s attention scores over a large validation set for any $\\beta\\phi(Q_i)\\phi(K_j)^\\top - \\gamma \\le 0$; finding one would falsify the positivity premise used in the proof of $p_m>p$. A more direct test is to fix a query, scale its magnitude by $a>1$, and verify that the ratio of its two highest positive scores strictly increases as the paper’s Equation (12) predicts.","tokens_in":16006,"feed_emoji":"🎯","tokens_out":12897,"duration_ms":121463,"temperature":0.7,"pith_summary":"Softmax attention gets sharper as the query vector grows, because larger magnitudes make already-favored keys win even harder. Standard linear attention cannot do this: its normalization cancels the magnitude of the transformed query, so the score distribution stays smooth no matter how the query scales. The paper identifies this “magnitude neglect” as a key cause of linear attention’s underperformance and proposes Magnitude-Aware Linear Attention (MALA), which rewrites each score as $\\beta\\phi(Q)\\phi(K)^\\top - \\gamma$ with $\\beta$ and $\\gamma$ fixed by normalization. MALA makes attention concentrate like softmax as the query magnitude grows, but at a gentler fractional rate, and the paper reports gains over both softmax and prior linear attention on image classification, detection, segmentation, NLP, speech, and generation.","feed_headline":"Linear attention matches softmax by restoring query magnitude","feed_subtitle":"A rescaling plus offset makes scores sharpen like softmax; the paper reports gains across seven task families.","key_machinery":"The load-bearing object is the affine attention score $$\\text{Attn}(Q_i,K_j) = \\$\\beta$\\,\\$\\varphi$(Q_i)\\$\\varphi$(K_j)^\\top - \\gamma,$$ with $$\\$\\beta$ = 1 + \\frac{1}{\\$\\varphi$(Q_i)\\sum_{m=1}^N \\$\\varphi$(K_m)^\\top},\\qquad \\gamma = \\frac{\\$\\varphi$(Q_i)\\sum_{m=1}^N \\$\\varphi$(K_m)^\\top}{N},$$ chosen so the $N$ scores for query $i$ sum to $1$ by addition rather than division. Under a magnitude scaling $a>1$ of $\\phi(Q_i)$, $\\beta$ and $\\gamma$ update as $\\beta_{\\mathrm{new}}=(\\beta+a-1)/a$ and $\\gamma_{\\mathrm{new}}=a\\gamma$, so the ratio between a high-scoring key and a lower-scoring key becomes $$\\frac{A_m - \\frac{a\\$\\beta$}{a+\\$\\beta$-1}\\gamma}{A_n - \\frac{a\\$\\beta$}{a+\\$\\beta$-1}\\gamma},$$ which the paper proves is larger than the unscaled ratio whenever all scores are positive. This closed-form update is what lets MALA mimic softmax attention’s sharpening trend while changing the ratio only fractionally rather than exponentially.","core_discovery":"The paper’s central claim is that linear attention’s performance gap to softmax comes not from the kernel approximation per se but from discarding a single scalar: the query’s magnitude. In softmax attention, scaling the query by $a>1$ raises the score ratio between two keys to the power $a$, so attention concentrates sharply on already-favored keys. In linear attention, the normalization cancels $\\|\\phi(Q)\\|$ exactly, so the distribution is fixed. MALA re-introduces the magnitude with an affine normalization: each score is $\\beta\\phi(Q_i)\\phi(K_j)^\\top-\\gamma$, with $\\beta$ and $\\gamma$ chosen so scores sum to 1. When $\\phi(Q_i)$ is scaled by $a$, the new $\\beta$ and $\\gamma$ are $(\\beta+a-1)/a$ and $a\\gamma$, and the ratio of two positive scores grows with $a$, along a fractional curve rather than an exponential one. The paper argues this reproduces softmax’s variation trend while staying more balanced, and attributes its empirical improvements across seven task families to that property.","pith_inferences":["An extension the paper leaves implicit is that the fractional-growth rate provides a dial for attention sharpness: replacing the fixed $a\\beta/(a+\\beta-1)$ factor with a learned temperature would let a model tune spikiness continuously, which is testable but not explored.","The positivity assumption could be made unconditional by clipping scores or re-parameterizing $\\gamma$; this variant is not examined in the paper and would turn the concentration proof into a guarantee.","Because MALA retains the associative linear-time computation, it should transplant into autoregressive and long-context settings, but the paper only reports small-scale NLP and speech results, so that reach is an inference rather than a demonstrated claim."],"forward_implications":["In the paper’s ablations, MALA beats both the softmax baseline and every listed linear-attention baseline while keeping the same parameter counts and FLOPs.","The attention maps become locally concentrated like softmax’s but without collapsing to a few tokens, so MALA recovers local perception without a separate convolution branch.","The operator still computes as $\\beta\\phi(Q)(\\sum_j \\phi(K_j)^\\top V_j) - \\gamma \\sum_j V_j$, so it retains linear complexity in token count and never forms the quadratic attention matrix.","The paper reports that this single mechanism transfers across image classification, detection, segmentation, NLP, speech recognition, and image generation.","Ablations show that removing either $\\beta$ or $\\gamma$, or learning them, collapses accuracy, identifying the closed-form rescaling and offset as the operative components."],"supporting_citations":[{"why":"Provides a convolution-compensated linear-attention baseline and the locality criticism that MALA addresses.","marker":"[4]"},{"why":"Introduces focused linear attention as a prior attempt to sharpen linear attention.","marker":"[20]"},{"why":"Supplies the strongest linear-attention baseline in the ablation that MALA outperforms.","marker":"[21]"},{"why":"Supplies the linear-attention backbone baseline for the classification and detection comparisons.","marker":"[22]"},{"why":"Documents the over-smoothing problem of linear attention that motivates the paper’s analysis.","marker":"[42]"},{"why":"Supplies the efficient/kernel attention formulation that the paper’s analysis starts from.","marker":"[44]"},{"why":"Supplies the small softmax baseline and training recipe used to isolate the effect of removing query magnitude.","marker":"[49]"}],"fun_headline_variants":["Linear attention's gap traced to forgotten query magnitude","Restoring query magnitude closes linear attention gap","MALA: query magnitude rescues linear attention","Why linear attention underperforms: it ignores query magnitude"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that MALA’s attention gets spikier as the query magnitude grows assumes every attention score stays positive; if any score becomes negative or zero, the monotonicity argument no longer applies, and the paper offers only an empirical statement that such scores did not appear.","fun_headline_variants_meta":{"raw":{"variants":["Linear attention's gap traced to forgotten query magnitude","Restoring query magnitude closes linear attention gap","MALA: query magnitude rescues linear attention","Why linear attention underperforms: it ignores query magnitude"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3585,"prompt_tokens":1005,"completion_tokens":2580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":2529}},"tokens_in":621,"tokens_out":2580,"duration_ms":21226,"temperature":1.0,"reasoning_tokens":2529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:08:52.422874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scan a trained MALA model’s attention scores over a large validation set for any $\\beta\\phi(Q_i)\\phi(K_j)^\\top - \\gamma \\le 0$; finding one would falsify the positivity premise used in the proof of $p_m>p$. A more direct test is to fix a query, scale its magnitude by $a>1$, and verify that the ratio of its two highest positive scores strictly increases as the paper’s Equation (12) predicts.","supporting_citations":[{"cited_title":"Efficientvit: Lightweight multi-scale attention for high- resolution dense prediction","cited_arxiv_id":null,"evidence_quote":"Provides a convolution-compensated linear-attention baseline and the locality criticism that MALA addresses."},{"cited_title":"Flatten transformer: Vision transformer using fo- cused linear attention","cited_arxiv_id":null,"evidence_quote":"Introduces focused linear attention as a prior attempt to sharpen linear attention."},{"cited_title":"Bridging the divide: Reconsidering softmax and linear attention","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest linear-attention baseline in the ablation that MALA outperforms."},{"cited_title":"Demystify mamba in vision: A linear attention perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the linear-attention backbone baseline for the classification and detection comparisons."},{"cited_title":"The devil in linear transformer","cited_arxiv_id":null,"evidence_quote":"Documents the over-smoothing problem of linear attention that motivates the paper’s analysis."},{"cited_title":"Efficient attention: Attention with linear complexities","cited_arxiv_id":null,"evidence_quote":"Supplies the efficient/kernel attention formulation that the paper’s analysis starts from."},{"cited_title":"Train- ing data-efficient image transformers & distillation through attention","cited_arxiv_id":null,"evidence_quote":"Supplies the small softmax baseline and training recipe used to isolate the effect of removing query magnitude."}],"review_version":1}