{"id":"d11b52b5-cf77-4a0f-9caa-408a21fdc201","arxiv_id":"2507.16018","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A training-free Nyström attention variant with farthest-point sampling preserves ViT performance at lower cost, and masking attention-sink tokens yields modest downstream gains.","lead":"The paper finds that vision transformers create a small number of massive tokens that absorb most attention and a reserve of artifact tokens that can replace them, and it uses this structure to build a faster approximate attention. It reports that the approximation preserves accuracy across retrieval, classification, segmentation, and visual question answering, and that masking these tokens in late layers gives small accuracy gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own complexity analysis in §4.2 shows the full FNA forward pass is O(N^2D + sLND), not linear; the abstract's 'linear time and space' claim holds only for the attention module after FPS landmarks are fixed, so the headline efficiency claim is internally contradicted.","rationale":"I read the paper in good faith. The empirical core is credible: FNA with FPS preserves retrieval, classification, and segmentation performance on CLIP and DINOv2 at sample size 32–64, the masking trick yields small consistent gains, and Table 4 shows FPS outperforms uniform and clustering sampling. However, the central efficiency claim is undermined by the paper's own asymptotic accounting. FPS costs O(N^2D) time and O(N^2) space; sampling once per forward pass leaves a quadratic term in the total, so 'linear time and space' is inaccurate. This is not a matter of consensus; it is an internal contradiction between the abstract and Section 4.2. The concrete test above would settle whether the quadratic FPS term is empirically negligible at the tested scales; if it is not, the headline needs revision. The reader's weakest assumption about FPS representing sink tokens is also valid, and Appendix C.3 explicitly shows MAE ViT L-16 does not produce massive activations, narrowing scope; but the complexity contradiction is the more decisive issue because it directly contradicts the main claim. I therefore retain the reader's CONDITIONAL verdict.","tokens_in":18834,"tokens_out":4028,"duration_ms":40563,"concrete_test":"Instrument the FNA forward pass to record wall-clock time and peak memory with FPS included versus excluded, at sequence lengths 512, 1024, 2048, 4096, and 8192 on the same GPU used for Table 1. Fit total time as a function of N; if total time scales superlinearly (approximately O(N^2)) while the attention-only time scales linearly, the 'linear time and space' headline fails for the full method. Also report peak memory with FPS included to check the O(N^2) space claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 states: 'While the FPS subroutine itself requires O(N^2D) time and O(N^2) space to compute the pairwise distance matrix, we find that we can produce comparable results by sampling once after massive token formation and reusing those samples in the subsequent layers. This results in an overall reduction from O(LN^2D) to O(N^2D + sLND) in time, and while the peak memory consumption remains O(N^2 + ND), it is reduced to O(sN + ND) after Fast Nyström Attention is applied.' This is a direct admission that the full method is not linear in sequence length N: FPS is O(N^2D) time and O(N^2) space. The abstract and introduction nevertheless claim FNA 'approximates self-attention in linear time and space' and 'reducing computational overhead.' That claim is only true for the attention block itself once landmarks are fixed; the full forward pass retains a quadratic term. Table 1 reinforces the concern: at sequence length 256, FNA (1.2 ms) is slower than standard attention (0.8 ms), which is consistent with FPS overhead being excluded from the FNA timing. Since inference efficiency is the central contribution, this internal inconsistency is the most load-bearing issue; the empirical performance-preservation results could still hold, but the efficiency claim as stated is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies massive tokens (attention sinks) and artifact tokens in pretrained vision transformers, argues that these tokens mutually suppress one another through attention, and introduces Fast Nyström Attention (FNA), a training-free method that approximates self-attention using Nyström extension with farthest point sampling (FPS). It also proposes masking sink tokens in later layers to denoise features. Experiments on CLIP and DINOv2 ViT-L14 cover zero-shot retrieval, classification, segmentation, and LLaVA-based VQA, reporting competitive performance with reduced memory and time; additional ablations compare sampling strategies, sample sizes, and finetuned efficient-attention baselines.","tokens_in":19122,"tokens_out":6189,"duration_ms":69219,"significance":"If the empirical claims hold, FNA would be a practical drop-in efficiency upgrade for CLIP/DINOv2-style backbones, and the masking analysis would connect to existing attention-sink phenomena in a useful way. The paper's strengths are its systematic ablation of all 27 sampling configurations, the sample-size sweep, the finetuned comparison with Linformer and Performer, and the LLaVA extension. The paper does not provide code and reports no repeated-run statistics, and the headline complexity claim is contradicted by the paper's own complexity analysis in Section 4.2. The theoretical mechanism in Section 7 is plausible but largely qualitative. As an empirical systems contribution the result is potentially significant, but the current presentation overstates the efficiency and scope claims.","major_comments":[{"comment":"The paper's own complexity accounting states that the FPS subroutine requires O(N^2D) time and O(N^2) space and that the total forward pass costs O(N^2D + sLND) in time with peak memory O(N^2 + ND); only the attention block after landmarks are fixed is O(sND). The abstract and introduction nevertheless claim that FNA approximates self-attention 'in linear time and space' and reduces computational overhead. These statements are internally inconsistent. The efficiency claim should be corrected to the honest bound, and Table 1's timings should clarify whether FPS overhead is included, since the 256-token row (FNA 1.2 ms vs standard attention 0.8 ms) is otherwise unexplained.","section":"Section 4.2 and abstract"},{"comment":"The performance-preservation claim rests on point estimates without error bars, repeated runs, or significance tests. For example, Table 2 shows CLIP+FNA+resample COCO image R@1 of 35.58 versus 35.33 baseline, and Table 3 shows ImageNet top-1 of 75.81 versus 75.96; these differences are small enough that run-to-run variability could change the conclusions. The manuscript should add variance estimates or bootstrap intervals and specify the evaluation protocol (number of seeds, any stochasticity in FPS initialization, and whether the same sample is reused across images).","section":"Tables 2, 3, 5, and 7"},{"comment":"The sampling justification assumes that massive and artifact tokens are statistical outliers on the feature manifold so that FPS naturally represents the sink-token population. Appendix C.3 explicitly shows that MAE ViT L-16 does not produce massive activations, and the paper does not test FNA on MAE or on any other backbone without sink tokens. The title and abstract refer to 'vision transformers' generally, so the scope is overclaimed. Either evaluate a no-sink backbone and characterize the failure mode, or explicitly restrict the claims to models with established sink-token structure such as CLIP and DINOv2.","section":"Section 4.2 and Appendix C.3"}],"minor_comments":[{"comment":"The notation 'SFsink(w,m)= md SF(w)' is undefined; it appears to mean elementwise multiplication of the softmax output by the mask, but this should be stated explicitly.","section":"Appendix B.1, Definition B.2"},{"comment":"The sentence 'For operand tensors of multiple dimensions, these operations similarly to SF will only be relevant on the last dimension' is grammatically incomplete and should be rephrased.","section":"Appendix B.1"},{"comment":"The reported baselines (43.28% text-to-image R@1 and 51.44% image-to-text R@1) do not match Table 5's CLIP COCO baselines (56.06% text-to-image R@1 and 35.33% image-to-text R@1); please clarify the evaluation setting or correct the labels.","section":"Figure 11"},{"comment":"The masking experiments do not state whether they use the iterative Algorithm 1 or the non-iterative spectral clustering from Section 3.4; if iterative detection is used, the claim that masking is 'at virtually no cost' requires a wall-clock breakdown of the multiple forward passes.","section":"Section 6 and Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The paper would be substantially easier to evaluate if code and timing scripts were released. Without code and without a corrected complexity statement, the main efficiency contribution is currently unsupported as advertised. The empirical trend is plausible, but the headline 'linear time and space' claim and the missing variance/reproducibility information are the key blockers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about arXiv:2507.16018. First, the empirical trick is real: you can take pretrained CLIP or DINOv2, swap the attention block for a Nyström approximation using farthest-point sampling on token features, and get near-identical performance on retrieval, classification, segmentation, and VQA, with speedups once sequences get long. The masking of sink tokens gives a small but consistent accuracy bump. Second, the headline efficiency claim is not what it appears: FNA is not linear in sequence length for the full forward pass. Section 4.2 gives the honest accounting — the FPS subroutine costs O(N^2D) time and O(N^2) space — so the total is O(N^2D + sLND), and peak memory stays O(N^2 + ND) before the approximation is applied. The abstract and intro say \"linear time and space.\" That's flatly contradicted by the paper's own equations. Table 1 shows FNA slower than standard attention at sequence length 256 (1.2 ms vs 0.8 ms), consistent with FPS overhead being left out of the comparison.\n\nWhat's genuinely new: the specific training-free combination of Nyström with FPS landmark selection, plus the artifact-token redundancy analysis. The Nyström formula is Xiong et al., and attention sinks were already documented by Darcet, Sun, and others. But the sampling scheme and the demonstration that masked-out massive tokens have latent replacements are new and worth citing.\n\nSoft spots beyond the complexity claim: no error bars or significance tests on any of the retrieval/classification numbers, so the \"matches baseline\" finding rests on small deltas; no code is released, which makes independent verification harder; and the method only works for backbones that actually form massive tokens — the appendix shows MAE ViT L-16 does not, and FNA is not evaluated there. The mutual-suppression analysis in Section 7 is more narrative than proof; the math describes a mechanism but doesn't derive it from the model weights.\n\nWho should read it: anyone working on training-free attention approximations for vision transformers, or on understanding attention sinks. It deserves a serious referee — the empirical contribution is real and the fixable complexity overstatement is not fatal — but it needs a revision that corrects the abstract, reports variance, and ideally ships code.","headline":"Solid empirical trick, broken headline complexity claim — the FNA forward pass is quadratic in N and the paper admits it in §4.2.","tokens_in":19677,"tokens_out":3228,"would_cite":true,"duration_ms":31037,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Structured attention sinks in vision transformers can be exploited for a training-free Nyström approximation that matches full attention while cutting compute.","keywords":["attention sinks","massive tokens","artifact tokens","Nyström attention","farthest point sampling","training-free approximation","vision transformers","efficient inference"],"falsifier":"Run FNA with sample size 64 on a pretrained MAE ViT L-16—a backbone the paper's own appendix shows produces no massive activations—and compare ImageNet linear-probe or retrieval accuracy against full attention; if the gap grows much larger than the sub-half-point gaps reported for CLIP and DINOv2, the assumption that sink tokens are statistical outliers is what carries the method, not the Nyström approximation itself.","tokens_in":18627,"feed_emoji":"⚡","tokens_out":6758,"duration_ms":67448,"temperature":0.7,"pith_summary":"Vision transformers running on images develop a handful of 'massive' tokens whose activation norms explode and which soak up attention like sinks; when those are masked, other 'artifact' tokens grow to replace them. The paper argues that this redundancy is systematic—massive and artifact tokens mutually suppress rival tokens during a distinct emergence phase—and that it can be exploited at inference time without any training. It introduces Fast Nyström Attention, which replaces the quadratic attention matrix with a low-rank Nyström approximation whose landmark tokens are chosen by farthest-point sampling, and shows that on pretrained CLIP and DINOv2 ViT L-14 the approximation matches full attention on retrieval, classification, and segmentation while cutting time and memory. A second, nearly free masking procedure removes the sink tokens in final layers and gives small consistent accuracy gains. If correct, the result turns a quirk of pretrained vision transformers into a drop-in efficiency upgrade.","feed_headline":"Farthest-point sampling matches full ViT attention at lower cost","feed_subtitle":"A Nyström approximation built on attention-sink tokens keeps CLIP and DINOv2 accuracy while cutting time and memory.","key_machinery":"The central object is the structured attention pattern formed by massive and artifact tokens: in middle-to-late layers of CLIP and DINOv2, a handful of 'sink' tokens absorb a disproportionate share of attention, and masked-out sinks are replaced by dormant artifact tokens in a roughly fixed priority order. FNA exploits this by writing the attention matrix as a Nyström approximation $A \\approx SF(QK^T/\\sqrt{d})\\,SF(qk^T/\\sqrt{d})^{-1}\\,SF(qK^T/\\sqrt{d})$ with $s$ landmark features, where landmarks are selected by farthest-point sampling (FPS) with the CLS token guaranteed. The mutual-suppression analysis—tokens' value subspaces project negatively onto rivals during the emergence phase—explains why only a few tokens become sinks and why FPS, which spreads sampled points over the feature manifold, naturally captures the outlier sinks. Sampling once after massive tokens form and reusing the landmarks in later layers avoids recomputing FPS per layer.","core_discovery":"Fast Nyström Attention (FNA) is a training-free replacement for self-attention in pretrained vision transformers. Standard Nyström attention approximates softmax attention as a product of three softmax matrices built from a small set of landmark tokens; FNA chooses those landmarks by farthest-point sampling over token features, guaranteeing only the CLS token, because massive and artifact tokens are statistical outliers and therefore naturally fall into the sample. With a sample size of 64 on CLIP and DINOv2 ViT L-14, FNA reproduces full-attention results to within a few tenths of a point on COCO and Flickr30k retrieval, ImageNet zero-shot classification, and VOC2012/ADE20k linear-probe segmentation, and it speeds up LLaVA-NeXT-7B VQA generation by about 10% at unchanged BERTScore. The paper also shows that masking detected sink tokens in the final layers consistently improves retrieval and modestly improves classification and segmentation. The stated complexity reduction is from $O(N^2D)$ to $O(sND)$ per attention block in time and from $O(N^2+ND)$ to $O(sN+ND)$ in memory; the farthest-point sampling subroutine itself costs $O(N^2D)$ time and $O(N^2)$ space, though sampling once and reusing landmarks across layers makes total cost $O(N^2D + sLND)$.","pith_inferences":["An implication not drawn in the paper: the 'linear time and space' claim applies to the attention module once landmarks are fixed; the full forward pass still includes an $O(N^2D)$ farthest-point pass unless sampling is amortized or replaced by a cheaper landmark detector.","Because FNA treats landmark selection as a purely geometric problem on token features, it should transfer to any attention-based model whose activations exhibit the same outlier structure, including multimodal and long-sequence settings; the paper only demonstrates CLIP, DINOv2, and one LLaVA variant.","The artifact-token redundancy suggests a direct test of register-trained models: if registers reduce massive tokens, FNA's sample may no longer find them, so the method may need to guarantee register or sink tokens explicitly.","Masking gains hint at a training-free denoising pass that could be combined with FNA: if sink tokens are excluded from the landmark set as well as from final-layer attention, the approximation and the denoising might compound."],"forward_implications":["Pretrained CLIP and DINOv2 ViT L-14 can run with a sample size of 64 landmarks and keep retrieval, classification, and segmentation metrics within about half a point of full attention, with no fine-tuning.","A one-time farthest-point sample taken after massive tokens form can be reused across later layers, so the per-layer cost drops to $O(sND)$ after one FPS pass.","Masking sink tokens in the final layers improves CLIP COCO text-to-image Recall@1 from 35.33 to 37.47 and gives small consistent gains on ImageNet, VOC2012, and ADE20k.","Applied to LLaVA-NeXT-7B, approximating the causal attention over image tokens in the LLM raises generated-token throughput by about 10% without lowering BERTScore.","The mutual-suppression account predicts that removing massive tokens early simply promotes artifact tokens, so efficiency must leave the sink population intact; only end-stage masking can safely remove them."],"supporting_citations":[{"why":"Supplies the Nyström low-rank decomposition of self-attention that FNA adapts to be training-free.","marker":"[25]"},{"why":"Supplies farthest-point sampling, the landmark-selection subroutine that FNA uses to pick tokens.","marker":"[17]"},{"why":"Documents massive activations and attention sinks in pretrained transformers, the phenomenon FNA exploits.","marker":"[21]"},{"why":"Shows massive tokens in vision transformers can be reduced with register tokens, providing background evidence of sink structure.","marker":"[8]"},{"why":"Shows artifacts in self-supervised ViT features can be removed by denoising, motivating FNA's masking and analysis.","marker":"[28]"},{"why":"Provides the pretrained CLIP model FNA is evaluated on.","marker":"[18]"},{"why":"Provides the pretrained DINOv2 model FNA is evaluated on.","marker":"[16]"},{"why":"Identifies attention-sink behavior in language models, the conceptual basis for treating massive tokens as sinks.","marker":"[12]"}],"fun_headline_variants":["Training-free attention trick: sample 64 tokens, keep ViT accuracy","64 sink tokens replace full ViT attention, no training needed","FNA: sample 64 landmarks, cut ViT cost, keep accuracy","Farthest-point sampling finds attention sinks, speeds ViT","Want ViT speed? Sample 64 tokens, skip training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that attention-sink tokens are statistical outliers in token-feature space, so farthest-point sampling picks them up automatically; the paper's own appendix shows MAE ViT L-16 has no massive activations, so for that backbone the premise fails and FNA's guarantees are not established.","fun_headline_variants_meta":{"raw":{"variants":["Training-free attention trick: sample 64 tokens, keep ViT accuracy","64 sink tokens replace full ViT attention, no training needed","FNA: sample 64 landmarks, cut ViT cost, keep accuracy","Farthest-point sampling finds attention sinks, speeds ViT","Want ViT speed? Sample 64 tokens, skip training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2958,"prompt_tokens":1011,"completion_tokens":1947,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1856}},"tokens_in":627,"tokens_out":1947,"duration_ms":14754,"temperature":1.0,"reasoning_tokens":1856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:20:16.697920+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run FNA with sample size 64 on a pretrained MAE ViT L-16—a backbone the paper's own appendix shows produces no massive activations—and compare ImageNet linear-probe or retrieval accuracy against full attention; if the gap grows much larger than the sub-half-point gaps reported for CLIP and DINOv2, the assumption that sink tokens are statistical outliers is what carries the method, not the Nyström approximation itself.","supporting_citations":[{"cited_title":"Nys- trömformer: A nyström-based algorithm for approximating self-attention","cited_arxiv_id":null,"evidence_quote":"Supplies the Nyström low-rank decomposition of self-attention that FNA adapts to be training-free."},{"cited_title":"Qi, Li Yi, Hao Su, and Leonidas J","cited_arxiv_id":null,"evidence_quote":"Supplies farthest-point sampling, the landmark-selection subroutine that FNA uses to pick tokens."},{"cited_title":"Vision transformers need registers","cited_arxiv_id":null,"evidence_quote":"Shows massive tokens in vision transformers can be reduced with register tokens, providing background evidence of sink structure."},{"cited_title":"Denoising vision transformers","cited_arxiv_id":null,"evidence_quote":"Shows artifacts in self-supervised ViT features can be removed by denoising, motivating FNA's masking and analysis."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained CLIP model FNA is evaluated on."}],"review_version":1}