{"id":"cff376a2-6828-40c1-b847-71d33f3cdb9b","arxiv_id":"2507.19409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MAELRE combines progressive token merging with a switch from cheap linear attention to standard attention, reporting lower memory and FLOPs than existing long-context encoders on four modality classification benchmarks.","lead":"This paper presents MAELRE, a transformer-style encoder that merges tokens and uses cheaper attention in early layers, letting a single GPU process much longer text, audio, time series, or images. It reports accuracy on par with or better than several efficient long-context models while using far less memory and computation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed O(M^2) memory bound is not what Eq. 2 delivers: linear attention still materializes N×D feature maps, so memory is O(ND+D^2), and Table 1's batch-size halving with N confirms it.","rationale":"The reader's weakest_assumption concerns the faithfulness of the finite nonnegative feature map as a surrogate for dot-product attention, with the all-approximate ablation as evidence. That is a legitimate risk, but the paper partially addresses it through the Proposed-ELU versus All-dot-base comparison in Tables 5 and 6, where the proposed switching rule is close to or better than all-dot attention while being cheaper. I therefore do not see the surrogate-faithfulness issue as the single most load-bearing threat to the central claim. Instead, the paper's own stated contribution, and the reader's strongest_claim, include a precise theoretical efficiency property: memory cost O(M^2) with M = min(N, D). Section 3.1.2's own complexity expression, O(D_b^2 + N_b D_b), contradicts that property for the regime N_b > D_b, which is exactly the regime the paper targets. The empirical batch-size scaling in Table 1 confirms the linear-in-N term: when N doubles from 2048 to 4096, the fit batch size halves from 180 to 90 despite D remaining 96. This is an internal inconsistency rather than an external disagreement with consensus, and it can be settled by direct profiling. If the profiling confirms linear memory growth, the theoretical framing must be revised to O(ND + D^2), which weakens the 'quadratic on D instead of N' selling point while still leaving a useful linear-memory architecture. The accuracy claims may survive, but the central efficiency claim as stated does not. The reader's CONDITIONAL verdict already captures the need for revision, so I do not move the verdict; I would keep it conditional with the complexity claim explicitly corrected. I also note that the abstract's 'superior accuracy' is not uniformly supported by Table 4 on ImageNet, but that is a secondary overstatement relative to the complexity flaw.","tokens_in":17301,"tokens_out":13706,"duration_ms":142622,"concrete_test":"Instrument the approximate-attention module from Eq. 2 with torch.cuda.max_memory_allocated around the module for fixed D = 96 and N = 2048, 4096, 8192, 16384 at batch size 1. If peak attention memory grows approximately linearly with N, the O(D^2) claim is false. As a second check, use torch.profiler or a manual allocation count to verify that phi(Q) and phi(K) tensors of shape (N, D) are allocated before the D×D product, yielding O(ND + D^2), and compare this against the O(M^2) formula.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.2 states that the approximate attention 'can be computed with a memory and computation cost of O(D_b^2 + N_b D_b)' and that 'the costs are quadratic on D_b instead of N_b'; the Introduction and Figure 1 translate this into a headline memory cost of O(M^2) with M = min(N, D). The O(D_b^2) accounting is incomplete. Even for a single block, Eq. 2 must materialize phi(Q) and phi(K), both N_b × D_b, before the D_b × D_b product (phi(K)^T V) exists, so the attention block requires O(N_b D_b + D_b^2) memory, and for N_b > D_b the O(N_b D_b) term dominates and grows linearly with N_b. The paper's own Table 1 is consistent with this linear-in-N dependence rather than O(D^2): doubling the token count from 2048 to 4096 halves the maximum batch size from 180 to 90 at the same D = 96. Thus the central efficiency claim as stated, memory quadratic in min(N, D) rather than in N, is unsupported. The method is still linear-memory in N, which is valuable, but the theoretical selling point needs to be restated as O(ND + D^2) memory and O(ND^2) compute, not O(M^2).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAELRE, a transformer encoder that combines progressive token merging with two attention modes: an approximate 'linear attention' in shallow layers where the token count exceeds the feature dimension, and standard dot-product attention in deeper layers after merging has reduced the token count. The authors argue that this hybrid gives O(M^2) memory with M = min(N, D), enabling long-context classification on a single device. They evaluate the architecture on four modalities: MeSH multilabel text classification, UCR MosquitoSound time series, VGGSound audio, and ImageNet vision, comparing against RoBERTa, Longformer, BigBird, OPT, mViTv1/v2, and other efficient transformers, and report large reductions in memory and FLOPs with competitive or superior accuracy in most settings. Ablations examine the attention combination, the choice of feature map, and the token merging operator.","tokens_in":17596,"tokens_out":5864,"duration_ms":56307,"significance":"If the empirical results hold, the contribution is valuable: a simple, modality-agnostic recipe that couples progressive token merging with a switch between approximate and exact attention, with meaningful efficiency gains on long inputs and a broad evaluation across four modalities. The paper's strength is its scope: MeSH with over twelve thousand labels, a 280k-sample time-series dataset, VGGSound, and ImageNet, plus ablations on the feature map, the attention combination, and the merging operator. However, the central complexity claim is overstated, and the abstract-level accuracy claim is contradicted by the paper's own tables. The underlying design is still likely salvageable, but the paper as written overclaims in two load-bearing places.","major_comments":[{"comment":"The claimed O(M^2) memory cost with M = min(N, D) is not what Eq. (2) delivers. Computing phi(Q), phi(K), and phi(V) requires O(N_b D_b) memory before the D_b x D_b product is formed, so a single shallow block costs O(N_b D_b + D_b^2) memory, not O(D_b^2). For N_b > D_b the O(N_b D_b) term dominates and grows linearly in N_b. Table 1 is consistent with this: doubling the token count from 2048 to 4096 at D = 96 halves the maximum batch size from 180 to 90, which is a linear-in-N scaling, not a quadratic-in-D scaling. Please restate the per-block memory as O(N_b D_b + D_b^2) and the compute as O(N_b D_b^2), and remove the O(M^2) formulation from the abstract, introduction, and Figure 1 unless a different accounting is intended.","section":"§3.1.2, Eq. (2); Introduction and Fig. 1"},{"comment":"The abstract's 'superior accuracy' claim is contradicted by Table 4: MAELRE-base reaches 82.97% on ImageNet versus 83.78% for mViTv2-base, and MAELRE-small reaches 82.21% versus 83.23% for mViTv2-small. The text in §4.4 acknowledges inferiority to mViTv2, but the paper-level claims in the abstract and introduction are not qualified accordingly. Similarly, Table 7 shows that removing token merging improves text EBF from 43.28 to 45.47, so the claim that token merging uniformly maintains or improves accuracy is also overstated. The defensible claims are lower memory/FLOPs and accuracy advantages on MeSH, MosquitoSound, and VGGSound; the 'superior accuracy' language should be restricted to those comparisons.","section":"§4.4, Table 4; also Abstract and §1"},{"comment":"The approximate attention relies on the unproven assumption that a finite nonnegative feature map with scaling tau = sqrt(N_b) faithfully replaces dot-product attention in shallow layers. The paper states that tau was chosen because it 'led to training convergence' and provides no error bound, unbiasedness guarantee, or sensitivity analysis. Since the all-approximate ablations in Tables 5 and 6 lose 3.48 EBF and 1.84 accuracy points respectively, this assumption is load-bearing. Please add a sensitivity analysis over tau and, ideally, a formal or empirical bound on the approximation error relative to the softmax kernel in the regimes used.","section":"§3.1.2 and Tables 5-6"},{"comment":"All accuracy results appear to be single runs, with no error bars, standard deviations, or seed counts reported. Since many claimed gains are around one point (e.g., 95.42% vs 94.45% on MosquitoSound, 57.85% vs 57.30% on VGGSound), run-to-run variance could change the conclusions. Moreover, the proposed and baseline models are trained with different learning rates and batch sizes by design (§4.2, §4.4), so part of the observed difference may be attributable to training protocol rather than architecture. Please report multiple seeds with intervals and a short hyperparameter search for baselines under a comparable protocol.","section":"Tables 1-4; §4.2 and §4.4"}],"minor_comments":[{"comment":"The dataset description says '1.2M training and 50K training images from 100 categories'; it should read '1.2M training and 50K validation images from 1000 classes.'","section":"§4.4"},{"comment":"The stride is called tau after tau was already defined as the attention scaling factor in §3.1.2; rename the stride to avoid notation collision.","section":"§4.5.2"},{"comment":"The normalization matrix Z' is not defined precisely in the factored form; please state how the row sums are computed without materializing the N x N attention matrix.","section":"Eq. (2)"},{"comment":"The All-dot-base memory entry '1347.38G' appears to mix units with the other entries given in 'M'; also state the memory units explicitly in the caption.","section":"Table 6"},{"comment":"PoolFormer rows lack memory and FLOPs entries; consider filling these for a complete comparison.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a solid engineering paper that combines two known ideas—linear attention and convolutional token merging—into a single modality-agnostic encoder, and it demonstrates real efficiency gains on four public benchmarks. It deserves a serious referee, but the abstract is too strong on accuracy and the asymptotic memory claim is wrong as stated.\n\nWhat's new: the N-versus-D switch is a clean design choice. They use the linear-attention surrogate when N > D and standard dot-product when N < D, and they show in ablations that this hybrid beats both all-approximate and all-dot variants. Convolution-based merging, with stride 4 and width doubling per block, works for text, time series, audio spectrograms, and images. The efficiency numbers are believable—180 samples of 2048 tokens fit in 16GB, and inference is fast. The feature-map robustness check (ELU, ReLU, Softplus) is a nice touch.\n\nNow the soft spots, in order of importance.\n\n1. The O(M^2) claim. The paper says memory is quadratic in min(N,D). That doesn't hold up against their own Equation 2: the approximate attention still materializes phi(Q) and phi(K) as N×D matrices, so memory is O(ND+D^2). For N > D, that's linear in N, not quadratic in D. The stress-test note is right about this. The method is still a good deal—linear memory instead of N²—but the theoretical selling point needs to be restated.\n\n2. The accuracy claim. The abstract says 'superior accuracy.' Table 4 shows MAELRE-base at 82.97% on ImageNet vs 83.78% for mViTv2-base. The authors admit this in the text but not in the abstract. Similarly, Table 7 shows a no-merging text variant at 45.47 EBF vs 43.28 for the proposed model. So the claim is not uniformly true. A revision should temper it.\n\n3. Evaluation rigor. No error bars, no multiple seeds. Several wins are close (e.g., VGGSound 57.85 vs 57.3). The linear-attention surrogate also lacks a theoretical error bound; tau=sqrt(N_b) is empirical. And there's no code. All fixable, but important.\n\nWho is this for? Researchers working on efficient transformers, especially long-context classification on modest hardware. It's not a breakthrough, but it's a sensible architecture with broad experiments. I'd send it to review, with a request to correct the complexity claim, soften the accuracy language, and add seeds/error bars/code. I wouldn't cite it as authoritative until those numbers are reproduced.","headline":"Useful hybrid of linear attention and token merging with real efficiency wins, but the accuracy and memory-complexity claims are overstated; deserves review after correction.","tokens_in":18126,"tokens_out":6684,"would_cite":false,"duration_ms":62569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAELRE claims that a single transformer encoder can process very long inputs across text, audio, time series, and vision on one commodity GPU by merging tokens progressively and switching from O(D^2) kernel attention to exact O(N^2)…","keywords":["long-context transformer","token merging","linear attention approximation","modality-agnostic encoder","efficient attention","single-device inference","multimodal classification","O(min(N,D)) memory"],"falsifier":"Replace every shallow-layer approximate attention in the MeSH or VGGSound training runs with exact dot-product attention, keeping token merging and all other settings fixed. If the exact-attention model's EBF or top-1 accuracy exceeds the approximate version by more than the margin seen in the all-approximate ablation (about 3.5 points on MeSH, 1.8 on VGGSound), the claim that the finite feature map faithfully approximates early-layer attention is falsified.","tokens_in":17082,"feed_emoji":"⚡","tokens_out":7543,"duration_ms":69283,"temperature":0.7,"pith_summary":"The paper tries to show that a single transformer encoder can handle very long inputs on one ordinary GPU by attacking both quadratic costs at once: shrink the token count as computation deepens and use a cheaper attention whenever tokens still outnumber feature dimensions. It claims this combination, not token merging alone and not approximation alone, preserves or improves accuracy while using a fraction of the memory and FLOPs of existing long-context transformers. If true, long-context classification becomes feasible on a single 16GB device, and the same encoder applies to text, time series, audio, and images with only the input convolution changed.","feed_headline":"Long-context transformer runs on one GPU with O(min(N,D))^2 memory","feed_subtitle":"Token merging plus a cheap-attention switch keeps accuracy high across four modalities on a single device.","key_machinery":"The load-bearing object is the attention switch rule and the token-reduction schedule that makes it possible. Each of four blocks contains MHSA/MLP layers, then a strided-convolution reduction (factor $\\nu=4$, kernel $2\\nu+1$) that cuts token count and doubles dimensionality; when $N_b>D_b$ the block uses kernel attention with a nonnegative feature map $\\psi\\in\\{1+\\mathrm{ELU},\\mathrm{ReLU},\\mathrm{Softplus}\\}$ and scaling $\\tau=\\sqrt{N_b}$, and when $N_b<D_b$ it uses exact dot-product softmax attention. This makes the attention matrix cost $O(D_b^2)$ early and $O(N_b^2)$ late, so the overall memory is $O(\\min(N,D)^2)$ rather than $O(N^2)$.","core_discovery":"The paper's central claim is that the quadratic cost of self-attention can be tamed by making the attention type follow the token-to-dimension ratio. In shallow blocks, where $N>D$, MAELRE replaces softmax attention with a kernel-based linear attention whose memory is $O(D^2)$; in deeper blocks, after each block's strided-convolution token reduction has cut $N$ by a factor of 4 and doubled $D$, it switches back to standard dot-product attention with $O(N^2)$ memory. The combined memory cost is $O(M^2)$ with $M=\\min(N,D)$. The paper reports that this switch, with token merging, gives higher accuracy than sparse-attention long-context baselines and lower cost than multiscale vision transformers, with accuracy above mViTv1 and below mViTv2 on ImageNet, on classification tasks in four modalities.","pith_inferences":["Because $\\tau=\\sqrt{N_b}$ was chosen by training convergence rather than derived, a natural extension is to sweep or learn $\\tau$ per block; if accuracy is sensitive to it, transfer to new modalities would need per-task tuning.","The same $O(M^2)$ attention switch can be stacked under sequence-parallel or ring-attention schemes, so the single-device claim is probably a floor, not a ceiling, for context length.","The all-approximate ablation loses about 3.5 EBF points on MeSH and about 1.8 points on VGGSound, suggesting exact deep-layer attention is the accuracy anchor; a testable hypothesis is that the deep-layer switch, not the merging, preserves accuracy.","The learned-projection token reduction performs worse than convolution, but this may be a training-difficulty artifact; with longer schedules or warm starts, learned projections could close the gap and remove the convolution prior."],"forward_implications":["A single 16GB GPU can hold 24 sequences of 17,000 tokens under MAELRE's base configuration, so long-context classification no longer requires high-end hardware or multi-device parallelism.","Doubling the input length beyond $D$ no longer squares attention memory in shallow layers; cost grows only through the linear projection terms until merging brings $N$ below $D$.","Using approximate attention over all tokens in early layers yields higher accuracy on MeSH text classification than sparse attention over a subset, supporting the paper's hypothesis about early-layer information consolidation.","The same four-block encoder reaches state-of-the-art accuracy on VGGSound audio (57.85%) and MosquitoSound time series (95.42%) while using roughly half the memory and FLOPs of the closest competitor.","On ImageNet the design matches pooling-based vision transformers at lower cost, but closing the gap to mViTv2 would require adding relative position embedding and residual pooling."],"supporting_citations":[{"why":"Supplies the kernel-based linear attention with nonnegative feature maps and associative regrouping that MAELRE uses when N > D.","marker":"[33]"},{"why":"Establishes the multiscale vision transformer design with progressive token reduction, the main vision/audio baseline and source of block head counts.","marker":"[18]"},{"why":"Provides the mViTv2 baseline whose relative position embedding and residual pooling MAELRE does without, and which edges out MAELRE on ImageNet.","marker":"[36]"},{"why":"Provides the MeSH full-text article dataset and the EBF/MiF evaluation measures used for the text classification experiments.","marker":"[58]"},{"why":"Supplies the UCR MosquitoSound time-series dataset used for species classification from wingbeats.","marker":"[14]"},{"why":"Provides the VGGSound audio dataset used for audio classification from spectrograms.","marker":"[6]"},{"why":"Sparse-attention long-context baseline whose lower accuracy supports the claim that approximate attention over all tokens beats attention over a subset.","marker":"[3]"},{"why":"Sparse-attention long-context baseline that MAELRE outperforms in text accuracy at much lower cost.","marker":"[65]"},{"why":"Pooling-based vision baseline that MAELRE matches or exceeds at similar cost, isolating the value of the attention switch.","marker":"[63]"}],"fun_headline_variants":["Attention flips to linear when tokens exceed dims","Transformer cuts memory to O(min(N,D)^2) for long context","Token merging plus adaptive attention powers single-GPU long context","MAELRE: quadratic attention tamed by switching to linear"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a cheap, nonnegative kernel feature map with an empirically chosen scaling factor can stand in for exact dot-product attention in the early layers without losing the information later layers need.","fun_headline_variants_meta":{"raw":{"variants":["Attention flips to linear when tokens exceed dims","Transformer cuts memory to O(min(N,D)^2) for long context","Token merging plus adaptive attention powers single-GPU long context","MAELRE: quadratic attention tamed by switching to linear"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3454,"prompt_tokens":953,"completion_tokens":2501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2431}},"tokens_in":569,"tokens_out":2501,"duration_ms":18011,"temperature":1.0,"reasoning_tokens":2431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:53:15.514801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace every shallow-layer approximate attention in the MeSH or VGGSound training runs with exact dot-product attention, keeping token merging and all other settings fixed. If the exact-attention model's EBF or top-1 accuracy exceeds the approximate version by more than the margin seen in the all-approximate ablation (about 3.5 points on MeSH, 1.8 on VGGSound), the claim that the finite feature map faithfully approximates early-layer attention is falsified.","supporting_citations":[{"cited_title":"Katharopoulos, A","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel-based linear attention with nonnegative feature maps and associative regrouping that MAELRE uses when N > D."},{"cited_title":"Multiscale vision transformers","cited_arxiv_id":null,"evidence_quote":"Establishes the multiscale vision transformer design with progressive token reduction, the main vision/audio baseline and source of block head counts."},{"cited_title":"Mvitv2: Im- proved multiscale vision transformers for classification and detec- tion","cited_arxiv_id":null,"evidence_quote":"Provides the mViTv2 baseline whose relative position embedding and residual pooling MAELRE does without, and which edges out MAELRE on ImageNet."},{"cited_title":"MeSHup: A Corpus for Full Text Biomedical Document Indexing","cited_arxiv_id":"2204.13604","evidence_quote":"Provides the MeSH full-text article dataset and the EBF/MiF evaluation measures used for the text classification experiments."},{"cited_title":"The ucr time series classifica- tion archive, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the UCR MosquitoSound time-series dataset used for species classification from wingbeats."},{"cited_title":"Vggsound: A large-scale audio-visual dataset","cited_arxiv_id":null,"evidence_quote":"Provides the VGGSound audio dataset used for audio classification from spectrograms."},{"cited_title":"Big bird: Trans- formers for longer sequences","cited_arxiv_id":null,"evidence_quote":"Sparse-attention long-context baseline that MAELRE outperforms in text accuracy at much lower cost."},{"cited_title":"Metaformer is actually what you need for vision","cited_arxiv_id":null,"evidence_quote":"Pooling-based vision baseline that MAELRE matches or exceeds at similar cost, isolating the value of the attention switch."}],"review_version":2}