{"id":"ec6f86b5-c533-413e-92de-680ab26bd2f1","arxiv_id":"2608.09468","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A new linear attention architecture with memory experts is described, but its equations are internally inconsistent and the reported results lack the detail needed to verify them.","lead":"This paper proposes MixFormer, a Transformer variant that stores long-range context in multiple 'memory experts' with a time-decay attention rule. The reported benchmark wins and sample images are not backed by a consistent mathematical formulation or reproducible experiments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The recurrent TALA update (Eq. 15) is not equivalent to its parallel form (Eq. 14) under the paper's own definition of G, leaving the claimed constant-memory recurrent inference unsupported.","rationale":"Reading the paper in good faith, the intended contribution is a linear attention with a learned decay plus a mixture of memory experts, with both parallel training and recurrent inference. For that contribution to be valid, the recurrent update must compute the same attention output as the parallel causal form. The paper's own equations do not satisfy this: Eq. 14 uses a full positional-bias matrix G, while Eq. 15 applies the same G as an elementwise decay to a d×d state. Since G is T×T and S is d×d, the operation is undefined unless G is silently redefined. If G is instead intended as a scalar or per-dimension decay, that is a different architecture from Eq. 14, whose entries depend on both n and m. The normalization mismatch with Eq. 12 reinforces that the two forms are not the same computation. This is precisely the reader's weakest assumption, and it is a genuine correctness risk: the headline experimental claims of 'significant performance gains' rest on an architecture whose core module is not formally specified. The experimental section would need substantial additional detail to support the claims even if the math were fixed, but the mathematical inconsistency alone is sufficient to reject the current preprint. Accordingly, my stress-test pass does not change the reader's verdict.","tokens_in":14215,"tokens_out":2964,"duration_ms":31387,"concrete_test":"Write out Eq. 14 for a concrete case with T=3 and d=2, using G as a full 3×3 learnable matrix, and expand Eq. 15 recursively from S0 to S3 with the same G. Attempt to compute S_n = ωG ⊙ S_{n−1} + K_n^T V_n; if the shapes do not match or the resulting O_3 differs from the row-3 output of Eq. 14, the equivalence claim fails. For an executable check, implement both forms in a short script with random Q, K, V and a random 3×3 G, and compare the outputs; the run either raises a shape error or produces a nonzero difference that cannot be removed by reparameterizing G as a scalar or per-channel decay.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the parallel TALA of Eq. 14 and the recurrent TALA of Eq. 15 compute the same function, so that MixFormer can be trained in parallel and then run autoregressively with O(1) memory per step. In Eq. 14, D_{nm} = ω^{n−m} G_{nm}, where G is defined in Table 1 as a learnable pair-wise positional bias and in the text as G ∈ R^{T×T}. In Eq. 15, the recurrence is S_n = ωG ⊙ S_{n−1} + K_n^T V_n, with the statement 'where G is the same as in Equation (14).' This is dimensionally inconsistent: S_n is a d×d state matrix, while G is a T×T matrix, so the elementwise product is undefined. To make Eq. 15 a valid constant-memory recurrence, G must be restricted to a scalar or a per-head/per-dimension decay factor that is independent of absolute positions, or the recurrence must use a different relative-position operator. The paper states no such restriction and gives no proof that the two forms coincide. Moreover, Eq. 12 includes a normalizing denominator that is absent from Eqs. 14–16, so even the scalar-decay reading does not make the parallel and recurrent forms match as written. Because the claimed O(1) inference, the complexity analysis of Sec. 4.6, and the novelty over prior SSMs all depend on this recurrent form, the inconsistency is load-bearing rather than cosmetic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MixFormer, a linear Transformer that combines a Time-Aware Linear Attention (TALA) mechanism with a Mixture-of-Memory-Experts (MoME) module. TALA is intended to add exponential decay and learnable positional biases to the SSM memory state, and MoME is designed to maintain several memory states selected by a router. The authors claim that this yields state-of-the-art results on LRA and qualitative improvements on image generation while retaining linear complexity in the sequence length.","tokens_in":14522,"tokens_out":6323,"duration_ms":53405,"significance":"If the claims were correct, the paper would offer a useful architectural addition to the linear-attention/SSM literature, and the idea of routing to multiple memory states is worth exploring. The manuscript, however, contains dimensional inconsistencies in the core equations, an unproven equivalence between parallel and recurrent formulations, an underspecified router, and uncontrolled experimental comparisons. These issues prevent the results from being validated, so the significance cannot be assessed on the current evidence. The paper does not provide code or detailed experimental logs, and the complexity analysis contains errors.","major_comments":[{"comment":"The recurrent form in Eq. (15) is not a valid constant-memory reformulation of the parallel form in Eq. (14). In Eq. (14), D_{nm}=ω^{n−m}G_{nm} with G a T×T pair-wise positional bias, while in Eq. (15) the same G is used in an elementwise product with the d×d state S_{n-1}; these dimensions are incompatible. If G is instead meant to be a scalar or per-head decay, then it is no longer 'the same as in Equation (14)', and the pair-wise positional dependence of the parallel form cannot be reproduced by a fixed-size recurrent state without additional approximations. No proof of equivalence is given, so the claimed O(1)-memory autoregressive inference is unsupported.","section":"4.3, Eqs. (14)-(15)"},{"comment":"The denominator in Eq. (12) is dropped in the parallel representation (Eq. 14) and in the recurrent and chunkwise forms (Eqs. 15-16). As written, Eq. (14) computes Q⊙(K^T⊙D)V without the normalization by Σ φ(K_i)(ωG_i)^{t-i}, so the parallel output scale differs from the recurrent output scale even if G is interpreted as a scalar decay. This discrepancy means the training and inference modes do not compute the same function, and it also changes the interpretation of the state S_n.","section":"4.3, Eq. (12) vs. Eqs. (14)-(16)"},{"comment":"The router is underspecified: E_i=Softmax(K_n V_n W_g) is undefined because K_n V_n is d×d while W_g is declared as 1×k_e, so the matrix product has incompatible shapes. In addition, the sum in Eq. (17) runs over n, which is the sequence length, rather than over the number of memory experts k_e. Without a consistent definition of the expert state M S_i and the routing weights, the MoME mechanism cannot be implemented or evaluated.","section":"4.4, Eqs. (17)-(18)"},{"comment":"The complexity calculation is incorrect: the FFN term is written as 2dd′ without the factor N, although the FFN is applied to every token; it should be 2Ndd′. The MoME cost Nk also conflicts with the d×d router computation of Eq. (18), which would add a per-token O(d^2) cost. Consequently, Eq. (21) does not establish the claimed O(N) total complexity.","section":"4.6, Eq. (21)"},{"comment":"The empirical evaluation is not adequately controlled. MixFormer is a large model (2B or 7B total parameters) trained on 15B-30B tokens, while the LRA baselines in Table 3 are taken from the original benchmark setup and are orders of magnitude smaller in parameter count and training data. The image-generation experiments (Sections 5.3-5.4) report only qualitative results with no quantitative metrics. The claim of 'consistently achieves significant performance gains' is therefore not supported by the evidence presented.","section":"5.2-5.4, Table 3"}],"minor_comments":[{"comment":"The notation φ(Q)=e^{Q/||Q||} should specify that the exponential is applied elementwise and that ||·|| is the row-wise L2 norm; the current notation is ambiguous for matrices.","section":"4.3, Eq. (11)"},{"comment":"The definition ω=1−e^{−h} with h∈[1,...,d/d_head] is unclear: is h a head index and are these decay values learnable or fixed? The relationship to the per-head decay parameters used in Eqs. (14)-(15) should be stated.","section":"4.3, Eq. (13)"},{"comment":"The index in R_{n−1} should be R_{i−1} to refer to the previous chunk; as written, it refers to the full sequence length.","section":"4.3, Eq. (16)"},{"comment":"Several entries have concatenated numbers (e.g., '53.8242.77 77.0551.41' for Performer), which obscures the reported values.","section":"5, Table 3"},{"comment":"The phrase 'reflection attention' appears to be a typo for 'linear attention'.","section":"5.5"},{"comment":"The reference list cites Han et al. twice for RetNet, but the correct authors are Sun et al.; the MoE-related reference [35] (Narayan et al.) is on summarization and does not support the MoE background.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central equations need substantial corrections before the claims can be evaluated, and the experimental comparisons need to be redone with matched baselines. Given the number of load-bearing issues, I recommend rejection. The underlying idea of multiple memory states is interesting enough that a properly specified sequel would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the idea is worth five minutes, but the paper is not in a publishable state. The mix of memory-state MoE with decay-based linear attention is a real combination I haven't seen in the cited prior work, and the general direction—multiple specialized recurrent states for linear attention—is reasonable. But the central math doesn't close, and the experiments as reported can't support the claims.\n\nWhat's new: TALA is essentially RetNet-style decay plus GLA-style gating plus a pair-wise bias, and MoME grafts an MoE router onto the state update. That combination is novel on the surface. The paper also does the right thing by laying out parallel, recurrent, and chunkwise forms, and by discussing complexity.\n\nWhere it falls apart: Eq. 14 defines D_{nm}=ω^{n-m} G_{nm} with G∈R^{T×T}, while Eq. 15 writes the recurrence as S_n = ωG ⊙ S_{n-1} + K_n^T V_n with \"G is the same as in Equation (14).\" That is dimensionally impossible: G is a T×T matrix and S_n is d×d. For the recurrence to be constant-memory, G has to be a scalar or a per-head/per-dimension decay independent of absolute positions. The paper never states that restriction or proves the two forms are equivalent. Also, Eq. 12 has a normalizing denominator that disappears from Eqs. 14–16, so even a scalar-decay reading does not make the parallel and recurrent forms compute the same function. This is load-bearing: the O(1) inference claim, the chunkwise formulation, and the novelty over SSMs all depend on that equivalence.\n\nEq. 18 has shape problems too: K_n V_n is d×d, W_g is described as R^{1×k_e}, so the product is undefined, and E_j^i is never clearly defined. That is minor relative to the recurrence issue, but it compounds.\n\nExperiments: LRA results appear without error bars, seeds, or task-specific setups; baselines are borrowed from an earlier paper; and the model sizes and hardware don't add up (2B/7B parameters on four RTX 4090s, with Table 4 contradicting the text on training tokens). Image results are qualitative only. The complexity analysis in Eq. 21 also drops the N factor from the FFN term. Citation sloppiness—RetNet attributed to the wrong authors, a duplicated sentence—is minor but not a good sign.\n\nBottom line: if someone fixed the recurrence and ran real ablations, the memory-expert direction might be worth a workshop paper. As it stands, the core equivalence is invalid on its own terms and the empirical support is too thin. I would not send this to referees; I'd return it with a clear list of what needs to change.","headline":"A plausible memory-expert idea undermined by an invalid parallel/recurrent equivalence and thin experiments; not ready for referees.","tokens_in":15063,"tokens_out":2467,"would_cite":false,"duration_ms":23292,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"MixFormer claims that replacing a single state-space memory with many time-aware memory experts yields the best long-range accuracy among efficient Transformers at linear cost.","keywords":["MixFormer","linear attention","state space models","mixture of experts","long-range dependency","time-aware decay","Long-Range Arena","image generation"],"falsifier":"Train MixFormer with an explicitly absolute-position-dependent $G$ and measure the memory footprint of the recurrent state as sequence length grows; if memory grows with $L$, the $O(1)$ claim fails. Alternatively, compute the outputs of Eqs. (14) and (15) on the same input; any mismatch for a fixed-weight model shows the two modes are not computing the same function.","tokens_in":13962,"feed_emoji":"🧠","tokens_out":5119,"duration_ms":99882,"temperature":0.7,"pith_summary":"The paper sets out to establish that a linear Transformer can keep pace with or beat softmax attention on long-range tasks if it replaces the usual single memory state with many specialized memory states gated by time. Existing state space models store all history in one fixed-size matrix, so early information gets diluted; MixFormer instead keeps sixty-four memory experts, each with its own exponential decay and learned positional bias, and routes inputs to a subset of them. On the Long-Range Arena benchmark it reports an average accuracy of 56.72 percent, above every efficient-Transformer baseline it compares against, and it shows image completions on MNIST and CIFAR-10. The reason this matters is that the model retains linear-time training and constant-memory autoregressive inference, so long contexts could be processed at lower cost than with quadratic softmax attention.","feed_headline":"MixFormer tops long-range benchmark among efficient transformers","feed_subtitle":"A 56.72 average on LRA beats every baseline listed, using linear attention with 64 memory experts.","key_machinery":"The load-bearing object is the TALA recurrence and its parallel matrix form. In parallel, attention is computed as $Q \\odot (K^T \\odot D)V$, where $D$ is a causal mask with entries $\\omega^{n-m} G_{nm}$; in recurrent form, the same rule becomes $S_n = \\omega G \\odot S_{n-1} + K_n^T V_n$. The MoME layer then aggregates multiple such states with router weights $E_i = \\text{softmax}(K_n V_n W_g)$. This combination is what lets the model keep constant-memory inference while having more than one place to store history.","core_discovery":"The central claim, stated by the authors, is that memory dilution, not kernel approximation, is the main bottleneck for linear Transformers on ultra-long sequences. MixFormer addresses it with Time-Aware Linear Attention (TALA), where the recurrent state update is $S_n = \\omega G \\odot S_{n-1} + K_n^T V_n$, with $\\omega$ an exponential decay weight and $G$ a learnable pairwise positional bias, and with a Mixture-of-Memory-Experts layer that routes queries to $K_n V_n$ through a softmax router. Each expert keeps an independent memory state, so different experts can preserve short-term and long-term patterns. The paper reports that this design achieves the best average score on LRA and produces high-quality image generations, and it claims these gains come with linear complexity and constant memory during inference.","pith_inferences":["The paper does not show that the parallel and recurrent forms in Eqs. (14) and (15) are mathematically equivalent; if $G$ is truly pairwise and absolute-position dependent, the recurrent state would have to grow with sequence length, so the constant-memory claim needs an extra argument. That is an editorial caution, not a result in the paper.","The MoME router cost is $O(Nk)$; at 64 experts this is negligible next to $N d^2$, but if $k$ were scaled with $d$, the linear-complexity claim could break unless the router is made sparse.","A natural extension would be to make the decay $\\omega$ or the bias $G$ input-dependent per token, turning TALA into a selective state space model; that would test whether the gains come from time-awareness or from the multi-expert memory alone.","The 'sustainable web infrastructure' conclusion is an extrapolation: the paper reports accuracy and complexity, not wall-clock time or energy per token, so that claim remains untested."],"forward_implications":["Long-context language modeling could be done with linear instead of quadratic attention without the accuracy drop usually reported for kernel-based linear Transformers.","Because MixFormer supports both parallel training and recurrent inference, the same weights can switch between GPU-parallel training and constant-memory generation.","Routing to multiple memory experts gives a mechanistic handle on what the model remembers: the paper's visualizations show long-term experts holding distant entities and short-term experts handling recent tokens.","The reported LRA average of 56.72 percent would put MixFormer ahead of every baseline in its comparison table, including BigBird and Performer.","Image completion on CIFAR-10 suggests the architecture can be applied to pixel-sequence generation, where long-range dependencies across channels matter."],"supporting_citations":[{"why":"Supplies the fixed-decay recurrent linear attention and chunkwise parallel training that TALA extends.","marker":"[16]"},{"why":"Supplies the input-dependent gating linear attention that MixFormer contrasts with its own time-aware mechanism.","marker":"[17]"},{"why":"Provides the kernel-based linear attention formulation and autoregressive recurrence that TALA builds on.","marker":"[18]"},{"why":"Establishes the structured state space model formulation that motivates the memory-state design.","marker":"[11]"},{"why":"Defines the Long-Range Arena benchmark that supplies all the long-range accuracy numbers.","marker":"[36]"},{"why":"Introduces learned pairwise positional bias and exponential decay ideas used in TALA.","marker":"[28]"},{"why":"Provides RWKV, a token-shifted decay baseline that MixFormer compares against.","marker":"[29]"},{"why":"Supplies the sparse mixture-of-experts routing idea adapted for memory experts.","marker":"[34]"},{"why":"Provides Mamba, a selective state space model baseline with input-dependent dynamics.","marker":"[26]"},{"why":"Provides Mamba-2 and the structured state space duality view that motivates efficient SSM training.","marker":"[27]"}],"fun_headline_variants":["MixFormer: mixture of memory experts beats all LRA baselines","Mixture of memory experts in linear attention tops LRA","Memory experts solve long-range bottleneck in linear transformers","Time-aware attention with 64 experts tops LRA average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The constant-memory recurrent form works only if the learned pairwise positional bias $G$ depends on relative position or is a per-head scalar; the paper does not prove that the parallel form with absolute-position $G$ is equivalent to the recurrent update.","fun_headline_variants_meta":{"raw":{"variants":["MixFormer: mixture of memory experts beats all LRA baselines","Mixture of memory experts in linear attention tops LRA","Memory experts solve long-range bottleneck in linear transformers","Time-aware attention with 64 experts tops LRA average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2341,"prompt_tokens":873,"completion_tokens":1468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":1401}},"tokens_in":489,"tokens_out":1468,"duration_ms":11939,"temperature":1.0,"reasoning_tokens":1401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:58:10.173725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MixFormer with an explicitly absolute-position-dependent $G$ and measure the memory footprint of the recurrent state as sequence length grows; if memory grows with $L$, the $O(1)$ claim fails. Alternatively, compute the outputs of Eqs. (14) and (15) on the same input; any mismatch for a fixed-weight model shows the two modes are not computing the same function.","supporting_citations":[{"cited_title":"Gated linear attention transformers with hardware-efficient training","cited_arxiv_id":null,"evidence_quote":"Supplies the input-dependent gating linear attention that MixFormer contrasts with its own time-aware mechanism."},{"cited_title":"Transformers are rnns: fast autoregressive transformers with linear attention","cited_arxiv_id":null,"evidence_quote":"Provides the kernel-based linear attention formulation and autoregressive recurrence that TALA builds on."},{"cited_title":"Efficiently modeling long sequences with structured state spaces","cited_arxiv_id":null,"evidence_quote":"Establishes the structured state space model formulation that motivates the memory-state design."},{"cited_title":"Long range arena: a benchmark for efficient transformers","cited_arxiv_id":null,"evidence_quote":"Defines the Long-Range Arena benchmark that supplies all the long-range accuracy numbers."},{"cited_title":"Rwkv: Reinventing rnns for the transformer era","cited_arxiv_id":null,"evidence_quote":"Provides RWKV, a token-shifted decay baseline that MixFormer compares against."},{"cited_title":"Outrageously large neural networks: the sparsely-gated mixture-of-experts layer","cited_arxiv_id":null,"evidence_quote":"Supplies the sparse mixture-of-experts routing idea adapted for memory experts."},{"cited_title":"Mamba: linear-time sequence modeling with selective state spaces","cited_arxiv_id":null,"evidence_quote":"Provides Mamba, a selective state space model baseline with input-dependent dynamics."},{"cited_title":"Transformers are ssms: generalized models and efficient algorithms through structured state space duality","cited_arxiv_id":null,"evidence_quote":"Provides Mamba-2 and the structured state space duality view that motivates efficient SSM training."}],"review_version":1}