{"id":"1ab8b416-4fa1-4c61-9d3b-1dffedb3a16a","arxiv_id":"2602.03922","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OVQ-attention delivers linear-time constant-memory sequence mixing via sparse Gaussian-mixture-based memory updates, matching self-attention performance on tasks up to 64k length while using far less memory.","lead":"This paper introduces online vector-quantized attention, a sequence mixing layer that uses linear compute and constant memory with a sparse update rule to support larger memory states for long contexts. A smart generalist might read it because it targets the core efficiency bottleneck in scaling language models to longer sequences without quadratic costs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Sparse memory update rule from Gaussian mixture regression lacks direct ablation against alternatives for long-range retention","rationale":"The reader's weakest assumption directly identifies the untested memory-management component that the strongest claim depends on. Full-text experiments appear to rely on the same unablated comparison, so the concern remains load-bearing and the provisional UNVERDICTED stance should be retained pending the isolation test.","tokens_in":1724,"tokens_out":305,"duration_ms":27376,"concrete_test":"Re-run the 64k long-context LM evaluation (and the synthetic tasks) with an ablated variant that replaces the GMM sparse update by a FIFO queue of identical state size; if the FIFO variant closes most of the gap to OVQ-attention, the update rule is not the primary driver of the reported long-range capacity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the sparse update (derived from GMM regression) maintains long-range information at 64k lengths without catastrophic interference or forgetting, while keeping constant memory. This is the least-secured step: the paper reports gains over linear/VQ baselines and parity with self-attention on synthetic and LM tasks, but does not compare the specific update rule to other constant-memory strategies (e.g., FIFO, reservoir sampling, or learned compression) of matched capacity. Without that isolation, performance differences could stem from codebook size, quantization, or training details rather than the GMM-derived sparsity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces online vector-quantized (OVQ) attention, a sequence mixing layer for language models that achieves linear compute and constant memory via a sparse memory update rule derived from Gaussian mixture regression. It reports significant gains over linear attention and original VQ-attention baselines, along with competitive (sometimes identical) performance to strong self-attention models on synthetic long-context tasks and language modeling up to 64k sequence lengths, while using a small fraction of the memory.","tokens_in":1890,"tokens_out":394,"duration_ms":51717,"significance":"If the results hold under further verification, OVQ-attention offers a promising compromise between the efficiency of linear attention/SSMs and the long-context performance of full self-attention. The explicit derivation from Gaussian mixture regression supplies a theoretical basis that strengthens the construction relative to purely empirical alternatives.","major_comments":[{"comment":"§4 (Experiments): the central performance claims at 64k lengths rest on the sparse update rule preserving long-range information without catastrophic forgetting or interference, yet the manuscript provides no ablation comparing this GMM-derived rule to other constant-memory strategies of matched capacity (e.g., FIFO, reservoir sampling, or learned compression). Without such isolation, gains could be attributable to codebook size, quantization details, or training procedure rather than the proposed sparsity mechanism.","section":"§4"}],"minor_comments":[{"comment":"The abstract contains the phrase 'up 64k sequence length'; this should be corrected to 'up to 64k sequence length' for clarity.","section":"Abstract"},{"comment":"Notation for the sparsity/update-rate hyperparameter and its relation to the number of mixture components could be made more explicit in the methods section to aid reproducibility.","section":"Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback and recommendation for major revision. The suggested ablation will help isolate the contribution of the GMM-derived update rule, and we will incorporate it in the revised manuscript.","responses":[{"response":"We agree that additional ablations would strengthen the experimental claims by isolating the effect of the GMM-derived sparse update. The update rule is derived from Gaussian mixture regression, which supplies a probabilistic justification for sparse, quantized memory that preserves long-range information without explicit forgetting mechanisms. This theoretical basis distinguishes OVQ-attention from heuristic constant-memory approaches. In the revision we will add experiments comparing OVQ-attention against FIFO and reservoir sampling baselines with matched memory capacity on the 64k-length tasks. We will also include a brief discussion of learned compression, noting that it typically requires extra parameters and training overhead not present in our fixed codebook approach. These changes should clarify that performance differences arise from the sparsity mechanism rather than codebook size or training details alone.","revision_made":"yes","referee_comment":"[§4] §4 (Experiments): the central performance claims at 64k lengths rest on the sparse update rule preserving long-range information without catastrophic forgetting or interference, yet the manuscript provides no ablation comparing this GMM-derived rule to other constant-memory strategies of matched capacity (e.g., FIFO, reservoir sampling, or learned compression). Without such isolation, gains could be attributable to codebook size, quantization details, or training procedure rather than the proposed sparsity mechanism."}],"tokens_in":1295,"tokens_out":324,"duration_ms":42967,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper builds an online version of vector-quantized attention. It uses a sparse memory update rule taken from Gaussian mixture regression so the model can keep constant memory while expanding its effective state size for longer sequences. This sits between full self-attention and standard linear attention or SSMs on the efficiency curve.","headline":"OVQ-attention adds a sparse GMM-derived update to vector-quantized attention for constant-memory long contexts, but the gains are not isolated from other design choices.","tokens_in":2370,"tokens_out":145,"would_cite":false,"duration_ms":26174,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"We develop a theoretical basis for OVQ-attention based on Gaussian mixture regression... sparse online update rules... growth function Nt = t N / (t + N)"}],"headline":"OVQ-attention's GMR-based sparse clustering and online dictionary growth are orthogonal to RS distinction-to-J-cost forcing","alignment":"orthogonal","rationale":"Paper develops online vector-quantized attention via Gaussian mixture regression, nearest-neighbor sparse updates, and plateauing growth Nt = tN/(t+N) for constant-memory long-context modeling. RS framework (reality_from_one_distinction, Jcost uniqueness via Aczél, phi-ladder constants, 8-tick periodicity) derives physics from bare distinguishability with no adjustable parameters or ML-specific clustering. No shared structure (cosh-cost, ratio symmetry, recognition lattices) and domain is cs.LG sequence mixing; RS has no opinion.","tokens_in":59033,"confidence":"high","tokens_out":250,"duration_ms":20651,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Online vector-quantized attention matches self-attention performance on sequences up to 64,000 tokens while using linear compute and constant memory.","keywords":["attention mechanisms","vector quantization","long context modeling","language models","sequence mixing","efficient transformers","gaussian mixture models","memory efficient attention"],"falsifier":"An experiment in which OVQ-attention produces substantially lower accuracy than full self-attention on a task that requires recalling a specific fact introduced only at the start of a 64,000-token sequence would indicate that the memory capacity increase does not deliver the claimed long-range retention.","tokens_in":2625,"feed_emoji":"","tokens_out":759,"duration_ms":37766,"temperature":0.7,"pith_summary":"The paper presents online vector-quantized attention as a new sequence mixing layer designed to improve the trade-off between computational efficiency and the ability to handle long contexts in language models. It achieves linear computation costs and constant memory usage by maintaining a memory state that grows through sparse updates rather than remaining fixed in size like standard linear attention methods. This design draws on Gaussian mixture regression to guide how new information is incorporated without overwriting distant details. A reader would care because current approaches either scale poorly in memory for long inputs or lose the capacity to connect information across long distances. The reported experiments indicate that this method improves on prior linear and vector-quantized attention variants and reaches parity with full self-attention in several long-context settings.","feed_headline":"Attention method rivals self-attention at 64k length with constant memory","feed_subtitle":"Sparse updates expand memory capacity while keeping compute linear and overall memory fixed across long sequences.","key_machinery":"The sparse memory update rule derived from Gaussian mixture regression, which expands the effective memory state size while keeping overall memory usage constant.","core_discovery":"OVQ-attention requires linear compute costs and constant memory, but unlike linear attention and SSMs it uses a sparse memory update that allows it to greatly increase the size of its memory state and consequently memory capacity. A theoretical basis is developed from Gaussian mixture regression, and the method is evaluated on synthetic long-context tasks as well as long-context language modeling, where it shows significant gains over linear attention baselines and the original VQ-attention while remaining competitive with strong self-attention baselines up to 64k sequence length despite using only a small fraction of the memory.","pith_inferences":["Hybrid models that combine this sparse memory state with existing state-space models might further reduce compute while preserving recall over even longer ranges.","The approach suggests a path toward constant-memory architectures that still support retrieval of arbitrary distant tokens if the Gaussian mixture update can be made more selective.","Real-world document and code completion tasks that span tens of thousands of tokens would be a natural next test to measure whether the synthetic-task gains translate to practical utility."],"forward_implications":["Language models could process sequences up to at least 64k tokens with performance close to standard self-attention but at a small fraction of the memory cost.","Linear attention and SSM approaches could be improved by incorporating similar sparse memory expansion for better long-range dependency handling.","Training and inference on longer contexts become feasible without quadratic memory growth or the information loss typical of fixed-size memory states.","The same mechanism could extend the effective context length in existing transformer architectures with minimal additional hardware requirements."],"fun_headline_variants":["OVQ attention rivals self-attention to 64k length","Sparse updates enable expanded memory state in OVQ attention","OVQ attention improves on linear attention baselines"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The sparse memory update rule derived from Gaussian mixture regression is assumed to retain enough long-range information across updates without interference or loss of earlier details.","fun_headline_variants_meta":{"raw":{"variants":["OVQ attention rivals self-attention to 64k length","Sparse updates enable expanded memory state in OVQ attention","OVQ attention improves on linear attention baselines"]},"model":"grok-4.3","cost_usd":0.014999,"raw_usage":{"total_tokens":6371,"prompt_tokens":689,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":149990500,"prompt_tokens_details":{"text_tokens":689,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5634,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":689,"tokens_out":48,"duration_ms":73385,"temperature":1.0,"reasoning_tokens":5634,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-21T13:38:14.239167+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment in which OVQ-attention produces substantially lower accuracy than full self-attention on a task that requires recalling a specific fact introduced only at the start of a 64,000-token sequence would indicate that the memory capacity increase does not deliver the claimed long-range retention.","supporting_citations":[],"review_version":1}