{"id":"99ad8b07-e145-4b9e-896f-ef11534e8397","arxiv_id":"2506.01963","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A proposed non-attention LLM combining state-space, convolution, recurrence, and retrieval claims near-linear scaling for million-token contexts, with reported benchmark gains unsupported by released artifacts.","lead":"This paper describes an LLM architecture that removes token-to-token attention entirely, using state-space layers, dilated convolutions, a recurrent chunk supervisor, and retrieval memory. It claims competitive language modeling and near-linear scaling on very long contexts, but supplies no code, data, or training logs to back the reported numbers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that one GRU hidden state per chunk plus top-k pooled-chunk retrieval (Eqs. 10-12, 19) preserves enough long-range information; the paper provides no theory or released experiment showing this, so the ultra-long-context claim is unsupported.","rationale":"The reader's weakest_assumption is correct: the entire ultra-long-context claim rests on the compressed cross-chunk channel. The architecture is a synthesis of known components; the complexity analysis in §4.8 is standard and sub-quadratic by construction, so that part is not the issue. The unverified part is whether a single GRU state plus top-2 chunk summaries can replace attention's direct token-to-token access. The paper's own ablations (§7.8) show that removing retrieval or the global RNN hurts cross-chunk performance, which confirms these components carry the long-range burden, but it says nothing about whether their capacity is sufficient. The proposed probe directly measures the channel's information throughput as a function of distance and retrieval depth; if it passes, the central claim gains support, and if it fails, the claim collapses regardless of the unverified PPL numbers. I therefore keep the reader's REJECT verdict unchanged, while noting that the reproducibility problems (no code, inconsistent Table 2 baselines) are a separate but reinforcing ground for rejection.","tokens_in":22521,"tokens_out":6045,"duration_ms":59960,"concrete_test":"Run a controlled long-range dependency probe: using the released code (or a reimplementation), train the proposed 120M model on sequences of 100k tokens containing a unique planted token at position p, probed at position q=p+L for L in {1k, 10k, 50k, 100k}, with distractor tokens, for k=1 and k=2 retrieval. If probe accuracy degrades with L or collapses when the planted token's chunk is not among the top-k retrieved chunks, the single-GRU/top-k channel is insufficient and the ultra-long-context claim fails. This test isolates information sufficiency from the unresolved PPL and baseline questions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the model handles hundreds of thousands to millions of tokens with competitive quality rests on cross-chunk information flow. In §4.5/§4.16 (Eqs. 12 and 19), all cross-chunk communication is compressed into a single GRU hidden state h_g^(m+1), updated once per chunk from the pooled embedding c'_m; in §4.4/§4.15, external memory returns only top-k (k=1-2 per §7.4) pooled chunk embeddings. No information-theoretic bound, capacity analysis, or released experiment shows that this roughly 512-dim-per-chunk channel is sufficient for the tasks claimed in §7.6, such as recalling a pattern 100,000 tokens earlier. The reported 18.7 PPL / 1.04 bpc cannot be checked because no code or checkpoints are released, and Table 2's baseline numbers are inconsistent with published results (e.g., GPT-2 small at 20.5 PPL on WikiText-103), so the empirical support for that sufficiency is also unverified. If the compressed channel is not demonstrably sufficient, the architecture reduces to a chunked local model that loses the long-range coherence defining a long-context LLM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a non-attention LLM architecture for ultra-long contexts. The input is split into chunks of length c; each chunk is processed by an S4-inspired state-space block and multi-resolution dilated convolutions, then pooled into a single chunk embedding. Cross-chunk information is carried by a GRU-based recurrent supervisor and by an external memory that retrieves the top-k most similar pooled chunk embeddings. The authors claim this avoids the QK^T V operation entirely, achieves O(n) or O(n log c) complexity in sequence length n, and report competitive perplexity on WikiText-103 (18.7 PPL) and Enwik8 (1.04 bpc), with qualitative claims of successful recall across up to 100,000 tokens and scaling to hundreds of thousands of tokens.","tokens_in":22645,"tokens_out":5680,"duration_ms":64574,"significance":"If the empirical claims were substantiated, the paper would present a useful modular alternative to attention for long-context language modeling, combining SSM blocks, multi-scale convolution, recurrence, and retrieval. The complexity analysis is transparent and the near-linear scaling is true by construction for chunked O(c) blocks. However, the central contributions depend on experimental results that are not reproducible from the manuscript, and the claimed training memory footprint omits a component that grows with the number of chunks. The paper is therefore best read as a conceptual architecture proposal; as a research contribution with quantitative claims, it is not currently supported.","major_comments":[{"comment":"The two headline numbers, 18.7 PPL on WikiText-103 and 1.04 bpc on Enwik8, are not backed by reproducible evidence. No training configuration, data split description, seed, hyperparameter sweep, or evaluation log is provided; the GitHub URL in §8 is not accompanied by code, checkpoints, or a README in the manuscript. In addition, the baseline values in Table 2 are not sourced: the GPT-2 Small entry of 20.5 PPL on WikiText-103 is not a standard published number for a 124M GPT-2 on that dataset, and BigBird's 19.2 PPL is reported without a training setup. These inconsistencies make the claim of 'competitive or superior' performance impossible to verify.","section":"§7.5, Table 2"},{"comment":"The claimed training memory footprint of approximately O(B*c*d) ignores the recurrent supervisor's backpropagation cost. Equation (12) and Equation (19) update the global hidden state once per chunk, so training through M = n/c chunks requires storing that hidden state at each chunk boundary (or using truncated BPTT, which is not stated). The additional memory is O(B*M*h) and the additional compute is O(M*h^2); for the million-token scenario with c = 1024, M is roughly 1000, and this is not a negligible overhead. The paper should state the actual training-time memory and specify whether full BPTT, truncated BPTT, or gradient checkpointing is used.","section":"§4.10, §4.5, §4.16"},{"comment":"The long-context capability rests on an unexamined information bottleneck. All cross-chunk information must pass through a single GRU hidden state updated once per chunk, plus top-1 or top-2 pooled chunk embeddings retrieved from external memory (§7.4). The paper provides no capacity analysis, information-theoretic bound, or controlled experiment demonstrating that this compressed channel is sufficient to maintain coherent dependencies across 100k+ tokens. The claim in §7.6 of recalling a pattern introduced 100,000 tokens earlier is anecdotal: no task definition, accuracy numbers, or comparison protocol is given.","section":"§4.4, §4.5, §4.16, §7.6"},{"comment":"The abstract and conclusion claim scaling to 'hundreds of thousands to potentially millions of tokens,' but the described experiments cover sequences of 50,000 to 200,000 tokens, with no scaling plot, wall-clock measurements beyond a coarse relative statement, or peak-memory curve. The million-token figure is explicitly speculative ('potentially'), so the central scalability claim is not empirically established at the stated scale.","section":"§7.6, §8, Abstract"}],"minor_comments":[{"comment":"Equation numbers restart between Section 3 and Section 4: Equations (1)-(5) are reused as (6)-(20). This makes cross-references such as 'Eq. 4' and 'Eq. 12' ambiguous and should be fixed with continuous numbering.","section":"General"},{"comment":"The text says 'Table 1 summarizes the test perplexity...' but the actual results table is Table 2; Table 1 in §2.9 is a qualitative comparison table and does not report the benchmark numbers claimed in the sentence.","section":"§7.5"},{"comment":"The entry 'Proposed (SSM+Conv 120M 32k (chunk))' is unclear because §7.4 reports chunk sizes of 1024, 2048, or 4096 tokens; the meaning of '32k (chunk)' should be clarified.","section":"Table 2"},{"comment":"The related-work discussion of 'DeepSeek LLM' cites DeepSeek-R1 (arXiv:2501.12948), which is a reasoning-model report and not an architecture paper on long-context extension. The citation does not support the claims made about DeepSeek's chunk-processing and caching mechanisms.","section":"§2.1, Reference [11]"},{"comment":"The FAISS citation is given as 'Johnson et al., 2017' but has no entry in the reference list; similarly, 'Wang et al., 2020' is used in the text for Linformer while the reference list item [21] is incomplete. All citations should be matched to complete bibliography entries.","section":"§6.4, References"}],"recommendation":"reject","confidential_remarks":"The manuscript's central quantitative claims cannot be checked because the experiments are not reproducible from the submitted material, and the baseline numbers in Table 2 appear inconsistent with known results. The architectural idea is not unreasonable, but the submission is not yet at the standard required for publication. If the authors supply code, data, experimental logs, and a corrected training-memory analysis that accounts for recurrent BPTT, a resubmission could be considered on the merits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a coherent proposal for a non-attention LLM (S4-style SSM + dilated convs + GRU chunk supervisor + FAISS retrieval), and the complexity analysis is standard and true by construction. But the paper is not a position piece: it reports trained-model numbers (18.7 PPL WikiText-103, 1.04 bpc Enwik8) with no code, no data, no logs, and a GitHub link that is not verifiable from the manuscript. The empirical core is unsupported, and the baseline table doesn't line up with published numbers (GPT-2 small at 20.5 PPL on WikiText-103 is off from the commonly cited ~29). So the headline result—competitive quality at near-linear cost—cannot be checked.\n\nWhat's genuinely here: the four-component combination is a legitimate synthesis, and the authors are honest that each part has precedent. The chunked architecture has a real chance of avoiding quadratic blowups, and the per-chunk cost analysis is fine as far as it goes. The ablation logic (retrieval helps cross-chunk recall, RNN helps coherence) is sensible, but it is reported as narrative, not as numbers anyone can inspect.\n\nThe soft spots run deeper than missing code. The load-bearing assumption is that one GRU hidden state per chunk plus top-1 or top-2 retrieved pooled chunk embeddings is enough cross-chunk information flow to support the 100k-token recall claims in Section 7.6. The paper offers no capacity bound, no analysis, and no released experiment to show that. The training-memory claim of O(B*c*d) also glosses over backprop through the recurrent supervisor, which needs per-chunk hidden states stored; the real footprint is O(B*c*d + M*B*h) or similar. That's not a fatal flaw in the design, but it is a hole in the stated complexity.\n\nThe writing is clear, the related work is mostly on point, and the authors avoid overclaiming novelty at the component level. If they released code, checkpoints, and a corrected experimental table, the architecture would be worth a look. As submitted, the central claim rests on assertion. I'd advise engaging with it only to extract the proposal and complexity sketch, not to trust the numbers. A serious editor should desk-reject pending evidence; there is not enough reproducible content here to justify referee time on the empirical claims.","headline":"A plausible architecture proposal undone by unverifiable empirical claims; the synthesis is real but the paper asks to be taken on faith.","tokens_in":23326,"tokens_out":3318,"would_cite":false,"duration_ms":32494,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a fully attention-free LLM can match efficient transformers on standard benchmarks while scaling near-linearly to million-token contexts.","keywords":["non-attention LLM","state-space models","multi-resolution convolution","recurrent supervisor","retrieval-augmented memory","ultra-long context","near-linear scaling","language modeling"],"falsifier":"Train the model on long synthetic documents in which a unique token appears exactly once and must be recalled more than one chunk later, then measure exact-match recall in next-token prediction as the chunk size grows. If recall collapses as $c$ increases or falls well below a same-size token-level attention model, the claim that one pooled vector per chunk plus top-$k$ retrieval preserves long-range information is falsified.","tokens_in":22181,"feed_emoji":"🧠","tokens_out":11417,"duration_ms":112600,"temperature":0.7,"pith_summary":"The paper tries to establish that a language model can eliminate token-to-token self-attention entirely and still match efficient transformers on standard benchmarks, while extending to contexts of hundreds of thousands or millions of tokens. The proposed architecture splits the input into chunks, mixes tokens inside each chunk with state-space and multi-resolution convolution layers, and bridges chunks with a recurrent hidden state plus retrieval from an external memory of chunk summaries. If true, this matters because the main obstacle to ultra-long-context modeling—the quadratic cost of the $QK^\\top V$ attention matrix—would be removed rather than approximated or sparsified. The paper reports 18.7 perplexity on WikiText-103 and 1.04 bits per character on Enwik8, and describes scaling experiments on sequences up to about 200,000 tokens.","feed_headline":"Attention-free LLM reaches million-token contexts at near-linear cost","feed_subtitle":"State-space mixing, chunk recurrence, and retrieval replace attention with competitive perplexity on standard benchmarks.","key_machinery":"The central object is the chunked processing pipeline, with the chunk summary vector as the unit of global memory. Inside each chunk, an S4-inspired state-space block provides near-linear long-range mixing, and parallel dilated convolutions refine local patterns at several scales; after mean pooling, one vector per chunk becomes both the input to a recurrent supervisor and the key stored in an external key-value memory. A gated fusion combines the current chunk summary with retrieved summaries, and the resulting vector updates the recurrent hidden state that carries coherence across chunks. This machinery carries the argument because no step ever forms a pairwise token interaction, so total cost scales with the number of chunks rather than with the square of the token count.","core_discovery":"The central claim is that a non-attention architecture can serve as a general-purpose language model, not merely a long-context special case. The defining move is to never construct an $n\\times n$ token-to-token attention map: each chunk is transformed by state-space kernels inspired by S4 and by dilated convolutions at multiple resolutions, then pooled into a single chunk embedding; a recurrent supervisor updates one global hidden state per chunk, and a retrieval-augmented memory returns a small number of past chunk summaries that are gated into the current representation. Token-level predictions come from an LM head applied to the locally mixed representations, with no attention anywhere in the pipeline. The paper argues that because every component costs $O(c)$ or $O(c\\log c)$ per chunk of size $c$, the total cost is $O(n)$ or $O(n\\log c)$, and it presents benchmark numbers intended to show that removing attention does not cost quality.","pith_inferences":["Editorial extension: if the claim holds, the practical ceiling shifts from model capacity to the retrieval index; scaling to millions of tokens becomes primarily an engineering problem of chunk summarization and nearest-neighbor search.","Editorial extension: a direct test of the information bottleneck would vary chunk size $c$ at fixed total context. If the recurrent supervisor truly compresses history, perplexity should degrade smoothly as $c$ grows; a sharp collapse would point to the per-chunk pooling step as the operative limitation.","Editorial extension: the external memory could be read hierarchically—first retrieving whole chunks, then local windows within them—which might recover token-level precision that a single pooled vector loses while keeping near-linear cost."],"forward_implications":["With the paper's reported numbers, a roughly 120M-parameter attention-free model reaches 18.7 perplexity on WikiText-103 and 1.04 bpc on Enwik8, which it presents as competitive with or better than efficient transformer baselines.","Context length stops being the main driver of memory: the active training and inference footprint is set by chunk size and batch size rather than by total tokens, so contexts of 100,000 to 1,000,000 tokens become feasible on a single GPU.","The paper reports stable chunked training and accurate recall of patterns planted up to 100,000 tokens earlier in synthetic and book-length inputs, attributing that recall to the recurrent supervisor and retrieval memory.","Because there is no attention, the effective context is bounded by the external memory index and by chunked decoding rather than by a fixed window, so processing an entire corpus becomes one pass instead of a pipeline of overlapping slices."],"supporting_citations":[{"why":"Defines the $QK^\\top V$ self-attention operation whose quadratic cost the paper sets out to eliminate.","marker":"[5]"},{"why":"Supplies the state-space convolution approach that the paper adapts for near-linear intra-chunk mixing.","marker":"[19]"},{"why":"Provides segment-level recurrent caching of hidden states, the direct predecessor of the cross-chunk recurrent supervisor.","marker":"[9]"},{"why":"Gives the sparse-attention baseline whose perplexity and context length the proposed model claims to match or surpass.","marker":"[6]"},{"why":"Gives the long-document sparse-attention model used for runtime and memory comparisons at extended contexts.","marker":"[7]"},{"why":"Supplies the kernel-linearized attention baseline that the paper compares against on perplexity and bpc.","marker":"[8]"},{"why":"Shows retrieval-augmented language modeling with external passages, the lineage for the paper's external memory module.","marker":"[14]"},{"why":"Provides a recent recurrent non-attention baseline that the paper compares against on standard benchmarks.","marker":"[16]"}],"fun_headline_variants":["No attention, no problem: LLM scales to million-token contexts","Scaling LLMs to millions of tokens without attention","Non-attention LLM hits million-token windows at near-linear cost","Million-token LLM without attention, near-linear cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that compressing each chunk to one compact summary vector, then passing only that vector and a few retrieved summaries across chunk boundaries, preserves enough information that removing token-to-token attention does not degrade language modeling on long contexts.","fun_headline_variants_meta":{"raw":{"variants":["No attention, no problem: LLM scales to million-token contexts","Scaling LLMs to millions of tokens without attention","Non-attention LLM hits million-token windows at near-linear cost","Million-token LLM without attention, near-linear cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2289,"prompt_tokens":860,"completion_tokens":1429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":1356}},"tokens_in":476,"tokens_out":1429,"duration_ms":11840,"temperature":1.0,"reasoning_tokens":1356,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:59:15.678196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the model on long synthetic documents in which a unique token appears exactly once and must be recalled more than one chunk later, then measure exact-match recall in next-token prediction as the chunk size grows. If recall collapses as $c$ increases or falls well below a same-size token-level attention model, the claim that one pooled vector per chunk plus top-$k$ retrieval preserves long-range information is falsified.","supporting_citations":[{"cited_title":"Gomez, Łukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Defines the $QK^\\top V$ self-attention operation whose quadratic cost the paper sets out to eliminate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides segment-level recurrent caching of hidden states, the direct predecessor of the cross-chunk recurrent supervisor."},{"cited_title":"Zaheer and et al","cited_arxiv_id":null,"evidence_quote":"Gives the sparse-attention baseline whose perplexity and context length the proposed model claims to match or surpass."},{"cited_title":"Choromanski and et al","cited_arxiv_id":null,"evidence_quote":"Supplies the kernel-linearized attention baseline that the paper compares against on perplexity and bpc."},{"cited_title":"Borgeaud and et al","cited_arxiv_id":null,"evidence_quote":"Shows retrieval-augmented language modeling with external passages, the lineage for the paper's external memory module."},{"cited_title":"Rwkv-lm: Reinventing rnns for the transformer era.https://github","cited_arxiv_id":null,"evidence_quote":"Provides a recent recurrent non-attention baseline that the paper compares against on standard benchmarks."}],"review_version":1}