Pith. sign in

REVIEW 3 major objections 5 minor 75 references

Overflow Prevention Enhances Long-Context Recurrent LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Recurrent LLMs trained on long contexts still overflow their fixed memory; a chunk-based inference method that decodes only the most relevant chunk fixes the failure and beats full-context processing.

desk verdict The overflow finding is real and OPRM helps consistently; the selection mechanism is under-validated and the benchmark gains include some test-set tuning, but the core result holds. read the letter →

arxiv 2505.07793 v2 pith:FX2I2EEV submitted 2025-05-12 cs.LG cs.AI

Assaf Ben-Kish , Itamar Zimerman , M. Jehanzeb Mirza , Lior Wolf , James Glass , Leonid Karlinsky , Raja Giryes This is my paper · ORCID
classification cs.LGcs.AI
keywords recurrentLLMsmemoryoverflowassociativerecallchunk-basedinferencelong-contextreasoningselectivedecodingLongBenchcontextextension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recurrent language models keep a fixed-size memory as they read, and this paper argues that even state-of-the-art models trained on long contexts do not actually use those contexts: the memory overflows, so retrieval accuracy collapses as the number of facts grows. The paper proposes OPRM, a training-free inference procedure that splits the prompt into chunks, processes each chunk in parallel, and decodes only from the chunk whose output distribution is most confident. In experiments across LongBench, LongBench v2, and context-extension tasks, this single-chunk strategy often beats full-context processing, improving scores by 14% to 51% on LongBench and reaching a reported state-of-the-art 30.8 on LongBench v2 for Falcon3-Mamba-Inst-7B. The result matters because it suggests recurrent LLMs' long-context weakness is a memory-capacity problem, not a length-generalization problem, and it can be mitigated without retraining.

What carries the argument

The load-bearing object is the fixed-size recurrent hidden state that recurrent layers update token by token; its capacity limits how many facts can be retrieved. In a Mamba layer this state has dimension $d \times d_{\mathrm{state}}$. OPRM (Overflow Prevention for Recurrent Models) is the method built around that limitation: it splits the context $C$ into $b$ chunks of length $L$, runs each chunk as a separate prompt $[P, C_i, S]$ in parallel, filters out chunks whose first predicted token is an "Error" IDK token, then selects the chunk with the lowest entropy over the next-token distribution and decodes from its recurrent state. This avoids overflow because each chunk's information load is bounded, and the associative-recall curve serves as the diagnostic that reveals the overflow in the first place.

What would settle it

Build a benchmark where every question can be answered only by combining facts from two chunks placed far apart, so that no single chunk contains the full answer; if OPRM scores lower than full-context decoding on those questions while matching it on single-chunk questions, the locality assumption is the ceiling.

Watch

Extended reading notes

Core claim

The central discovery is a bounded-memory overflow failure in recurrent LLMs: when a prompt contains more key-value facts than the fixed recurrent state can hold, retrieval accuracy falls toward zero, and the amount of information matters more than sequence length. The paper establishes this with associative-recall curves on Falcon-Mamba-Inst-7B and with trained 2-layer Mamba models, and then shows that OPRM, which turns the prompt into chunks small enough to fit memory and selects the most relevant one by minimum entropy after discarding "don't know" chunks, prevents the overflow. The same method that avoids overflow often improves over full-context decoding even on tasks that seem to require cross-context relations, and it extends usable context lengths well beyond training. The paper states this as evidence that recurrent models underuse their long contexts and that single-chunk decoding can be state-of-the-art.

Load-bearing premise

OPRM assumes that everything needed to answer a question sits inside one fixed-size piece of the context, and that the model's least-uncertain prediction identifies that piece; tasks that require joining evidence from several distant pieces would break it.

Editorial extensions

If this is right

  • On LongBench, OPRM raises overall scores by 14% for Falcon3-Mamba-Inst-7B, 28% for Falcon-Mamba-Inst-7B, 50% for RecurrentGemma-IT-9B, and 51% for RWKV6-Finch-7B, with the largest gains appearing on the longest contexts.
  • On LongBench v2, Falcon3-Mamba-Inst-7B plus OPRM reaches 30.8, the reported state of the art at its size class, and it beats comparable 7B Transformers on the 32K-128K and 128K+ length groups.
  • OPRM extends contexts far beyond training length without fine-tuning: a Mamba-130m trained on 2K tokens keeps high needle-in-a-haystack accuracy at 512K tokens, 256 times its training length, versus 64 times for a dedicated extension method.
  • Multi-hop reasoning scores double or more on several LongBench benchmarks, so overflow prevention helps tasks that require chaining evidence, not just single-fact retrieval.
  • Because chunks are processed in parallel and each stored state is tiny, OPRM is faster than full-context decoding on long inputs, for example 26.9 seconds versus 36.2 seconds at 128K tokens for Falcon3-Mamba-Inst-7B, with a small memory overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If recurrent models underuse trained long contexts because their memory overflows, then the advertised "context length" is not the operative capacity; the useful context is bounded by memory capacity, and evaluations should report information density per input rather than token count alone.
  • OPRM's success suggests that a recurrent LLM's own next-token confidence can act as a cheap relevance oracle, a signal that could be reused for retrieval-augmented generation or for routing long prompts without an external retriever.
  • A natural extension the paper does not build is to select multiple chunks and merge their states with an overflow-aware aggregation, since its multi-chunk ablation shows gains on short contexts; such a hybrid could recover tasks that genuinely need cross-chunk evidence.
  • The method predicts that OPRM's advantage grows with context length and information density, so applying it to larger recurrent or hybrid models should show smaller overflow but still monotone gains until memory scales with the task.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how fixed-size recurrent memory limits long-context performance in recurrent LLMs. It introduces an associative recall (AR) diagnostic showing that accuracy degrades as the number of key-value facts grows, even when the sequence is padded to fixed length, and it reproduces this overflow-like pattern in small 2-layer Mamba models trained from scratch across different channel and state dimensions. The paper then proposes OPRM, a training-free inference method that splits the context into chunks, processes each chunk with the prefix and suffix in parallel, removes chunks predicting an IDK/error token, selects the chunk with lowest output entropy, and decodes the answer from that chunk's recurrent state. On LongBench the method improves average scores by 14-51% over full-context baselines depending on the model, substantially improves multi-hop QA, extends Mamba-1.4b to longer contexts than dedicated context-extension methods, and achieves a LongBench v2 score of 30.8 with Falcon3-Mamba-Inst-7B, which the paper identifies as state of the art for the 7B size class. The manuscript includes ablations of chunk selection and the IDK filter, comparisons to RAG and agentic baselines, and an efficiency analysis.

Significance. The overflow phenomenon is convincingly demonstrated in the controlled setting: AR accuracy decays with the number of facts, is largely insensitive to padded sequence length, and improves with state size without disappearing. This is a useful diagnostic and the paper's main empirical contribution. OPRM is simple, training-free, parallelizable, and clearly better than full-context decoding on most tested benchmarks, so the method itself is valuable even if the selection mechanism is not fully understood. The paper releases code and detailed experimental configurations, which strengthens reproducibility. However, the most provocative interpretation, that recurrent LLMs do not exploit cross-chunk dependencies, is not established by the current experiments, and the headline gains depend on per-benchmark selection of the chunk size. With additional validation of the selection mechanism and a stricter evaluation protocol, this would be a solid contribution to the long-context efficiency literature.

major comments (3)
  1. [§4.2, Algorithm 1, Table 5] The central claim that OPRM 'identifies and processes only the most relevant portion' is not directly validated. No experiment measures whether the min-entropy criterion selects the chunk containing the gold evidence; the evidence is indirect, consisting of final task accuracy plus one qualitative example in Figure 7. This matters because Table 5 shows that random chunk selection already outperforms full-context decoding in the 4K-8K and 8K+ groups (23.02 vs 21.18 and 27.62 vs 18.4), so a large part of the gain is attributable to overflow avoidance rather than to localization. Min entropy is stronger than random in those groups (37.1 and 35.18), which shows the selection signal has value, but the aggregate numbers remain consistent with the weaker hypothesis that any short chunk is better than an overflowing full context. Please report selection accuracy against a gold-chunk oracle (e.g., the chunk containing the supporting evidence in HotPotQA or MuSiQue), entropy calibration statistics, and variance of the selection step; this is needed to support the localization claim and the related claim about cross-chunk dependencies.
  2. [Appendix A.3 and Figure 1 (right)] The headline improvements are computed with per-benchmark oracle selection of the chunk size: Appendix A.3 states that L∈{1000,2000,3000} is used and 'we select the best scoring chunk size.' Since the abstract and Figure 1 (right) quote improvements of 14-51%, these numbers are upper bounds over L, not the performance of a fixed configuration. Table 7 shows that scores are fairly robust across L for many LongBench tasks, but the selection is still made on the test set. Please report results for a single fixed L, or for a validation-based selection rule, for all headline aggregates, and give the spread of the aggregate scores across L rather than only the per-task sigma/mu values.
  3. [§1/§7 and Appendix B.4, Table 11] The abstract and Section 7 say the results 'raise questions about whether recurrent models genuinely exploit long-range dependencies,' but the evidence does not discriminate between 'recurrent models cannot use cross-chunk information' and 'full-context decoding is impaired by overflow, so any chunking helps.' The paper's own multi-chunk ablation (Table 11) shows that combining the top-k chunks (CC) outperforms OPRM at 0-4K (40.27 vs 37.41) and is comparable at 8K+ (35.98 vs 36.25), with a higher overall average (37.27 vs 36.05), and no task is annotated for whether it actually requires cross-chunk evidence. Please either soften the conclusion to an explicitly open question supported by a task-level analysis, or add a controlled experiment in which the answer requires evidence from two different chunks and compare OPRM, CC, and full-context decoding under matched token budgets.
minor comments (5)
  1. [Section 3 and Appendix A.1] The zero-shot AR curve is averaged over only 5 sampled contexts per number of facts; please add error bars or report additional seeds, since this curve is the primary motivation for the method.
  2. [Appendix A.3 and Figure 1 (right)] The experimental configuration is heterogeneous across models: the IDK filter is applied to Falcon-Mamba-Inst-7B, RecurrentGemma-IT-9B, and RWKV6-Finch-7B but not to Falcon3-Mamba-Inst-7B, and summarization tasks use a different decoding mode; this should be stated in the main text so the gains are not read as a single uniform intervention.
  3. [Tables 1 and 2] The 'new SOTA for this size class' claim is made on the full LongBench v2 dataset, while Table 1 compares to baselines only on the <150K-token subset; please clarify that the full-dataset improvement over the unmodified Falcon3-Mamba-Inst-7B is not directly measured because the baseline runs out of memory, and that the comparison to Transformer models therefore uses different evaluation subsets across the two tables.
  4. [Section 5.4, Table 8] The sentence 'OPRM outperforms vanilla inference in speed' is true only for contexts of 8K and above in Table 8; at 2K the OPRM time is 2.2s vs 2.0s and at 4K the times are equal, so the speed claim should be qualified as applying to longer contexts.
  5. [Appendix A.5] There is a typo in 'During evaluation the same setting is usedm' which should read 'used'; please fix it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the overflow diagnostic and the OPRM heuristic are externally validated, and the paper's self-citations are not load-bearing.

full rationale

The paper's derivation chain is not circular. The overflow phenomenon is diagnosed with associative-recall experiments that are external to the proposed method: the AR curves measure a model's retrieval accuracy as a function of the number of facts, and the controlled setup trains independent 2-layer Mamba models. OPRM is then introduced as an inference-time heuristic that chunks the context, speculatively prefills each chunk, and decodes from the chunk with minimum output entropy. The selection criterion is an assumption about the informativeness of the model's own output distribution, stated explicitly in Appendix C, and its effectiveness is tested on held-out benchmarks rather than derived from the target result. The benchmark gains are therefore empirical evidence, not consequences of the definitions. The paper does cite prior work by overlapping authors (Ben-Kish et al. 2025) for length-generalization findings and for experimental setups, but these citations are used as baselines or as motivation, and the present paper's own AR experiments and ablations independently support the central claims. No uniqueness theorem is imported from the authors' prior work, no parameter is fitted to the target benchmark and then renamed a prediction, and the method is not a renaming of a known result. Concerns about whether min-entropy selection truly identifies the most relevant chunk, or whether the gains are partly due to context shortening, are legitimate experimental and interpretability questions, but they are not circularity: the paper does not define its conclusions in terms of its assumptions or fit its central claim to its own outputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical observations and a few heuristic assumptions. The main free parameters are tuned on the test benchmarks, which weakens the generality of the reported numbers. No new theoretical entities are introduced. The axioms are reasonable for an empirical paper, but the locality assumption and the confidence-as-relevance assumption are the most fragile.

free parameters (3)
  • chunk size L = 1000, 2000, or 3000 (selected per benchmark as best)
    Appendix A.3: 'We apply OPRM inference with L in {1000, 2000, 3000} and select the best scoring chunk size.' This is a test-set-tuned hyperparameter that directly affects all reported benchmark improvements.
  • IDK filter usage = Enabled for all models except Falcon3-Mamba-Inst-7B
    Appendix A.3: 'We apply the IDK filter for all models, except for Falcon3-Mamba-Inst-7B.' This per-model choice is made after ablations (Table 6) and is post-hoc.
  • Summarization mode = Concatenate all chunk predictions for summarization tasks
    Appendix A.3: 'For the summarization tasks, we found that decoding all chunks in parallel and then concatenating the predictions into a single summary yields better performance.' This is a per-task adaptation.
assumptions (4)
  • domain assumption Natural language has strong local structure, so relevant information fits within a single chunk.
    Appendix C states this as the basis for the chunking design. If false for a task, OPRM cannot recover needed distant information.
  • domain assumption Minimum entropy of the output distribution is a reliable indicator of chunk relevance.
    Section 4.2 uses min entropy as the default selection criterion. The paper provides an ablation (Table 5) showing it outperforms alternatives, but the assumption is heuristic.
  • domain assumption The associative recall task is a valid proxy for real-world long-context memory behavior.
    Section 3 uses AR to motivate the overflow hypothesis. The controlled experiments support the pattern, but transfer to natural language is assumed.
  • domain assumption The IDK token instruction makes the model reliably indicate when a chunk lacks the answer.
    Section 4.2 introduces the IDK filter. Its effectiveness varies by model (Table 6), and for Falcon3-Mamba it is disabled, showing the assumption is not universally satisfied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Overflow Prevention Enhances Long-Context Recurrent LLMs." pith.science (2026). https://pith.science/paper/FX2I2EEV

@misc{pith2026250507793,
  author       = {Pith},
  title        = {Pith review of: Overflow Prevention Enhances Long-Context Recurrent LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FX2I2EEV}},
  note         = {Machine review of arXiv:2505.07793}
}
read the original abstract

A recent trend in LLMs is developing recurrent sub-quadratic models that improve long-context processing efficiency. We investigate leading large long-context models, focusing on how their fixed-size recurrent memory affects their performance. Our experiments reveal that, even when these models are trained for extended contexts, their use of long contexts remains underutilized. Specifically, we demonstrate that a chunk-based inference procedure, which identifies and processes only the most relevant portion of the input can mitigate recurrent memory failures and be effective for many long-context tasks: On LongBench, our method improves the overall performance of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%, RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Surprisingly, this simple approach also leads to state-of-the-art results in the challenging LongBench v2 benchmark, showing competitive performance with equivalent size Transformers. Furthermore, our findings raise questions about whether recurrent models genuinely exploit long-range dependencies, as our single-chunk strategy delivers stronger performance - even in tasks that presumably require cross-context relations.

Figures

Figures reproduced from arXiv: 2505.07793 by the authors.

Figure 1
Figure 1. Limited recurrent memory capacity limits leading long-context LLMs. (Left) We quantify this behavior by measuring zero-shot associative recall curves: The x-axis represents the number of facts (key-value pairs) in the context, while the y-axis shows the retrieval accuracy of the correct value from the context. To mitigate the memory problem, we propose OPRM, a chunk-based inference strategy that does not force the m… view at source ↗
Figure 2
Figure 2. Memory overflows in a controlled setup. (Left) AR curves of 2-layer models with different hidden state sizes. d is the channels dimension and dstate is the state size. The x-axis represents the number of facts in the context, and the y-axis shows the retrieval accuracy of correct values from the context. (Right) Memory capacity as a function of channel and state dimensions. Each point shows the ratio between the cap… view at source ↗
Figure 3
Figure 3. Visualization of OPRM: Given a prompt structured as Prefix (P), Context (C), and Suffix (S), we first split the context into chunks (C1, · · · , Cb ). All chunks are wrapped with the Prefix (P) and Suffix (S) and are processed independently in a speculative manner (in parallel). During decoding, tokens are predicted auto-regressively, conditioned exclusively on the selected chunk. (ii) Probability-based Criteria: Fo… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Context Extension - Needle in a Haystack. The x-axis is the context length in tokens, and the y-axis is the depth in which the passkey is hidden inside the context. The color indicates the success rate of the needle retrieval. We show that overflow-prevention mechanism…
Figure 5
Figure 5. Figure 5: Sensitivity of recurrent memory capacity to input length. We repeat the zero-shot AR experiment (Section 3) with varying input sequence lengths L ∈ {300, 600, 1200, 2400, 4800}. We find that the model is not sensitive to the input sequence length, but rather to the amo…
Figure 6
Figure 6. Figure 6: Positional sensitivity to recurrent memory overflows. We analyze the position of successfully retrieved key-value (KV) pairs from the zero-shot AR experiment (Section 3). Each plot shows a normalized histogram of the locations of successfully retrieved KV pairs (facts)…
Figure 7
Figure 7. Figure 7: IDK Filter ablation - qualitative example We provide 3 samples from the Hot￾PotQA benchmark (LongBench e). In each one of the samples we present the question, ground truth answer, and the response of three Recurrent-Gemma-IT-9B models: the base￾line, + OPRM, and + OPRM…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 20 canonical work pages

  1. [1]

    Mechanistic evaluation of transformers and state space models

    Aryaman Arora, Neil Rathi, Nikil Roashan Selvam, R \'o bert Csord \'a s, Dan Jurafsky, and Christopher Potts. Mechanistic evaluation of transformers and state space models. arXiv preprint arXiv:2505.15105, 2025

    arXiv 2025

  2. [2]

    Zoology: Measuring and improving recall in efficient language models, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

    arXiv 2023

  3. [3]

    Simple linear attention language models balance the recall-throughput tradeoff, 2024

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2024. URL https://arxiv.org/abs/2402.18668

    arXiv 2024

  4. [4]

    Mambaextend: A training-free approach to improve long context extension of mamba

    Seyedarmin Azizi, Souvik Kundu, Mohammad Erfan Sadeghi, and Massoud Pedram. Mambaextend: A training-free approach to improve long context extension of mamba. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  5. [5]

    Neural machine translation by jointly learning to align and translate, 2016

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. URL https://arxiv.org/abs/1409.0473

    arXiv 2016

  6. [6]

    Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL https://arxiv.org/abs/2308.14508

    arXiv 2024

  7. [7]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, 2025. URL https://arxiv.org/abs/2412.15204

    arXiv 2025

  8. [8]

    xlstm: Extended long short-term memory, 2024

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL https://arxiv.org/abs/2405.04517

    arXiv 2024

Show all 75 references
  1. [9]

    o ppel, Phillip Lippe, Richard Kurle, Patrick M Blies, G \

    Maximilian Beck, Korbinian P \"o ppel, Phillip Lippe, Richard Kurle, Patrick M Blies, G \"u nter Klambauer, Sebastian B \"o ck, and Sepp Hochreiter. xlstm 7b: A recurrent llm for fast and efficient inference. arXiv preprint arXiv:2503.13427, 2025

    2025 arXiv

  2. [10]

    Graph mamba: Towards learning on graphs with state space models

    Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp.\ 119--130, 2024

    2024

  3. [11]

    Decimamba: Exploring the length extrapolation potential of mamba, 2025

    Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein, Nadav Cohen, Amir Globerson, Lior Wolf, and Raja Giryes. Decimamba: Exploring the length extrapolation potential of mamba, 2025. URL https://arxiv.org/abs/2406.14528

    2025 arXiv

  4. [12]

    Recurrentgemma: Moving past transformers for efficient open language models

    Aleksandar Botev, Soham De, Samuel L Smith, Anushan Fernando, George-Cristian Muraru, Ruba Haroun, Leonard Berrada, Razvan Pascanu, Pier Giuseppe Sessa, Robert Dadashi, et al. Recurrentgemma: Moving past transformers for efficient open language models. arXiv preprint arXiv:240...

    2024 arXiv

  5. [13]

    Longlora: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023

    2023 arXiv

  6. [14]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024. URL https://arxiv.org/abs/2405.21060

    2024 arXiv

  7. [15]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George-Cristian Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. CoRR, 2024

    2024

  8. [16]

    Hymba: A hybrid-head architecture for small language models, 2024

    Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models, 2024. URL h...

    2024 arXiv

  9. [17]

    Vision- RWKV : Efficient and scalable visual perception with RWKV -like architectures

    Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision- RWKV : Efficient and scalable visual perception with RWKV -like architectures. In The Thirteenth International Conference on Learning Representations,...

    2025

  10. [18]

    Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models

    Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, and Junshi Huang. Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models. arXiv preprint arXiv:2404.04478, 2024

    2024 arXiv

  11. [19]

    Splade v2: Sparse lexical and expansion model for information retrieval, 2021

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval, 2021. URL https://arxiv.org/abs/2109.10086

    2021 arXiv

  12. [20]

    Mamba: Linear-time sequence modeling with selective state spaces, 2024

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024. URL https://arxiv.org/abs/2312.00752

    2024 arXiv

  13. [21]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R \'e . Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021 a

    2021 arXiv

  14. [22]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R \'e . Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34: 0 572--585, 2021 b

    2021

  15. [23]

    Realm: Retrieval-augmented language model pre-training, 2020

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training, 2020. URL https://arxiv.org/abs/2002.08909

    2020 arXiv

  16. [24]

    Mambavision: A hybrid mamba-transformer vision backbone

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. arXiv preprint arXiv:2407.08083, 2024

    2024 arXiv

  17. [25]

    Decision mamba: Reinforcement learning via hybrid selective sequence modeling

    Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision mamba: Reinforcement learning via hybrid selective sequence modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://op...

    2024

  18. [26]

    Unsupervised dense information retrieval with contrastive learning, 2022

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2022. URL https://arxiv.org/abs/2112.09118

    2022 arXiv

  19. [27]

    How can we know when language models know? on the calibration of language models for question answering

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9: 0 962--977, 2021

    2021

  20. [28]

    Dense passage retrieval for open-domain question answering, 2020

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020. URL https://arxiv.org/abs/2004.04906

    2020 arXiv

  21. [29]

    The impact of positional encoding on length generalization in transformers

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  22. [30]

    Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, ...

    2025

  23. [31]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  24. [32]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing...

    2020

  25. [33]

    Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach, 2024

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach, 2024. URL https://arxiv.org/abs/2407.16833

    2024 arXiv

  26. [34]

    How to train your dragon: Diverse augmentation towards generalizable dense retrieval, 2023

    Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval, 2023. URL https://arxiv.org/abs/2302.07452

    2023 arXiv

  27. [35]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 a

    2024

  28. [36]

    Vmamba: Visual state space model

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024 b

    2024 arXiv

  29. [37]

    Focus your attention (with adaptive iir filters)

    Shahar Lutati, Itamar Zimerman, and Lior Wolf. Focus your attention (with adaptive iir filters). arXiv preprint arXiv:2305.14952, 2023

    2023 arXiv

  30. [38]

    Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl

    Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, and Liqiang Nie. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. Advances in Neural Information Processing Systems, 37: 0 22827--22849, 2024

    2024

  31. [39]

    Uncertainty estimation in autoregressive structured prediction

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations

  32. [40]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/abs/1609.07843

    2016 arXiv

  33. [41]

    Exploring the capability of mamba in speech applications

    Koichi Miyazaki, Yoshiki Masuyama, and Masato Murata. Exploring the capability of mamba in speech applications. In Proc. Interspeech 2024, pp.\ 237--241, 2024

    2024

  34. [42]

    Landmark attention: Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023

    2023 arXiv

  35. [43]

    Webgpt: Browser-assisted question-answering with human feedback, 2022

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webg...

    2022 arXiv

  36. [44]

    Revisiting associative recall in modern recurrent models

    Destiny Okpekpe and Antonio Orvieto. Revisiting associative recall in modern recurrent models. In First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models, 2025

    2025

  37. [45]

    In-context learning and induction heads, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022 arXiv

  38. [46]

    Resurrecting recurrent neural networks for long sequences, 2023

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences, 2023. URL https://arxiv.org/abs/2303.06349

    2023 arXiv

  39. [47]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  40. [48]

    Eagle and finch: RWKV with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Gregory Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Kranthi Kiran GV, Haowen Hou, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Ruic...

    2024

  41. [49]

    Mechanistic design and scaling of hybrid architectures

    Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Bj \"o rn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher R \'e , et al. Mechanistic design and scaling of hybrid architectures. In Proceedings of the 41st International Conference...

    2024

  42. [50]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021

    2021

  43. [51]

    Large language models are effective text rankers with pairwise ranking prompting, 2024 a

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting, 2024 a . URL https://arxiv.org/abs/2306.17563

    2024 arXiv

  44. [52]

    Hgrn2: Gated linear rnns with state expansion, 2024 b

    Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion, 2024 b . URL https://arxiv.org/abs/2404.07904

    2024 arXiv

  45. [53]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  46. [54]

    Know what you don't know: Unanswerable questions for squad, 2018

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, 2018

    2018

  47. [55]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024. URL https://arxiv.org/abs/2406.07522

    2024 arXiv

  48. [56]

    A study of branch prediction strategies

    James E Smith. A study of branch prediction strategies. In 25 years of the international symposia on Computer architecture (selected papers), pp.\ 202--215, 1998

    1998

  49. [57]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  50. [58]

    Learning to (learn at test time): Rnns with expressive hidden states, 2025

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rnns with expressive hidden states, 2025. URL https://arxiv.org/abs/2407.04620

    2025 arXiv

  51. [59]

    The falcon 3 family of open models, December 2024

    Falcon-LLM Team. The falcon 3 family of open models, December 2024

    2024

  52. [60]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024 a

    2024 arXiv

  53. [61]

    Jamba-1.5: Hybrid transformer-mamba models at scale, 2024 b

    Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal...

    2024 arXiv

  54. [62]

    Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021. URL https://arxiv.org/abs/2104.08663

    2021 arXiv

  55. [63]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  56. [64]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    2024 arXiv

  57. [65]

    Mambabyte: Token-free selective state space model

    Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, and Alexander M Rush. Mambabyte: Token-free selective state space model. In First Conference on Language Modeling

  58. [66]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Findings of the Association for Computational Linguistics ACL 2...

    2024

  59. [67]

    Retrieval meets long context large language models, 2024

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models, 2024. URL https://arxiv.org/abs/2310.03025

    2024 arXiv

  60. [68]

    Gated delta networks: Improving mamba2 with delta rule, 2025

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2025. URL https://arxiv.org/abs/2412.06464

    2025 arXiv

  61. [69]

    Longmamba: Enhancing mamba's long-context capabilities via training-free receptive field enlargement

    Zhifan Ye, Kejing Xia, Yonggan Fu, Xin Dong, Jihoon Hong, Xiangchi Yuan, Shizhe Diao, Jan Kautz, Pavlo Molchanov, and Yingyan Celine Lin. Longmamba: Enhancing mamba's long-context capabilities via training-free receptive field enlargement. In The Thirteenth International Confe...

    2025

  62. [70]

    Useful confidence measures: Beyond the max score

    Gal Yona, Amir Feder, and Itay Laish. Useful confidence measures: Beyond the max score. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications

    2022

  63. [71]

    bench: Extending long context evaluation beyond 100k tokens, 2024

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. bench: Extending long context evaluation beyond 100k tokens, 2024. URL https://arxiv.org/abs/2402.13718

    2024 arXiv

  64. [72]

    Llm mapreduce: Simplified long-sequence processing using large language models, 2024

    Zihan Zhou, Chong Li, Xinyi Chen, Shuo Wang, Yu Chao, Zhili Li, Haoyu Wang, Rongqiao An, Qi Shi, Zhixing Tan, Xu Han, Xiaodong Shi, Zhiyuan Liu, and Maosong Sun. Llm mapreduce: Simplified long-sequence processing using large language models, 2024. URL https://arxiv.org/abs/2410.09342

    2024 arXiv

  65. [73]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024

    2024 arXiv

  66. [74]

    Falcon mamba: The first competitive attention-free 7b language model

    Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2410.05355, 2024

    2024 arXiv

  67. [75]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.