Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Smooth Reading closes the long-context gap between recurrent and self-attention LLMs, with a 3B recurrent model beating its 3B attention counterpart by 3.61% on LongBench.

desk verdict Smooth Reading is a plausible, well-controlled chunkwise inference recipe for recurrent LLMs, but the headline gap-closing claim is only shown on the two benchmarks used to build the training data. read the letter →

arxiv 2507.19353 v1 pith:YKMAY5N6 submitted 2025-07-25 cs.CL cs.AI

Kai Liu , Zhan Su , Peijie Dong , Fengran Mo , Jianfei Gao , ShaoTing Zhang , Kai Chen This is my paper · ORCID
classification cs.CLcs.AI
keywords SmoothReadingrecurrentLLMlong-contextinferencesliding-windowattentionlinearlengthextrapolationcontextualsummarizationneedle-in-a-haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Recurrent LLMs were thought to lag behind self-attention LLMs on long-context tasks because their memory is small and fixed. This paper argues the real problem is the inference method: feeding the whole context in one forward pass overwhelms that memory. It proposes Smooth Reading, which splits the context into chunks, processes them one by one while keeping the recurrent hidden state alive, and after each chunk writes a short contextual summary (target, clues, reason, continue/stop) that the hidden memory then carries forward. The paper reports that a 3B sliding-window model with a 4k window, trained on a 48,856-sample curated dataset, surpasses a 3B self-attention model on LongBench by 3.61% and reaches 99.6% on a four-needle retrieval test at 256k tokens, while keeping linear-time training and inference. If correct, this gives recurrent architectures a practical path to long contexts without quadratically scaling attention.

What carries the argument

The load-bearing object is Smooth Reading itself, formalized as $I_i, M_i = f_R(C_i, M_{i-1})$: the recurrent model processes chunk $C_i$ with the previous hidden memory $M_{i-1}$, producing an updated hidden memory and a contextual summary. The summary's four fields—Target, Clues, Reason, Continue—tell the model what to attend to, what to remember, why, and whether to keep reading via a special <CONTINUE> or <STOP> token. Because the hidden memory carries the summary forward, the model never re-reads accumulated text, so cost stays linear: the paper derives $T = (1 + g\beta/c) \times l \times p_r$ for total inference time. The ablations show the method's operating constraint: chunk size must be smaller than window size, with a 1:2 chunk-to-window ratio giving the best accuracy-efficiency trade-off.

What would settle it

Run SWA-3B-4k-SR on a 256k-token task whose answer requires every occurrence of a rare pattern, such as all numbers greater than 1000 or every mention of a specific name spread across many chunks; if accuracy falls well below the 99% four-needle retrieval level, the fixed-size summary is discarding the needed detail. The paper's own ablation—accuracy collapses when chunk size exceeds window size—points to exactly where that limit lives.

Watch

Extended reading notes

Core claim

The central claim is that a recurrent LLM equipped with Smooth Reading can match or beat a self-attention LLM on long-context benchmarks while retaining linear complexity. The method works chunk-wise: each chunk is read through the recurrent model, the updated hidden memory $M_i$ replaces the need to re-feed any external summary, and the model emits a structured contextual summary with four components—Target, Clues, Reason, and a <CONTINUE>/<STOP> decision—so the model knows what to keep and when to stop. The paper's headline result is SWA-3B-4k-SR improving from 5.68% below to 3.61% above Qwen-2.5-3B-OS on LongBench (50.99 vs 47.38), and reaching 99.93% average on NIAH up to 32k and 99.80% average from 64k to 256k with four needles, despite training on only 32k contexts. It also reports a 3x training speedup and a 2x inference speedup at 64k context compared with the self-attention baseline, and, with early stopping, a 4x inference speedup. The authors state this is the first work to bring recurrent LLMs to parity with self-attention LLMs on long-context tasks.

Load-bearing premise

The load-bearing premise is that a fixed-size hidden state can compress an ever-growing context into a faithful summary without losing the exact details an answer needs; the paper tests this only on NIAH and LongBench, not on tasks where the answer depends on precise details scattered across many chunks.

Editorial extensions

If this is right

  • If the claims hold, any recurrent LLM—sliding-window attention, RWKV, or linear-attention variants—can be turned into a competitive long-context model by changing only the inference loop and fine-tuning on summary-style data, not the architecture.
  • The 4k-window SWA-3B-4k-SR model's 99.6% accuracy on a four-needle NIAH task at 256k tokens, trained at 32k, implies recurrent models can extrapolate far beyond their training length when information is funneled through summaries.
  • Because Smooth Reading retains hidden memory, it beats Unsmooth Reading (re-feeding compressed summaries with reset memory) by 2.62% on LongBench and is far more stable on NIAH, suggesting the hidden state is a more faithful carrier than explicit re-input.
  • The method's linear complexity and early stopping make it attractive for deployment: at 64k context, training is 3x faster and inference 2x (4x with early stopping) than a self-attention baseline.
  • The method is orthogonal to memory-expanding architectures, so combining Smooth Reading with bigger or more expressive recurrent states should stack further gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not test: tasks where the answer requires exact aggregation over many chunks (e.g., 'list every transaction above $100' across a 256k document) would stress whether the fixed-size hidden summary retains precise details, not just the presence of one needle.
  • Since the training data is generated by a self-attention teacher performing Unsmooth Reading, the student's ceiling is the teacher's summary quality; training the recurrent model on its own rollouts or with reinforcement learning on the <CONTINUE>/<STOP> decision could push beyond the teacher and reduce the summary overhead.
  • The chunk-size < window-size constraint suggests the effective context is governed by the quality of the iterative summary rather than the physical window; this recasts long-context understanding as a recursive compression problem, connecting to state-space and memory-augmented models.
  • If early stopping is reliable, Smooth Reading could yield adaptive compute that scales with query difficulty, which has implications for serving cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Smooth Reading, a chunk-wise inference procedure for recurrent LLMs that maintains hidden memory across chunks and generates a structured contextual summary at each step. The authors construct an SFT dataset from LongBench and NIAH raw data using DeepSeek-V3 or rule-based teachers, fine-tune Qwen2.5-derived sliding-window models and RWKV-7 in three formats (One-Step, Unsmooth Reading, Smooth Reading), and report accuracy on LongBench and NIAH plus training/inference time. The headline claims are that SWA-3B-4k-SR beats Qwen-2.5-3B-OS by 3.61% on LongBench, achieves 99.8% average accuracy on a four-needle NIAH task at 256k tokens, and is about 3x faster to train and 2x faster to infer at 64k context than the self-attention baseline. The paper also includes ablations on chunk and window sizes, comparisons with RAG-style baselines, and 7B-model experiments.

Significance. The idea is timely and potentially useful: showing that an inference protocol, rather than an architectural change, can narrow the recurrent/self-attention gap would be a meaningful contribution. The paper's strengths include the fair within-format comparison (all models are fine-tuned on the same raw data in different formats), a clear formulation of Smooth Reading, a useful back-of-envelope efficiency calculation, detailed appendices, a frank limitations section, and a stated plan to release code and data. The NIAH length-extrapolation results, if reproducible under a clearly disclosed protocol, are striking. However, the significance is currently limited by the in-distribution evaluation and by presentation issues around the modified NIAH task and the absence of statistical reliability information.

major comments (3)
  1. [Section 3.1.2, Table 5, and A.6] The supervised fine-tuning dataset is built from the same two benchmark families used for evaluation, and A.6 explicitly concedes that the collected dataset is "not generalizable across a wide range of tasks." All compared models, including the self-attention baselines, are fine-tuned on this in-distribution data, so the reported LongBench gains (e.g., 50.99 vs 47.38 in Table 2) and the near-perfect NIAH results reflect the models' ability to reproduce the benchmark-specific teacher protocol (contextual summaries, <CONTINUE>/<STOP> decisions, and clue formats) rather than a demonstrated general closing of the recurrent/self-attention gap. This is load-bearing because the abstract and Section 4.2 make a general claim about long-context tasks. A held-out evaluation on a benchmark or task family not used in Table 5, or a claim explicitly scoped to the two benchmark families, is required before the headline can be accepted.
  2. [Section 4.3 and Table 3, with A.1.4] The near-perfect NIAH results are obtained on a modified four-needle task, disclosed only in A.1.4 (“We use only one needle in our experiments by default, but four needles in Table 3”), while the main text and Table 3 present the results simply as NIAH. Because standard NIAH protocols use a single needle and results are sensitive to needle depth and position, the 256k extrapolation claim is not comparable to published NIAH numbers. Please report one-needle and four-needle results separately with needle placement details, or clearly label Table 3 and all related text as a four-needle variant.
  3. [Tables 2, 3, and 6; A.1.4; Section 4.5] All benchmark tables report single runs without error bars or significance testing, and A.1.4 states that chunk sizes are set differently per model and task (1024 vs 2048 vs 512 vs 256). Given that Section 4.5 and Table 4 show accuracy ranging from 0.0% to 100.0% depending on the chunk/window combination, the reported advantages (e.g., LongBench 50.99 vs 48.37 vs 47.38; SWA-7B-SR within 0.74% of Qwen-7B-OS in Table 6) could be driven by hyperparameter or seed variance rather than by the method itself. Please provide multiple seeds or error bars and a sensitivity analysis for the chunk/window choice on at least one main benchmark, or temper the comparative claims accordingly.
minor comments (5)
  1. [Table 3 and Section 4.3] The model names are inconsistent: Table 3 uses "RWKV-3B-OS" and "RWKV-3B-SR" while the main text and Table 2 use "RWKV-7-3B-OS" and "RWKV-7-3B-SR"; please use one consistent naming convention throughout.
  2. [Section 3.2] The linearity conclusion depends on the assumption that the per-step generation length g is constant; the paper should state explicitly that the complexity is O(L + n·g), which is linear only if g is O(1), and should discuss tasks (e.g., summarization of very long documents) where g may grow with context length.
  3. [Section 4.4 and Figure 4] The efficiency comparison should report the exact experimental setup, including GPU type, batch size, the number of generated tokens per step, and the early-stopping policy, so that the claimed 2x/3x speedups are reproducible.
  4. [A.5 and Table 8] The RAG/RAG+/CompACT baselines use Qwen-2.5-3B-Instruct and, for RAG, external Wikipedia retrieval, while Smooth Reading uses the fine-tuned SWA-4k model; this mismatch should be stated in the main comparison or the baselines should be re-run on the same base model and input format.
  5. [A.1.1] The token-count approximation ntoken ≈ Int(1.5 × n_words) should state the tokenizer and language for which this ratio holds, since errors in this estimate can cause chunks to exceed the intended maximum size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation is self-contained, and the in-distribution evaluation is an external-validity limitation, not a circular step.

full rationale

The paper's central derivation is the Smooth Reading inference procedure and its efficiency analysis. The efficiency formula in Section 3.2 is a standard complexity calculation with explicitly stated assumptions (constant per-token time, constant summary length), and it is not fitted to the results. No parameter is fit to a subset of data and then reported as a prediction of a closely related quantity. The teacher model used to build the SFT dataset is external (DeepSeek-V3 or rule-based models, Section 3.1.2), and there is no load-bearing self-citation chain: the references to RWKV-7, sliding-window attention, and other recurrent architectures are external prior work, not claims whose validity depends on this paper's authors. The possible concern is that the SFT training data is constructed from the same two benchmark families (NIAH and LongBench) used for evaluation, so the reported LongBench and NIAH gains are in-distribution; the paper itself concedes in A.6 that the collected dataset is not generalizable across a wide range of tasks. That is a limitation on the scope of the empirical claim, but it is not circularity in the derivation: the models are trained on teacher-generated summaries and evaluated on held-out examples from the same benchmark families, which is standard supervised evaluation rather than a prediction that reduces to an input by construction. The ablation finding that chunk size should be smaller than window size (Section 4.5) is a consequence of the sliding-window architecture definition, but it is presented as a design guideline and is not used to derive the headline performance claim. No equation in the paper equates a predicted quantity with a fitted input, and no uniqueness theorem or prior self-citation is invoked to force the method. The verdict is therefore no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims are empirical; the only analytical derivation is the efficiency model in Section 3.2, which rests on standard complexity assumptions. The teacher-student dataset construction is an external dependence whose quality is not independently verified.

free parameters (2)
  • Chunk size = SWA: 1024 (LongBench), 2048 (NIAH); RWKV: 512 (LongBench), 256 (NIAH)
    Chosen per model and task; Table 4 ablation shows accuracy and inference time depend strongly on chunk size, so headline results apply only to these hand-selected values.
  • Window size = 4096 for SWA models in main experiments; 8192 suggested in ablation for efficiency
    Sliding-window width is a model configuration choice; the main results use a 4k window, and performance is sensitive to this memory size.
assumptions (4)
  • domain assumption Per-token prefill and decode times are constant and independent of context length.
    Section 3.2 uses this to derive T = (1 + g*beta/c)*l*p_r; true asymptotically for recurrent architectures but not exact on real hardware.
  • domain assumption The per-chunk summary decoding length g is constant and independent of context length.
    Section 3.2 states this assumption explicitly and uses it to conclude linear complexity.
  • domain assumption A teacher LLM (DeepSeek-V3 or rule-based) can produce contextual summaries accurate enough to serve as training targets.
    Dataset construction in Section 3.1.2 relies entirely on teacher outputs; no independent quality check of summaries is reported beyond task metrics.
  • domain assumption Sliding-window LLMs have strong length extrapolation ability.
    Appendix A.2 empirically supports this for the tested model; the 256k extrapolation result depends on this property persisting under Smooth Reading training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks." pith.science (2026). https://pith.science/paper/YKMAY5N6

@misc{pith2026250719353,
  author       = {Pith},
  title        = {Pith review of: Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKMAY5N6}},
  note         = {Machine review of arXiv:2507.19353}
}
read the original abstract

Recently, recurrent large language models (Recurrent LLMs) with linear computational complexity have re-emerged as efficient alternatives to self-attention-based LLMs (Self-Attention LLMs), which have quadratic complexity. However, Recurrent LLMs often underperform on long-context tasks due to their limited fixed-size memory. Previous research has primarily focused on enhancing the memory capacity of Recurrent LLMs through architectural innovations, but these approaches have not yet enabled Recurrent LLMs to match the performance of Self-Attention LLMs on long-context tasks. We argue that this limitation arises because processing the entire context at once is not well-suited for Recurrent LLMs. In this paper, we propose Smooth Reading, a chunk-wise inference method inspired by human reading strategies. Smooth Reading processes context in chunks and iteratively summarizes the contextual information, thereby reducing memory demands and making the approach more compatible with Recurrent LLMs. Our experimental results show that this method substantially narrows the performance gap between Recurrent and Self-Attention LLMs on long-context tasks, while preserving the efficiency advantages of Recurrent LLMs. Our Smooth Reading boosts SWA-3B-4k (a Recurrent LLM) from 5.68% lower to 3.61% higher performance than Self-Attention LLMs on LongBench. Besides, our method maintains the high efficiency, training 3x faster and inferring 2x faster at 64k context compared to Self-Attention LLMs. To our knowledge, this is the first work to achieve comparable performance using Recurrent LLMs compared with Self-Attention LLMs on long-context tasks. We hope our method will inspire future research in this area. To facilitate further progress, we will release code and dataset.

Figures

Figures reproduced from arXiv: 2507.19353 by the authors.

Figure 1
Figure 1. Comparison of Architectures: (a) Self-Attention LLM: A new token is ap￾pended to the memory. (b) Sliding-Window LLM: A new token is appended to the mem￾ory, and the earliest token is evicted. (c) RWKV-7: “Transform”, “k”, and “v” are gen￾erated from the new token and used to update memory. The memory size of Self-Attention LLM is virtually unlimited, while the other two are fixed-size [PITH_FULL_IMAGE:figures/full_… view at source ↗
Figure 3
Figure 3. Illustration of Smooth Reading. White boxes indicate the information gathered by the model and the decision points for continuing reading. Blue boxes represent chunks of the query context [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Efficiency Comparison across Context Lengths (x-axis) vs. Inference Time (y-axis). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Inference Time Comparison with a Fixed Ratio of Chunk Size to Window Size on NIAH. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Sliding-Window LLM and RWKV-7 on NIAH. The label for each line is [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning

    cs.CV 2025-09 conditional novelty 5.0 of 10

    CARVE contrasts attention maps from a general prompt and a specific question to mask out visual noise, then re-asks the question on the cropped and enlarged image, improving VQA accuracy by up to 75% on some benchmarks.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  3. [3]

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024

    2024

  4. [4]

    Beltagy, M

    I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer, 2020. URLhttps://arxiv.org/abs/2004.05150

    arXiv 2020

  5. [5]

    Y . Chen, X. Zhang, S. Hu, X. Han, Z. Liu, and M. Sun. Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling

    work page

  6. [6]

    Contributors

    L. Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https: //github.com/InternLM/lmdeploy, 2023

    work page 2023

  7. [7]

    Contributors

    X. Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/ InternLM/xtuner, 2023

    work page 2023

  8. [8]

    Dao and A

    T. Dao and A. Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

    work page

Show all 47 references
  1. [9]

    Dasigi, K

    P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner. A dataset of information- seeking questions and answers anchored in research papers. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2021

  2. [10]

    DeepSeek-AI and D. G. et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2501.12948

    2025 arXiv

  3. [11]

    J. Du, W. Sun, D. Lan, J. Hu, and Y . Cheng. Mom: Linear sequence modeling with mixture-of- memories, 2025. URLhttps://arxiv.org/abs/2502.13685

    2025

  4. [12]

    D.-A. et al. Deepseek-v3 technical report, 2024

    2024

  5. [13]

    A. R. Fabbri, I. Li, T. She, S. Li, and D. Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, 2019. 10

    2019

  6. [14]

    Gliwa, I

    B. Gliwa, I. Mochol, M. Biesek, and A. Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. InProceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China, Nov. 2019. Association for Computational Linguisti...

    2019 doi

  7. [15]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces

  8. [16]

    D. Guo, C. Xu, N. Duan, J. Yin, and J. McAuley. Longcoder: A long-range pre-trained language model for code completion, 2023. URLhttps://arxiv.org/abs/2306.14893

    2023 arXiv

  9. [17]

    Ho, A.-K

    X. Ho, A.-K. D. Nguyen, S. Sugawara, and A. Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, 2020

    2020

  10. [18]

    Hsieh, S

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y . Zhang, and B. Ginsburg. Ruler: What’s the real context size of your long-context language models?, 2024

    2024

  11. [19]

    Huang, S

    L. Huang, S. Cao, N. Parulian, H. Ji, and L. Wang. Efficient attentions for long document summarization. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419– 1436, Online, ...

    2021 doi

  12. [20]

    Joshi, E

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv.org/abs/1705. 03551

    2017

  13. [21]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

    2020

  14. [22]

    Koˇciský, J

    T. Koˇciský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge, 2017. URL https://arxiv.org/abs/ 1712.07040

    2017 arXiv

  15. [23]

    Li and D

    X. Li and D. Roth. Learning question classifiers. InCOLING 2002: The 19th International Con- ference on Computational Linguistics, 2002. URL https://www.aclweb.org/anthology/ C02-1150

    2002

  16. [24]

    P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating wikipedia by summarizing long sequences, 2018. URL https://arxiv.org/abs/1801. 10198

    2018

  17. [25]

    T. Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code auto- completion systems.arXiv preprint arXiv:2306.03091, 2023

    2023 arXiv

  18. [26]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL https: //arxiv.org/abs/1711.05101

    2019 arXiv

  19. [27]

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV , X. He, H. Hou, J. Lin, P. Kazienko, J. Kocon, J. Kong, B. Koptyra, H. Lau, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, B. Wang, J. S. Wind, S. Woz...

  20. [28]

    B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV , J. Koco ´n, B. Koptyra, S. Krishna, R. M. Jr., J. Lin, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, C. Wirawan, S. Wo´ zniak, R. Zhang, ...

    2024

  21. [29]

    B. Peng, R. Zhang, D. Goldstein, E. Alcaide, X. Du, H. Hou, J. Lin, J. Liu, J. Lu, W. Merrill, G. Song, K. Tan, S. Utpala, N. Wilce, J. S. Wind, T. Wu, D. Wuttke, and C. Zhou-Zheng. Rwkv-7 "goose" with expressive dynamic state evolution, 2025. URL https://arxiv.org/ abs/2503.14456. 11

    2025 arXiv

  22. [30]

    H. Qian, Z. Liu, P. Zhang, K. Mao, Y . Zhou, X. Chen, and Z. Dou. Are long-llms a necessity for long-context tasks?ArXiv, abs/2405.15318, 2024. URL https://api.semanticscholar. org/CorpusID:270045856

    2024 arXiv

  23. [31]

    Z. Qin, S. Yang, W. Sun, X. Shen, D. Li, W. Sun, and Y . Zhong. HGRN2: Gated Linear RNNs with State Expansion

  24. [32]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

    2025 arXiv

  25. [33]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  26. [34]

    Y . Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y . Dubois, X. Chen, X. Wang, S. Koyejo, T. Hashimoto, and C. Guestrin. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. doi: 10.48550/ARXIV .2407.04620

    doi arXiv

  27. [35]

    Trivedi, N

    H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  28. [36]

    A. Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017

    2017

  29. [37]

    Waleffe, W

    R. Waleffe, W. Byeon, D. Riach, B. Norick, V . Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayanan, G. Kulshreshtha, V . Singh, J. Casper, J. Kautz, M. Shoeybi, and B. Catanzaro. An Empirical Study of Mamba-based Language Models. 2024. doi: 10.48550/ ARXIV .2406...

    2024 doi arXiv

  30. [38]

    S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim. Gated Linear Attention Transformers with Hardware-Efficient Training. doi: 10.48550/ARXIV .2312.06635

    doi arXiv

  31. [39]

    S. Yang, J. Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    2024 arXiv

  32. [40]

    S. Yang, B. Wang, Y . Zhang, Y . Shen, and Y . Kim. Parallelizing linear transformers with the delta rule over sequence length, 2025. URLhttps://arxiv.org/abs/2406.06484

    2025 arXiv

  33. [41]

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    2018 arXiv

  34. [42]

    C. Yoon, T. Lee, H. Hwang, M. Jeong, and J. Kang. Compact: Compressing retrieved documents actively for question answering.ArXiv, abs/2407.09014, 2024. URL https: //api.semanticscholar.org/CorpusID:271161642

    2024 arXiv

  35. [43]

    Zhang, W

    H. Zhang, W. Du, J. Shan, Q. Zhou, Y . Du, J. B. Tenenbaum, T. Shu, and C. Gan. Building cooperative embodied agents modularly with large language models, 2024

    2024

  36. [44]

    Zhang, R

    Y . Zhang, R. Sun, Y . Chen, T. Pfister, R. Zhang, and S. Ö. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024. URL https://arxiv.org/abs/ 2406.02818

    2024 arXiv

  37. [45]

    Zheng, D

    Y . Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. 2025. URL https://api. semanticscholar.org/CorpusID:277596185

    2025

  38. [46]

    Zhong, D

    M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. Hassan Awadallah, A. Celikyilmaz, Y . Liu, X. Qiu, and D. Radev. QMSum: A New Benchmark for Query-based Multi-domain Meet- ing Summarization. InNorth American Association for Computational Linguistics (NAACL), 2021. 12

    2021

  39. [47]

    ” delimiter, and the merging process is repeated at the sentence level. In practice, we utilize a prioritized list of delimiters—“\n\n\n

    Z. Zhong, H. Liu, X. Cui, X. Zhang, and Z. Qin. Mix-of-granularity: Optimize the chunking granularity for retrieval-augmented generation, 2025. URL https://arxiv.org/abs/2406. 00456. 13 0.5k / 1k 1k / 2k 2k / 4k 4k / 8k 8k / 16k Chunk Size vs. Window Size 450 500 550Inference ...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.