Pith. sign in

REVIEW 3 major objections 5 minor 66 references

A trained recurrent side branch carries an LLM's memory across context turnover, so questions can be answered after their evidence leaves the window.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 05:42 UTC pith:R63WHGXQ

load-bearing objection A sensible recipe and clear framing, but the state-continuity claim isn't supported by the paper's own controls: the eviction analyses omit the Trunc arm, and the training mixture overlaps the eval benchmarks. the 3 major comments →

arxiv 2608.02515 v1 pith:R63WHGXQ submitted 2026-08-03 cs.CL cs.LG

LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

Zhichen Liu , Ruihan Sun , Hengjie Yang , Zipeng Wu , Zhaohan Chen , Xiaofan Zhang , Yang Xu This is my paper
classification cs.CL cs.LG
keywords state continuitycontext turnoverrecurrent memoryGated DeltaNet-2long-context LLM inferencememory-oriented post-trainingLongMemEvalfixed-capacity memory state
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LiveMem sets out to prove that an LLM can maintain a continuous memory across context switches: as old tokens are evicted from the bounded attention window, a fixed-capacity recurrent state trained alongside the frozen backbone continues to influence later answers. The paper argues that this state continuity is a distinct capability missing from retrieval, summarization, and KV-eviction systems, and that it can be created by post-training a memory side branch under context turnover. If true, long-running agents no longer need to keep or re-fetch the whole past; the model itself carries a compressed, updateable trace of its entire lifecycle. The experiments show the trained model answering questions even when all supporting evidence has been removed from the visible window, with accuracy staying measurable as the evidence moves farther away.

Core claim

The paper's central claim is that a fixed-capacity memory state, updated online by a Gated DeltaNet-2 side branch attached to every layer of a frozen full-attention LLM, can remain load-bearing after the tokens that wrote it are evicted from the KV cache. Context turnover is applied during post-training so the model cannot learn to bypass the state by keeping key history visible; it must write, preserve, and read through the memory. On LongMemEval, LiveMem answers questions with the supporting evidence fully outside the active window at accuracy more than ten percentage points above a base model with no memory state, and the accuracy stays measurable as the evidence is placed up to 64K token

What carries the argument

The memory state is a per-layer recurrent tensor of shape heads × 128 × 128, updated at every token. Its update uses GDN2: a learned log-decay forget gate, an erase gate that removes old associations before writing, and a write gate; readout is S^T q added to the attention output. Because the side branch's output projection is zero-initialized, the model starts as the original LLM and the branch is opened by post-training. The turnover mechanism—a FIFO chunk queue with the system prompt retained as an attention sink—is applied identically in training and serving, so the memory state is the only path to evicted information.

Load-bearing premise

The load-bearing premise is that the post-training and evaluation sets are disjoint: Section C.4 names MuSiQue, 2WikiMultiHopQA, and NarrativeQA as post-training data while Section D.1 uses the same benchmarks for evaluation, and the paper never states that evaluation instances were excluded from training.

What would settle it

A decisive check is to compare the post-training packs against the evaluation instance IDs for MuSiQue, 2WikiMultiHopQA, and NarrativeQA; any overlap means the fully-evicted accuracy numbers could be memorized answers rather than memory-state continuity. A cleaner experiment would re-run the same recipe on fresh synthetic long documents with no public benchmark representation in pretraining and measure the fully-evicted setting from scratch.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If a fixed-capacity memory state carries evicted information, long-running agents can keep working with a bounded KV window instead of reprocessing or re-retrieving their full history.
  • The model can answer multiple questions about the same past evidence after that evidence has left attention; the largest gains in the paper appear on multi-question tasks.
  • Retrieval systems and recurrent memory are complementary: retrieval selects what to read, while state continuity determines how the model stays coherent between reads.
  • Because the state is query-independent and updated online, facts that later become relevant can still influence predictions even when no query was known at eviction time.
  • LiveMem's accuracy decays only mildly with evidence-to-window distance, so the state's forgetting mechanism does not immediately dump old evidence at the eviction boundary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the result transfers to longer streams, the practical serving cost of a lifelong assistant could become independent of total history length—only a fixed recurrent tensor plus bounded KV pages are carried between turns.
  • Editorial inference: one can test how much of the memory effect is a learned summary versus parametric prior knowledge by repeating the fully-evicted LongMemEval analysis on synthetic documents generated after training; if accuracy collapses, the state is not the thing doing the work.
  • Editorial inference: the 256K positional horizon of the base architecture is the main barrier to true infinite-context use; combining LiveMem with position-free attention would be the natural next experiment.
  • Editorial inference: the train/eval overlap question—the same benchmark families appear in both post-training and evaluation—should be resolved by releasing instance IDs or code; otherwise a reader cannot separate state continuity from benchmark fitting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LiveMem, a method for adding a fixed-capacity recurrent memory state to a frozen full-attention LLM so that information remains usable after its source tokens leave the active KV window. A GDN2 side branch is attached at every layer, the model is trained with a context-turnover mask and memory-oriented SFT/RL, and a paged serving implementation maintains the state across chunk evictions. The evaluation covers Wiki QA, conversation, test-time learning, and long-document QA under bounded 8K/32K contexts, plus a 256K reference condition. The central claim is that the recurrent state provides state continuity under context turnover—specifically, that LongMemEval evidence removed from the active context can still be answered from the memory state (Figures 3–4).

Significance. If the central claim were fully supported, LiveMem would be a useful step toward continual inference in long-running assistants: it offers a clear formulation of state continuity as distinct from retrieval, a concrete architecture on a frozen backbone, and an honest discussion of limitations (NIAH failure, positional horizon). The paper is commendable for reporting a State-minus-Trunc internal control in Table 2 and for acknowledging that the memory state is lossy. However, the load-bearing evidence for the headline claim is weakened by the absence of the same-checkpoint Trunc control in exactly the LongMemEval analysis used to motivate the paper, and by post-training data that overlaps evaluation benchmarks. The claim remains plausible, but the experimental support as written is not yet sufficient.

major comments (3)
  1. [§4.3, Table 2 vs Figures 3–4] The paper's own internal control shows that on LongMemEval-S, LiveMem-SFT scores 0.286 in the State condition vs 0.296 in Trunc (Δ=-0.010), and LiveMem-RL scores +0.008. Yet Figures 3 and 4—the headline evidence for state continuity—use LongMemEval-S and compare LiveMem against Qwen3-4B, δ-Mem, and Context2LoRA, not against LiveMem-Trunc. The +10-point advantage under full eviction in Figure 3(A) could therefore be attributable to post-training effects (better instruction following, calibration, or question use) rather than to information carried in the recurrent state. Please add the same-checkpoint Trunc curve/delta to Figures 3 and 4 and report State-minus-Trunc with confidence intervals, or explicitly restrict the continuity claim to datasets where the State condition outperforms Trunc.
  2. [§C.4 vs §D.1] The post-training mixture in C.4 includes MuSiQue, 2WikiMultiHopQA, and NarrativeQA, all of which are also used as evaluation benchmarks in D.1. The paper never states whether the evaluation instances/splits were excluded from the post-training data. Without a disjoint-split guarantee, the Wiki QA and Long QA gains in Table 1 (and the MQ variants constructed from the same datasets) may be partly fitted to the test benchmarks rather than demonstrating a general memory capability. Please state explicitly whether evaluation splits were excluded, and if not, re-run the affected experiments with the overlapping benchmark families removed from the training mixture or with disjoint held-out splits.
  3. [§4.3, Table 2] No error bars or repeated-seed results are reported for the State/Trunc comparisons or for Figures 3–4. Many deltas in Table 2 are small (e.g., +0.000 for NarrQA-SFT, +0.006 for ∞Bench-QA-SFT, -0.010 for LME-S-SFT) and could be within sampling noise. To make the central claim load-bearing, the paper should provide variance estimates, confidence intervals, or at least multiple evaluation seeds for the representative State-vs-Trunc and evidence-distance analyses.
minor comments (5)
  1. [§4.3, text near Figure 3] The sentence 'leveraging historical information from LiveMem's memory state can achieve higher accuracy than other methods' compares against other systems, not against the Trunc control. Please rephrase to avoid implying a within-model state comparison that is not made.
  2. [§3.2, Eq. (10)] The predicate is written 'in_mem(chunk(i), j)' but the surrounding text refers to 'in_window(c(i), j)'. Please unify the notation.
  3. [Table 2] The header 'w./w.o.' should be spelled out (with/without); also consider adding the number of evaluation instances per dataset so the small deltas can be interpreted.
  4. [§4.1] All reported numbers are point estimates with no variance. Even for the main table, a note on evaluation stability or a small repeated-seed check would help, especially because many headline differences are a few points.
  5. [§C.4] The dataset list for post-training is given in prose; a table with source, split, and whether that split was used in evaluation would improve transparency and reproducibility.

Circularity Check

1 steps flagged

Post-training mixture includes the Wiki QA and Long QA evaluation benchmarks, so several headline 'predictions' reduce partly to fitted data; the LongMemEval continuity claim also lacks the paper's own Trunc control.

specific steps
  1. fitted input called prediction [Section C.4 (post-training recipe) vs Section D.1 (evaluation construction) and Table 1]
    "The second runs for 500 steps on a mixture of MuSiQue, 2WikiMultiHopQA, NarrativeQA, Qasper, long-form instruction data, AG News, and DBpedia ... For the Wiki QA suite, we used 2WikiMultiHopQA, HotpotQA, and MuSiQue as evaluation sets ... For Long QA, we adopted the English QA task from ∞Bench, EventQA from MAB, and NarrativeQA."

    The same benchmark names appear in both the post-training mixture (C.4) and the evaluation suites (D.1). Table 1's Wiki QA and Long QA gains (e.g., LiveMem-RL overall Wiki 0.560 vs Qwen3-4B 0.498; Long QA 0.376 vs 0.282) are therefore not independent predictions of a state-continuity capability: the memory branch was fit to instances from these exact datasets/benchmark families. The paper never states that evaluation instances/splits were excluded from this mixture, so the observed improvements can be attributed, at least in part, to memorized training signal rather than to the recurrent state carrying evicted evidence. This is a fitted input being reported as a prediction on the same benchmarks.

full rationale

The paper's deepest claim is that a fixed-capacity recurrent state carries evicted evidence across context turnover. Its own operational definition (Section 2.2) and control (Table 2) are the State-minus-Trunc comparison. That control shows positive deltas on most reported datasets, but on LongMemEval-S, the exact benchmark used for the headline continuity plots, the deltas are −0.010 (SFT) and +0.008 (RL). Figures 3 and 4 nevertheless attribute accuracy to the memory state without a Trunc control, comparing instead to Qwen3-4B and δ-Mem, which also lack post-training. This is a serious missing-control/correctness issue, but it is not circularity: it is an unsupported attribution, not a reduction of the claim to its inputs. The genuine circular step is the train/eval overlap. Section C.4 lists MuSiQue, 2WikiMultiHopQA, and NarrativeQA in the post-training mixture, while Section D.1 lists the same datasets as evaluation sets. Because no split exclusion is stated, the Wiki QA and Long QA columns in Table 1 are partly fitted to the evaluation benchmarks rather than independent tests of memory-state continuity. The LongMemEval evidence is not in the training mixture, so the central mechanism still has partially independent support. The NIAH limitation in Section E.1 is acknowledged by the authors and is a limitation, not a circularity. No load-bearing self-citation chain or imported uniqueness theorem is present. Overall, the circularity is partial: several 'predictions' reduce by construction to the training data, but the core architectural claim retains independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are postulated; 'living memory state' is a label for an existing recurrent state trained with a new recipe. The load-bearing assumptions are architectural compatibility of the side branch and the validity of the evaluation benchmarks; both are reasonable but not independently certified.

free parameters (3)
  • GRPO clip values and gamma = epsilon_low=0.20, epsilon_high=0.28, gamma=3
    Chosen by hand in Eq. (13)-(14); affect RL training dynamics and final memory behavior, not derived.
  • Turnover chunk sizes / live-token budgets = Table 5: chunk 64-512 tokens, live budget 512-8192 tokens
    Hand-picked operational schedule used identically in training and serving; determines what enters memory.
  • Memory state shape / heads = 32 heads x 128 x 128 per layer, FP32
    Architectural choice (Sec. C.1) that fixes capacity; no ablation of state size provided.
axioms (5)
  • domain assumption A pretrained full-attention LLM (Qwen3-4B) can be augmented with a parallel GDN2 side branch while the backbone stays frozen without destroying capabilities.
    Used throughout Sec. 3; relies on the pretrained model's representations being compatible with a recurrent readout; not proven for other backbones.
  • domain assumption GDN2 update (Eqs. 7-9, 15) is an adequate learned transition for carrying task-relevant information over many tokens.
    Taken from Hatamizadeh et al. 2026; the paper does not compare against other recurrent kernels except in passing.
  • domain assumption Dynamic attention masks during training faithfully simulate inference-time KV page release.
    Sec. C.2/C.3 asserts train-inference consistency; actual page release may interact differently with memory reuse/drop.
  • domain assumption The frozen backbone's rotary position embeddings remain valid for 256K inputs.
    Sec. E.2 admits finite positional horizon; experiments cap at 256K, so state continuity beyond that horizon is untested.
  • domain assumption Evaluation benchmarks (LongMemEval, LoCoMo, TTL, ∞Bench) are valid proxies for state continuity.
    Task design and judge-based scoring (Qwen3.6-35B-A3B) are taken as ground truth without human agreement checks.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference." pith.science (2026). https://pith.science/paper/R63WHGXQ

@misc{pith2026260802515,
  author       = {Pith},
  title        = {Pith review of: LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R63WHGXQ}},
  note         = {Machine review of arXiv:2608.02515}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Long-running assistants and agents consume interaction streams that eventually outgrow the context. Existing context retention, summarization, and retrieval preserve access to selected history, but do not provide a persistent state over the full lifecycle when working context changes. We formulate this missing inference capability as \emph{state continuity under context turnover}: carrying computation forward through a fixed-capacity memory state whose lifetime is independent of the active context. We introduce an intrinsic memory method, \textbf{LiveMem}, which augments a pretrained full-attention LLM with a memory state that preserves the historical information over the whole lifecycle while the main attention path retains a bounded KV window. Context turnover and memory state maintaining, memory-oriented post-training, and state-aware serving jointly make this memory state load bearing after its originating tokens are released. Our experiments show that LiveMem achieves leading overall performance among evaluated systems and other intrinsic memory methods. Experiments on LongMemEval show that LiveMem is able to answer the question based on the memory state, even when the supporting evidence has been removed from the current context, and evidence-distance analysis shows that useful information persists beyond the active window. LiveMem thus establishes state continuity as a distinct and complementary abstraction for continual LLM inference.

Figures

Figures reproduced from arXiv: 2608.02515 by Hengjie Yang, Ruihan Sun, Xiaofan Zhang, Yang Xu, Zhaohan Chen, Zhichen Liu, Zipeng Wu.

Figure 1
Figure 1. Figure 1: LiveMem enables stateful continual inference under context turnover. A bounded working [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of LiveMem. A full-attention main path maintains a bounded working context, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: LongMemEval-s accuracy grouped by whether the supporting evidence is fully or partially [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LongMemEval-S accuracy as supporting evidence is placed progressively farther from the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 1 canonical work pages

  1. [1]

    Hatamizadeh, Ali and Choi, Yejin and Kautz, Jan , journal =. Gated. 2026 , url =

    2026

  2. [2]

    2025 , url =

    Yang, An and others , journal =. 2025 , url =

    2025

  3. [3]

    2024 , url =

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and others , journal =. 2024 , url =

    2024

  4. [4]

    2025 , url =

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and others , journal =. 2025 , url =

    2025

  5. [5]

    and Stoica, Ion and Gonzalez, Joseph E

    Packer, Charles and Wooders, Sarah and Lin, Kevin and Fang, Vivian and Patil, Shishir G. and Stoica, Ion and Gonzalez, Joseph E. , journal =. 2023 , url =

    2023

  6. [6]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , year =

  7. [7]

    arXiv preprint arXiv:2501.00663 , year =

    Titans: Learning to Memorize at Test Time , author =. arXiv preprint arXiv:2501.00663 , year =

    Pith/arXiv arXiv

  8. [8]

    Advances in Neural Information Processing Systems , year =

    Nested Learning: The Illusion of Deep Learning Architectures , author =. Advances in Neural Information Processing Systems , year =

  9. [9]

    arXiv preprint arXiv:2504.07952 , year =

    Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory , author =. arXiv preprint arXiv:2504.07952 , year =

    Pith/arXiv arXiv

  10. [10]

    2026 , url =

    Yang, Xintong and Gu, Hao and Xu, Binxing and Li, Lujun and Liu, Bei and Liu, Jiacheng and Zhu, Qiyuan and Guo, Yike and Han, Sirui , journal =. 2026 , url =

    2026

  11. [11]

    2025 , url =

    Yu, Hongli and Chen, Tinghong and Feng, Jiangtao and Chen, Jiangjie and Dai, Weinan and Yu, Qiying and others , journal =. 2025 , url =

    2025

  12. [12]

    Back, Seungju and Lee, Dongwoo and Kang, Naun and Lee, Taehee and Hong, S. K. and Gwon, Youngjune and Ahn, Sungjin , booktitle =. Understanding. 2026 , url =

    2026

  13. [13]

    arXiv preprint arXiv:2605.12357 , year =

    -mem: Efficient Online Memory for Large Language Models , author =. arXiv preprint arXiv:2605.12357 , year =

    Pith/arXiv arXiv

  14. [14]

    2025 , url =

    Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and others , journal =. 2025 , url =

    2025

  15. [15]

    Constructing a Multi-hop

    Ho, Xanh and Duong Nguyen, Anh-Khoa and Sugawara, Saku and Aizawa, Akiko , booktitle =. Constructing a Multi-hop. 2020 , url =

    2020

  16. [16]

    and Salakhutdinov, Ruslan and Manning, Christopher D

    Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle =. 2018 , url =

    2018

  17. [17]

    2022 , url =

    Trivedi, Harsh and Balasubramanian, Niranjan and Khot, Tushar and Sabharwal, Ashish , journal =. 2022 , url =

    2022

  18. [18]

    Evaluating Very Long-Term Conversational Memory of

    Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei , booktitle =. Evaluating Very Long-Term Conversational Memory of. 2024 , url =

    2024

  19. [19]

    2025 , url =

    Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong , booktitle =. 2025 , url =

    2025

  20. [20]

    Evaluating Memory in

    Hu, Yuanzhe and Wang, Yu and McAuley, Julian , booktitle =. Evaluating Memory in. 2026 , url =

    2026

  21. [21]

    2024 , url =

    Zhang, Xinrong and Chen, Yingfa and Hu, Shengding and Xu, Zihang and Chen, Junhao and Hao, Moo and others , booktitle =. 2024 , url =

    2024

  22. [22]

    Ko. The. Transactions of the Association for Computational Linguistics , year =

  23. [23]

    International Conference on Learning Representations , year =

    Memorizing Transformers , author =. International Conference on Learning Representations , year =

  24. [24]

    Advances in Neural Information Processing Systems , year =

    Augmenting Language Models with Long-Term Memory , author =. Advances in Neural Information Processing Systems , year =

  25. [25]

    2024 , url =

    Xiao, Chaojun and Zhang, Pengle and Han, Xu and Xiao, Guangxuan and Lin, Yankai and Zhang, Zhengyan and Liu, Zhiyuan and Sun, Maosong , booktitle =. 2024 , url =

    2024

  26. [26]

    2023 , url =

    Zhong, Wanjun and Guo, Lianghong and Gao, Qiqi and Ye, He and Wang, Yanlin , journal =. 2023 , url =

    2023

  27. [27]

    2024 , url =

    Wang, Yu and Gao, Yifan and Chen, Xiusi and Jiang, Haoming and Li, Shiyang and Yang, Jingfeng and others , booktitle =. 2024 , url =

    2024

  28. [28]

    2025 , url =

    Wang, Yu and Krotov, Dmitry and Hu, Yuanzhe and Gao, Yifan and Zhou, Wangchunshu and McAuley, Julian and Gutfreund, Dan and Feris, Rogerio and He, Zexue , booktitle =. 2025 , url =

    2025

  29. [29]

    Proceedings of the 41st International Conference on Machine Learning , year =

    Larimar: Large Language Models with Episodic Memory Control , author =. Proceedings of the 41st International Conference on Machine Learning , year =

  30. [30]

    Learning to (

    Sun, Yu and Li, Xinhao and Dalal, Karan and Xu, Jiarui and Vikram, Arjun and Zhang, Genghan and others , booktitle =. Learning to (. 2025 , url =

    2025

  31. [31]

    Transformer-

    Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc and Salakhutdinov, Ruslan , booktitle =. Transformer-. 2019 , url =

    2019

  32. [32]

    arXiv preprint arXiv:2207.06881 , year =

    Recurrent Memory Transformer , author =. arXiv preprint arXiv:2207.06881 , year =

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2404.07143 , year =

    Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention , author =. arXiv preprint arXiv:2404.07143 , year =

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2312.00752 , year =

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author =. arXiv preprint arXiv:2312.00752 , year =

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2403.19887 , year =

    Jamba: A Hybrid Transformer--Mamba Language Model , author =. arXiv preprint arXiv:2403.19887 , year =

    Pith/arXiv arXiv

  36. [36]

    Proceedings of the 38th International Conference on Machine Learning , year =

    Linear Transformers Are Secretly Fast Weight Programmers , author =. Proceedings of the 38th International Conference on Machine Learning , year =

  37. [37]

    Advances in Neural Information Processing Systems , year =

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length , author =. Advances in Neural Information Processing Systems , year =

  38. [38]

    Gated Delta Networks: Improving

    Yang, Songlin and Kautz, Jan and Hatamizadeh, Ali , booktitle =. Gated Delta Networks: Improving. 2025 , url =

    2025

  39. [39]

    arXiv preprint arXiv:2510.26692 , year =

    Kimi Linear: An Expressive, Efficient Attention Architecture , author =. arXiv preprint arXiv:2510.26692 , year =

    Pith/arXiv arXiv

  40. [40]

    International Conference on Learning Representations , year =

    Efficient Streaming Language Models with Attention Sinks , author =. International Conference on Learning Representations , year =

  41. [41]

    2023 , url =

    Zhang, Zhenyu and Sheng, Ying and Zhou, Tianyi and Chen, Tianlong and Zheng, Lianmin and Cai, Ruisi and others , booktitle =. 2023 , url =

    2023

  42. [42]

    2024 , url =

    Li, Yuhong and Huang, Yingbing and Yang, Bowen and Venkitesh, Bharat and Locatelli, Acyr and Ye, Hanchen and Cai, Tianle and Lewis, Patrick and Chen, Deming , booktitle =. 2024 , url =

    2024

  43. [43]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle =. 2022 , url =

    2022

  44. [44]

    arXiv preprint arXiv:2412.05496 , year =

    Flex Attention: A Programming Model for Generating Optimized Attention Kernels , author =. arXiv preprint arXiv:2412.05496 , year =

    Pith/arXiv arXiv

  45. [45]

    and Zhang, Hao and Stoica, Ion , booktitle =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with. 2023 , url =

    2023

  46. [46]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , year =

    A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers , author =. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , year =

    2021

  47. [47]

    Advances in Neural Information Processing Systems , year =

    Character-level Convolutional Networks for Text Classification , author =. Advances in Neural Information Processing Systems , year =

  48. [48]

    2024 , url =

    Hsieh, Cheng-Ping and Sun, Simeng and Kriman, Samuel and Acharya, Shantanu and Rekesh, Dima and Jia, Fei and Zhang, Yang and Ginsburg, Boris , journal =. 2024 , url =

    2024

  49. [49]

    arXiv preprint arXiv:2506.05176 , year=

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author=. arXiv preprint arXiv:2506.05176 , year=

    Pith/arXiv arXiv

  50. [50]

    2026 , month = jul, howpublished =

    2026

  51. [51]

    2025 , eprint =

    Xu, Wujiang and Liang, Zujie and Mei, Kai and Gao, Hang and Tan, Juntao and Zhang, Yongfeng , booktitle =. 2025 , eprint =

    2025

  52. [52]

    2025 , doi =

    Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj , booktitle =. 2025 , doi =. 2504.19413 , archivePrefix =

    Pith/arXiv arXiv 2025

  53. [53]

    2025 , doi =

    Li, Zhiyu and Xi, Chenyang and Li, Chunyu and Chen, Ding and Chen, Boyu and Song, Shichao and Niu, Simin and Wang, Hanyu and Yang, Jiawei and Tang, Chen and Yu, Qingchen and Zhao, Jihao and Wang, Yezhaohui and Liu, Peng and Lin, Zehao and Wang, Pengyuan and Huo, Jiahao and Chen, Tianyi and Chen, Kai and Li, Kehang and Tao, Zhen and Lai, Huayi and Wu, Hao ...

    Pith/arXiv arXiv 2025

  54. [54]

    arXiv preprint arXiv:2512.12818 , year =

    Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects , author =. arXiv preprint arXiv:2512.12818 , year =. doi:10.48550/arXiv.2512.12818 , eprint =

    doi:10.48550/arxiv.2512.12818

  55. [55]

    Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI , pages =

    Efficient Intent Detection with Dual Sentence Encoders , author =. Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI , pages =. 2020 , month = jul, address =. doi:10.18653/v1/2020.nlp4convai-1.5 , url =

    doi:10.18653/v1/2020.nlp4convai-1.5 2020

  56. [56]

    An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction , author =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages =. 2019 , month = nov, address =. doi:10.18653/v1/D19-1131 , url =

    doi:10.18653/v1/d19-1131 2019

  57. [57]

    arXiv preprint arXiv:1903.05566 , year =

    Benchmarking Natural Language Understanding Services for Building Conversational Agents , author =. arXiv preprint arXiv:1903.05566 , year =. doi:10.48550/arXiv.1903.05566 , url =. 1903.05566 , archivePrefix =

    Pith/arXiv arXiv doi:10.48550/arxiv.1903.05566 1903

  58. [58]

    2002 , url =

    Learning Question Classifiers , author =. 2002 , url =

    2002

  59. [59]

    Advances in Neural Information Processing Systems , volume =

    Towards Deep Conversational Recommendations , author =. Advances in Neural Information Processing Systems , volume =. 2018 , url =

    2018

  60. [60]

    2026 , url =

    Wei, Rubin and Cao, Jiaqi and Wang, Jiarui and Kai, Jushi and Guo, Qipeng and Zhou, Bowen and Lin, Zhouhan , booktitle =. 2026 , url =. 2508.01832 , archivePrefix =

    arXiv 2026

  61. [61]

    2024 , month = nov, address =

    Bai, Yushi and Lv, Xin and Zhang, Jiajie and He, Yuze and Qi, Ji and Hou, Lei and Tang, Jie and Dong, Yuxiao and Li, Juanzi , booktitle =. 2024 , month = nov, address =. doi:10.18653/v1/2024.findings-emnlp.74 , url =

    doi:10.18653/v1/2024.findings-emnlp.74 2024

  62. [62]

    2024 , note =

    Chen, Yukang and Qian, Shengju and Tang, Haotian and Lai, Xin and Liu, Zhijian and Han, Song and Jia, Jiaya , booktitle =. 2024 , note =. 2309.12307 , archivePrefix =

    Pith/arXiv arXiv 2024

  63. [63]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    What Are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2025 , month = jul, address =. doi:10.18653/v1/2025.acl-long.1316 , url =

    work page doi:10.18653/v1/2025.acl-long.1316 2025

  64. [64]

    2023 , howpublished =

    2023

  65. [65]

    2023 , eprint=

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. 2023 , eprint=

    2023

  66. [66]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =

    Dense Passage Retrieval for Open-Domain Question Answering , author =. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages =. 2020 , month = nov, address =. doi:10.18653/v1/2020.emnlp-main.550 , url =

    doi:10.18653/v1/2020.emnlp-main.550 2020

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.